Documentation

Everything you need to build, schedule, and integrate workflows in webHarvest.

Getting started

Every workflow lives inside a project. Projects are just folders for organizing related workflows — create one per site, client, or use case.

Inside a project, use the "Generate a workflow with AI" box: give it a URL and describe what you want in plain English. It builds the extraction, runs it against the live page immediately, and shows you real data before you save anything. If it looks right, name it and save — you're done.

Writing good prompts

The AI planner works best when your instruction is specific about what fields you want, not just the general topic. A few examples:

get the title, price as a number, and the link to every product
get the job title, location, and posted date for every listing
get the headline and link of every article on the page

If the page requires searching first (like a real estate or job board homepage), point the tool at an actual results/listing page — not the homepage — since the AI can only see the page you give it, and a homepage usually has no real data on it yet.

Workflow features

Pagination. If a page has a "next" link, describe wanting "every" item or "all pages" and the AI will include pagination automatically, following links until there's nothing left (capped at 20 pages by default for safety).

Multiple URLs. After generating a workflow, use the "Run this on multiple pages?" section to either paste a list of URLs directly, or use "Grab links" to pull a list of URLs automatically from a listing page.

Typed fields. The AI automatically types fields as text, numbers (with currency symbols stripped), or booleans, and can pull attributes like image or link URLs instead of just visible text.

Tags. Add comma-separated tags when saving a workflow to organize and filter across projects on the All Workflows page.

Scheduling

Every workflow has a Schedule dropdown — Off, hourly, every 6 hours, or daily. Once set, it runs automatically without you triggering anything. You can also set a custom cron expression via the API.

Change detection

Every run is automatically compared against the workflow's last successful run. If anything changed — a new item appeared, one disappeared, or a field's value changed — the run is flagged with a "changed" badge on the dashboard, project page, and run history, and the change details are included in the webhook payload.

A run is only compared against the last successful run — failed or broken runs are never used as the comparison baseline, so a stale-selector failure won't generate false "everything changed" alerts.

When a site blocks you

Some sites show a CAPTCHA or bot-check. When this happens, the run pauses and shows a "Take over" button on the dashboard and project page. Click it to see the live browser in your own tab, solve the challenge yourself, then click Resume — the workflow continues from exactly where it left off.

Sessions stay open for 5 minutes waiting for takeover, then close automatically if untouched. A run that pauses this way doesn't retry automatically, since retrying won't solve a CAPTCHA on its own.

API reference

Generate an API key from Settings → API Keys. Include it as a Bearer token on every request.

POST /api/v1/workflows/:id/run
Authorization: Bearer YOUR_API_KEY

Returns a run object with a pending status. Check its progress with:

GET /api/v1/runs/:id
Authorization: Bearer YOUR_API_KEY

Once the run finishes, this returns the run's status, any error, and the extracted records.

Webhooks

Set a webhook URL on any workflow to get a POST request the moment a run finishes — no polling required.

{
  "event": "run.completed_with_changes",
  "run_id": "...",
  "workflow_id": "...",
  "status": "success",
  "record_count": 20,
  "records": [...],
  "has_changes": true,
  "changes": {
    "added": 1,
    "removed": 0,
    "changed": 2,
    "details": [...]
  }
}

Plans & limits

All plans include unlimited workflows, scheduling, the API, and webhooks — the only difference between tiers is how many runs you get per month. If a run technically completes but returns no real data (a hollow success — usually a sign selectors went stale after a site redesign), it doesn't count against your usage.