Core
Extractions
An extraction is the one resource that does everything. The request is three orthogonal choices — what, scope, and how much — plus optional output shaping, a session, and delivery.
Request fields
Which site to extract from — e.g. linkedin, instagram, google_maps, or generic for any URL.
What to pull, from the connector's catalog — e.g. company.posts, search.posts, place.reviews.
Operation-specific scope — the entity or search terms (a company, a profile, keywords, a URL).
How much to gather. See collect modes. Defaults to { "mode": "limit", "limit": 100 }.
Shape the result — pick fields, set a schema, choose json / ndjson / csv.
A session id to run under, using a logged-in session you supply.
Where results go — sync, webhook, store, or a destination. See delivery.
One connector, two very different needs
Because operation and collect are independent, one connector serves opposite use cases without special endpoints.
Customer A — N posts on a topic
{
"connector": "linkedin",
"operation": "search.posts",
"input": { "keywords": "cold brew" },
"collect": { "mode": "limit", "limit": 200 }
}Customer B — every post from a page
{
"connector": "linkedin",
"operation": "company.posts",
"input": { "company": "microsoft" },
"collect": { "mode": "all" }
}Same envelope. operation picks search-vs-listing; collect decides the depth. Which internal search or pagination path satisfies it is our concern, not yours.
Synchronous and asynchronous
Most extractions run in the background. A real crawl — paginated, defended, thousands of records — cannot finish inside one HTTP request, so by default a POST /v1/extractions returns 202 Accepted with a job id and status: "queued". You then poll the job or receive a webhook.
Small jobs can opt into an inline response with delivery: { "type": "sync" } — the request blocks briefly and returns the records directly. Sync is a convenience for jobs up to 1,000 records; above that it is rejected and you must run the job asynchronously.
{
"id": "ext_9f2a1c",
"status": "queued",
"connector": "blinkit",
"operation": "search.products",
"request_id": "req_a1b2c3"
}Checking a job
Poll a job to see where it stands. status moves through queued → running → succeeded (or failed), and counts.collected rises as records land, so you can show progress before the job is done.
| status | Meaning |
|---|---|
queued | Accepted, not yet started. |
running | In progress. counts.collected is a partial total. |
succeeded | Complete. Fetch records via store, or they were sent to your webhook / destination. |
failed | Stopped. See errors for the generic reason; full detail is keyed by request_id. |
{
"id": "ext_9f2a1c",
"status": "running",
"counts": { "collected": 1840, "target": 5000 },
"usage": { "credits": 1840, "tier": "rendered" },
"request_id": "req_a1b2c3"
}Once status is succeeded, fetch the records with GET /v1/extractions/{id}/records?cursor=… (see delivery), unless you chose a webhook or destination — in which case they are already on their way.