JavaScript Scraping API
JavaScript Scraping API — scrape JS-rendered pages with headless Chrome. Click, scroll, fill forms, wait for elements, and run custom JavaScript before capture. $0.02 per call.
Pricing, live playground and use cases: JavaScript Scraping API on YepAPI
/v1/scrape/jsThe JavaScript Scraping API always loads the page in headless Chrome, so content rendered client-side by React, Vue, Angular, or any other framework is there when the page is captured. On top of rendering, you can script the browser with js_scenario — click "load more", scroll, fill a search box, wait for an element, or run your own JavaScript and get its return value — then receive the page as Markdown, HTML, or text.
For pages that do not need a browser, the Web Scraping API is cheaper. If the site blocks headless browsers, use Stealth Scrape.
Usage
const res = await fetch('https://api.yepapi.com/v1/scrape/js', {
method: 'POST',
headers: {
'x-api-key': 'YOUR_API_KEY',
'Content-Type': 'application/json',
},
body: JSON.stringify({
url: 'https://example.com/products',
format: 'markdown',
wait_for_selector: '.product-card',
js_scenario: {
instructions: [
{ click: 'button.load-more' },
{ wait_for: '.product-card:nth-child(40)' },
{ scroll_y: 2000 },
{ wait: 1000 },
{ evaluate: 'document.querySelectorAll(".product-card").length' },
],
},
}),
});
const { data } = await res.json();
console.log(data.content);
console.log(data.evaluateResults); // ["40"]curl -X POST https://api.yepapi.com/v1/scrape/js \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/products",
"format": "markdown",
"wait_for_selector": ".product-card",
"js_scenario": {"instructions": [
{"click": "button.load-more"},
{"wait_for": ".product-card:nth-child(40)"},
{"scroll_y": 2000},
{"wait": 1000},
{"evaluate": "document.querySelectorAll(\".product-card\").length"}
]}
}'import requests
res = requests.post(
"https://api.yepapi.com/v1/scrape/js",
headers={"x-api-key": "YOUR_API_KEY"},
json={
"url": "https://example.com/products",
"format": "markdown",
"wait_for_selector": ".product-card",
"js_scenario": {"instructions": [
{"click": "button.load-more"},
{"wait_for": ".product-card:nth-child(40)"},
{"scroll_y": 2000},
{"wait": 1000},
{"evaluate": "document.querySelectorAll('.product-card').length"},
]},
},
)
data = res.json()["data"]
print(data["content"], data.get("evaluateResults"))Request Body
| Parameter | Type | Required | Description | Default |
|---|---|---|---|---|
url | string | Yes | Full http(s) URL of the page to scrape (max 4096 characters). Localhost and private IPs are rejected | — |
format | string | No | Output format. One of: markdown, html, text | markdown |
selector | string | No | CSS or XPath selector. Returns only the first matching element — its HTML when format is html, otherwise its text (max 500 characters) | — |
wait_for | number | No | Extra milliseconds to wait after the page loads, max 15000 | — |
wait_for_selector | string | No | CSS selector to wait for before capturing. Returns 400 if it never appears (max 500 characters) | — |
js_scenario | object | No | Browser actions to run before capture: {"instructions": [...]} or the bare array. See JS Scenario (max 8000 characters) | — |
device | string | No | Browser profile. One of: desktop, mobile | desktop |
block_ads | boolean | No | Block ads and trackers — faster loads and cleaner content | false |
camelCase aliases are also accepted: waitFor, waitForSelector, jsScenario, blockAds. For backward compatibility, a non-numeric string in wait_for is treated as wait_for_selector.
JS Scenario
js_scenario is a list of instructions the browser runs in order after the page loads and before it is captured:
| Instruction | Example | What it does |
|---|---|---|
click | {"click": "#load-more"} | Click the element matching a CSS selector |
wait | {"wait": 1000} | Pause for a number of milliseconds |
wait_for | {"wait_for": ".results"} | Wait until an element appears |
wait_for_and_click | {"wait_for_and_click": "#accept-cookies"} | Wait for an element, then click it |
scroll_x | {"scroll_x": 1000} | Scroll horizontally by N pixels |
scroll_y | {"scroll_y": 1000} | Scroll vertically by N pixels |
fill | {"fill": ["#search", "running shoes"]} | Type a value into an input: [selector, value] |
evaluate | {"evaluate": "document.title"} | Run JavaScript; its result is returned in evaluateResults |
infinite_scroll | {"infinite_scroll": {"max_count": 0, "delay": 1000}} | Keep scrolling to load lazy content. max_count is the number of scrolls (0 = until the page stops growing), delay the ms between scrolls |
A typical scenario accepts the cookie banner, clicks "load more", waits for the new items, and reads a value with evaluate. An invalid instruction returns a 400 VALIDATION_ERROR naming the problem. The whole call has to finish within about 25 seconds, so keep waits short.
Response
{
"ok": true,
"data": {
"url": "https://example.com/products",
"statusCode": 200,
"content": "# Products\n\n## Trail Runner 3\n$89.00\n\n## Road Racer Pro\n$129.00\n\n...",
"format": "markdown",
"evaluateResults": ["40"],
"jsScenarioReport": {
"total_duration": 3.48,
"task_executed": 5,
"task_success": 5,
"task_failure": 0,
"tasks": [
{ "task": "click", "params": "button.load-more", "duration": 0.21, "success": true },
{ "task": "wait_for", "params": ".product-card:nth-child(40)", "duration": 1.12, "success": true },
{ "task": "scroll_y", "params": 2000, "duration": 0.1, "success": true },
{ "task": "wait", "params": 1000, "duration": 1.0, "success": true },
{ "task": "evaluate", "params": "document.querySelectorAll(\".product-card\").length", "duration": 0.02, "success": true }
]
}
}
}Response Fields
| Field | Type | Description |
|---|---|---|
ok | boolean | Whether the request succeeded |
data.url | string | Final URL after redirects |
data.statusCode | number | HTTP status code the target page returned |
data.content | string | The page (or the selector match) in the requested format, captured after the scenario ran. Empty string if selector matched nothing |
data.format | string | The format used: markdown, html, or text |
data.evaluateResults | array | One result per evaluate instruction, in order. Only present when the scenario has evaluate steps |
data.jsScenarioReport | object | Per-step report: total_duration (seconds), task_executed, task_success, task_failure, and tasks[] with each step's task, params, duration, and success. Only present when js_scenario was sent |
Caching
Results are cached for 1 hour per URL and option set (including the scenario). Repeating the same request within that window returns the cached page.
Errors
| Status | Code | When |
|---|---|---|
400 | VALIDATION_ERROR | url is missing, invalid, or private; format / device is invalid; wait_for is over 15000; js_scenario is malformed, too long, or has an invalid instruction; wait_for_selector never matched an element; country was sent (stealth only); or the URL is a Google page (use /v1/serp/google) |
404 | NOT_FOUND | The target page returned 404 Not Found |
429 | RATE_LIMITED_UPSTREAM | Too many scrapes in flight — retry in a few seconds |
502 | UPSTREAM_ERROR | The site blocked the request or failed to load — retry, or use /v1/scrape/stealth |
503 | UPSTREAM_TIMEOUT | The page (plus scenario) took too long — about 25 seconds |
Failed calls are not charged.
Related Endpoints
- Web Scraping API — cheaper smart-mode scrape for pages that do not need scripting
- Stealth Scrape API — the same browser options on stealth residential proxies
- Screenshot API — capture the rendered page or one element as a PNG
Standard Scrape
Web Scraping API — scrape any URL to clean Markdown, HTML, or plain text. Smart mode escalates to JS rendering and premium proxies automatically, one flat $0.01 per call.
Stealth Scrape
Stealth Scraping API — scrape heavily protected sites with headless Chrome on stealth residential proxies, with country targeting and browser scripting. $0.03 per call.