Web Scraping API
Web Scraping API — scrape any URL to clean Markdown, HTML, or plain text. Smart mode escalates to JS rendering and premium proxies automatically, one flat $0.01 per call.
Pricing, live playground and use cases: Web Scraping API on YepAPI
/v1/scrapeThe Web Scraping API fetches any public URL and returns the page as clean Markdown (ideal for LLMs and RAG), raw HTML, or plain text. It runs in smart mode: each request starts with the cheapest fetch that can work and automatically escalates — to a headless browser with JavaScript rendering, then to premium residential proxies — until the page loads. You always pay one flat $0.01, whatever it took.
Need to click, scroll, or wait for an element before the page is captured? Use the JavaScript Scrape API. For sites behind heavy anti-bot protection, use the Stealth Scrape API.
Usage
const res = await fetch('https://api.yepapi.com/v1/scrape', {
method: 'POST',
headers: {
'x-api-key': 'YOUR_API_KEY',
'Content-Type': 'application/json',
},
body: JSON.stringify({ url: 'https://example.com', format: 'markdown' }),
});
const { data } = await res.json();
console.log(data.content);curl -X POST https://api.yepapi.com/v1/scrape \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "format": "markdown"}'import requests
res = requests.post(
"https://api.yepapi.com/v1/scrape",
headers={"x-api-key": "YOUR_API_KEY"},
json={"url": "https://example.com", "format": "markdown"},
)
print(res.json()["data"]["content"])Request Body
| Parameter | Type | Required | Description | Default |
|---|---|---|---|---|
url | string | Yes | Full http(s) URL of the page to scrape (max 4096 characters). Localhost and private IPs are rejected | — |
format | string | No | Output format. One of: markdown, html, text | markdown |
selector | string | No | CSS or XPath selector. Returns only the first matching element — its HTML when format is html, otherwise its text (max 500 characters) | — |
wait_for | number | No | Extra milliseconds to wait after the page loads, max 15000 | — |
waitFor is accepted as an alias of wait_for.
Browser-only options — js_scenario, wait_for_selector, device, and block_ads — are not accepted here because smart mode may answer from a plain HTTP fetch. Sending one returns a 400 that points you to /v1/scrape/js. country is only accepted on /v1/scrape/stealth.
Response
{
"ok": true,
"data": {
"url": "https://example.com/",
"statusCode": 200,
"content": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
"format": "markdown"
}
}Response Fields
| Field | Type | Description |
|---|---|---|
ok | boolean | Whether the request succeeded |
data.url | string | Final URL after redirects |
data.statusCode | number | HTTP status code the target page returned |
data.content | string | The page (or the selector match) in the requested format. Empty string if selector matched nothing |
data.format | string | The format used: markdown, html, or text |
Smart Mode
/v1/scrape picks the cheapest way to load each page:
- A plain HTTP fetch — fast, and enough for most blogs, docs, and static pages.
- If the page needs JavaScript or the fetch is blocked, a headless Chrome render.
- If it is still blocked, premium residential proxies.
Escalation happens inside the same request; the price stays $0.01. If the site still refuses every attempt, you get a 502 UPSTREAM_ERROR and are not charged — retry, or switch to Stealth Scrape for heavily protected sites.
Caching
Results are cached for 1 hour per URL and option set. Repeating the same request within that window returns the cached page.
Errors
| Status | Code | When |
|---|---|---|
400 | VALIDATION_ERROR | url is missing, not a full http(s) URL, or points to localhost / a private IP; format is invalid; wait_for is over 15000; a browser-only option or country was sent; or the URL is a Google page (use /v1/serp/google for Google results) |
404 | NOT_FOUND | The target page returned 404 Not Found |
429 | RATE_LIMITED_UPSTREAM | Too many scrapes in flight — retry in a few seconds |
502 | UPSTREAM_ERROR | The site blocked the request or failed to load — retry, or use /v1/scrape/stealth |
503 | UPSTREAM_TIMEOUT | The page took too long to load (about 25 seconds) — retry |
Failed calls are not charged.
Related Endpoints
- JavaScript Scrape API — headless Chrome with clicks, scrolling, and custom JavaScript
- Stealth Scrape API — stealth residential proxies and country targeting
- AI Extraction API — describe the data you want and get JSON back
Place Photos
Resolve a Google Maps photo reference into a real, fetchable image URL — sized up to 4800px, sourced live from the official Google Place Photos endpoint.
JavaScript Scrape
JavaScript Scraping API — scrape JS-rendered pages with headless Chrome. Click, scroll, fill forms, wait for elements, and run custom JavaScript before capture. $0.02 per call.