How to Turn Any Website into an API: cURL, Python & Webhooks (2026)

Most of the data you want lives on websites that will never ship an API. Pricing pages, government portals, inventory pages, competitor changelogs: the information is right there in the HTML, and there is no endpoint to call.

You have three realistic ways to get that data into your code. You can build a scraper, you can pay a scraping API to extract structured data on demand, or you can turn the page into a change feed that pushes JSON to you when something actually happens. This guide covers all three, then walks through the third approach end to end with working cURL and Python examples, the real response schema, webhook setup, and the rate limits you'll hit. Every request and response in this post was run against the live API in July 2026.

Scraper vs. Scraping API vs. Change Feed: Which One Do You Need?

The honest decision rule: if you need to extract thousands of pages into a dataset, use a scraping API like Firecrawl or Apify (we compare them in our web scraping API roundup). If you need to know when one page changes, and you want the diff rather than the whole page, a change feed is less code and less babysitting. That's what we're building here.

What You'll Build

One POST request turns any public URL into a monitored endpoint. Visualping checks the page on a schedule, runs visual and text diffing, and exposes every change as JSON: a percentage diff, a plain-English AI summary, and links to the before-and-after evidence. You can poll for changes with GET requests or have each change POSTed to your own endpoint as a webhook.

API access is included on every plan, including Free. There is no separate API credit meter; API usage draws on your plan's normal check quota.

Step 1: Create a Free API Key

Sign up at visualping.io/sign-up, then generate a key at visualping.io/account/developer. Every request authenticates with a Bearer header:

export VP_API_KEY="your_key_here"

Step 2: Turn the Page into a Monitored Endpoint (cURL)

Create a job. This example watches a pricing page every hour, targets the pricing table with a CSS selector, and uses a plain-language condition so you only get alerted when something you care about changes:

curl -X POST 'https://job.api.visualping.io/v2/jobs' \
  -H "Authorization: Bearer $VP_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com/pricing",
    "description": "Competitor pricing page",
    "interval": "60",
    "xpath": "#pricing-table",
    "summalyzer": {
      "importantDefinitionType": "custom",
      "importantDefinition": "A plan price changes or a new plan tier appears"
    }
  }'

Three details that will save you debugging time:

• interval is a string in requests, in minutes. "60" is hourly, "1440" is daily. (Responses return it as an integer. You get used to it.)
• The create response's id field is a confirmation token, not the job ID. It looks like "KFe5GPz0dEgwnht". The numeric job ID you'll use for every later call is in the jobid field, returned as a string.
• xpath accepts CSS selectors too. Scope the monitor to the element you care about and you'll cut false positives dramatically.

Business accounts also pass a workspaceId (get yours from GET /describe-user). Personal accounts can omit it.

Step 3: Read Your New API

List your jobs to grab the job ID:

curl 'https://job.api.visualping.io/v2/jobs' \
  -H "Authorization: Bearer $VP_API_KEY"

Then fetch the job with its change feed:

curl 'https://job.api.visualping.io/v2/jobs/8675309' \
  -H "Authorization: Bearer $VP_API_KEY"

The response is your page, as structured data. Key fields, trimmed for readability:

{
  "id": "8675309",
  "url": "https://example.com/pricing",
  "interval": 60,
  "active": true,
  "mode": "ALL",
  "next_run": "2026-07-27T18:00:00.000Z",
  "changes": [
    {
      "created": "2026-07-27T16:02:11.000Z",
      "mode": "ALL",
      "PercentDifference": 0.6369,
      "englishSummary": "The Pro plan price dropped from $49 to $39 per month.",
      "analyzerAlertTriggered": true,
      "htmlDiffUrl": "https://visualping.io/diff/1046594335?...",
      "thumb_diff_full": "https://s3.us-west-2.amazonaws.com/..."
    }
  ],
  "history": [
    {
      "created": "2026-07-27T15:00:04.000Z",
      "mode": "ALL",
      "PercentDifference": 0,
      "notification_send": false,
      "initial": false
    }
  ]
}

The fields that matter for building on top of this:

Step 4: The Same Thing in Python

A complete script: create the monitor, find its ID, and poll the change feed.

import os
import requests

API_KEY = os.environ["VP_API_KEY"]
BASE = "https://job.api.visualping.io/v2"
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}

# 1. Turn the page into a monitored endpoint
resp = requests.post(f"{BASE}/jobs", headers=HEADERS, json={
    "url": "https://example.com/pricing",
    "description": "Competitor pricing page",
    "interval": "60",
    "summalyzer": {
        "importantDefinitionType": "custom",
        "importantDefinition": "A plan price changes or a new plan tier appears",
    },
})
resp.raise_for_status()

# 2. The numeric job ID is in "jobid" (the "id" field is a token)
job_id = resp.json()["jobid"]

# 3. Read the change feed
job = requests.get(f"{BASE}/jobs/{job_id}", headers=HEADERS).json()
for change in job.get("changes", []):
    if change.get("analyzerAlertTriggered"):
        print(change["created"], "-", change["englishSummary"])

Run it on a schedule, or skip polling entirely with a webhook.

Step 5: Skip Polling, Get Pushed JSON (Webhooks)

Add a notification object to the job (at create time, or with an update) and Visualping POSTs structured JSON to your endpoint on every alert:

curl -X POST 'https://job.api.visualping.io/v2/jobs' \
  -H "Authorization: Bearer $VP_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com/pricing",
    "interval": "60",
    "notification": {
      "enableEmailAlert": false,
      "onlyImportantAlerts": true,
      "config": {
        "webhook": {
          "url": "https://your-app.com/hooks/visualping",
          "active": true,
          "notificationType": "slack",
          "channels": []
        }
      }
    }
  }'

(Yes, notificationType is "slack" even for a plain webhook. It's a quirk of the config schema; the delivery is a standard HTTP POST to your URL.)

The webhook payload is a 20-field JSON object. The fields you'll actually use: datetime, change (the diff), added_text, removed_text, summarizer (the AI summary), and important (whether your condition matched). Set onlyImportantAlerts to true and your endpoint only hears about changes that matched your plain-language condition.

Webhooks are available on every plan, and they're also how you connect Visualping to Zapier, n8n, or Make without writing a server. Setup details are in our Zapier integration guide.

Rate Limits and Quotas

There is no separate API rate card. Your plan's check quota is the limit, and API-created jobs draw from the same pool as jobs created in the app:

The math worth doing before you pick an interval: one page checked hourly consumes about 720 checks a month. On the Free plan, "1440" (daily) across a few pages fits comfortably; hourly on one page does not. Match the interval to how often the page actually changes, and let the importantDefinition condition handle the filtering.

Your Condition Is the Query Language

With a traditional API you filter data with query parameters. With a change feed, the filter is the plain-language condition on the monitor:

• "A plan price changes or a new plan tier appears"
• "A new RFP is posted with NAICS code 541511"
• "The API documentation adds or removes an endpoint"
• "The product goes back in stock"

Visualping AI evaluates every detected change against your condition, marks matches as important, and summarizes them in English. Your code reads analyzerAlertTriggered and englishSummary instead of diffing HTML. If you're building with AI agents, the same capability is exposed over MCP at visualping.io/mcp/sse, so ChatGPT and Claude can create and query monitors directly.

Frequently Asked Questions

How do I get data from a website that has no API? Three options: build a scraper (full control, high maintenance), use a scraping API for bulk extraction, or point a change-monitoring API at the page and consume the JSON change feed. For "notify my system when this page changes" use cases, the change feed is the least code: one POST request, then webhooks.

Can I turn a website into an API without writing code? Yes. Create the monitor in Visualping's web app instead of via the API, then add a webhook or Zapier connection to receive structured JSON. The Zapier route needs no server at all.

What's the difference between a web scraping API and a change-monitoring API? A scraping API extracts a page's content each time you call it; you get the full page as structured data and diff it yourself. A change-monitoring API watches the page for you and emits only the changes, with the diff and an AI summary already computed. Extraction problems want scrapers; "did anything change?" problems want change feeds.

Does API access cost extra on Visualping? No. Every plan, including Free, includes REST API access and webhook delivery. API usage draws on the same monthly check quota as the web app.

Related Resources:

• How to Monitor Website Changes with an API
• What 100,000+ Users Monitor with the Visualping API
• Top Web Scraping APIs for Extracting Data
• How to Use the Visualping Zapier Integration