Schedules & Runs API

A read-only API for your organization’s schedules (your cron-scheduled spiders) and runs (every job, with its metrics) — the same data the dashboard shows, available to your own systems and to your spiders.

Use it to build dashboards, alerting, or a metrics-export spider that ships run stats to your own warehouse.

Every credential here is scoped to your organization and grants read access only. You see your own schedules and runs, never anyone else’s, and nothing here can start, stop, or change a spider.

Authentication

Two ways to authenticate, depending on where you’re calling from.

From your own systems — use a deploy API key. Create one in Deploy → API Keys (it’s shown once, starts with sp_). Send it as a bearer token:


curl -H "Authorization: Bearer sp_your_key" \
  "https://<your-arachne-host>/api/v1/client/schedules"

X-API-Key: sp_your_key works too. The key returns all of your organization’s schedules and runs; narrow it with ?project= when you want one project.

From inside a spider — no setup. Every job is handed a short-lived, org-scoped, read-only token and the base URL as environment variables:


import os, urllib.request, json
 
base = os.environ["INSIGHT_METRICS_API_URL"].rstrip("/")
token = os.environ["INSIGHT_METRICS_API_TOKEN"]
req = urllib.request.Request(
    f"{base}/api/v1/client/jobs?limit=20",
    headers={"Authorization": f"Bearer {token}"},
)
runs = json.load(urllib.request.urlopen(req, timeout=30))["jobs"]

See Inside a Job for the full list of injected variables.

Endpoints

All endpoints are GET under /api/v1/client and return JSON with a "status": "ok" field.

Endpoint	Returns
`GET /api/v1/client/schedules`	Your schedules (`schedules[]`, `total`).
`GET /api/v1/client/schedules/{id}`	One schedule (`schedule`).
`GET /api/v1/client/schedules/{id}/executions`	A schedule’s recent fires (`executions[]`).
`GET /api/v1/client/jobs`	Runs, newest first (`jobs[]`, `total`).
`GET /api/v1/client/jobs/{id}`	One run (`job`).
`GET /api/v1/client/jobs/{id}/stats`	A run’s detailed stats (`stats`, or `null` until it finishes).
`GET /api/v1/client/jobs/stats`	Run counts by status.
`GET /api/v1/client/metrics/dashboard`	Counts + active schedules + online nodes.

Query parameters

schedules and jobs are paginated and sortable:

Parameter	Applies to	Notes
`limit`, `offset`	both	Page size (default 50, no hard cap) and offset. Responses include `total`.
`project`	both	Exact project name.
`spider`	jobs	Substring match on the spider name.
`status`	jobs	Exact run status: `pending` / `running` / `finished` / `error` / `cancelled`.
`search`	both	Substring over the spider name (and, on either endpoint, the schedule name).
`sort`, `dir`	both	Sort column (see below) + `asc` / `desc`.

Sortable columns — schedules: next_run_at, last_run_at, run_count, name, project, spider, status, cron_expression, timezone, created_at. jobs: created_at (default, newest first), start_time, end_time, items_scraped, pages_scraped, error_count, status, priority, project, spider, schedule_name.

The schedules endpoint has no status filter (only jobs does). To see just active or just paused schedules, filter client-side on the status field — see the note under The next scheduled spiders.

The next scheduled spiders

Sort schedules by next_run_at ascending to get what’s coming up:


GET /api/v1/client/schedules?sort=next_run_at&dir=asc&limit=30


{
  "status": "ok",
  "total": 42,
  "schedules": [
    {
      "id": 7,
      "name": "Nightly catalogue",
      "project": "shop",
      "spider": "catalogue",
      "cron_expression": "0 3 * * *",
      "timezone": "America/Mexico_City",
      "status": "active",
      "next_run_at": "2026-07-08T09:00:00Z",
      "last_run_at": "2026-07-07T09:00:00Z",
      "run_count": 128
    }
  ]
}

Each schedule also carries args, settings, tags, priority, version, jitter_seconds, misfire_grace_seconds, coalesce, max_instances, retry_on_failure, retry_count, start_date, end_date, and pause_reason.

⚠️

Paused schedules can sort first. A paused schedule keeps its last next_run_at — a time in the past that doesn’t advance while paused. So sorting by next_run_at ascending can surface paused schedules ahead of the ones that will actually fire. Since the endpoint has no status filter, keep only status == "active" client-side to get “what will run next”:


page = get("/api/v1/client/schedules", sort="next_run_at", dir="asc", limit=200)
upcoming = [s for s in page["schedules"] if s["status"] == "active"][:30]

Recent runs & their metrics

Runs come back newest-first by default:


GET /api/v1/client/jobs?limit=20


{
  "status": "ok",
  "total": 5310,
  "jobs": [
    {
      "id": "b3f1a2c4-...",
      "project": "shop",
      "spider": "catalogue",
      "status": "finished",
      "items_scraped": 12840,
      "pages_scraped": 1310,
      "error_count": 2,
      "warning_count": 5,
      "start_time": "2026-07-07T09:00:03Z",
      "end_time": "2026-07-07T09:14:51Z",
      "finish_reason": "finished",
      "schedule_id": 7,
      "schedule_name": "Nightly catalogue"
    }
  ]
}

status is one of pending, running, finished, error, cancelled. The per-run counters (items_scraped, pages_scraped, error_count, warning_count) update live while a job runs and are final once it’s finished. schedule_name is present when the run came from a schedule.

Each run returns only these client-relevant fields (id, project, spider, version, status, priority, args, settings, tags, the counters, start_time, end_time, finish_reason, error_message, schedule_id, schedule_name, created_at, updated_at) — internal execution details are not exposed.

For the deeper breakdown of a single run — response codes, retries, dropped items, runtime — call its stats:


GET /api/v1/client/jobs/{id}/stats


{
  "status": "ok",
  "stats": {
    "runtime_seconds": 888.4,
    "items_scraped": 12840,
    "items_dropped": 3,
    "pages_crawled": 1310,
    "response_200": 1290,
    "response_404": 12,
    "response_429": 4,
    "response_500": 0,
    "retry_count": 18,
    "error_count": 2,
    "finish_reason": "finished"
  }
}

stats is null until a run finishes (the detailed row is written at job end). While a job is running, read the live counters from the run object itself (/jobs/{id}).

Build a metrics-export spider

Put the two together to export everything on a schedule of your own: a spider that reads your schedules + runs and writes them wherever you want. A working end-to-end example ships in the reference project as metrics_export_demo — it logs the next 30 scheduled spiders and the last 20 runs with their metrics. The core of it:


import os, urllib.request, urllib.parse, json
import scrapy
 
class MetricsExportSpider(scrapy.Spider):
    name = "metrics_export"
 
    def __init__(self, *a, **k):
        super().__init__(*a, **k)
        self.base = os.environ["INSIGHT_METRICS_API_URL"].rstrip("/")
        self.token = os.environ["INSIGHT_METRICS_API_TOKEN"]
 
    def get(self, path, **params):
        url = self.base + path
        if params:
            url += "?" + urllib.parse.urlencode(params)
        req = urllib.request.Request(url, headers={"Authorization": f"Bearer {self.token}"})
        return json.load(urllib.request.urlopen(req, timeout=30))
 
    async def start(self):
        upcoming = self.get("/api/v1/client/schedules", sort="next_run_at", dir="asc", limit=30)
        for s in upcoming["schedules"]:
            self.logger.info("next: %s %s @ %s", s["spider"], s["cron_expression"], s["next_run_at"])
 
        runs = self.get("/api/v1/client/jobs", limit=20)
        for r in runs["jobs"]:
            self.logger.info("run: %s %s items=%s errors=%s",
                             r["spider"], r["status"], r["items_scraped"], r["error_count"])
            # …ship `r` to your own warehouse / dashboard here.
        return
        yield  # marks start() as an async generator

Because the token is injected automatically, the same spider works unchanged on any schedule — point it at your destination in settings.py (see Getting your data out).