Running & Scheduling

Once a project has a successful build, you can run its spiders — one-off from the dashboard, or on a recurring schedule. Every run is an isolated container.

Run a spider

In the Run Spider tab, pick:

Project → Version (defaults to Latest — the newest successful build) → Spider.
Priority — higher-priority jobs are pulled from the queue first.
Arguments (-a) — spider arguments passed to your spider’s __init__ (e.g. country=mx, start_page=1).
Settings (-s) — per-run Scrapy setting overrides (see the allowlist below).

That’s equivalent to running:


scrapy crawl <spider> -a country=mx -s DOWNLOAD_DELAY=1.0

The job enters the queue, a worker picks it up, and it runs in its own container. You can also start a run programmatically instead of clicking:


POST /api/v1/jobs
Content-Type: application/json
 
{ "project": "my-project", "spider": "my-spider",
  "args": { "country": "mx" }, "settings": { "DOWNLOAD_DELAY": "1.0" } }

What you can override per run

Arguments (-a) are spider-specific and free-form — they’re your spider’s own parameters.

Settings (-s) are restricted to a safe allowlist — the knobs that tune how a crawl runs, not where its traffic goes. Allowed settings include:

Throughput & politeness: CONCURRENT_REQUESTS, CONCURRENT_REQUESTS_PER_DOMAIN, DOWNLOAD_DELAY, AUTOTHROTTLE_*, DEPTH_LIMIT
Retries & timeouts: RETRY_ENABLED, RETRY_TIMES, DOWNLOAD_TIMEOUT
Stop conditions: CLOSESPIDER_ITEMCOUNT, CLOSESPIDER_PAGECOUNT, CLOSESPIDER_TIMEOUT, CLOSESPIDER_ERRORCOUNT
Behavior: ROBOTSTXT_OBEY, COOKIES_ENABLED, USER_AGENT, LOG_LEVEL, HTTPCACHE_*

⚠️

Settings that reroute traffic or load code — *_PROXY, DOWNLOADER_MIDDLEWARES, ITEM_PIPELINES, EXTENSIONS, TELNETCONSOLE_* — cannot be set per run. These belong in your project’s settings.py, which you deploy with your code. It keeps every run’s proxy routing, pipelines, and output config fixed and auditable.

Controlling a running job

From a job’s detail view (or the API) you can:

Cancel — a graceful stop (Scrapy shuts down cleanly, finishing in-flight work).
Force stop — an immediate kill.
Reschedule — re-run the same spider with the same parameters.

Schedules (cron)

Open Schedules to run a spider recurrently. Each schedule targets one project + spider (version defaults to Latest at fire time, so schedules pick up new builds automatically).

Build the timing two ways:

Basic — point-and-click presets: every N minutes/hours, daily, weekly, or monthly.
Expert — a raw 5-field cron expression (minute hour day-of-month month day-of-week) with a live human-readable preview, in the timezone you pick.


0 3 * * *      →  every day at 03:00
*/30 * * * *   →  every 30 minutes
0 6 * * 1      →  Mondays at 06:00

Each schedule also supports the knobs you’d expect for reliable recurring runs:

Option	What it does
Timezone	Interpret the cron in your local time, DST-aware.
Jitter	Spread fires by a random offset so many schedules don’t all fire on the same second.
Max instances	Cap how many runs of this schedule may overlap.
Coalesce	If runs were missed (downtime), collapse them into one catch-up run instead of a burst.
Misfire grace	How late a missed fire may still run before it’s skipped.
Retry on failure	Automatically retry a failed run, with a count and delay.
Start / end date	Bound the window in which the schedule is active.

You can pause, resume, or fire now any schedule, and view its recent executions.

Concurrency & the queue

Jobs run through a shared worker pool, pulled in priority order — so a burst of runs queues and drains rather than overwhelming anything. If you need many spiders running at once, stagger them with schedule jitter and use priority to make time-sensitive runs jump the line.

Next: what your spider can reach once it’s running — see Inside a Job.