Skip to Content
CrawlersDeploying a Project

Deploying a Project

A project is a named home for one Scrapy codebase. You create it once, deploy your code into it, and the platform builds that code into a runnable image. This page takes a plain Scrapy project to a first successful build.

1. Create a project

In the Projects tab, create a project and give it a name (letters, digits, ., - and _, up to 64 characters). It starts empty — code arrives in the next step.

2. Meet the runtime contract (your Dockerfile)

The platform builds your project from a Dockerfile at the root of your repo — so you choose the base image, Python version, and dependencies. Your image only has to honor a few rules the runtime enforces:

  • It runs as a non-root user (uid 65534, “nobody”).
  • The root filesystem is read-only — the only writable path is /tmp.
  • Your default command should be scrapy list (used to discover your spiders). The platform overrides it with scrapy crawl <spider> when it runs a job.
  • Never bake secrets into the image — inject them at runtime instead (below).

A minimal, correct Dockerfile:

FROM python:3.12-slim
ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=/app/src
WORKDIR /app
 
RUN pip install --no-cache-dir "scrapy>=2.12,<3"
 
COPY scrapy.cfg ./
COPY src/ ./src/
 
# Runs as a non-root user on a read-only filesystem; only /tmp is writable.
USER 65534:65534
 
# Spider discovery. The platform overrides this with `scrapy crawl <spider>`.
CMD ["scrapy", "list"]
⚠️

Because the filesystem is read-only, anything your spider writes at runtime — caches, temp files, a browser’s profile dir, feed buffers — must live under /tmp. A spider that tries to write elsewhere will fail with a permissions error.

Your project is otherwise an ordinary Scrapy project — a scrapy.cfg pointing at your settings module, and your package under src/ (or wherever your Dockerfile copies it).

3. Deploy your code

Open the Deploy tab. It gives you everything to ship code, generated for your specific project:

  1. Deploy key — generate one; the private key is shown once, so copy it then. It authorizes uploads into this project only.
  2. CI workflow — the tab hands you a ready-to-paste GitHub Actions workflow that ships your repository to the platform on every push (or tag). Add it to your repo along with the secrets the instructions list.
  3. Push — commit and push. Each push uploads your project directory (with the Dockerfile), and the platform builds it automatically a few seconds after the upload settles.

Not on GitHub? The deploy is just an authenticated file upload using the deploy key — the generated workflow is a convenience, not a requirement. Any CI that can ship your directory with the key works.

⚠️

Don’t upload a .env file. Runtime configuration comes from secrets (below), not from files in your repo. A committed .env won’t reach your running spider and shouldn’t be shipped.

4. Set your secrets

Anything sensitive your spider needs at runtime — proxy credentials, a database DSN, an API key for your data destination — is stored as a secret on the project, not baked into the image. Manage secrets in Deploy → Secrets (or via the API for automation).

  • Names are uppercase letters, digits, and _ (e.g. INSIGHT_PROXY_USERNAME).
  • Secrets are write-only — you can set and replace them, but never read them back — and are encrypted at rest.
  • At run time each secret is injected into your spider’s container as an environment variable. Read them with os.environ.
  • A few names the platform injects itself (such as SCRAPY_JOB) are reserved.

See Inside a Job for the full list of what’s available in the environment at run time.

5. Enable add-ons (optional)

On each project you can enable the capabilities your spiders need. Enabling an add-on makes the platform inject its access into every job automatically:

Add-onWhat it gives your spider
playwright, nodriver, camoufox, insightbrowserA managed browser to connect to — BROWSER_WS_URL_* + a BROWSER_TOKEN are injected.
postgresA ready-to-use database — a POSTGRES_URL DSN is injected.

With no add-ons, your project builds from a slim base with no extras. Add-ons are set per project (in Projects, or the project’s add-on settings).

6. Builds and versions

  • Every deploy (and manual Rebuild) produces a build; a successful build becomes a version you can run. Build logs are visible per project.
  • Your spiders are discovered from the built image, so a new spider appears once its build succeeds.
  • The platform keeps your most recent builds and prunes older images; when you run a spider, “Latest” always means the newest successful build.
  • Builds have a 15-minute limit — keep image builds lean (pin dependencies, lean base image) so they finish comfortably inside it.

Once you have a green build, head to Running & Scheduling.

Last updated on
OverviewRunning & Scheduling