Media Fetch

Scraping a page gives you image URLs — but downloading those images from your own servers often gets blocked (wrong IP reputation, missing browser headers, hotlink protection). The Media Fetch API solves that: send it an image URL, and it returns the image bytes, fetched through the same managed network your proxies use, with anti-blocking and automatic retries handled for you.

Quick start

Download an image and save it to a file:

curl -u USERNAME:PASSWORD \
  "https://media.insightscrap.com/v1/fetch?url=https%3A%2F%2Fexample.com%2Fproduct.jpg" \
  -o product.jpg

You get the raw image bytes back, with the original Content-Type. The url query parameter must be URL-encoded.

Authentication

Send your proxy username and password as standard HTTP Basic credentials in the Authorization header (this is -u user:pass in curl). These are the same credentials you create under Proxy Users in the dashboard — the same ones you use for the proxy gateway.

• Invalid credentials return 401.
• Usage is metered and billed to your organization, alongside your other services in analytics.

The endpoint

GET  https://media.insightscrap.com/v1/fetch?url=<encoded-url>
POST https://media.insightscrap.com/v1/fetch     (same fields as a JSON body)

Use POST with a JSON body when your URL is very long or awkward to encode.

Responses

Binary (default)

The raw image bytes, streamed back with the upstream Content-Type. Useful response headers:

Responses are served with Content-Disposition: attachment and a strict Content-Security-Policy, so an image is never rendered inline in a browser tab — it’s data for your pipeline, not a page.

JSON

Add format=json (or send Accept: application/json) to get an envelope with the image base64-encoded — handy when you’re embedding the image directly into a JSON pipeline:

{
  "data": "iVBORw0KGgoAAAANSUhEUg...",
  "content_type": "image/png",
  "bytes": 8090,
  "duration_ms": 312,
  "truncated": false
}

Geo-targeting

Pass country to fetch the image as if from that country — useful when a CDN serves region-specific media or blocks foreign traffic.

curl -u USERNAME:PASSWORD \
  "https://media.insightscrap.com/v1/fetch?url=<encoded>&country=mx" \
  -o product.jpg

Supported countries: us, mx, br, ar (case-insensitive). An unsupported code returns 400. Country targeting applies when a request needs a residential IP; the fastest path is a single US pool, so a plain fetch always tries there first regardless of country and only applies your country when it escalates.

Handling blocks

You don’t manage any of this — it’s the point of the service. For each request we make a best effort from cheapest to most robust, and you’re billed the same flat amount no matter how many attempts it takes:

1. A fast datacenter fetch with realistic browser headers and TLS fingerprint.
2. A residential IP in your chosen country, if the first attempt is blocked.
3. Only if you pass allow_browser=true — a real, stealth browser that executes JavaScript, for CDNs behind a bot-challenge (“checking your browser…”).

The retries are invisible to you: you get one clean result, or one clean error.

Billing

Media Fetch is billed per request, at a flat rate — the same whether the image is 5 KB or 5 MB, and no matter how many internal attempts we made to deliver it. A request that we ultimately could not deliver (a 502, below) is not charged. Spend rolls up under Media in your analytics alongside your other services.

Errors

Errors are returned as JSON: { "error": "<code>", "message": "..." }.

Limits

• Maximum image size is 25 MB; larger downloads are cut short and flagged with the X-Insightscrap-Truncated trailer.
• Each attempt has a short timeout; a full request (including the optional browser path) is bounded to keep latency predictable.

Examples

Save an image (binary):

curl -u USERNAME:PASSWORD \
  "https://media.insightscrap.com/v1/fetch?url=$(python3 -c 'import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1]))' 'https://example.com/a.jpg')" \
  -o a.jpg

Python (binary):

import requests
from urllib.parse import quote
 
def fetch_image(image_url, *, country=None, allow_browser=False):
    params = {"url": image_url}
    if country:
        params["country"] = country
    if allow_browser:
        params["allow_browser"] = "true"
    r = requests.get(
        "https://media.insightscrap.com/v1/fetch",
        params=params,
        auth=("USERNAME", "PASSWORD"),
        timeout=60,
    )
    r.raise_for_status()
    return r.content, r.headers["Content-Type"]
 
data, ctype = fetch_image("https://example.com/product.jpg", country="mx")
open("product.jpg", "wb").write(data)

In a Scrapy pipeline — turn scraped image URLs into downloaded files without your crawler’s IP ever touching the target:

import requests
 
class MediaFetchPipeline:
    def open_spider(self, spider):
        self.session = requests.Session()
        self.session.auth = ("USERNAME", "PASSWORD")
 
    def process_item(self, item, spider):
        if url := item.get("image_url"):
            resp = self.session.get(
                "https://media.insightscrap.com/v1/fetch",
                params={"url": url},
                timeout=60,
            )
            if resp.ok:
                item["image_bytes"] = resp.content
                item["image_type"] = resp.headers["Content-Type"]
        return item