Crawling 70,722 artworks past a firewall

Color Walk lets you pick a colour and browse the public-domain art that shares it. Neither the Met nor the Cleveland Museum of Art stores a colour field, so I had to measure it myself: download every thumbnail, find its dominant colours, and write a static index the site can read without a backend.

The measuring turned out to be easy. Getting the images was not.

The documented limit wasn't the real one

The Met's Collection API documents a generous 80 requests per second. In practice a firewall sits in front of it, and after roughly 75 quick requests it started refusing me outright, regardless of the documented number.

So the crawler slowed down to about 1.25 requests per second, and when refused it backs off for 90 seconds before trying again. At that pace the Met alone took around 16 hours of requests, for about 89,800 thumbnails across both museums.

Search stops at 10,000

The Met's search endpoint quietly stops returning results past an offset of 10,000. Large departments like European Paintings have more objects than that, so relying on search alone would have silently dropped thousands of works.

The fix was to take the union of two sources of object IDs: the search results and the uncapped per-department listing. Cleveland's open-access API had no such cap and could be filtered to cc0=1&has_image=1 directly.

A crawl that can run for 16 hours

A 16-hour crawl will be interrupted: sleep, network drops, me pressing Ctrl+C. Everything about the crawler is built so that stopping is cheap:

• Append-only cache. Every raw record goes into a JSONL file as it arrives. My first version rewrote one big JSON file on every save; at 148 MB per save that would have meant about 93 GB of writes over the run. Appending a line costs nothing, and a restart simply skips IDs already in the file.
• Timeouts on everything. Once I found the process hung for an hour with sockets stuck in SYN_SENT. Now every request has an AbortSignal timeout: 30 seconds for JSON, 60 for images.
• Pause, don't die. Ctrl+C pauses immediately and saves state; a --minutes flag time-boxes a run.
• One crawler at a time. A pid lock stops two runs from writing the same cache.
• Say what you're doing. A heartbeat every 5 seconds lets a crawl:status command report running, stalled, silent or not running, which beats guessing from a terminal that stopped scrolling.

Simplified, a single request looks like this:

const res = await fetch(url, { signal: AbortSignal.timeout(30_000) })
if (res.status === 403) {
    await sleep(90_000) // refused by the firewall: back off, then retry the same id
    return fetchObject(id)
}

Only keep what you're allowed to show

Licensing is enforced in code, not by hand. A normaliser turns each raw record into an artwork or into nothing:

• Met records must have isPublicDomain === true; Cleveland records must have share_license_status === 'CC0'.
• Image and page URLs must be https and on an allowlist of museum hosts.
• Every string is capped at 300 characters.

It returns null instead of throwing, so one odd record can't stop a 16-hour run. It dropped 22,542 records, which is exactly the point.

Measuring colour

Each thumbnail is shrunk to 48 pixels on its long edge with sharp and converted to HSL. Pixels that are nearly black, nearly white or unsaturated are ignored as canvas, varnish or frame. The rest are counted into 24 hue buckets weighted by saturation, with circular means so reds near 0° and 360° land together.

Every work is filed under each colour it holds, so 70,722 works become 128,799 index entries, written as 600-item pages of about 330 KB each. The site fetches a page only when you scroll to it.

What I'd do again

• Measure the real rate limit first, then design the crawl around it.
• Append, never rewrite, anything a long job saves as it goes.
• Give every network call a timeout; a hang is worse than a failure.
• Make "is it still running?" answerable without reading logs.
• Encode licensing rules in the pipeline so the output is clean by construction.