Bot control

A discovery API returns structured, complete catalog data — prices, stock levels, assortment — on demand, which makes it a target for scrapers. Several independent layers sit in front of the API, each catching what the ones above it miss.

The layers

Layer	Stops
Edge protection	Volumetric attacks and large botnets
Proof of work	Cheap, high-volume crawling
Connection fingerprinting	Clients that are not browsers
Behavioural checks	Sessions that do not behave like shopping
Model scoring	Whatever survives the layers above

Edge protection. API traffic is routed through a CDN that absorbs denial-of-service traffic and known botnets. It handles volumetric cases and does not stop small, low-rate scrapers.

Proof of work. A browser has to compute a short-lived token before it can call the API. Paid once per page load it is imperceptible to a shopper; paid per request at crawling volume it is slow and expensive. It is a deterrent, not an authorization mechanism: it does not identify the caller, it prices bulk access. See What the proof of work costs.

Connection fingerprinting. The properties of the TLS connection identify the kind of client making it, so a script using an HTTP library is recognized as not a browser and refused. It fingerprints the technology making the connection, not the person using it, and is not user fingerprinting in the sense the ePrivacy Directive governs.

Behavioural checks. A shopping session is coherent: people type, see suggestions, click and navigate. A session that does not behave that way is treated as automated, whatever its connection looks like. These checks read a session as a whole, so a single request never triggers one.

Model scoring. A model trained on session behaviour, and retrained on an ongoing basis, catches what the fixed layers do not.

Each layer is individually beatable. They combine so that defeating all of them costs more than the data is worth to most scrapers.

What the proof of work costs

The Luigi’s Box integration library solves the challenge once, when it loads — before the first search and before the first analytics event — in a Web Worker, so it does not compete with page rendering. The worker ships inside the library bundle rather than as a separate request, so it adds nothing to the critical path. A storefront running the library gets this without configuring anything.

Build a custom integration the same way: solve on page load rather than on the first search, run the solver in a Web Worker, and cache the token instead of minting one per request.

Measured solve times, at the difficulty Luigi’s Box runs in production:

Device	Median	95th percentile
Current laptop or desktop	35 ms	160 ms
Mid-range phone	130 ms	640 ms
Low-end phone	370 ms	870 ms

Connection speed does not change these figures: they are CPU time on a single core. The exchange around them costs two sequential round trips to the auth host — one for the challenge, one for the token — and on a high-latency mobile connection those can add more than the solve does. Both are paid once, while the page is loading.

A token is valid for five minutes, and the library refreshes it in the background at 80% of that. A shopper who lands on a page and then searches never waits on the challenge, and a long visit never pays it again part-way through.

Difficulty is a per-account setting and solve time scales geometrically with it. If the cost you measure on your own device mix is higher than you want, ask Luigi’s Box to lower it.

If a device takes longer to solve than the challenge’s validity window, the token request is refused, and the integration library fetches a fresh challenge and retries once. At the figures above this is rare, and it costs a second challenge-and-token exchange.

What this means for your integration

A normal storefront passes every layer with no special handling. Four cases need attention.

Browser code solves the challenge; server code does not

The proof of work belongs to the browser flow. A server-side integration authenticates with an OAuth client ID and secret instead, and never touches a challenge.

The two are not interchangeable. A backend script that uses a publishable key and the browser flow is refused by connection fingerprinting, because it is indistinguishable from a scraper using a stolen key. If your code runs on a server, use server-to-server tokens.

Where your code runs	Flow
A shopper’s browser	Publishable key + proof of work
Your server	OAuth client credentials

Mint the token early

Solve the challenge while the page is loading, not when the shopper starts typing. Paid in advance the cost is invisible; paid on the first keystroke it is a visible delay. See A reusable token provider.

Reporting behaviour helps you here too

The behavioural layer reads a session as a whole. An integration that searches but never reports what shoppers clicked produces sessions indistinguishable from automated querying.

Reporting clicks, cart additions and purchases improves ranking and also marks your traffic as shopper traffic.

Register your origins, and keep keys per storefront

A publishable key is bound to the origins you register and to one channel. Keep one key per storefront and register every origin it is served from, staging included. A key used from an unregistered origin is refused, so a key copied from your page source is useless elsewhere.

Crawlers and comparison engines

Bot control only applies to pages whose content comes from Luigi’s Box: a search results page, and a category listing if you build one from the Discovery API. Product pages, the category pages you render yourself and the feeds comparison engines read are served by you, so a crawler never meets any of these layers there. There is no crawler allowlist because that traffic never needs one.

On a page that does come from Luigi’s Box, a crawler that executes JavaScript runs the same browser flow a shopper does, challenge included.

Legitimate automation

Not every non-browser client is hostile. Monitoring, uptime checks, internal test suites, your own data exports and approved partner integrations all have reasons to call the API without a browser.

In every case, authenticate as a service with client credentials, scoped to what that integration needs. A service credential tells the API that the caller is not a browser and is not meant to be one, so connection fingerprinting does not refuse it. Automation that instead disguises itself as a browser is indistinguishable from a scraper using a stolen key, and is refused on that basis.

Authorization here is by credential, not by IP address. A credential travels with the system that holds it, while the addresses monitoring and CI run from change without notice.

If a case fits neither flow, contact Luigi’s Box.

When traffic is refused

Bot control failures surface as 403. There is no CAPTCHA and no interstitial: a refusal is an ordinary failed API call, so what a shopper sees is whatever your storefront does when a search request fails. Handle it as you would any other failure.

A refusal from the behavioural layer is scoped to one session — not to an IP address, a device, or a shopper across sessions — and expires by itself after one hour. Nothing accumulates between sessions, and there is no manual unblock to wait for.

The detection pipeline fails open. When it is degraded or slow, requests are served rather than refused.

Common causes:

Symptom	Likely cause
403 on /browser-token, mentioning origin	The page’s origin is not registered for the key
403 on /browser-token from server-side code	Connection fingerprinting — use client credentials instead
403 mentioning the nonce or the address	The challenge was reused, or redeemed from a different address
Transient 403 right after a DNS or CDN change	The challenge and the token call must resolve to the same address; retry once

Report sustained 403s on legitimate traffic with the X-Request-Id of a failing response; it identifies which layer refused the request.

Rate limiting is a separate mechanism with its own headers and a 429; see Errors and rate limits.

What is recorded

While a session is active, Luigi’s Box holds a small set of counters describing the shape of that session — how much it asked for and how it was paced, never what it asked for. These expire one hour after the session’s last request.

Counters from finished sessions are archived for tuning the layers above. No request content is recorded — no IP address, no user agent, no query text, no cookie value, no product identifiers and no shopper identifier. The session key is a one-way hash, so an archived row is pseudonymous, and Luigi’s Box retains nothing that would re-identify it or reconstruct what was searched for.

Archived counters are stored in the region your account runs in, encrypted at rest, and deleted after 90 days.

See also

• Browser tokens — the proof-of-work flow in detail
• Server-to-server tokens — the flow for automation
• Sending events — reporting behaviour
• Errors and rate limits