Feeds

Feeds are batch exports of catalog data that Luigi’s Box AI downloads and indexes on a schedule. Use feeds when your platform already produces catalog files and you would rather not build a push integration. The Content API is the other way in, and is the better fit when your platform can emit what changed — pick one path per object type rather than mixing them.

How feed ingestion works

1. Export a full snapshot for each object type your catalog needs.
2. Host each feed at a stable URL.
3. Provide the URLs and source schema to your Luigi’s Box implementation contact or support.
4. Luigi’s Box configures the record path, object type, and source-to-catalog field mapping.
5. Luigi’s Box AI checks the registered feeds for changes and indexes valid snapshots.

Hosting a file does not register it automatically. Confirm the configuration and first successful import with your implementation contact before relying on scheduled updates.

Update frequency

Luigi’s Box AI checks registered feeds every 60 seconds. It uses a conditional HEAD request with ETag or Last-Modified; an unchanged feed is skipped without downloading it again.

Feed types

• Product feeds: searchable and recommendable products.
• Category feeds: categories and hierarchy source data.
• Brand feeds: brand pages and metadata.
• Article feeds: editorial and content pages.

Source fields and catalog fields

A feed contains source fields from your system. During setup, a configurable mapping tells Luigi’s Box AI where records are located, which object type they contain, which source field provides identity, and how source values become canonical catalog fields. The mapping lets your source names and structure differ from the examples. A record without a value for the configured identity field is skipped.

The catalog object model owns the rules for registered @ fields, typed IDs, references, and field visibility. The feed type pages linked above focus on the files you export, and field mapping covers the mapping document itself — renaming, type conversion and defaults — including a dry run you can use before saving a change.

Serving your feed

Host each feed at a stable URL that Luigi’s Box AI can fetch on a schedule.

• Use a stable HTTPS URL with a valid TLS certificate.
• Keep the feed reachable without interactive login.
• Return an accurate ETag or Last-Modified header so unchanged feeds can be skipped and changed feeds can be synchronized promptly.
• Serve the feed with gzip compression to reduce download time. Conditional-update metadata and gzip are strongly recommended for timely, near-real-time synchronization.
• Prefer a direct URL over a redirect chain.

Supported formats

Luigi’s Box AI supports XML, JSON, JSON Lines, and CSV feeds.

Format	Notes
XML	Supports repeated and nested values. Encode special characters with entities or CDATA.
JSON	Use a top-level array or an object containing the records at an agreed path.
JSON Lines	Write one complete JSON object per line with no enclosing array.
CSV	Include a header row and use the delimiter configured during setup. CSV is suitable for flat records with one value per column.

All feeds must be valid for their format.

XML represents repeated values with repeated elements and structured values with nested elements:

<product>
  <id>product/sku-1001</id>
  <image_url>https://cdn.example.com/sku-1001-front.jpg</image_url>
  <image_url>https://cdn.example.com/sku-1001-back.jpg</image_url>
  <dimensions>
    <width>42</width>
    <height>64</height>
  </dimensions>
</product>

JSON feeds can contain a top-level array or wrap records under an agreed key:

{
  "products": [
    {
      "id": "product/123",
      "title": "Blue Cotton T-Shirt",
      "url": "https://example.com/products/blue-cotton-t-shirt"
    }
  ]
}

JSON Lines feeds contain one object per line:

{"id":"product/123","title":"Blue Cotton T-Shirt","url":"https://example.com/products/blue-cotton-t-shirt"}
{"id":"product/124","title":"Red Cotton T-Shirt","url":"https://example.com/products/red-cotton-t-shirt"}

XML encoding

Text with special characters must be valid XML.

Invalid XML

<title>Black & White Shirt</title>

Entity-encoded text

<title>Black &amp; White Shirt</title>

CDATA section

<title><![CDATA[Black & White Shirt]]></title>

Do not entity-encode text inside CDATA, because the encoded text would be indexed literally. XML attributes are ignored, so put catalog values in elements.

Compression

Use gzip compression. Smaller feeds download faster, and download speed is normally the main constraint on total feed processing time. Luigi’s Box AI detects gzip from the file content and decompresses it automatically.

Feed rules

• Provide one object type per feed: products, categories, brands, and articles use separate files.
• Export the full current state, not only records changed since the previous run.
• Encode the feed as UTF-8.
• Keep source IDs stable for the lifetime of each object.
• Type-prefix every source ID and ensure the prefix matches the configured object type, such as product/123 for a product. The mapping does not repair a missing or mismatched prefix.
• Keep field names and data types consistent across records and exports.
• Omit values you do not have instead of changing their type or meaning.
• Do not put credentials, personal data, or other secrets in a feed.

See Types and IDs for the complete identity rules.

How multiple feeds form one catalog

A catalog can combine separate full-snapshot feeds for products, categories, brands, and articles. Each feed has its own stable URL, source mapping, and update lifecycle.

• Provide a product feed for searchable and recommendable products.
• Add a category feed when products use category references.
• Add a brand feed when brand pages or metadata should be indexed.
• Add an article feed when editorial content should appear in discovery.

Do not mix object types in one file. If your platform splits one object type across several files, coordinate consolidation during onboarding.

Source fields are mapped independently for each feed. References can be ingested before the referenced object exists, so do not depend on a processing order between separate feeds.

Updates and deletions

Each feed is processed as a full snapshot:

• New and changed records are indexed.
• Existing records that are absent from the currently downloaded snapshot are deleted.

Large drops are rejected

A feed run that would remove more than 30% of the existing objects is rejected to protect the catalog from broken or truncated exports. Coordinate an intentional reduction of that size with Luigi’s Box support.

Performance benchmark

Luigi’s Box AI parses and indexes a feed while it downloads, so the two overlap. Which of the two dominates depends on how fast the feed is served.

Test feed: 1.32 GB compressed (10.4 GB uncompressed) · 207,993 products

Feed link speed	Download alone	End to end	Processing beyond the download
100 Mbps (≈12.5 MiB/s)	105.5 s	105.5 s	≈ 0 s (under 1%)
1 Gbps (≈125 MiB/s)	10.6 s	13.2 s	2.6 s (20%)
2.5 Gbps (≈300 MiB/s)	4.2 s	13.3 s	9.1 s (68%)
Unthrottled (loopback)	0.5 s	14.0 s	13.5 s (96%)

Two regimes, crossing a little under 1 Gbps:

• Below it, the run is download-bound. Halving the transfer time halves the run, and processing adds nothing measurable. Most feeds sit here.
• Above it, the run is processing-bound. End to end does not drop below roughly 13 seconds for this feed however fast the bytes arrive. Going from 1 Gbps to unthrottled saves under a second.

Serve the feed compressed and unthrottled. Below about 1 Gbps, a reduction in transfer size reduces the total run by the same amount. Above it, the transfer is no longer the constraint, and additional bandwidth does not change the total.

Troubleshooting

See Feed troubleshooting for common parsing, identity, update, and reference problems.

See also

• Field mapping — the mapping document, its transforms, and a dry run
• Feed management API — registering feeds and triggering a sync yourself
• Catalog metadata — checking what actually got indexed, and with which types
• Content API — pushing changes as they happen, instead of on a schedule
• Object history — tracing one object’s changes