Life of a query: QDFeeds (query-defined feeds)

Design document, 2026-10-30. Covers NeuralWeb-dev/qdfeeds (main bbbfab7) as deployed in revision qdrss-00049-yet (image qdrss:0fc88b7). Every claim carries a file and line reference; the measured numbers in section 4 were read from the running service on 2026-10-09. Sibling documents: ARD, NeuralKG2, NLWeb samples.

0. How to read this

QDFeeds (at rss.neuralweb.dev) turns a standing interest written as a sentence into an RSS feed: the recent items (episodes, or transcript passages) of a collection that a model judged to match it. You subscribe to the topic, not to a show. It is used by the NYT, NPR and This American Life pages and by any feed reader.

QDFeeds does not call ARD or NeuralKG2. It shares only the model provider (OpenRouter) and the principle that a model judges and code does everything else.

Words used throughout

• Collection: a named group of sources in sources.yaml (nytimes, npr, tal, ai, ...). A feed can be restricted to one.
• Item: one stored record: an episode (title plus description), or for This American Life one passage of a transcript.
• Ingest time: when the service first saw an item. A feed's default window uses it; its order uses the publication date (ingest time when an item has none).
• Strong / relevant / exclude: the model's verdict on one candidate against the interest.

1. Where QDFeeds sits

flowchart LR
  R[Feed reader or page] -->|GET /feed.xml?q=| Q[QDFeeds<br/>rss.neuralweb.dev]
  Q --> COS[(Cosmos DB<br/>items + vectors)]
  Q <-->|embed, classify| OR[(OpenRouter)]
  Q --> MEM[(TAL in memory<br/>GCS snapshot)]
  MEM -.->|catch up every 5 min| COS
  ING[Ingest job qdrss-ingest<br/>daily 13:00 UTC] -->|RSS, TAL transcripts| COS
  ING <-->|embed new items| OR

One Starlette app on Cloud Run (qdrss, us-central1; 1 CPU, 2 GiB, at most 3 instances). Items and vectors are in Cosmos DB in production, except that the tal collection is searched in process memory; locally, SQLite plus an in-memory numpy index. Ingestion runs as a separate Cloud Run job, not in the service. Models: embeddings openai/text-embedding-3-small (1536 dimensions), classification openai/gpt-oss-20b.

2. The life of a feed query

QDFeeds turns a standing interest into an RSS feed. The interest is a sentence; the feed is the recent items (or passages) of a collection that a model judged to match it. Source: NeuralWeb-dev/qdfeeds (src/qdrss/).

There are two halves: ingestion keeps the corpus current, and /feed.xml answers an interest against it.

2.1 The corpus (ingestion)

Sources. sources.yaml maps collection names to lists of {name, url[, segmenter]}. Twelve collections: ai, history, finance, international, cricket, american_football, soccer, nonprofit, war, nytimes (73 shows), npr (97 shows) and tal (This American Life, one segmented source). /collections reports sources and items per collection (for example npr 55,302 items, tal 28,444 passages, nytimes 8,776 items).

Fetch (ingest.py:94-148). All sources are fetched concurrently (semaphore of 4, 30 s timeout). Requests are conditional (stored ETag and Last-Modified); a 304 or an unchanged body hash is skipped. A new source's RFC 5005 archive pages (rel="next", up to 200 pages) are followed once, so the full back catalogue is ingested. A feed is parsed before its validators are saved, so a malformed body never poisons the next fetch. A source that fails keeps its old items and records last_error; one failing source never blocks the others.

Parse (feeds.py). RSS 2.0 and Atom. Text is content:encoded or description, tags stripped. The item id is "{source}:{guid or url or hash(title,date)}". Items are append-only: an upstream edit is ignored. ingested_at is the time of first sight.

Transcript passages (tal.py). For the tal source the feed is only a list of episode numbers and audio links. For each episode the service fetches the transcript page (thisamericanlife.org/<n>/transcript), the episode page (air date) and the audio URL; a 5xx on the transcript falls back to the newest Internet Archive capture that carries timestamps. The transcript is cut into acts, and each act into windows of at least 280 words that never cross an act (a tail under 80 words merges into the previous window; a speaker label is added when the speaker changes). Each window is one item: id tal_this_american_life:<episode>:<act id>:<nnn>, title "episode title — act" (tal.py:187), published_at the air date, and an extra record with episode, actId, act, start, end (seconds), the transcript anchor and audio as <mp3>#t=<start>. That extra is what lets a result say "which episode, which act, and the exact time to press play". The back catalogue (episodes 1 to 899) was loaded with scripts/backfill_tal.py, back-dating ingested_at to the air date.

Embed (providers.py, app.py:115-129). Text is title + "\n" + content, first 8000 characters, embedded with openai/text-embedding-3-small (1536 dimensions, L2-normalised) through OpenRouter. In production items are stored in Cosmos DB (one container, partitioned by collection, vector index diskANN, cosine); a background pass embeds items that have no vector yet. Locally the store is SQLite with an in-memory numpy index.

When. In production the service does not ingest (QDRSS_INGEST_IN_APP=false, app.py:188-190). The Cloud Run job qdrss-ingest (same image, ingest_main, app.py:380-400) runs one refresh daily at 13:00 UTC and exits. The job cannot clear the service's feed cache, so cached feeds expire only after QDRSS_REFRESH_MINUTES (1440, a day; app.py:78), and a new item can take up to a day to reach a cached feed. Locally, or with in-app ingest on, the app refreshes at start and then every QDRSS_REFRESH_MINUTES (in a child process when Cosmos is used, so a slow ingest cannot block queries) and clears the feed cache after each refresh.

TAL in memory (memory.py). In production tal is served from process memory: its passages and vectors are loaded at startup from a snapshot in gs://neuralweb-qdrss-snapshots (written by scripts/export_snapshot.py), and whatever Cosmos holds from after the snapshot is added at startup and then at most every 5 minutes, triggered by queries (memory.py:114-120). If the snapshot cannot be loaded, tal falls back to Cosmos and /health says so.

2.2 A /feed.xml request

GET /feed.xml?q=<interest>[&collection=npr][&limit=40][&since=2000-01-01][&threshold=relevant|strong][&fresh=<days>|on]

The NYT, NPR and TAL pages send collection, limit=40, since=2000-01-01 ("Include past episodes", on by default) and fresh=365 ("Favor recent episodes", on by default); the index page sends fresh=365 only when "Favor recent" is ticked (nyt.html:180-182, npr.html:186-188, tal.html:135-137, index.html:255-256).

1. Validate (app.py:235-265). q required. since RFC 2822 or ISO 8601. limit 1-100 (default 20). threshold relevant or strong. fresh 0 (off) or 1-3650 days; on means 365. collection must be one of the twelve. 503 while the index is not ready (with Cosmos the service is ready once setup finishes, app.py:90-91). There is no length cap on q and no rate limit in the code (any limit comes from infrastructure).

2. Cache. If since is absent the answer is cached by (q, limit, threshold, collection, fresh, host) for the refresh interval (app.py:271-274), and the response carries an ETag and Cache-Control: public, max-age=900, so feed readers poll cheaply. With since the answer is not cached.

3. Window. since defaults to now minus 7 days (QDRSS_WINDOW_DAYS): a feed shows what has been ingested recently, not what was published recently. The example pages pass since=2000-01-01 ("include past episodes"), which is how back-dated archive items and transcript passages are reached.

4. Embed the interest. One embedding call per distinct interest and model (cached, up to 5000 entries). The raw sentence is embedded as written; nothing rewrites it.

5. Retrieve 40 candidates (QDRSS_CANDIDATE_COUNT). Cosmos: SELECT TOP @k ... ORDER BY VectorDistance(c.embedding, @v) with ingested_ts > @since and, when a collection is named, a single-partition filter. The date and collection masks are applied before the top-k, so the 40 are the best items among the eligible ones. Candidates come back closest first; with cosine, VectorDistance is a similarity, so the merge across partitions sorts by descending score (cosmos.py:209-212). For collection=tal the same search runs over the in-memory index instead.

5a. Freshness boost (optional, fresh=<half-life days>; feed.py:71-74, 86-105). Retrieval takes 3 × 40 = 120 candidates. Each is scored as its closeness rank (1 for the closest, 0 for the last) plus 0.5 × 0.5^(age / half-life), with age taken from the publication date, and the best 40 go on in closeness order. The boost only changes which items the model sees; it still judges every one, so a recent item that does not fit is not returned.

6. Classify (rank.py). The 40 candidates are cut into batches of 5 and sent to the ranking model (openai/gpt-oss-20b) in parallel (8 calls). The prompt asks it to act as a relevance filter, not a ranker: for each record say strong (directly satisfies the interest), relevant (substantially useful but partial) or exclude, with a one-sentence why of at most 25 words naming the specific topic. Input per record: title and the first 1500 characters. Output JSON {"results":[{"i":0,"m":"strong","why":"..."}]}, temperature 0, reasoning effort low. A bad index, a duplicate or an unknown category is dropped (treated as exclude). Retries: one retry of the provider call with the failed provider ignored, one retry of the batch, and the HTTP client's own retry; a provider that fails 3 times with a 30% failure share within an hour is ignored for routing.

7. Threshold, order, limit. strong keeps only strong; relevant keeps both. The survivors are sorted by publication time, newest first (ingestion time for an item without one; the id breaks ties; feed.py:78-82) (the model's verdict and the similarity do not affect order), then cut to limit. So a feed is a chronological stream of the matching items, not a relevance ranking, and it may hold fewer than limit items because only 40 candidates are considered.

8. Render RSS 2.0. Channel title rssnlweb: <q>. Each item has the title, link, guid (the item id), pubDate, source (the show), a description [strong] <why> followed by the first 1000 characters of the text, and a qd: extension (namespace https://qdrss.dev/ns): qd:category, qd:ingestedAt, qd:source, and for transcript passages qd:episode, qd:act, qd:actId, qd:offsetSeconds, qd:endSeconds, qd:transcript and qd:audio.

2.3 Where QDFeeds is deterministic and where it is not

Deterministic: parsing, ids, windowing of transcripts, retrieval for a fixed embedding, ordering, ETags. Model-driven: the embeddings and every strong/relevant/exclude verdict and why. The verdicts vary between providers even at temperature 0, which is why the status ledger (the status ledger at neuralweb.dev/status) judges the feeds independently with a stronger model.

2.4 Failure behaviour

3. Configuration (config.py)

4. Measured behaviour

From the status ledger (neuralweb.dev/status, 2026-10-09) and /health:

5. Invariants

1. A ranking failure fails the request (503); a partial feed would look like "nothing new" (rank.py:3-5, app.py:285-287).
2. Items are append-only: an upstream edit never rewrites a stored item (store.py:71-93, cosmos.py:107-123).
3. A malformed feed never poisons the next fetch: validators are saved only after the body parses (ingest.py:127-139).
4. A feed is a chronological stream of matching items, ordered by publication date, not by relevance (feed.py:78-82).
5. since filters on ingest time, so back-dated archive items and transcript passages need a far-back since.
6. Transcript passages never cross an act (tal.py:87-111).

6. Open issues

• No rate limit and no query-length cap in the code.
• A query-embedding failure on a cache miss is an unhandled 500.
• The README says the vector index is quantizedFlat; the code creates diskANN.
• The ETag covers item ids only, so a changed why for the same ids does not change it.
• Cosmos search does not check that stored vectors were made by the current embedding model.
• Only 40 candidates are classified, so a feed can hold fewer than limit items.
• The ingest job does not clear the service's feed cache, so a cached feed can lag new items by up to a day.
• Transcript ingestion exists only for This American Life. NPR publishes speaker-labelled transcripts for Weekend Edition, Up First, Consider This, Throughline, Invisibilia and TED Radio Hour, with no timestamps (Fresh Air has none); nytimes.com refuses scripted fetches.
• /health counters are per instance; /healthz is answered by Cloud Run's front end, so use /health.

7. File map