Life of a query: the NLWeb samples

Design document, 2026-10-30 (first written 2026-10-09). Covers the nlweb-samples repository (main d451cb0, 2026-10-29) as deployed to five Cloud Run services, all running image nlweb-samples:20261030-172716 built from d451cb0 (revisions nlweb-recipes-00037-cit, nlweb-reviews-00021-4gh, nlweb-movies-00019-st7, nlweb-trails-00019-9r7, nlweb-homes-00019-6jq), and the static front end at neuralweb.dev/samples/. Paths are relative to the repo root; line references are into that revision. Sibling documents: ARD, NeuralKG2, QDFeeds.

0. How to read this

The NLWeb samples let a person ask for things in natural language (a robot vacuum for two cats and a long-haired dog; a showstopper Thanksgiving main that is vegan and gluten-free; a 4-bedroom house with a pool) and get back a list of real items, each with a one-sentence reason, optionally with a short written summary. There are five datasets, each its own service:

The five counts sum to 73,736, matching src/askhub/corpus/.manifest.json (snapshot 58d55061fe9b, built 2026-10-02).

What "NLWeb" means here. This is not Microsoft's NLWeb codebase. The server is a Python/Starlette app (package askhub, console script ask-samples) that the README describes as ask-hf-hub "with the corpus swapped" (README.md:28-30; the parent repo rvguha/HFNLW is not on the VM). It is NLWeb-compatible by protocol: the /ask and /mcp endpoints and the server-sent-event message types (begin-nlweb-response, nlws, end-nlweb-response) follow NLWeb (pipeline.py:67,309,315; models.py:134-150).

Words used throughout

• Site / kind: one of the five datasets. A request names the one to search (site=recipes).
• Record / document: one item (a recipe, a movie, a trail...), stored as a schema.org JSON object.
• Fields: each record is split into three searchable fields, name, description and body.
• BM25F: keyword scoring over those fields. Vector score: similarity of the query embedding to each field's embedding.
• Candidate: a record retrieved for ranking. Ranker: the model that judges candidates strong, relevant or neither.
• Decision ranker: a ranker that only scores records (OpenRouter's Decisions API) and writes nothing; offered as Jev 1.14.
• Describer: a second model that writes the shown one-sentence descriptions and the summary (recipes and reviews only, and not under a decision ranker).
• Triage / restrictions: a cheap model call that decides how hard a recipe request is and extracts dietary constraints.
• Mode: list (results only), summarize (results plus a short summary), generate (results plus an answer).

1. System map

flowchart LR
  B[Browser<br/>neuralweb.dev/samples/KIND/] -->|POST /ask, SSE| S[nlweb-KIND<br/>Cloud Run, 1 instance]
  W[wake page] -->|GET /health x5| S
  S --> MEM[(In-memory index<br/>recipes, reviews)]
  S --> COS[(Cosmos DB<br/>trails, movies, homes)]
  S <-->|embed, decontextualize, rank, describe| OR[(OpenRouter)]
  GCS[(GCS bucket<br/>snapshot per site)] -->|download at start| S

Each service is the same image (python:3.12-slim, code only); it downloads its data from gs://neuralweb-nlweb-<kind> at start (Dockerfile, tools/deploy_cloudrun.sh). Cloud Run settings: --cpu 1 --cpu-boost --min-instances 0 --max-instances 1 --concurrency 20 --timeout 300; memory 3 GiB for recipes and 1 GiB for the other four (a Cosmos-backed service always gets 1 GiB, deploy_cloudrun.sh:34,50). Because the minimum is zero, an idle service sleeps (a cold start took 27 to 30 s against about 3 s warm, deploy_cloudrun.sh:36-39); the wake page pings /health on all five to warm them (static/wake.html).

Store per site: recipes and reviews are held in memory from a snapshot; trails, movies and homes use Azure Cosmos DB (database samples, container items, one partition per site). Recipes and reviews moved off Cosmos on 2026-10-04 because it was slower (deploy_cloudrun.sh:23-26).

2. What exists before any query (ingestion and index)

1. Crawl to JSONL (tools/convert.py). Crawled pages hold schema.org JSON-LD. The converter keeps one main entity per page by type (Recipe; Movie, TVSeries, TVEpisode; Article, NewsArticle; LocalBusiness; SingleFamilyResidence, RealEstateListing, House), drops images and publisher boilerplate, caps review at 3 entries and strings at 1,500 characters, folds Wirecutter's sibling Product and ItemList names into hasPart so product names are searchable, keeps one image URL for the card, and de-duplicates by URL (convert.py:23-152).
2. Split into fields (adapters.py). name from name, alternate name and headline; description from description and alternative headline; body from every other string or number, each prefixed by its property name (for example recipeCuisine Italian), URLs removed (adapters.py:200-343).
3. Embed (tools/embed.py). Each record gets three embeddings, one per field, with openai/text-embedding-3-small through OpenRouter (1536 dimensions, L2-normalised; empty fields are zero vectors). Batches of 64, inputs cut at 24,000 characters, concurrency 8, up to 6 retries. About 31 million tokens, about $0.60 for all five sites (README).
4. Index (catalog.py). Keyword scoring is BM25F computed in numpy: k1 = 1.5, field weights name 32, description 1, body 0.5, b = 0.75 (config.py:55-56); query tokens are lowercase [a-z0-9]+ with stopwords removed. Vector scoring is the weighted sum of the three per-field dot products with weights 1/1/1 (catalog.py:167-180).
5. Snapshot (tools/build_snapshot.py). catalog.pkl plus per-field .npy vectors, so a service starts in about 1 s instead of about 41 s for recipes (Dockerfile:8-10). For the Cosmos sites a keyword-only snapshot is built and the vectors live in Cosmos (tools/load_cosmos.py, one quantizedFlat vector index per field).

3. A query, from click to card

3.1 In the browser (static/app.js, cards.js, threadstore.js, config.js)

1. Scope. /samples/<kind>/ serves ask.html; the scope is the last path segment. config.js maps each kind to its Cloud Run URL (window.NLWEB_API); the endpoint is <service>/ask (app.js:21-24). The repository's config.js is empty (same origin); tools/build_site.py writes the deployed mapping. A "How it works" link in the top bar opens this document (ask.html:33); on phones (760 px or narrower) the answer mode, the model picker and that link move into the menu drawer (app.js:647-666).
2. Request. A fetch POST with Accept: text/event-stream and a JSON body (app.js:163-167, 309-311):

   {"query": "...", "site": "recipes", "mode": "summarize",
    "previous_queries": ["...up to 5 earlier turns..."],
    "previous_results": ["...names of the top 10 shown last..."],
    "ranking_model": "..."}

   previous_queries carries each earlier turn as its interpreted form; ranking_model is sent only if the user picked one in the model picker (Auto sends none; see §5). The mode picker offers list and summarize; generate is not reachable from the page.
3. Streaming. The reader splits the stream on blank lines and handles each event by type (app.js:179-197, 317-357): candidate draws provisional cards the moment the ranker names them; the first result clears them and each result adds a final card; nlws sets the summary; restrictions shows "diet: ... · ranked by ..."; decontextualized_query shows "interpreted as: ..."; usage fills the cost dialog; end-nlweb-response closes with "N results · tokens · $cost".
4. Cards (cards.js:128-275). The layout follows the schema.org @type (recipe, movie, review, trail, home). Trail length, loop shape, difficulty and dog rules are read from the description text by regular expressions; homes show price, beds and baths, size, lot and schools. The shown description is the model's one-sentence reason, or under a decision ranker the record's own description; a card with an empty description shows no reason line (cards.js:270). Links allow http(s) only.
5. State is in the browser. Conversations live in IndexedDB (ask-samples, stores threads and turns, gzip per turn) and only the ranking-model choice and sidebar state are in localStorage. The server keeps nothing between requests: context is whatever the client sends (threadstore.js:3-6).

3.2 On the server (app.py, pipeline.py)

Front door. Middleware rate-limits /ask and MCP ask calls (rate_limit.py:154-172) and applies CORS for https://neuralweb.dev. ask_route accepts GET or POST, merges query-string and JSON parameters, and validates them: mode (default summarize), retrieval (vector, bm25 or compare, default compare), max_results 10, min_score 70, previous_results capped at 10, optional ranking_model and counts. Bad values are a 400; a service that has not loaded yet is a 503 "Service is starting" (app.py:86-180). The reply is a server-sent-event stream; each event is {message_id, message_type, query_id, sequence, timestamp, content}.

search() is a generator that emits events as it goes (pipeline.py:57-315):

1. begin-nlweb-response. An empty query ends with an error event.
2. Decontextualize (only when previous_queries is non-empty). One call to the general model (gpt-oss-120b, effort low) with the earlier queries and the last results, returning {standalone_query} so "now only the vegan ones" becomes a complete query. Timeout 15 s. Whenever the rewrite differs from the typed query it is searched and announced with decontextualized_query; the model is not asked whether it changed anything (gpt-oss-120b wrote correct rewrites and still said changed=false 8 times in 20). It fails open: on any error the original query is searched (pre_retrieval.py:25-83).
3. Fast track. On a first turn, retrieval and ranking start immediately on the raw query in parallel with the other steps, and are cancelled and restarted only if a rewrite or a restriction changed the plan (pipeline.py:98-151).
4. Triage (recipes only). A cheap call (timeout 4 s) returns a tier (simple or hard) and typed constraints. A failed or slow triage is treated as hard (triage.py:122-134). Triage does not change the ranking in production (no restriction ranking model is configured): the ranker reads any dietary restriction from the request itself, told to check hidden ingredients and allow a simple substitution (pipeline.py:464-476). Triage's restrictions go to the describer, which must name the substitution or drop the item (pipeline.py:210-222). The events triage and, when there are restrictions, restrictions are emitted.
5. Retrieve (_retrieve_and_rank, pipeline.py:318-706). BM25 returns 20 candidates; vector search returns 20 more, excluding the ones BM25 already found, so about 40 candidates (Dockerfile:26-27; the code default is 10 + 10). The query embedding runs concurrently with BM25. If the embedding or the vector search fails, a notice says so and the BM25 results are ranked (pipeline.py:567-588).
6. Rank with a chat model. The default (Auto) is openai/gpt-oss-20b (effort low, 800 output tokens), routed to the providers coreweave, novita, groq in order.
   • The candidates are dealt round-robin into 3 batches of about 13 and ranked in parallel (pipeline.py:676-697).
   • Each batch sees a projection of every record limited to the fields that matter for the site (lists cut to 20 items, strings to 1,000 characters, a nested Person or Organization such as an author reduced to @type and name; config.py:16-33, pipeline.py:869-895) and the instruction "Act as a relevance filter, not a ranker": pick at most 3 records and mark each strong or relevant. On movies, trails and homes it also gives a "why" of at most 25 words; on recipes and reviews it writes nothing else, because the describer writes what is shown (pipeline.py:409-452).
   • Verdicts are categorical, not numeric: strong maps to 90, relevant to 70. The result order is strong first, then relevant, with input order kept inside each band; duplicates (same site and name) are removed and the list is cut to 10 (pipeline.py:819-859, 698-706).
   • Each verdict is streamed to the browser as a candidate event the moment its JSON object closes, so the first cards appear while the other batches are still thinking (pipeline.py:488-548).
7. Or rank with a decision model (Jev 1.14, typesafe/jev-1.14, chosen in the picker or by ranking_model). One POST to OpenRouter's Decisions API (…/api/alpha/decisions) scores every candidate at once on a five-level 0 to 4 scale. Each record is sent as its name, its own description (up to 1,200 characters) and the ranking projection without what repeats them. Below 1.9 is excluded; 3 or more is strong (90), otherwise relevant (70); results are ordered by score. A request too large for the model is split in half; up to 3 attempts, 90 s each (decision_models.py:26-44, 129-265). Nothing is written: each result shows the record's own description, the describer is skipped, and verdicts arrive together rather than one by one (pipeline.py:230-238, 626-658).
   • Fallback. If the decision model keeps nothing or fails, ASKHUB_DECISION_FALLBACK_MODEL (default openai/gpt-oss-20b; empty turns it off) re-ranks the same candidates in the usual batches, silently. With the fallback off, a failure ranks by retrieval score with the notice "Decision ranking unavailable; results use retrieval scores." (config.py:104-105, 250-252; pipeline.py:659-675).
8. Expand (on by default). One call (timeout 6 s) proposes search terms and up to 10 well-known item names. Names are searched by BM25 (top 5) and kept only when the record's name matches exactly after normalisation; terms yield up to 10 more candidates. The new candidates go through the same ranker (a decision model, with its fallback, scores them in a second call) and the results are merged. Errors are swallowed (expansion.py, pipeline.py:735-808).
9. Describe (recipes and reviews, chat rankers only). One streamed call to google/gemini-3.8-flash over the top 8 results writes a one-sentence description for each (30 words or fewer, only from the record's own JSON) and a 2 to 4 sentence summary, and may drop items that are not the requested kind of thing or fail a stated requirement. Only those 8 are ever shown. It exists because the 20B ranker stated unsupported product facts in about a quarter of its descriptions in a 30-question Wirecutter test on 2026-10-05 (pipeline.py:225-292, 949-1022). If the describer fails, the results show as ranked (with no reason line, since this ranker writes none); if it drops everything, the ranked top items are restored.
10. Simple tier (recipes). If triage says the request is simple, expansion and the describer are skipped; the top 8 results show the page's own markup description (cut at 280 characters) and a short summary from google/gemini-2.5-flash-lite.
11. Movies, trails and homes have no describer: the ranker's "why" is the description.
12. Mode. list stops at the results. summarize uses the describer's summary, or composes one with the general model. generate composes an answer to the question. Both emit nlws {answer, items}. If the model's JSON is unreadable, a template lists the top 4 names with their descriptions (pipeline.py:1025-1117).
13. Closing events: notices (intermediate_message), usage (a per-model token, cost and latency ledger; only the decision ranker records latency so far, usage.py), complete, end-nlweb-response.

Result items on the wire: {"@type":"Item", url, name, site, score, relevance, description, vector_score, bm25_score, retrieval_source, schema_object}, where schema_object is the full schema.org record (models.py:117-130).

4. Endpoints, limits and state

No authentication (Cloud Run is --allow-unauthenticated; the model key lives only on the server). Rate limits (rate_limit.py:56-96): 50 requests per minute per client, 1,000 per hour and 10,000 per day overall, 4 concurrent paid queries; a violation is a 429 with retry_after. The code defaults are 10, 60 and 500 (config.py:312-314); the Dockerfile sets 1,000 and 10,000 (Dockerfile:28-29), and the 50 is an environment variable set on the live services only (see §8). Per-request caps: 200 candidates for retrieval and for ranking. Because the instance count is capped at 1, all of this is per service.

5. Models and timeouts

Explicit timeouts: decontextualize 15 s, triage 4 s, expansion 6 s. The chat ranking, describer and compose calls have no explicit timeout (the SDK default applies); the decision ranker allows 90 s per attempt, 3 attempts. The Cloud Run request limit is 300 s. The page's ranking-model picker offers Auto (gpt-oss-20b) and nine models: Jev 1.14, GPT-OSS 20B and 120B, Gemma 4 26B and 31B, Gemini 2.5, 3.1 and 3.5 Flash Lite, and Gemini 3.8 Flash (app.js:43-54). Only three of those are serving models in ranking_models.py; the server accepts every id listed there, including Phi-4 and the other reference models (ranking_models.py:25-63).

6. Deterministic and model-driven parts

Deterministic: parsing and validation, rate limiting, BM25F, vector scoring, merging the two candidate lists, banding, de-duplication, the result cut, name-matching of expansion suggestions, card rendering and the trail regexes. Model-driven: the rewrite, triage, expansion, the relevance judgment (there is no numeric threshold on retrieval scores; the ranker is the filter, and a decision model's 1.9 cut applies to its own scores), the descriptions, the summary and the answer.

There is no server-side memory, tool router or "is this query relevant to this site" gate. The homes sample has no real-estate tools either: no price or bedroom parsing, no geography, no school lookup, no sort by field. "Under $2M with 4 bedrooms" and "newest construction over 2,500 sq ft" are judged by the ranker from each record's JSON alone.

7. Failure behaviour

8. Open issues

• A ranking batch can fail without telling the reader, and a second failure can end the stream without its closing event, so the page cannot tell an empty answer from a broken one.
• The chat ranking, describer and compose calls have no explicit timeout.
• With Jev 1.14 on recipes or reviews, when the fallback fires, gpt-oss-20b ranks with the describer-site prompt (no "why"), but the describer is still skipped because a decision model was chosen. The cards show no reason line at all (pipeline.py:233-236, 438-452; cards.js:270). A describer failure on those sites has the same effect.
• The per-client limit of 50 a minute exists only as an environment variable on the live services. tools/deploy_cloudrun.sh deploys with --set-env-vars, which replaces every variable, and sets no rate limit, so the next scripted deploy drops it back to the code default of 10. It belongs in the Dockerfile or the script.
• Numeric and sorting questions (price, bedrooms, year built) rely entirely on the model; the homes page itself notes that "newest construction" is sorting, not filtering.
• Dockerfile sets the trusted proxy list to *, so a client can influence the rate-limit key through X-Forwarded-For.
• One instance per service means cold starts and a shared limit; the wake page is a workaround.
• Data refresh is disabled in production: a catalog update means copying files into the bucket and restarting.
• Dormant code from the parent project (hf:Model handling, benchmark comments) remains.
• There is no /health activity report like the other services (no counts, last request time or latency); only item counts and readiness.
• The corpus files and embeddings are not in the repository, so item counts in this document come from the README and manifest, not from inspecting the buckets.

9. File map