Slides: How ARD works (PDF)

Life of a query: the ARD agent finder

Design document, 2026-10-09, revised 2026-10-30. Covers ard-finder main 8d8d3d68 as deployed in revision ard-00082-dar: 118,703 entries (built-in 25,049 + open-data catalog 93,654). Every claim carries a file and line reference (on main 8d8d3d68); the measurements in section 4 are from 2026-10-09. Sibling documents: NeuralKG2 (the caller that depends on ARD most), QDFeeds (QDFeeds), NLWeb samples (the NLWeb samples).

0. How to read this

ARD (the Agent Resource Discovery finder, at ard.neuralweb.dev) answers one question: which published tables describe this quantity? You give it a description ("median household income", "gdp india") and it returns the OKF table documents that describe it, ranked, each naming a variable, a source and how to ask for it. It never fetches data itself.

Callers: NeuralKG2 (one search per piece of data a question needs, plus exact /variable and /variables lookups), the ARD browser page at neuralweb.dev/4dc/ard/, and any MCP client (POST /mcp).

Words used throughout

• Entry / document / leaf: one OKF markdown file describing one published table or variable from one source. For Data Commons, one (variable, source) pair.
• Source family: the publisher a document belongs to (datacommons, sec-edgar, census, worldbank, ...). The five BigQuery directories collapse into one family, bigquery.
• Phrasing: one text the caller asks about. A request carries up to 4; ARD adds a place-stripped version of each.
• Pool / head / slice: the candidates gathered by embedding similarity, and the first part of them that the model actually sees.
• Rerank: scoring candidates 0 to 100 with a model. The only model judgment on the search path besides the embedding.
• Family (of a Data Commons variable): (populationType, measuredProperty, statType), with its constraints (the dimensions it is split by: age, race, gender...). Used by /variables.

1. Where ARD sits

flowchart LR
  NK[NeuralKG2<br/>catalog client] -->|POST /search<br/>GET /variable, /variables| ARD[ARD finder<br/>ard.neuralweb.dev]
  PG[ARD page<br/>neuralweb.dev/4dc/ard] -->|POST /search| ARD
  MCP[MCP clients] -->|POST /mcp, /ask| ARD
  ARD --> IDX[(Index in memory<br/>118,703 vectors + metadata)]
  ARD --> LEAF[(Leaf files on disk<br/>OKF markdown)]
  ARD <-->|embed query, rerank| OR[(OpenRouter)]

The service is one Starlette app on Cloud Run (ard, us-central1). The index and the leaf files ship inside the image; there is no database. Models: embeddings openai/text-embedding-3-small (1536 dimensions), rerank openai/gpt-oss-20b, both through OpenRouter.

2. The life of a search

ARD answers one question: which published tables describe this quantity? It never fetches data. Its output is a ranked list of OKF table documents, each naming a variable, a source, and how to ask for it. Source: ard-finder (agent_finder.py, registry/index.py, registry/places.py, filters.py, llm.py).

2.1 What exists before any query (the build)

The index is built offline by registry/index.py:build (212-309) and shipped inside the image as registry/current/.

1. Collect every *.md under sources/ and corpora/gcp-bigquery/okf/ whose front matter has representativeQueries (_collect_docs, 158-203). These leaves are written by the tools/gen_*_okf.py generators from each publisher's catalog (Data Commons variables, SEC filings, World Bank indicators, Census tables and so on).
2. Embed one text per entry (index_text, 120-131): title, all representative queries, "Describes {scope}" (the source's entity type), then the full description, joined with ". ". Model openai/text-embedding-3-small, 1536 dimensions, 8000-character input cap.
3. Publish vectors.npy (float16, 76 MB on disk, about 150 MB as float32 in memory), meta.json (26.9 MB) and manifest.json to registry/current. Vectors are reused when an entry's sig (md5 of text plus model) is unchanged, so a rebuild embeds only what changed. Batches of 96 are checkpointed.

Built-in index (built 2026-10-11): 25,049 entries. Data Commons 11,595; SEC EDGAR 8,096; Census 2,000; World Bank 1,498; UN SDG 713; BigQuery public 356 and marketplace 240; USAspending 223; Treasury 186; nonprofit 990 57; CDC PLACES 40; 45 others.

Extra catalogs. ARD_CATALOGS="name=/path,..." mounts more indexes beside the built-in one (registry/catalogs.py:31-45); each holds index/{vectors.npy,meta.json,manifest.json} and its own sources/. Its identifiers carry a name: prefix. catalogs.combine (120-143) refuses to start when catalogs were embedded with a different model or dimension, or share a source directory, unless the later manifest declares extends. Production mounts open-data=/app/open-data-catalog (93,654 entries, built by tools/build_open_catalog.py), which declares extends: ["census", "datacommons"]: its ACS detailed and subject tables join the census family and its India supplement joins datacommons. Live total: 118,703.

At server start the service verifies the built-in release is current (agent_finder.py:924-931; extra catalogs get only the combine checks), then (lifespan, 873-877) loads the vectors and metadata of every catalog (index._store), projects every row into an ARD entry (_catalog), and builds two exact-lookup tables from the Data Commons leaves (_variable_index, 630-652): variable -> leaf paths and variable -> family where a family is (populationType, measuredProperty, statType, constraints). Those two tables are what /variable and /variables read. If no model credentials are present, startup fails (llm.py:47).

2.2 Endpoints

There is no authentication. CORS allows any origin. Body limit 64 KB (413 above). /search has a per-IP daily quota of 10,000 held in process memory (429 with Retry-After); the quota is charged after the body is parsed but before the query is validated.

2.3 A /search request, step by step

NeuralKG2 sends (neuralkg2/catalog.py:160-170):

{"query": {"text": "Find tables for: gdp india. Background only, not the search: the user's full question is \"What is India's GDP?\".",
           "texts": ["gdp india", "What is India's GDP?", "gdp"]},
 "pageSize": 10, "rerank": true, "sources": ["datacommons"]}

Step 1: validate (agent_finder.py:793-824). query.text must be a non-empty string; query.texts a list of 1 to 4 non-empty strings (default [text]); pageSize clamped to 1-100; sources/excludeSources lists of at most 50 names that must exist (400 with the valid names otherwise); query.filter may use publisher, type, tags (other keys are reported back in filter.ignored). Values in one key are ORed, different keys ANDed (filters.py). catalogs optionally names the catalogs to search (unknown names are a 400; agent_finder.py:758-773); the MCP tool does not pass it.

Step 2: phrasings (index.py:697-708). The caller's texts come first. For each, places.strip (places.py:98-101) adds a version with places removed ("gdp india" becomes "gdp") from a gazetteer of US states, countries, large cities, demonyms and "X County"-style patterns. Reason: documents describe measures, not places, and a place in the query drags in any document that merely mentions it. At most 8 phrasings result. No model rewrites the query.

Step 3: embed and score (search_many_async, 807-867). All phrasings are embedded in one call and L2-normalised. Each entry's score is its maximum cosine similarity over the phrasings (sims = vecs @ q.T, 118,703 rows across all catalogs, numpy). Entries are walked in descending score, the request's filters applied, until about 150 (ARD_PREFILTER_MAX) are collected. Each carries embed_score = round(score*100, 1).

Step 4: shape the pool before the model sees it. This is the part that decides recall.

The per-phrasing rule exists because a place-bearing phrasing and the stripped phrasing find different tables; keeping each phrasing's best means a table that either finds survives. The per-source rule exists so a large source (Data Commons has 11,595 entries) cannot push every other source out of the head.

Step 5: rerank with a model (_rerank_by_source, 870-900). Candidates are grouped by source family in order of first appearance; the first 5 families are scored, one model call each, in parallel. The first group is scored alone; the other four are prefixed with the first group's top 2 candidates as anchors (their scores are discarded) so a weaker source is not graded on a curve.

• Prompt (_rerank_messages, 482-503): judge each numbered card by title, description and example questions; prefer recall, keep siblings and alternative definitions; return the k (pageSize, plus the anchors) most relevant as {"ranked":[{"i":n,"score":0-100}]}; score every one returned (low rather than omitted); never return an empty list.
• Card (_card, 423-442): title, covers: <scope>, geography: <the tags' geographic levels in words> (registry/scope_text.py; a countries-only table adds that it is not published for any area inside a country), up to 6 people ask: queries. The description is left out unless ARD_RERANK_DESC=1.
• Model RERANK_MODEL, openai/gpt-oss-20b in production. Temperature 0, JSON mode, reasoning effort low, provider sort throughput.
• Output budget (_rerank_budget, 408-420): max(400, 16*n + 300) + 1500 tokens.
• Retries (llm.py:431-489): up to 4 attempts on 408/409/429/5xx with Retry-After or min(30, 2**attempt) back-off; an empty completion retries with max_tokens doubled, up to 8000 (this was the cause of the slow "choosing tables" step fixed on 2026-10-08, when a reasoning model spent its whole budget thinking). Per call 120 s; the whole query has a 180 s in-app deadline, but Cloud Run's 120 s request timeout ends a production request first.
• Parse (_ranked_candidates, 506-521): only the first k items; an item needs an integer i in range and a numeric score; bad items are skipped. A result is eligible when score >= 50 (ARD_MIN_RERANK_SCORE).

Step 6: merge (_source_first, 663-673). Eligible results are ordered with the best result of each source first (up to 5 sources), then the rest by score, cut to pageSize. The order is therefore not purely by score: it guarantees breadth across sources.

Step 7: widen when nothing qualifies (846-866). If no candidate scored 50 or more, the service looks at how spread the embedding scores of the head are. A spread (standard deviation) of 3.0 or more means the head already separates good from bad, and the answer is "no match". A narrow spread means the embedding cannot tell and the answer may be deeper, so it scores further windows of PREFILTER_SLICE (60 in production), one model call each with no per-source grouping or anchors, up to 120 examined. It stops when something qualifies or when a window's own spread is 3.0 or more; a failed widening call is a 503.

Step 8: respond. Each hit becomes a full ARD entry read from the leaf file (_entry_from_meta, 230-280): identifier (a URN urn:air:<publisher>:okf:<source>.<leaf>), displayName, description, representativeQueries, tags, the okf:* terms of the leaf merged over its source's _access.md (variable, facet, unit, populationType, measuredProperty, statType, constraints, coverage, latest observation, access operations), the markdown body, and score. The description is the leaf's own (see 6). The envelope also carries usage: model calls, tokens, dollars and a per-stage and per-model breakdown (llm.Ledger).

2.4 What NeuralKG2 does with the result

RemoteCatalog._search (neuralkg2/catalog.py:160-200) validates the shape strictly (a malformed response is a provider failure, never an empty result), maps each entry to a Resource through the directories table in ard-gcp.json (datacommons to the Data Commons plugin, and so on; an entry with no plugin becomes ard:<family> and is handled by the generic source), and turns no_match into an empty result. If the reranked search comes back empty, NeuralKG2 retries once with rerank: false (catalog.py:131-138) and the plugins judge every document themselves.

2.5 Where ARD is deterministic and where it is not

Deterministic: validation, filters, place stripping, the embedding walk, the slice, per-phrasing and per-source promotion, source-first ordering, the score threshold, the widening rule, and every /variable and /variables answer. Model-driven: the 0-100 rerank score (which decides what appears at all) and the embeddings themselves.

2.6 Failure behaviour

Silent behaviours worth knowing (all visible in the code, none visible in a response): sources beyond the fifth are never scored; a failure in any non-leading group drops that source without a signal; malformed ranking items are skipped without notice; rerank: false returns embedding order on a different score scale without saying so; a wide first slice short-circuits widening even when the answer is deeper; broken leaf files lose their okf:* terms without an error.

3. Tunables and defaults

4. Measured behaviour (service statistics, as of 2026-10-09)

Read from revision ard-00049-ciq (24,827 entries) on 2026-10-09; not re-measured since the open-data catalog was added.

The live /health shows the same counters for the current instance (searches.total, last_at, latency_seconds, failures_by_status); /stats.json has the per-day and per-model breakdown and the OpenRouter key's own spend.

5. Invariants

1. A failed rerank is never replaced by embedding order. The leading group's failure is a 503 that says so (agent_finder.py:838-844; a widening window's failure too, index.py:855-856).
2. Breadth across sources is guaranteed before depth by score: each source's best three enter the head, and each of up to five sources leads the results with its best table (index.py:642-673).
3. Either phrasing finds a table: a table found by the place-bearing or the place-stripped phrasing stays eligible (_head_with_each_phrasing).
4. /variable and /variables are exact: no embedding, no model (agent_finder.py:666-708).
5. Names are checked: an unknown source or publisher is a 400 listing the valid ones, never an empty result that looks like "no match" (filters.py).
6. The release is verified at start: a stale or incomplete built-in index refuses to serve (agent_finder.py:924-931); extra catalogs must match its embedding model and dimension and may not share a source directory unless declared (registry/catalogs.py:120-143).

6. Open issues

• Sources beyond the fifth are never scored, and a failed non-leading rerank group drops its source with no signal in the response.
• Malformed ranking items are skipped without notice; rerank: false returns embedding order on a different score scale without saying so.
• A wide first slice (standard deviation of 3.0 or more) stops the search with no_match even when the answer sits deeper.
• The daily quota is charged before the query is validated, and it is per process (it resets on restart and is per replica).
• Embedding or credential failures are unhandled and return 500 with no usage in the body.
• Place stripping is a regex and gazetteer; it misses places not listed and can strip words that are also names ("Georgia", "Jordan", "Austin"). strip_capitalized exists but is unused in production.
• Leaf-file errors are swallowed (_leaf_fm, _access_fm return {}), so a broken leaf silently loses its okf:* terms.
• Building search results reads leaf files synchronously inside the event loop; time grows with pageSize.
• /health and /stats counters are per instance and reset on restart.
• /healthz is answered by Cloud Run's front end, not the app; use /health.
• A search result's description is the leaf's own description. _search passes scope_text.described(hit) (geography and tags in words; agent_finder.py:859), but _entry_from_meta(..., full=True) replaces it with the leaf's front-matter description (273), so the tag wording reaches only the rerank card.
• /variable and /variables read only the built-in Data Commons leaves: _variable_index takes paths starting sources/datacommons/ (agent_finder.py:640), and the open-data India supplement's are open-data:sources/datacommons/.... Those variables are found by /search but not by the exact lookups.
• The 180 s query deadline is longer than Cloud Run's 120 s request timeout, so a slow query is cut by the front end, not answered with 504.

7. File map