How to Stop Your AI from Hallucinating Numbers

Why it happens

An LLM predicts the next token. Asked for "Apple's latest reported holdings value," it will produce a confident, well-formatted number whether or not it knows the answer. There is no internal "I'm not sure" gate unless you build one. For financial work, a wrong number that looks right is worse than no number.

Four failure classes look identical in the output:

1. Invented values. A figure that never existed: learned format, plausible magnitude, fictional number.
2. Stale values. Real once, but a 13F can land up to 45 days after quarter end, so the latest quarter always post-dates any training cutoff.
3. Entity confusion. Similar filers blended, or a CUSIP mapped to the wrong ticker; identifier resolution is lookup work, not language work.
4. Scale and unit slips. Filing conventions differ across forms and eras; a value off by a factor of a thousand reads as fluently as the right one.

Scale alone settles it. Q1 2026 saw 1,824 13F filers report 1.87M positions worth $53.7T, plus 51,000+ insider transactions. No model memorizes that corpus, keeps it current, and recalls it reliably. The honest architecture fetches.

What doesn't work

• "Be accurate" prompts. Instructions change tone, not knowledge; the cautious version is the same guess with hedging stapled on.
• Bigger models. Failures get rarer and more convincing at once. A model right nine times in ten trains you to stop checking, exactly when the tenth answer costs you.
• Trusting training data. Stale and uncited by definition. With 13F deadlines quarterly (in 2026: Feb 17, May 15, Aug 14, Nov 16), whole reporting periods appear after any cutoff. The model isn't lying; it's answering a present-tense question with past evidence.
• Temperature zero. Determinism makes the wrong number reproducible, not correct; the same hallucination every run can pass a naive consistency check.

What works: grounding + provenance

1. Give the model a tool, not a memory. Let it fetch the datapoint at answer time (via an API or MCP server) instead of recalling it. Its job shrinks to choosing the right query and explaining the result.
2. Return the source with the data. Every value should arrive with its origin, a timestamp, and ideally a link to the underlying filing. Bare numbers from a tool move the risk rather than removing it: the agent can still misattribute or mis-date what it fetched.
3. Require citation. Instruct the agent to answer only with fetched, sourced numbers and to show the source. If it can't fetch it, it says so.

This is exactly why provenance is a feature, not a footnote. When the data layer hands back { value, source, timestamp, url }, the agent can cite primary evidence and a human can verify it in one click.

The grounding checklist

This is the practical difference between a chatbot and an agent workflow. A chatbot can produce a fluent answer. An agent with a data tool can produce an answer you can audit.

A concrete pattern for market data

Start with a source-backed endpoint, then let the agent call it through MCP. For example, a developer can search Arkolith's public data API like this:

curl -H "Authorization: Bearer YOUR_KEY" \
  "https://arkolith.com/api/v1/search?q=NVDA"

The response resolves the query to a concrete entity, for example NVDA's institutional ownership, instead of free-associating from the name. The same data is exposed through the MCP quickstart, so the model never memorizes filing facts: it asks the tool, gets sourced data, and answers with evidence attached.

Ask an ungrounded assistant "what is Warren Buffett's largest disclosed position?" and you get a fluent answer of unknown vintage. A grounded agent fetches the tracked Berkshire portfolio and attaches the filing period. Both might name the same stock; only one tells you as of when, and in 13F work that distinction is the whole product.

Each SEC question runs on its own clock, and a grounded agent should know which applies:

An agent quoting a 13F position as "current" cites real data and still misleads: the snapshot can be weeks old on the day it files. Phrase 13F answers as "as of the latest reported quarter" and Form 4 answers as near-real-time.

For SEC ownership questions, this maps directly onto the data:

• 13F holdings should point back to the original SEC filing or an Arkolith filing page.
• Insider trades should point back to Form 4 source data.
• Identifier mappings, such as CUSIP to ticker, should be treated as resolved data, not guessed text.

The data pages for 13F ownership and Form 4 insider transactions are built for that pattern: agents can move from natural-language questions to sourced facts without inventing the missing middle.

The agent rule that prevents most bad answers

• Connect an MCP server that exposes sourced data (what is an MCP server?).
• The agent calls a tool, gets a value plus its provenance.
• The agent answers: "X, per [source], as of [date]" with a link.
• Anything it can't source, it flags rather than invents.

Four lines in a system prompt do more than any accuracy plea:

1. Every number must come from a tool call in this conversation.
2. State the source and as-of date next to the number.
3. If the tool cannot return it, say so and show the query you tried.
4. Never state a current value from memory, even when confident.

Use that as a hard rule in financial workflows: no fetched source, no stated number. If the agent cannot fetch the latest filing, holding, or transaction, the answer should say what is missing and show the query it tried. That is slower than a hallucination and much cheaper than trusting one.

How to test that the guessing actually stopped

Grounding is a behavior claim, so test behavior. Five probes catch most regressions:

1. The withheld-data probe. Ask about a real entity your tools do not cover. The only pass is a stated miss; a confident number means memory is leaking back in.
2. The stale-bait probe. Ask for a fund's "current" holdings; pass only if the answer carries the reporting period. A 13F snapshot presented as live fails even when every figure matches.
3. The provenance audit. Sample production answers and follow every citation; value, date, and entity should all match. Misses cluster fast and point at the tool output to tighten.
4. The identifier trap. Ask about a security by CUSIP only, or a ticker that changed hands; the agent should resolve through the tool, not pattern-match the name.
5. The tool-down drill. Revoke the key and re-run your standard questions. Correct degraded behavior is "unavailable, here is what I tried," not a quiet fallback to memory.

Run these before launch and after any prompt or model change. Hallucination is the model's default; the harness suppresses it, so re-test the harness.

Frequently asked questions

Does RAG solve hallucination?

Retrieval helps, but only if what you retrieve is sourced and the model is required to cite it. Retrieving unsourced text just moves the problem. Structured tool calls beat document retrieval for market numbers: the value arrives typed, dated, and attributable.

How is this different from a normal API?

A normal API returns a value. A provenance-first one returns the value and where it came from, so the answer is verifiable without redoing the research.

Is MCP required?

No. A REST API can ground a model if your code calls it and passes the result into the prompt. MCP is useful when the assistant itself should discover and call the right tool. For agent workflows, start with the quickstart.

How fresh can grounded SEC data actually be?

It depends on the form. Form 4 trades arrive within 2 business days and 13D disclosures within 5 business days, while 13F holdings can lag quarter end by up to 45 days. A grounded agent states that latency instead of implying everything is live.

What should the agent do when data is missing?

It should say the datapoint was unavailable, show the attempted lookup, and avoid filling the gap from memory. Missing data is an operational issue; invented data is a trust issue.

---

Arkolith stamps every datapoint with its source and timestamp so your agent cites ground truth. Get a key, read the MCP quickstart, or explore the 13F data layer.