Grounding LLM Responses: RAG, Tools, or Fine-Tuning?

Why language models get financial numbers wrong

A language model's weights are a lossy, frozen statistical compression of its training text. That representation is good at concepts and bad at point values. Ask a strong model what a 13F filing is and you get a clean explanation. Ask it for a specific fund's third-largest position last quarter and you are sampling from a distribution of plausible-sounding answers.

Three failure modes do most of the damage.

Staleness. Disclosure runs on a statutory clock that no training cutoff can keep up with. A 13F is due 45 days after quarter end (the 2026 deadlines fall on February 17, May 15, August 14, and November 16). A Form 4 insider transaction must be reported within two business days of the trade, and a Schedule 13D activist stake within five business days. Even the freshest model is permanently behind the filing stream.

Plausible interpolation. Models emit the most likely next token, and a confident, well-formatted share count is more likely than an admission of ignorance. Academic work on hallucination has generally found that error rates climb on long-tail numeric facts, precisely because training data offers thin and contradictory coverage there. Wrong numbers that look right are the most expensive kind of wrong; we walk through the concrete failure patterns in how to stop AI hallucinating market data.

Entity and version confusion. Funds file amendments that restate holdings, managers file for multiple vehicles, tickers change, and share classes blur together. A model that memorized an early version of a filing will happily blend it with the amended one. The disclosure regime itself has known limits too, covered in how accurate is 13F data.

RAG, fine-tuning, and tool calls do different jobs

The grounding conversation usually collapses three distinct techniques into one bucket. They are not substitutes, and picking the wrong one for the question type is the most common architecture mistake in agent systems.

Fine-tuning is the right tool for teaching a model domain behavior: what transaction codes mean, how to phrase a screening summary, how your team labels sectors. It is the wrong tool for facts. Weights cannot be updated per filing, and there is no way to ask a fine-tuned model where a number came from.

RAG shines on narrative documents. If the question is "what does this prospectus say about leverage," embedding a corpus and retrieving relevant passages works well. The retrieved text is real, the model paraphrases it, and you can display the passage as evidence.

Tool calls invert the flow. Instead of pushing data into the context window in advance, the model recognizes that a question needs external data, emits a structured function call, and receives typed JSON back. The number in the final answer was never sampled from weights at all; it passed through the model as payload. This is the architecture behind MCP servers, and the practical difference between exposing data over MCP versus plain REST comes down to who writes the call, which we compare in MCP vs REST API.

Where RAG breaks down for market data

RAG fails on financial numbers for structural reasons, not implementation ones, so better chunking or a bigger embedding model will not save you.

Chunking destroys tables. A 13F information table is thousands of rows of issuer, identifier, share count, and value. Split it into fixed-size chunks and the row structure dies; retrieve three chunks and you have a random slice of a portfolio, not the portfolio.

Similarity search cannot aggregate. "Which funds added semiconductor exposure last quarter" is a query over position deltas across filers. Nearest-neighbor retrieval returns passages that mention semiconductors; it cannot compute a sum, a rank, or a quarter-over-quarter change. Aggregation needs a database, not an index.

Scale makes it worse. Arkolith's Q1 2026 13F dataset alone covers 1,824 institutional filers and 1.87 million long positions totaling $53.7 trillion in reported value. Embedding all of that is expensive, and querying it by semantic similarity makes no sense when what you actually want is a filter and a sort.

As-of semantics. Filings get amended, and amendments restate the past. A vector index typically holds one version of a document, so a RAG pipeline will quietly serve superseded numbers unless someone builds version-aware retrieval, which almost nobody does. Structured stores can treat amendment supersession as a first-class data-model feature instead.

None of this means RAG is bad. It remains the right approach for prose-heavy material: risk factors, footnotes, call transcripts. The practical rule is simple. Ground narrative questions with retrieval, and ground numeric questions with tools.

Tool calls in practice: typed queries, structured answers

A tool-calling agent works a financial dataset the way an analyst works a terminal: resolve the entity first, then pull the table. Resolution comes first because names are ambiguous and identifiers are not.

curl -H "Authorization: Bearer YOUR_KEY" "https://arkolith.com/api/v1/search?q=berkshire%20hathaway"

That resolves a free-text name to a CIK, the SEC's permanent identifier for a filer. With the CIK in hand, the holdings call returns the structured table:

curl -H "Authorization: Bearer YOUR_KEY" "https://arkolith.com/api/v1/funds/1067983/holdings"

And the covered universe is itself enumerable, so an agent can discover what exists before querying it:

curl -H "Authorization: Bearer YOUR_KEY" "https://arkolith.com/api/v1/funds"

Each response is typed JSON: share counts as integers, values as exact figures, dates as dates. The model never reconstructs a number from tokens; it formats a payload. The same surface is exposed as an MCP server, so Claude or any MCP-capable agent discovers these endpoints as tools and calls them without custom glue code. Setup takes a few minutes via the quickstart, and there is a full walkthrough in connecting market data to Claude over MCP.

Two properties matter for grounding. First, freshness becomes the API's problem rather than the model's: when a new filing lands, the next tool call reflects it, with no re-embedding or retraining step. Second, refusal becomes possible. A model improvising from weights cannot know what it does not know, but a tool can return an empty result, and an empty result is honest in a way a hallucinated holding never is. Threshold effects make this concrete: 13F applies to managers above $100 million in covered US equities, so a small fund's absence from results is signal, not failure.

Provenance: the grounding layer most setups skip

A tool call moves the trust boundary; it does not remove it. The model now faithfully repeats whatever the API returns, which means the data layer's own sourcing becomes the weakest link. If your vendor silently merges entities or drops amendments, your agent is grounded in a defective ground truth, and grounded-but-wrong is arguably worse than ungrounded because it arrives with citations-shaped confidence.

The fix is per-datapoint provenance. Every institutional position and insider transaction in Arkolith carries the SEC EDGAR accession number of the filing it came from, so any figure an agent quotes can be traced to the primary document and checked by a human, or by a second agent, on EDGAR full-text search. That turns a citation into something that resolves, like a footnote you can actually click through. The SEC's own Form 13F FAQ is the authoritative reference for what these filings do and do not contain, and its limits are worth internalizing: long US equity positions, quarterly, on a 45-day lag.

Provenance also compounds across data types. Institutional positions answer "who owns it," while the 51,000+ insider transactions tracked from Forms 3, 4, and 5 answer "what are the people inside doing," with Form 4's two-business-day deadline making it the timeliest of the ownership disclosures. On a page like NVDA, both layers sit side by side, each row linked back to its filing.

The end state for a grounded agent answer is three layers: the claim, the structured datapoint behind it, and the accession number behind that. If any layer is missing, you are trusting. If all three are present, you are verifying. Grounding is finished only when verification is cheap.

Frequently asked questions about grounding LLM responses

What does grounding an LLM actually mean?

Grounding means constraining a model's factual claims to an external, verifiable source rather than its internal training weights. In practice the model either reads retrieved documents (RAG) or calls tools that return structured data, and every claim in the answer can be traced back to that source. Without grounding, factual output is a probability-weighted guess.

Is RAG the same as grounding LLM responses?

RAG is one grounding technique, not a synonym for grounding. It anchors answers in retrieved text chunks, which works well for prose and poorly for large numeric tables, aggregations, and time-sensitive values. Tool calls against a structured API are the stronger grounding method for numbers.

When is fine-tuning the right choice?

Fine-tune for behavior, not facts: domain vocabulary, output format, classification conventions, and tone. Facts baked into weights go stale immediately and cannot be cited or updated per filing. A common production pattern is a well-prompted or fine-tuned model for form, plus tool calls for every number.

How do I verify a number an LLM gives me?

Ask for the source identifier, not just the value. When the data layer carries provenance, each figure links to an SEC accession number you can open on EDGAR and compare against the original filing. If the model cannot produce a checkable source, treat the number as unverified.

This article explains public filings and data concepts. It is not investment advice.