Teaching language models to replicate expert judgment in financial document triage

Researchers report that a proprietary language model trained on expert investor annotations outperformed frontier models on financial document triage, scoring higher on information accuracy and recall at a fraction of the cost. The work targets information triage, the filtering and interpretation step that consumes substantial time in investors’ daily workflows and underpins higher-level synthesis. The team concludes that domain-specific training on curated expert labels can outperform larger general-purpose models.

Why information triage matters

Investors are bombarded every day with news articles, research reports, company filings, emails, and internal write-ups. Reading is the easy part. The real work sits on top of reading: filtering, interpreting, segmenting, and locating the useful signal. These judgments are repeated throughout the daily workflow and consume substantial time.

If that triage step could be automated, it would free attention for higher-level synthesis and decision-making, which is the part of investing that most resists automation.

The judgment problem in finance

Outperforming the market is hard. When investors share access to the same public information, alpha has to come from insight built on taste and judgment. That judgment is difficult to articulate and teach, whether the learner is human or a language model, because it comes from experience.

Even when an investor’s work is broken into its simplest constituent tasks, those tasks turn out to be surprisingly difficult for large language models. One illustrative slice is information triage: deciding which documents deserve attention.

The core question

Given that off-the-shelf language models perform poorly on simple financial tasks, the researchers asked whether it is possible to teach them financial judgment directly. Their answer is that with high-quality human annotations, language models can be trained to interpret text with expert-level taste.

According to the post, their proprietary model outperformed all frontier models they tested on information accuracy and recall, at a fraction of the cost.

How was the study set up?

The team evaluated models on six information-filtering tasks drawn from investors’ daily workflows. They reported that other internal tasks follow similar patterns to the six released tasks, with frontier models underperforming relative to internally trained ones.

Accuracy, the percentage of documents correctly labeled according to the firm’s investors, was the primary measure. For classification tasks, they also reported F1 score.

What does a sample task look like?

One of the released tasks asks a model to classify whether a financial article is relevant to a C-suite investment professional. Evaluation metrics for the task are F1 score and accuracy. The broader pattern across all six tasks is that judgment, not raw reading, is where general-purpose models fall short.

What do the results suggest?

The findings point toward what the authors call a vision of differentiated intelligence, in which models are tuned for specific organizational needs rather than treated as one-size-fits-all assistants. For knowledge work built on subtle judgment, including investing, domain-specific training on curated expert labels can outperform larger general models while costing less to run.

The practical lesson for teams building AI into research and analysis workflows is that the limiting factor is often label quality. Frontier capability matters, but expert annotation is what closes the gap between a model that reads and a model that judges.

FAQ

What did the researchers set out to test?

They asked whether language models could be taught financial judgment directly, given that off-the-shelf models perform poorly on simple financial tasks, and answered that high-quality human annotations let models interpret text with expert-level taste.

How did the proprietary model perform compared with frontier models?

According to the post, the proprietary model outperformed all frontier models they tested on information accuracy and recall, at a fraction of the cost.

What is the main practical takeaway for teams building AI into research workflows?

The limiting factor is often label quality. Expert annotation, not frontier capability alone, is what closes the gap between a model that reads and a model that judges.