Teaching language models to replicate expert judgment in financial document triage

The judgment problem in finance

Outperforming the market is hard. When investors share access to the same public information, alpha has to come from insight built on taste and judgment. That judgment is difficult to articulate and teach, whether the learner is human or a language model, because it comes from experience.

Even when an investor’s work is broken into its simplest constituent tasks, those tasks turn out to be surprisingly difficult for large language models. One illustrative slice is information triage: deciding which documents deserve attention.

Why information triage matters

Investors are bombarded every day with news articles, research reports, company filings, emails, and internal write-ups. Reading is the easy part. The real work sits on top of reading: filtering, interpreting, segmenting, and locating the useful signal. These judgments are repeated throughout the daily workflow and consume substantial time.

If that triage step could be automated, it would free attention for higher-level synthesis and decision-making, which is the part of investing that most resists automation.

The core question

Given that off-the-shelf language models perform poorly on simple financial tasks, the researchers asked whether it is possible to teach them financial judgment directly. Their answer is that with high-quality human annotations, language models can be trained to interpret text with expert-level taste.

According to the post, their proprietary model outperformed all frontier models they tested on information accuracy and recall, at a fraction of the cost.

How the study was set up

The team evaluated models on six information-filtering tasks drawn from investors’ daily workflows. They reported that other internal tasks follow similar patterns to the six released tasks, with frontier models underperforming relative to internally trained ones.

Accuracy, the percentage of documents correctly labeled according to the firm’s investors, was the primary measure. For classification tasks, they also reported F1 score.

A sample task: financial article relevancy

One of the released tasks asks a model to classify whether a financial article is relevant to a C-suite investment professional. Evaluation metrics for the task are F1 score and accuracy. The broader pattern across all six tasks is that judgment, not raw reading, is where general-purpose models fall short.

What the results suggest

The findings point toward what the authors call a vision of differentiated intelligence, in which models are tuned for specific organizational needs rather than treated as one-size-fits-all assistants. For knowledge work built on subtle judgment, including investing, domain-specific training on curated expert labels can outperform larger general models while costing less to run.

The practical lesson for teams building AI into research and analysis workflows is that the limiting factor is often label quality. Frontier capability matters, but expert annotation is what closes the gap between a model that reads and a model that judges.