---
title: "How Good Are LLMs at Sentiment Scoring?"
description: "We find that open-weight models, especially in an ensemble, approach frontier models’ performance. As with labeling, sentiment scoring is now accurate and cheap."
canonical_url: "https://www.vbase.com/blog/llm-sentiment-scoring-benchmark/"
last_updated: "2026-10-10T18:30:56+00:00"
---

# How Good Are LLMs at Sentiment Scoring?

> Benchmarking sixteen closed- and open-weight models and standard featurization tasks

- **Published:** 2026-10-10
- **Updated:** 2026-10-10
- **Author:** Dan Averbukh

![#image_title](https://www.vbase.com/wp-content/uploads/2026/10/sentiment-scoring-ensemble-accuracy.png)

## Overview

In a benchmark of 16 models scoring 660 company mentions, GPT-5.4 had the lowest mean error (0.060), while an open-weight flagship ensemble nearly matched the closed-weight light ensemble (0.082 vs. 0.080). The article compares accuracy, correlation, and cost, and explains why teams should benchmark on their own documents before choosing a production pipeline.

Turning large bodies of text into usable predictive data or investable signals used to be labor-intensive and thus often uneconomical. Our earlier work demonstrated how LLMs remove the barriers to labeling (tickerization) of financial data. This article extends the analysis to sentiment scoring, a workflow at the foundation of many alternative data (AltData) products, signals, and quant strategies.

We find that open-weight models, especially in an ensemble, approach frontier models’ performance. As with labeling, sentiment scoring is now accurate and cheap. The cheapest models in the study can score a thousand company mentions for cents, run on a workstation, and the results match those of commercial models that are orders of magnitude more expensive.

In this article, we show results for the June 2026 model cohort, which will be periodically updated to cover the latest model releases.

## Background

Part 1 of our series on LLM performance in financial data pipelines ([https://www.vbase.com/blog/llm-tickerization-benchmark/](https://www.vbase.com/blog/llm-tickerization-benchmark/)) benchmarked closed-weight models from OpenAI, Anthropic, Google, and xAI on tickerization. Tickerization tells you which companies are discussed in a body of text. Part 2 ([https://www.vbase.com/blog/local-llm-ticker-mapping/](https://www.vbase.com/blog/local-llm-ticker-mapping/)) covered open-weight models.

Featurization determines what the document says about them and transforms this content into quantitative data (features) suitable for statistical modeling and ML. Sentiment-related features are often the foundation of such datasets. This article uses the same data and pipeline as our previous tickerization analysis, while evaluating the ability of LLMs to score sentiment in investment-relevant text.

Legacy approaches to sentiment analysis used word lists such as the Loughran-McDonald dictionaries ([https://sraf.nd.edu/loughranmcdonald-master-dictionary/](https://sraf.nd.edu/loughranmcdonald-master-dictionary/)) or encoders such as FinBERT ([https://arxiv.org/abs/1908.10063](https://arxiv.org/abs/1908.10063)), went stale, and lacked deeper contextual reasoning of humans. Until LLMs, the human-level reasoning needed to find economically relevant sentiment scores in large text corpus was often prohibitively expensive.

## LLM Sentiment Scoring Performance Using Closed-Weight Models

Below are the results for the closed-weight model cohort. GPT-5.4 showed the best performance with a 0.060 mean error and scoring 95% of mentions accurately. GPT-5.3 Chat, its light sibling, beat two of the four flagship models.

These results match the findings for tickerization: model choice matters more than model tier, and flagship pricing does not always buy flagship performance. This fits the general pattern that attentive benchmarking, selection, and harnesses for these models are key to get the most value from their relatively higher costs.

| Model | Tier | Mean error | Accuracy | Correlation |
| --- | --- | --- | --- | --- |
| GPT-5.4 | Flagship | 0.060 | 0.951 | 0.966 |
| Claude Opus 4.6 | Flagship | 0.073 | 0.915 | 0.960 |
| Gemini 3.1 Pro Preview | Flagship | 0.098 | 0.829 | 0.925 |
| Grok 4.20 Beta | Flagship | 0.100 | 0.851 | 0.888 |
| GPT-5.3 Chat | Light | 0.091 | 0.883 | 0.931 |
| Gemini 3.1 Flash Lite | Light | 0.103 | 0.854 | 0.908 |
| Grok 4.3 | Light | 0.117 | 0.798 | 0.899 |
| Claude Haiku 4.5 | Light | 0.123 | 0.804 | 0.877 |

![Closed-weight model sentiment scoring accuracy](https://www.vbase.com/wp-content/uploads/2026/10/sentiment-scoring-closed-weight-accuracy-1024x482.png)

## LLM Sentiment Scoring Performance Using Open-Weight Models

Below are the results for the open-weight models. Qwen 3.6 Plus had the best performance with a 0.096 mean error. It beat Gemini 3.1 Pro Preview and Grok 4.20 Beta, two of the four flagship models that defined the reference datasets.

Intriguingly, the order ranking of open-weight models on sentiment scoring matches that of our earlier LLM tickerization benchmark.

These light models are remarkable commercial tools: Qwen 3.6 35B-A3B and DeepSeek V4 Flash had performance almost as strong as that of leading flagship models. They achieved this performance while fitting on a workstation.

This performance may be adequate for many applications while mitigating various regulatory, operating, and analytical problems with cloud-based closed-weight models.

| Model | Tier | Mean error | Accuracy | Correlation |
| --- | --- | --- | --- | --- |
| Qwen 3.6 Plus | Flagship | 0.096 | 0.862 | 0.918 |
| DeepSeek V4 Pro | Flagship | 0.105 | 0.851 | 0.895 |
| Mistral Large 2512 | Flagship | 0.128 | 0.818 | 0.859 |
| Llama 4 Maverick | Flagship | 0.134 | 0.777 | 0.850 |
| Qwen 3.6 35B-A3B | Light | 0.109 | 0.818 | 0.889 |
| DeepSeek V4 Flash | Light | 0.111 | 0.818 | 0.877 |
| Llama 4 Scout | Light | 0.133 | 0.769 | 0.820 |
| Mistral Small 2603 | Light | 0.188 | 0.681 | 0.765 |

![Open-weight model sentiment scoring accuracy](https://www.vbase.com/wp-content/uploads/2026/10/sentiment-scoring-open-weight-accuracy-1024x482.png)

## Overall LLM Sentiment Scoring Performance

In the overall error leaderboards, GPT-5.4 and Claude Opus 4.6 lead. Surprisingly, model tier does not predict where a model stands in the ranking, illustrating the competitiveness of light and open-weight models in basic tasks. Of course, results will vary based on the idiosyncrasies of the underlying data, so be sure to benchmark on your own documents before committing to a production pipeline.

![Sentiment scoring mean error by model](https://www.vbase.com/wp-content/uploads/2026/10/sentiment-scoring-error-by-model-1024x768.png)

## LLM Sentiment Scoring Using Model Ensembles

An ensemble is a set of models each given the same task – like a group of AI experts. Ensembles work well across all model groups in our study. Ensembles work for the same reason as in prior studies: Vendor errors are not perfectly correlated, so one model's mistake rarely repeats in another, and trimmed mean scoring discards outliers.

| Consensus | Mean error | Accuracy | Correlation |
| --- | --- | --- | --- |
| Closed-weight light | 0.080 | 0.906 | 0.946 |
| Open-weight flagship | 0.082 | 0.912 | 0.931 |
| Open-weight light | 0.098 | 0.867 | 0.901 |

![Consensus sentiment scoring by model tier](https://www.vbase.com/wp-content/uploads/2026/10/sentiment-scoring-ensemble-accuracy-1-1024x571.png)

The open-weight flagship consensus reaches 0.082 and matches the closed-weight light consensus at 0.080. We thus replicate the ensemble benefits from our earlier tickerization study on the harder task of sentiment scoring. Voting across open-weight models buys you the performance of the closed-weight light tier, losing only to the top frontier models. Open ensemble beats all except the very best frontier models (GPT 5.4 and Opus 4.6) in our study. The open-weight ensemble beats all except the very best frontier models (GPT 5.4 and Opus 4.6).

## The Cost of LLM Sentiment Scoring

Price does not strongly track quality within a tier or across tiers of models.

We priced every run’s cost using OpenRouter ([https://openrouter.ai/](https://openrouter.ai/)) at list rates, normalized as $ per 1,000 scored mentions. Accuracy is defined as the share of mentions scored within 0.20 of the reference value. We also benchmarked and priced the four-model ensemble for each model tier.

![Sentiment scoring cost versus accuracy](https://www.vbase.com/wp-content/uploads/2026/10/sentiment-scoring-cost-vs-accuracy-1024x675.png)

Self-hosting can further lower the costs. All four open-weight light models can run on a single workstation using vLLM ([https://docs.vllm.ai/](https://docs.vllm.ai/)) or Ollama ([https://ollama.com/](https://ollama.com/)), turning token cost into raw hardware and electricity. Self-hosting also removes a class of compliance blockers and lets you fix the model version to deal with headaches of point-in-time testing and reproducibility ([https://www.vbase.com/blog/financial-data-must-be-made-point-in-time/](https://www.vbase.com/blog/financial-data-must-be-made-point-in-time/)). Without such control, historical scoring and strictly causal backtests may be impossible.

## Conclusions

Given the cost and performance of LLM sentiment scoring, particularly given the effectiveness of open-weight models, cost and data custody need not decide what data products can be built. The barrier to a high-quality AltData product has moved from basic labeling and featurization to higher-order concerns: product design, context and harness, point-in-time integrity and reproducibility, completeness, production pipelines, and quality of service in live deployments.

Open-weight models score sentiment about as well as closed-weight light models, and their consensus matches the closed-weight light consensus. For many applications, the differences across tiers may now be immaterial, and the cost of cents per thousand articles unlocks fundamentally new levels of analysis, depth of context, and breadth of features. Critically, open-weight models support fully reproducible historical simulations and fully causal backtests that are generally not attainable with externally hosted and closed-weight models.

## Appendix: LLM Sentiment Scoring Benchmark Methodology

The corpus includes 100 financial articles, spanning earnings call transcripts to sector commentary, yielding 660 company mentions. Each model gets the article text and the tickers already resolved for it, then scores sentiment toward each company from -1.0 (most bearish) to +1.0 (most bullish). All runs are at temperature 0.

As for tickerization benchmarks, the four flagship closed-weight models act as an annotation committee. We use the scoring analog of supermajority voting, which is a trimmed mean. The score farthest from the average is dropped, and the rest are averaged to produce the gold/reference value. If the three retained scores span more than 0.50, the mention goes to a human arbiter.

The committee agreed nearly everywhere. Of 660 mentions, just one, a life-sciences company discussed in a complex pharmaceutical article, went to a reviewer.

The sweeps ran in June 2026, using the current model at the time. The tables below are a snapshot rather than a live leaderboard and will be updated periodically for new model releases.

The following numbers describe each model’s performance:

- **Mean error:** The average gap between the model's score and the reference score.
- **Accuracy:** The share of mentions scored within 0.20 of the reference.
- **Correlation:** Pearson correlation ([https://en.wikipedia.org/wiki/Pearson\_correlation\_coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient)) with the reference scores.

## References

- validityBase, "How good are LLMs at tickerization?" [https://www.vbase.com/blog/llm-tickerization-benchmark/](https://www.vbase.com/blog/llm-tickerization-benchmark/)
- validityBase, "Local LLMs can now handle ticker mapping for alternative data." [https://www.vbase.com/blog/local-llm-ticker-mapping/](https://www.vbase.com/blog/local-llm-ticker-mapping/)
- T. Loughran and B. McDonald, "Loughran-McDonald Master Dictionary with Sentiment Word Lists," University of Notre Dame. [https://sraf.nd.edu/loughranmcdonald-master-dictionary/](https://sraf.nd.edu/loughranmcdonald-master-dictionary/)
- D. Araci, "FinBERT: Financial Sentiment Analysis with Pre-trained Language Models," 2019. [https://arxiv.org/abs/1908.10063](https://arxiv.org/abs/1908.10063)
- P. Malo, A. Sinha, P. Takala, P. Korhonen, and J. Wallenius, "Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts," 2013. [https://arxiv.org/abs/1307.5336](https://arxiv.org/abs/1307.5336)