What Investors Want in a Data Provider’s Backtest
And why the most effective backtest is usually not the most impressive
·
·
Many providers assume the goal of creating a backtest package is to prove their dataset has “alpha,” so they build sophisticated outputs designed to show impressive performance. Paradoxically, this can have the opposite of the desired effect on data buyers.
Providers we talk to ask many of the same questions: How important is this package? What is it supposed to show? How should it be presented?
That confusion is understandable. Backtests are highly technical outputs that can feel more conclusive about a dataset’s value than they really are. A provider might know every detail of how their data comes together but be lost when it comes to putting together an investor-ready backtest package.
Moreover, even the best-intentioned backtests are considered suspect. A provider backtest is interested-party evidence: useful, but naturally viewed with skepticism.
Buyers know from experience how much can go wrong in the process, including overfitting, lookahead bias, revisions in the underlying data, and other choices and assumptions that would not survive live trading. Serious investors therefore test the data themselves, in their own environment, on their own universe, with their own market data, and using their own assumptions about liquidity, transaction costs and availability timing.
Jordan Hauer (co-author of this post) raised these questions in a recent LinkedIn post, prompting a lively discussion among data providers and investors.
So why do funds ask for an output they don’t trust? And how do you present a backtest that builds buyer interest without undermining credibility?
The real job of the data provider backtest
The provider’s backtest package isn’t meant to answer the buyer’s final question: “Should we buy this dataset?” but a much earlier one: “Is this data worth a serious look for what I’m trying to do?”
A successful backtest leaves the data buyer thinking four things:
- This data provider understands my use case. A thoughtful backtest package signals that the provider is adapting its product to the investor use case. For company-level datasets, mapping observations to publicly traded securities is one obvious example of making the data ready for investment research. Without a backtest or other evidence that the data is research-ready, the buyer is left unsure whether the provider has taken the investment use case seriously.
- The analysis looks credible. A strong backtest package reduces a fund’s skepticism by showing its work. Funds give more credence to results when the data appear to be point-in-time, the assumptions are reasonable, and the methodology addresses the most obvious sources of bias. Buyers also want to see that the data and results have been appropriately normalized and compared with relevant benchmarks. Some buyers care less about the reported performance than about the methodology behind it: the backtest gives them a starting point for their own research.
- If the result holds, the dataset could be valuable. The signal captures something economically meaningful, with an intuitive story and some chance of mattering in a real investment process. Storytelling is a key part of the process here (more on that below).
- We could begin testing this data quickly. Funds strongly prefer data that is easy to ingest, understand, and start testing. A properly-presented backtest can show that the data is ready for research without requiring a major engineering lift. It also does some of the buyer’s first-pass work: presenting basic diagnostics, decile or quintile spreads, factor exposure analysis, and results that are easy to reproduce and update.
A backtest like this will not earn the buyer’s full trust, but it can make the buyer feel their next data evaluation step is worth taking.
It’s important to note that not every dataset needs a trading backtest to accomplish this. Depending on the buyer and the use case, a well-documented dataset, a KPI study, a cross-sectional factor analysis or a reproducible research notebook may be more useful. Some funds even prefer datasets that haven’t been widely researched, seeing these as more likely to contain untapped edge. Evidence of how other institutional investors are already using the data can also be persuasive, where providers are able to share it.
This article combines validityBase’s perspective helping providers build credible backtests and making data easier for investors to evaluate and trust, with that of Amass Insights, an alternative data intelligence platform connecting providers with institutional investors.
What a credible package looks like
Provide evidence that the data is point-in-time
This is a precondition. If the data, mappings, or derived features are not point-in-time, the rest of the backtest does not matter very much. A beautiful result built on data that isn’t point-in-time is not reliable evidence of alpha.
If the pipeline uses LLMs, buyers will want assurance that the historical outputs were generated using only information available at the time and are free of LLM lookahead bias.
A provider’s goal should be to convincingly answer the question of whether a live strategy could realistically have acted on the underlying data. This means not only that the source data existed, but that the relevant mappings, derived features, and signals were available before the simulated trades were executed. At every step, the test is the same: “Would I have made that trade at that time, based strictly on the information I would have had then?” If the answer is no, or if the buyer can’t reliably tell, the backtest needs rethinking.
Choose signals with an economic intuition
Even when a backtest chart looks good, the absence of an intuitive story can leave investors wondering whether the result reflects genuinely predictive data, common risk exposures, modeling choices, or chance. A plausible economic mechanism makes it easier to believe that an investment signal could persist beyond the historical sample.
A strong use case often starts narrowly. The provider first shows that the data is related to a specific company KPI, operating outcome, or independently verifiable external metric that the data could plausibly help explain.
Take a hypothetical provider of store foot-traffic data. It might first show that its quarterly visit counts for a retailer track that retailer’s reported same-store sales, matched to the same stores and fiscal quarters. Only then would it ask whether the visit data available before each earnings release pointed the right way relative to consensus, and whether a simple long-short basket built on that gap earned a return.
The comparisons need to be meaningful: the geography, business segment, fiscal calendar, and measurement definitions should line up. Providers should also disclose the period tested, number of observations, and whether they explored other targets.
The provider can then widen the analysis to an investment use case, with the backtest linking the specific outcomes the data predicts to investment returns. These are distinct claims. Explaining a company’s reported results does not establish that the data could have predicted them in advance. And predicting a KPI does not necessarily establish a profitable trading opportunity, since market expectations, prices, timing, and implementation all matter.
Match the evidence to the buyer
What constitutes useful evidence depends on the buyer. A systematic investor may want to see simple signal diagnostics, while a fundamental investor may care more about whether the dataset can explain or predict a particular company KPI. A hypothetical trading strategy is not necessarily the most useful starting point for every dataset. Start with the claim your evidence supports, and let the buyer decide how it fits into their investment process.
Keep your signal simple
Simple signals leave fewer degrees of freedom for overfitting. Rank correlations, decile or quintile spreads, and simple long-short baskets are more useful than heavily optimized results. For KPI forecasts, a simple model compared with a seasonal baseline or the consensus estimate available at the time can be more informative than a sophisticated trading strategy.
Funds do not expect data providers to generate the actual trading strategy they’ll implement – they just want to see that the data has predictive value before they dedicate resources to it.
Show robustness
Funds want to know in which scenarios the signal stops working. Historically robust signals are more valuable because they are more likely to work in live trading.
Does the signal’s effectiveness hold across time periods? Is performance concentrated in a particular sector or liquidity bucket? Can rebalances be lagged by a day or 10 days without undermining the alpha? Is the signal correlated to known risk factors?
Does the result depend on only a handful of securities? And if the backtest presents investment returns, how sensitive are they to transaction costs, turnover, liquidity constraints, and shorting assumptions? Providers should be clear about which assumptions they’ve made and how changing them affects the result.
When presenting investment returns, distinguish gross returns from returns after estimated trading costs, and make those cost assumptions explicit.
Try enough variations and some will produce impressive historical results merely by chance. Overfit or highly sensitive signals often lose their alpha in response to slight variations in initial conditions. Robustness checks show your data’s predictive content is not a one-off artifact.
Make the result reproducible and easy to evaluate
A buyer should feel that backtesting your data is easy, and that they can quickly reproduce your results before iterating on them for their own research.
A useful package clearly documents the investment universe, the handling of timestamps, the handling of missing data, and the rules for signal construction and rebalance. Methodology notes and notebooks can matter just as much as the backtest performance report. The buyer should be able to follow how raw observations were transformed into derived features and ultimately into the signals being tested.
A good package should also make the next step straightforward: how to access the data for a trial, which examples or analyses to reproduce first, and who can answer questions about the methodology and underlying data.
Show live evidence
Live results are especially useful because they demonstrate both out-of-sample evidence of a signal’s effectiveness, and the existence of a functioning production pipeline. A backtest, however careful, is assembled with hindsight.
A properly maintained, verifiable live record provides something different: evidence that the signals and supporting data existed before outcomes were known, with any subsequent changes recorded. Even a few months of verifiable live record gives a buyer evidence that no backtest can.
Conclusion
The best provider backtest package is rarely the one with the highest Sharpe or the most sophisticated strategy. It is the one that leaves buyers thinking:
This provider understands my use case. The way they have tested the data looks plausible and intuitive. This result is worth investigating further, and we can start testing the data ourselves in hours, not weeks.
The bar for a successful package is having the buyer feel “this is worth a serious look,” not “this is proven alpha.”
If you are a data provider preparing an investor-facing backtest package, validityBase and Amass Insights are happy to compare notes.
Before vBase, Dan founded Clerkenwell Asset Management, a systematic hedge fund that bought and evaluated external data to build trading strategies. He brought that buyer-side perspective to vBase, which builds infrastructure for teams making credible predictive claims.
Jordan has spent more than a decade helping data providers commercialize their datasets and institutional investors find and use alternative data. Before Amass, he worked at Hunter Global Investors, where he added quantitative analysis to the fund’s investment process and improved its internal data systems.
