Give an artificial intelligence recent prices, interest rates, volatility and a few economic indicators. Then ask it one question. Will the market be higher or lower next week?
The answer arrives in seconds, and it reads well. The model names a trend, admits a risk, and delivers a forecast in the voice of a television strategist. None of that tells you whether the forecast is any good.
Language models are very good with language. They can summarize an earnings call, turn a filing into a list of risks, and explain an unfamiliar derivative without sending you to a textbook. Predicting a price is a different job. It is not a writing contest. It means pulling a weak, unstable signal out of a system where millions of people already hunt for the same advantage.
That makes market forecasting a useful reality check on the AI boom. The question is not whether an AI writes impressive commentary. It is whether its forecasts beat simple alternatives when the future is genuinely hidden.
A low bar that is hard to clear
Suppose the S&P 500 rose today. You could predict that tomorrow looks like today. You could predict the long-run average. You could make no call at all and give equal odds either way. These rules sound crude. They are also hard to beat.
Financial returns carry a great deal of noise. Patterns appear, disappear, and sometimes reverse once investors find them. A complex model can match past data almost perfectly while learning a relationship that never repeats. Every extra choice a researcher makes — which data, which period, which settings — makes it easier to mistake luck for insight.
So a fair test moves forward through time, the way a real forecaster lives. The model learns from the past, predicts the next period, and never sees a date it should not have seen. Shuffle those dates at random and tomorrow's market quietly shapes yesterday's model. The forecast that results is not a forecast at all.
Every competitor in the same room
The experiment gives every competitor the same information and the same deadline. Six markets that trade in size daily: broad equities, government bonds, oil, gold, foreign exchange, and Bitcoin. Three horizons: one day, one week, one month. Nine forecasters, from a literal coin flip up through simple rules, statistical models, two machine-learning models, and a language model asked the same question at every date.
Every answer the language model gives is archived with the exact model version that produced it. That is not bookkeeping. Commercial AI systems change without notice, so a result you cannot trace to one model on one date is not a durable experiment.
Four questions then score the forecasts. Did the model pick the right direction? How large was its typical error? Did it perform steadily, or did one short stretch supply the whole advantage? And would any improvement survive the cost of trading on it?
The result, and the arm that did not run
The disclosure comes first: the language-model arm did not run. Nobody supplied a credential to this run, so no forecast below came from ChatGPT or any other language model. The headline question is not answered here. What follows is the rest of the field, over the course the language model would have run — and it turns out to be the more interesting story.
The measured result
Every score carries a margin of error: the range luck alone could produce. Pooled across markets, no model's margin of error excluded 50% at any horizon. The best at 20 days was momentum, at 51.89% balanced accuracy, and its margin comfortably covers a coin flip.
At one day the best was a random forest, at 50.93%. At five days, a rolling mean, at 50.87%. On the size of its mistakes, the best model beat the baseline by 0.2 per cent at five days and by nothing at all at one day and twenty.
On this evidence, none of these models has been shown to forecast direction better than chance.
Notice which families won. At five days a rolling mean beat both machine-learning models. At twenty days momentum — a rule you can write on a napkin — beat everything. Once costs came out, the two machine-learning models finished bottom at −5.55% and −7.47% a year, with ridge regression at −5.36%. The most advanced tools in the study were the ones that lost money.
Raw accuracy is a trap, and a coin flip proves it
There are two ways to score a directional forecast, and the gap between them is where most published edges live. Ask only how often a forecast was right, and a model that says "up" every day scores well above 50% in a market that drifted upward — while knowing nothing. Score up days and down days separately, then average them, and that free win disappears. Every model here that looked impressive on the first measure fell back to roughly 50% on the second.
Some forecasts in this study really do clear 50% by more than luck explains. Nine of them belong to a baseline — a literal coin flip, or "whatever happened yesterday." On twenty-day equities the coin flip scores 61.99%. The measure that flatters a real model flatters a worthless one just as much.
Pick any forecaster and see where its interval lands
The bar is the 95% margin of error around the accuracy. The line through it is the score itself. The vertical rule is a coin flip. Start with the random forest on equities, then try the coin flip on the same market at twenty days.
This demonstration needs JavaScript. All 162 results, with their margins of error, are in the repository's reports/results.json.
A ranking that reshuffles when you change the asset is a ranking of noise.
And it does reshuffle. Across the 18 contests — six markets by three horizons — the winner was a different model eight times, and no model won more than six. The average model moved 7.6 places between its best market and its worst, out of nine. Ridge regression finished first in one contest and last in another. A model that is genuinely good at forecasting does not need the asset to cooperate.
What luck alone can produce
Pooled across all six markets, the best one-day model scored 51.54% raw accuracy. Here is that number as a picture: a fair coin, flipped as many times as the study made one-day equity forecasts, over and over.
1,781 flips of a fair coin, several hundred times over
This one is a simulation running in your browser, not a repository result. It is the standard every number above is measured against.
This demonstration needs JavaScript — it simulates fair coin tosses to show the range luck alone produces.
Roughly one run in ten reaches the best score in the study. That is the whole problem with a narrow margin of error: it turns an ordinary run of luck into a finding, and a finding into a product.
The large return that is not evidence of anything
One number looks like a refutation of everything above. After trading costs, the momentum rule returned 16.84% a year against 4.23% for the best baseline — a gap of 12.61 percentage points.
Look at where it came from. Only 12 of the 18 market-and-horizon combinations made money at all. The average is carried by oil at twenty days, which returned 114.16%, and Bitcoin at five days, at 43.46%. The same rule ranged from −4.30% to 114.16%, a standard deviation of 29.67% — several times the gap it is meant to demonstrate. These are full-size bets, with no borrowed money, in the two most volatile series in the study. Most of that return is volatility, not forecasting, and this model's accuracy still cannot be told apart from a coin flip.
A large return from a forecast that has not been shown to beat chance is what being right about a few big moves looks like. It is also, uncomfortably, what most published strategy backtests look like from the outside.
| model | family | net of costs | worst cell | best cell | positive cells |
|---|---|---|---|---|---|
| momentum | simple rule | 16.84% | −4.30% | 114.16% | 12/18 |
| rolling mean | simple rule | 11.81% | −4.24% | 77.29% | 13/18 |
| yesterday's direction | baseline | 4.23% | −51.16% | 28.45% | 12/18 |
| coin flip | baseline | 3.56% | −49.37% | 43.39% | 11/18 |
| autoregression | statistical | −0.64% | −30.50% | 22.70% | 9/18 |
| ridge | statistical | −5.36% | −31.41% | 11.54% | 6/18 |
| random forest | machine learning | −5.55% | −47.06% | 10.01% | 7/18 |
| gradient boosting | machine learning | −7.47% | −57.80% | 25.93% | 5/18 |
Every forecast becomes a full-size bet, long or short, charged two basis points. That is the crudest possible way to turn a forecast into a position, and it is deliberate. Sizing a position well is another place where a researcher's judgement can quietly improve the answer.
The difference between explanation and prediction
These results do not settle whether AI is useful in finance. Since the language model never ran, they do not even open the question the headline asks. What they do establish is how hard the task is. The models here are not broken and the test is not rigged against them. They are hunting the same weak signal as everyone else, and the right expectation for that job is close to no edge. Close to no edge is what the table shows.
An analyst does more than guess whether a chart will rise or fall. Analysts search documents, compare definitions, find inconsistencies, form hypotheses, and explain decisions. Those tasks are made of language and structure — what language models handle best. Raw price forecasting holds little language and enormous competition, and is probably the single task a language model is least suited to.
The distinction matters. A calculator can be valuable without writing an investment memo. A language model can be valuable without holding a secret map of next week's prices.
What this means for investors
The danger is not that an AI will always be wrong. It is that a polished answer makes an uncertain forecast feel more reliable than it is. So ask three questions of any AI-powered market prediction:
- What simple benchmark did it beat?
- Was it tested on data the model had never seen?
- Are the results stable across markets and time periods?
If those questions have no clear answers, the forecast is closer to a demonstration than evidence.
AI may change finance profoundly. It already changes how information is processed and how fast research appears. But the market does not pay for eloquence. It pays for being usefully right before everyone else, and that remains a much harder test.
What this study cannot support
This is one decade, six markets, and one narrow task. A negative result about forecasting the direction of a price is evidence about that task, not about language models in finance. The data series are imperfect too, in ways that change how large the returns look without changing the comparison between models, which all see the same data.
And when the language-model arm does run, its result will belong only to the exact model version archived with it. Commercial models change without notice. A number produced by "GPT" in 2026 is not a number about "GPT" in 2027, and any study reporting one without the archived version is reporting a rumour.
Companion repository ai-market-forecasting-test, published with the verified results.
Repository specification
Build a publication-quality, fully reproducible GitHub repository named ai-market-forecasting-test. Its purpose is to test whether a large language model or conventional machine-learning model can forecast liquid financial markets better than simple baselines. Do not assume or manufacture a positive result.
Use Python 3.12. Select six daily public market series representing broad U.S. equities, U.S. government bonds or yields, WTI oil, gold, a major foreign-exchange rate and Bitcoin. Prefer stable public sources with clear licensing; record the exact source URL, field, units, timezone, retrieval date and transformation in data/data_manifest.csv. If an API key is required, provide a documented alternative or cached derived fixture that is legally redistributable.
Evaluate three horizons: 1, 5 and 20 trading days. Compare: zero-return and historical-direction baselines; rolling mean and momentum rules; autoregression or regularized linear regression; random forest; histogram gradient boosting; and an OpenAI model called through the official Responses API. Keep the model selectable in configuration, but require a snapshot identifier rather than a moving alias for the published experiment; official OpenAI documentation explains that snapshots lock a specific model version. The model must receive a fixed, versioned prompt containing only information available at the forecast origin. Require a Structured Outputs JSON schema containing direction, expected return and confidence. Read credentials only from environment variables, never commit secrets, use temperature zero where supported, archive every raw response and record the exact model snapshot, response ID, request time, prompt hash and token usage. The repository must still run all non-AI experiments when no API credential is present.
Use expanding-window or rolling-window evaluation. Never randomly split observations. Keep feature engineering inside each training fold, add a gap where necessary, and prevent revised or future data from leaking backward. Report directional accuracy, balanced accuracy, MAE, RMSE, correlation, calibration by confidence bucket and a simple after-cost strategy return. Include confidence intervals or block-bootstrap uncertainty. Separate statistical accuracy from economic usefulness.
Produce these article outputs in reports/article_values.md: best model and baseline by horizon; directional accuracy with uncertainty; relative MAE; performance by asset; performance by market regime; result after a clearly disclosed trading-cost assumption; and a plain-language interpretation. Produce at least four publication-ready SVG and PNG charts: model-versus-baseline scorecard, accuracy by asset and horizon, cumulative after-cost strategy comparison, and rolling performance stability.
Repository structure: README.md, LICENSE, pyproject.toml, .env.example, Makefile, src/, tests/, configs/, data/raw/ ignored, data/processed/, reports/figures/, notebooks/, and GitHub Actions. Include deterministic seeds, data validation, unit tests for temporal splits and leakage, a model-card-style limitations section, and one command such as make reproduce that rebuilds every non-LLM result from source. Add make reproduce-llm for the credentialed portion. The README must lead with the research question and verified result, not installation instructions.
Discussion