Ask Claude which Hong Kong stocks worth more than $30 billion have gained the most this year, and with web search alone it does something sensible. It searches, reads, and searches again. In our test it used all ten of its searches, found the Hang Seng's return for the year and six-month figures for a few big banks, guessed that Lenovo was probably near the top, and stopped. "I can't give you a top list," it said.
Give the same model the same question with a financial data layer connected, and it comes back with a ranked table: Lenovo up 139%, WuXi AppTec up 94%, Swire Pacific up 68%, Techtronic up 46%, each with its start and end price.
We wanted to know how often that happens, and on which kinds of questions. So we asked Claude Opus 5.5 200 stock questions twice: once with web search, and once with web search plus Distilla, our research data layer, connected through MCP. Every answer was graded against a reference answer written before the test.
Key takeaways
- For one number from one company, web search is fine. It passed 77 of 90 single-figure and everyday questions. With Distilla, Claude passed 88.
- For anything that spans many companies or a stretch of history, web search mostly can't answer. Price moves across a market, estimate revisions, valuation against history, rankings and screens: 20 of 110 passed with web search, 105 with Distilla.
- Across all 200 questions, Claude with Distilla passed 193 (96%). With web search alone, it passed 97 (48%). Scoring a partial answer as half, that is 98% against 61%.
- When web search failed, it usually said so. Of its 103 answers that did not pass, 48 said the data could not be found and only 5 stated wrong facts.
- The gap held in every market. Web search passed 54% of US questions and 44% of Asian ones. With Distilla, Claude passed 96% and 98%.
Background
Large language models now reason well enough that, for financial research, the limit is usually the data. Can the model find the number? Is it current? Does it cover every company the question asks about, or only the ones that rank well in a search?
Most AI assistants answer those questions with web search. Web search works when the answer sits on a public page: a company's results release, a news report on a big move. It struggles when the answer has to be computed from data that no single page holds: a ranking across a market, a week-by-week path of estimates, a valuation multiple compared with its own two-year history.
Distilla is a read-only research data layer for AI agents. It covers listed companies in the US, Japan, Hong Kong, Korea and China A-shares, with financials, consensus estimates, valuation multiples, daily prices with written explanations of large moves, and summaries of sell-side research, investor podcasts and YouTube videos. An assistant connects to it through MCP and queries it like any other tool.
We tested the same model, Claude Opus 5.5, two ways:
- Web search: Claude with web search, up to 10 searches per question.
- With Distilla: Claude with the Distilla MCP server and the same web search.
Both runs got each question exactly as written, with no system prompt and no hint to use one tool or another. The 200 questions cover five markets and eight types of question, from a single figure in an annual report to a market-wide screen. Full details are in the methodology section at the end.
Accuracy
Each answer was graded PASS, PARTIAL or FAIL against a reference answer and a pass rule written before the test. The accuracy score counts PASS as 1, PARTIAL as 0.5 and FAIL as 0.
| Web search | With Distilla | |
|---|---|---|
| PASS | 97 | 193 |
| PARTIAL | 50 | 6 |
| FAIL | 53 | 1 |
| Accuracy score | 61% | 98% |
Compared question by question, Claude with Distilla scored higher on 103 questions, the same on 94 and lower on 3. Every tie was a question both setups passed. Of the 3 where web search did better, two came from errors in Distilla's own data: one quarterly revenue figure, and the cause given for one price move. On the third, asked how a stock reacted to its results, Claude with Distilla led with the big move the day after rather than the results-day reaction.
Accuracy by type of question
The overall score hides a split. On some types of question the two setups are level. On others the gap is wide.
| Type of question | Questions | Passed: web search | Passed: with Distilla |
|---|---|---|---|
| Single figures from filings | 45 | 42 | 45 |
| Everyday questions about one stock | 45 | 35 | 43 |
| Price moves and their causes | 32 | 6 | 30 |
| Valuation against its own history | 29 | 5 | 28 |
| Analyst estimate revisions | 15 | 4 | 14 |
| Rankings and screens | 19 | 1 | 18 |
| Financial comparisons across companies | 8 | 2 | 8 |
| Sell-side research | 7 | 2 | 7 |
Ask for one number, and web search does the job
Most of the time, a single figure from a filing is easy to find. Ask for a company's capital expenditure or net income for a given year, and the web search run usually found the annual report and read the number correctly: 42 of 45.
Everyday questions about one stock went almost as well: 35 of 45. These were reactions to earnings, monthly returns, revenue growth and distance from the 52-week high. Most of the misses were monthly returns, where the answer needs a closing price at each end of the month and web search often found only one.
If most of your questions look like these, a model with web search will serve you well.
Ask for a list or a history, and it runs out
The picture changes when the question needs data across many companies or over time:
- Which stocks moved more than 7% yesterday, and why?
- How has the consensus for this year's earnings changed since July?
- Which large companies trade furthest above their own two-year median valuation?
Web search passed 6 of 32 questions on price moves and their causes, 5 of 29 on valuation against history, 4 of 15 on estimate revisions and 1 of 19 on rankings and screens. With Distilla, Claude passed 30, 28, 14 and 18.
Sell-side research questions, such as how five brokers read one company's results, were closer. Web search passed 2 of 7 and Distilla 7. Broker reactions are often reported in the press, so the web finds some of them, though rarely all.
Error breakdown
When the web search answer did not pass, we sorted what went wrong into three kinds.
| What went wrong | Answers |
|---|---|
| Could not find the data and said so | 48 |
| Partial answer: some items right, not enough to pass | 50 |
| Wrong answer: stated facts that were wrong | 5 |
Most misses were incomplete answers or unavailable data. In 48 answers, Claude said it could not find the data, explained what it had found instead, and often pointed to Bloomberg or FactSet as the place to look. In 45 of those 48, it had used all ten searches first.
That is the right behaviour for a model without the data. It doesn't make anything up. It just doesn't answer the question.
Three things lie behind it:
- It found no page with the answer. For a ranking of valuation premiums, a week-by-week path of estimates or a complete list of the day's movers above a size cut, our web-search run did not find it. Answers like these have to be computed from data.
- Ten searches don't go far. The web search run used its full budget on 98 of the 110 questions about price moves, valuation, estimates, rankings and research (89%), against 18 of the 90 single-figure and everyday ones.
- Lists come back partial. When it did produce a list, it usually had the best-known names and missed the rest. Most of the 50 partial answers look like this.
Wrong answers were rare: 5 of 200. Claude with Distilla had 7 answers that did not pass, all of them partial except one.
Examining errors by kind
Could not find the data
"Which US companies worth more than $100B trade at the biggest premium to their own 2-year median forward price-to-sales as of October 23, 2026?"
Reference: Intel (+101%), CrowdStrike (+79%), Palo Alto Networks (+69%), Dell (+64%), Seagate (+45%).
The web search answer explained, correctly, that no public page ranks companies this way, and that the calculation needs two years of analyst sales forecasts that sit in paid terminals. Then it explained how to run the screen yourself.
With Distilla, Claude pulled two years of weekly readings for 98 companies, worked out exact medians for the 27 that could rank near the top, and ranked the premiums: Intel at about twice its median, then CrowdStrike, Palo Alto Networks, Dell and Seagate. It took 36 tool calls, the most of any question in the test.
I couldn't find a ranked answer to this. No published screen or article I found compares US companies worth over $100B on their forward price-to-sales against their own 2-year median as of October 23, 2026. I also can't work it out myself, because that needs the daily history of analyst sales forecasts for each company over two years. That data sits in paid tools like FactSet, Bloomberg, Koyfin or S&P Capital IQ, and my searches couldn't reach it. I'd rather not guess at a list.
Background I did find:
- Overall market: S&P 500 Price to Sales was 3.563 as of 2026-10-20, according to S&P Dow Jones Indices. That's 38.6% above its long-term average of 2.57. But this is the trailing ratio for the whole index, not forward and not for individual companies.
- Micron probably isn't near the top. In late October, at about $887 a share, Micron trades at a forward price-to-earnings (P/E) ratio of 7. Its revenue jumped to $133.19 billion versus $37.38 billion for the prior year, so its sales multiple has likely shrunk rather than grown.
How to run the screen yourself (e.g., in Koyfin, FactSet or Bloomberg):
- Who's included: US-listed companies with a market value above $100B on October 23, 2026.
- The ratio: company value divided by expected sales over the next 12 months, on October 23, 2026.
- The baseline: the median of that daily ratio from October 23, 2024 to October 23, 2026.
- Ranking: sort by today's ratio divided by the median, minus 1, from highest to lowest.
- Things to check: whether you use share-price value or enterprise value (which adds debt); leaving out banks and insurers, where sales figures mean less; and companies with under two years of trading history, like SpaceX, which listed in June 2026. Those need a shorter window or should be left out.
If you can export forward price-to-sales history from one of those tools, I can do the calculation and ranking for you.
Let me discover the available multiple types.
Intel trades at the biggest premium: its forward price-to-sales (P/S) is 6.46×, about twice its 2-year median. CrowdStrike, Palo Alto Networks and Dell follow, all more than 60% above their own medians.
How I measured it: forward P/S is price divided by expected sales over the next 12 months. For each company I took the median of the 104 weekly readings from Oct 23, 2024 to Oct 23, 2026 and compared it with the Oct 23, 2026 reading. The size cut is market cap above $100B on that date, which gave 98 US companies.
| # | Company | Fwd P/S now | 2-yr median | Premium |
|---|---|---|---|---|
| 1 | Intel (INTC) | 6.46× | 3.22× | +101% |
| 2 | CrowdStrike (CRWD) | 37.8× | 21.1× | +79% |
| 3 | Palo Alto Networks (PANW) | 21.3× | 12.6× | +69% |
| 4 | Dell (DELL) | 1.31× | 0.80× | +64% |
| 5 | Seagate (STX) | 6.92× | 4.63× | +49% |
| 6 | AMD (AMD) | 10.2× | 7.36× | +39% |
| 7 | Merck (MRK) | 5.18× | 3.76× | +38% |
| 8 | Fortinet (FTNT) | 14.9× | 10.9× | +37% |
| 9 | Western Digital (WDC) | 5.37× | 3.95× | +36% |
| 10 | Marvell (MRVL) | 11.2× | 8.36× | +34% |
| 11 | Coca-Cola (KO) | 8.04× | 6.20× | +30% |
| 12 | Cloudflare (NET) | 32.9× | 25.4× | +30% |
| 13 | Johnson & Johnson (JNJ) | 5.95× | 4.72× | +26% |
| 14 | Amgen (AMGN) | 5.92× | 4.74× | +25% |
What stands out:
- Two different kinds of premium.
- CrowdStrike, Palo Alto, Fortinet, Dell, Merck and Coca-Cola are at or near their 2-year highs. Coca-Cola's current reading is close to its 2-year peak.
- Intel, Seagate and Western Digital are well above their medians but have already fallen a lot from peaks earlier in the window (Intel about 11×, Seagate and Western Digital about 14.6–14.8×). The premium mostly comes from very low readings in the older part of the window.
- Cybersecurity stands out as a group. CrowdStrike, Palo Alto, Fortinet and Cloudflare are all in the top 12.
- Some chip and AI names are not stretched on this measure. Micron, SanDisk and Applied Materials are roughly at their medians, Lam Research is about 11% below, and Nvidia and Broadcom are at their 2-year lows. Their share prices have risen, but expected sales have risen about as fast.
Limits of the screen:
- Not every median was calculated. I computed exact medians for 27 companies. I left out the rest because their current multiple was below or close to their 2-year average, or at most about 30% above their 2-year low, so they couldn't beat roughly +30%. A few lower-volatility names were dropped as unlikely rather than strictly ruled out: Thermo Fisher, McKesson, Marriott, Apple, Altria and Philip Morris. Their premium to the 2-year average is 15–18%, and for steady stocks the median sits close to the average.
- Bank of America (about $350B) is excluded. It has price data but no company record or valuation multiples in the database. Novo Nordisk and BHP have the same gap.
- "US" follows the database's headquarters field. That field also puts some foreign companies listed in the US, such as TSMC, ARM, Rio Tinto and PDD, under the US. None of them made the top 14.
- A few series are shorter than two years. SanDisk, TSMC, BlackRock, American Express and Rio Tinto have 73–93 weekly readings instead of 104. None of them is in the table.
The same thing happened on a much simpler question. Asked how China Life's Hong Kong shares performed in August 2026, the web search answer found prices for 20 and 21 August, but not the closing prices on 31 July and 31 August, so it could not give the month's return. With Distilla the answer was one lookup: HK$29.20 to HK$30.30, up 3.8%.
Partial answer
"Which US large caps above $20B rose 7% or more on October 23, 2026, and what was the common theme?"
Reference: four cybersecurity stocks (CrowdStrike, Cloudflare, Zscaler, Palo Alto Networks) rallied after AI-found flaws in PHP were disclosed and US agencies warned about AI-driven hacking.
The web search answer found three of the four from a list of the day's top gainers and spotted the cybersecurity theme, but ran out of searches before it found the reason. With Distilla, Claude listed all four and gave the trigger.
"How have analysts' consensus estimates for Nintendo's fiscal year ending March 2027 revenue and EPS changed since early July 2026?"
The web search answer pieced together the revenue estimate from two sources and got the direction right, down about 1%. It could not find a current EPS estimate at all. With Distilla, Claude gave both from dated snapshots: revenue down 0.9%, EPS up about 13%.
Wrong answer
"Several large Japanese companies split their shares in 2026. For those worth more than $10 billion, when did each split take effect, what was the ratio, and how has the stock done since?"
The web search answer listed Fujikura's split as 5-for-1 and marked it done. It was 6-for-1, with an ex-rights date of 30 March. With Distilla, Claude gave the right ratio. A split ratio is what you use to adjust price history: with the wrong one, the pre-split price, the cost basis and every return since are off by 20%.
"What was Shandong Hongqiao Aluminum Industry Holding's net income attributable to shareholders for fiscal 2024?"
The web search answer said it could not find a listed company by that name, assumed the question meant the Hong Kong-listed China Hongqiao Group, and gave that company's profit of about RMB 22.4 billion. The company in the question is listed in Shenzhen, and its restated figure is about RMB 17.2 billion. With Distilla, Claude found the right listing and the right figure. Similar names, parent companies and dual listings are an easy trap, especially outside the US.
"How did Chugai Pharmaceutical's share price perform in August 2026?"
The web search answer could not find the 31 July close, so it started from its estimate of the 3 August close. That skipped a 3.5% fall on the first trading day of the month, and the answer reported a gain of about 3.4%. The stock actually ended August down 0.3%, from ¥6,932 to ¥6,913. With Distilla, Claude gave both month-end closes and the right direction.
All three mistakes are costly for the same reason: each answer cited its sources and gave a confident number, so nothing told the reader to check it.
Results by market
| Market | Questions | Passed: web search | Passed: with Distilla |
|---|---|---|---|
| US | 96 | 52 (54%) | 92 (96%) |
| Asia (Japan, Hong Kong, Korea, China A-shares) | 96 | 42 (44%) | 94 (98%) |
| Several markets | 8 | 3 (38%) | 7 (88%) |
Web search did worse on Asian questions than on US ones. English-language coverage of Japanese, Hong Kong, Korean and mainland Chinese stocks is thinner, and many of the figures sit in local-language filings and exchange pages. With Distilla, the pass rate was about the same in every market.
How much each setup searched
With web search alone, Claude averaged 7.4 searches a question: 4.5 on single-figure and everyday questions, and 9.7 on the rest, close to the cap.
With Distilla, it averaged 6.6 Distilla calls a question: 4.2 on the simple questions and 8.6 on the rest, with a maximum of 36. It rarely needed the web: 0.2 searches a question on average, and any search at all on only 11 of 200 questions.
The answers were similar in length. The median web search answer ran to 358 words and the median Distilla answer to 431. The bigger difference was form. 151 of the 200 Distilla answers included a table, against 37 of the web search answers: when Claude had the data, it showed the ranking or the time series rather than describing it.
What this means if you use AI for research
- For a single figure from a filing, web search is enough. Check the period, and check that it found the right company and listing.
- For screens, rankings, estimate revisions and valuation history, connect a data source. Without one, expect the model to stop partway or give you part of the list.
- Outside the US, expect web search to struggle sooner. Local tickers, fiscal years and thinner English coverage all add friction.
- Check what matters against the filing, whatever the source. A data layer can be wrong too, and in two of our questions ours was.
- Ask the model to say what it couldn't find. The most useful answers in our test, from both setups, said where their data stopped.
Conclusion
On questions a single web page can answer, Claude with web search is already good, and a data layer adds little. On questions that have to be computed from data across many companies or over time, web search runs out of searches, or out of pages to find, and says so. That is where a data layer makes the difference: 105 of 110 passed with Distilla, against 20 with web search alone.
Methodology
Setup. Both runs used Claude Opus 5.5 with no system prompt. Each got the question exactly as written, with no hint to use one tool or another. Web search was capped at 10 searches a question in both runs. Distilla calls were not capped. Each question ran once per setup, between 29 October and 2 November 2026.
Questions. 200 questions across the US, Japan, Hong Kong, China A-shares and Korea, in eight types (see the table under "Accuracy by type of question"). Examples:
- "What was Sysco's diluted EPS for the three months ended June 30, 2026?"
- "How did Snowflake's stock react to the quarterly results it reported around September 2026?"
- "Which Japanese stocks above $15B moved 6% or more in a single day between October 19 and October 28, 2026, and why?"
- "Where do the forward P/E multiples of Lam Research, KLA, Applied Materials and Tokyo Electron sit within their own two-year range?"
- "Compare how Goldman Sachs, Morgan Stanley, JPMorgan, Citi and Nomura read WuXi AppTec's Q3 2026 results."
Grading. Each question has a reference answer and a pass rule, both written before the test. The pass rule defines PASS, PARTIAL and FAIL for that question, with tolerances. A typical rule: "PASS = at least 3 of the top 5, with the premium within 10 percentage points. PARTIAL = 1 or 2 of the top 5. FAIL = none." Grading was done by Claude Opus 5.5 using Promptfoo's llm-rubric. The grader saw only the answer, the reference answer and the pass rule. It never saw the question or which run the answer came from. The head-to-head comparison uses the grades directly; the same grade counts as a tie.
Grading check. To check the grader, a member of our team graded 40 answers by hand, without seeing the automated grade or which run each answer came from. The answers were drawn from questions whose reference answers had been fully verified against outside sources. The hand grade matched the automated grade on 39 of 40; the other one differed by one step.
Reference answers. We built each one from Distilla's database, then checked the key facts against company filings, public price histories, news and other data providers. In all, we checked 724 facts. 492 were confirmed outside Distilla. 66 differed only in definition, and we accepted either figure. We corrected 64 where outside sources disagreed with our data. The other 102 could not be checked publicly, for example weekly snapshots of analyst estimates or two-year valuation ranges. All corrections were made before grading.
If you want to try your own questions, Distilla has a free plan
Try DistillaExplore all 200 questions
Every question in the test, with the reference answer, both answers and both grades. Filter by type, market or outcome, or search for a company. Click a question to open it.