JetBrains ran a benchmark proving its own search tools beat grep for AI agents. The approach is methodologically sound: same tasks, same models, tooling on vs off, compared task-by-task with p<0.05 significance. The results are real. The conflict of interest is also real, and no amount of transparent methodology makes the vendor disappear from the study.
On Codex 5.2, the primary test model, IDE-native search cut P95 latency 16.44%, from 268.71 seconds to 213.17 seconds. Median latency dropped 8.33% (83.11s to 79.03s). Total cost fell 5.60%, from USD 44.17 to USD 41.67. Budget overruns, tasks exceeding a USD 0.50 cap, dropped from 6.67% to 4.44%, a 33.28% reduction. These are not cherry-picked averages. The P95 improvement means the worst-case tasks got faster, not just the typical ones.
I counted the search steps in the traces: 12+ blind discovery calls without IDE search, versus 3 targeted calls with it. A task labeled "Update service and controller layers for comments and replies" took 472 seconds without IDE search. The agent ran 12+ search and explore steps: list files, grep, jar inspect, javap, curl download, decompile, before it found the relevant code. With IDE-native search, the same task took 127 seconds. Three search calls, done.
Another trace shows a Jackson deserializer task dropping from 150 seconds to 34 seconds. The pattern is consistent: agents with shell tools waste steps on blind discovery. IDE-native search points them at the right files immediately.
Cross-model results add credibility. GPT 5.4 on Kotlin saw a 13.48% total cost reduction, the largest cost improvement across any model-language pair. On Java, GPT 5.4 still improved by 4.07%. However, P95 latency on Kotlin was not statistically significant, meaning the gains concentrate in the typical case while tail behavior stays unchanged.
Here is the part the blog post buries in a footnote: JetBrains reports no statistically significant quality change. IDE-native search does not make agents write better code. It makes them find code faster and spend fewer tokens doing it. Speed, not substance. The agent still produces the same bugs. It just finds the right file to put them in faster.
The vendor problem is unavoidable. JetBrains designed the eval tasks, chose the models, and ran the benchmarks on its own product. The tasks come from JetBrains' internal test suite, heavily weighted toward Java and Kotlin, languages where IntelliJ's project model is deepest and its commercial interest is loudest.
Python, TypeScript, and .NET, the languages most actual agent users work in, are listed as "planned" but untested. The eval tasks also lean toward structured search scenarios (service/controller updates, Jackson deserializers) that favor IDE-native tools over grep. Tasks requiring web search or cross-repo analysis are absent. Nobody tests the races their horse would lose.
Adoption rates across models vary sharply. Codex adopted IDE-native search for 91% of its search calls. Claude Opus used it 53% of the time. Claude Haiku only 28%, preferring grep and find. Models with strong built-in search capabilities benefit less from IDE tooling, which suggests diminishing returns for already-capable models.
The methodology is transparent. Independent work at MorphLLM found that scaffolding matters more than the model for coding agents, which aligns with JetBrains' finding that tooling changes efficiency independently of model choice. JetBrains reports null results. Statistical significance is tested. Configuration variants are compared openly: the "4 Search Tools" variant had the lowest cost (USD 40.53) but JetBrains chose the "Unified Tool + Router" for best latency (79.03s), trading marginal savings for speed. Defensible for interactive sessions, but JetBrains is still choosing the winner from its own race.
The numbers are real. The method is rigorous. But JetBrains is both the referee and the runner. Wait for external replication on TypeScript and Python before restructuring your agent pipeline around IDE-native search.