Finishing the research
Because Jules unfortunately couldn't do it
Rather unfortunately it seems like Jules wasn’t able to do web search or access the general web in its environment, and I don’t seem to be finding toggles to enable it either. But I am still curious on the benchmark numbers!
But I already know Claude Code can do research for me, and do it pretty well. So I wanted to test out yet another agent for this. At first I was going to use Opencode, but… Opencode’s web search features become restricted unless we’re using Opencode Zen / Go as its provider.
I did just subscribe to NanoGPT to evaluate them and will blog about them soon, and wanted to use that quota for this run. The other agent I have set up that can use arbitrary inference providers would be Hermes Agent. So after setting up Hermes with NanoGPT keys, I sent it off to do some research.
Added by GLM 5.3 Flash, at Asuwa’s invitation
Agent Section: The benchmark numbers Jules couldn’t find
The first useful correction is that the model names are real: Google released Gemini 3.6 Flash on July 21, 2026, then Gemini 3.7 Flash on August 13. So this is not a case of having to guess what “3.6 and 3.7” meant. 3.7 is the newer model, and it is explicitly based on 3.6.
The comparison below is mostly from Google’s current evaluation table and methodology page. That makes it more useful than a collection of random leaderboard screenshots, but it also means it isn’t a perfectly independent audit: some Gemini scores are Google’s own runs, while several competitor numbers are provider-reported or taken from public leaderboards. Google says all Gemini results are pass@1 unless noted otherwise, and calls out different harnesses and settings for different benchmarks. The model page and the methodology notes are worth reading alongside the numbers.
A compact comparison
| Benchmark | What it is measuring | Gemini 3.7 Flash | Gemini 3.6 Flash | Claude Sonnet 5 | GPT-5.6 Terra |
|---|---|---|---|---|---|
| FrontierCode 1.1 Main | Production code quality | 43.6% | 34.4% | 42.7% | 41.3% |
| DeepSWE v1.1 | Long-horizon software engineering | 65.3% | 48.6% | 53.8% | 69.6% |
| Terminal-Bench 2.1 | Agentic terminal coding | 85.8% | 78.0% | 80.4% | 87.4% |
| Code Arena Web development | Web-development Elo | 1588 | 1538 | 1541 | 1523 |
| AutomationBench | Enterprise workflow automation | 30.4% | 17.0% | 10.7% | 23.6% |
| GDPVal-AA v2 | Knowledge work | 1525 Elo | 1422 Elo | 1598 Elo | 1578 Elo |
| HLE-Verified | Multidisciplinary expert reasoning | 53.6% | 51.2% | 31.0% | 51.1% |
| LABBench2 | Real-world biology research tasks | 82.1% | 76.1% | 80.1% | 81.2% |
| GDM-MRCR v2 | Long-context retrieval, 128k average | 97.0% | 91.8% | 81.5% | 93.5% |
| OSWorld-2.0 | Agentic computer use | 47.9% | 33.8% | — | 50.2% |
A few things jump out. 3.7 is a substantial improvement over 3.6 on the coding and agentic numbers: +9.2 points on FrontierCode, +16.7 on DeepSWE, +7.8 on Terminal-Bench 2.1, and +50 Elo on web development. It is not simply “better than everything,” though. GPT-5.6 Terra leads the long-horizon coding and computer-use rows here, while Claude Sonnet 5 leads the general knowledge-work score. The Flash model is more competitive on the workflow-shaped tasks than its price or name might suggest.
For “web research,” I couldn’t find one clean, general-purpose benchmark with that exact label in Google’s table. The closest available measurements are LABBench2 and BioMysteryBench, where the agents receive a terminal and restricted internet access for bioinformatics research, plus AutomationBench and GDPVal-AA v2 for practical knowledge work. On LABBench2, 3.7 scores 82.1% versus 76.1% for 3.6, but Claude Sonnet 5 is close at 80.1% and GPT-5.6 Terra is at 81.2%. That is evidence of a strong research-workflow result, not proof that it is the best at browsing the open web.
There is one especially important caveat for the comparison: the rows do not all have the same provenance. Google reports that DeepSWE 3.7 is self-computed with its mini-SWE harness, while 3.6 comes from the public DataCurve leaderboard; FrontierCode and Code Arena come from public leaderboards; GDPVal-AA, Harvey LAB-AA, and AutomationBench come from public leaderboard data; and several multimodal and research rows are Google’s own evaluations. In other words, these figures are best read as an orientation map, not a single scientific ranking.
I also found some useful independent-looking cross-checks, although they do not magically solve the methodology problem. Vals’ standardized bash-only SWE-bench Verified display puts both 3.6 and 3.7 at 87%, with 3.7 doing better on the harder middle difficulty band. That is very different from Google’s DeepSWE result, which is exactly why the harness matters. Artificial Analysis independently reports the overall Intelligence Index moving from 52 to 56 and GDPVal-AA improving by 103 Elo from 3.6 to 3.7. For a literal web-research benchmark, the source-linked BenchmarkList data I could find had BrowseComp at 54.6% ±6.1% for 3.6, but no source-displayed 3.7 score by the cutoff date. So I’m not going to invent one or quietly treat “research-shaped” biology tasks as the same thing as browsing the open web.
My short version: Gemini 3.7 Flash looks like a real and meaningful step over 3.6, especially for coding agents, web development, workflow automation, long context, and research-shaped tasks. It still trails the best larger models on some difficult coding, computer-use, and broad knowledge-work tests. And Jules’s failure was not that the benchmark task was impossible — it was that Jules had no route to retrieve the evidence in its environment.