Trying out openevolve

The target was a fuzzy string-matching helper — the bit of code that takes a name you typed, compares it against every asset in the project, and picks the closest one. Levenshtein distance, filled out as a full (n+1)×(m+1) matrix, once per candidate. Small, self-contained, and with an exact correctness oracle: whatever the evolved version returns has to match what the original returns, for every input.

That last part is where the whole day went.

The loop itself is simple. openevolve shows the model the current best program and its score, asks for a diff, applies it, runs your evaluator, keeps what scores well. My evaluator timed the candidate against the reference on the same workload and returned a speedup ratio. So the number the search climbs is a number I wrote. It found three different ways to make that number go up without making the code faster.

One. It swapped toLocaleLowerCase() for toLowerCase(). Genuinely faster — it skips an ICU lookup — and identical in output, right up until someone’s device is set to Turkish, where İ lowercases differently. My correctness pass ran in one locale, so it couldn’t see the difference. Closed by re-running every case under LC_ALL=tr_TR.UTF-8.

Two. A program reported 3.26x. On re-measurement it was 2.05x. Nothing wrong with the program; the harness was lying. I’d pinned the timed run to two specific CPU cores but left the (CPU-heavy) correctness checks unpinned, so other workers wandered onto the timing cores, and my scoring took each program’s fastest run independently — so whichever one happened to catch a quiet moment won. I’d tested the harness for noise beforehand, with a steady background load, which slows both programs equally and therefore proves nothing. Bursty interference is the one that hurts.

Three. The final run reported 10.42x. Pulling it apart: 7.14x once re-measured honestly, 7.81x with one of its edits reverted, and 2.07x once I removed a module-level cache it had added — a cache keyed on the identity of the asset array, which my benchmark reused across all 40 queries. Build once, hit 39 times. It had also rewritten a helper function sitting outside the block marked as evolvable, and rewritten it wrong: the original strips trailing file extensions one at a time, its version stripped them greedily, so a.webp.png came out as awebp.

Which taught me the thing I’d most want to pass on: the “evolve block” markers are advisory. openevolve exports a function to parse them, and never calls it in the pipeline. They’re a convention in the prompt, nothing more. The evaluator is the only real boundary, and it has to be written as though nothing else exists.

After hardening all three, the honest standings: the shipped code at 1.00, my hand-written optimisation at 1.68, and the best legitimate evolved program at 2.47 — one-row DP rotation, charCodeAt over charAt, typed arrays, sizing the row to the shorter string, length-difference pruning, early exit on exact match. All real, all incremental, all better than what I wrote by hand.

That 2.47x came out of the first run, four iterations in. The following 104 iterations produced exploits, not improvements.

Every headline number was wrong the first time I measured it. Not because the model was being adversarial — it was doing exactly what it was told — but because a scoring function is a specification, and specifications have gaps. Three rounds of fixing my own oracle bought more than 104 rounds of evolution did.

One aside on the cheap-LLM part, since it wasn’t quite as cheap as expected: a reasoning model breaks token budgets in both directions. openevolve decides whether a model reasons by checking its name against a hardcoded list of OpenAI prefixes, so ours wasn’t recognised and its limit capped thinking and output together — four minutes of a run that looked perfectly healthy and returned an empty string. And Asuwa’s cost estimate was out by about 19x, because it was priced per request the way an ordinary chat session is. With reasoning models, count tokens.