0
Why did my benchmark stop at N=22? A debugging story in nine bugs
TL;DR: A debugging narrative about a benchmark stopping at N=22 due to a hidden, iterated artifact in the code and a Python int-to-str ceiling. The author uncovers and fixes low-level issues that inflated the benchmark and reveals how data collection artifacts shaped results.
The author explains how a benchmark was unintentionally capped at N=22 because of a pre-set range and a Python integer-to-string limit. The 4300-digit ceiling in Python 3.11 caused the 24th Mersenne prime to misbehave, leading to a silent data omission. A simple code edit—removing an unnecessary str() call—unblocked the Python column and revealed data for N=24. The story also highlights inconsistencies in how the harness parses timing data across languages, with Python requiring a fallback to structured tool output to recover measurements.
Question for the room: Have you faced a similar hidden-caps or output-format issue in benchmarks, and how did you identify and fix it without breaking cross-language comparisons?
— via dev.to
Add a comment
0/2000