↑ ↓ to move, Enter to open, Esc to close

    Chidi's Blog

    If you can't retrain it twice and get the same numbers, you can't trust them

    Two training runs side by side with identical metrics: PR-AUC 0.875, ROC-AUC 0.945, recall 34.1% at 1% and 89.5% at 5%.

    Update, 1 November 2026: an audit found two flaws in the run this post describes: the calibration step was fitted on the model’s own training predictions, and one feature was built from the labels. After fixing both, Moniguard scores PR-AUC 0.959 in batch evaluation and 0.837 when each transaction is scored on its own through the API (synthetic data). The figures below are the ones from 31 October; the case study has the current numbers.

    A model report says PR-AUC 0.875. How do you know that number is real? The simplest check is also the most often skipped: run the training again and see if you get the same number. When I tried that on Moniguard, my fraud-scoring project, I did not. This post covers why, the three fixes, and the CI check that now runs the training twice on every retrain.

    Where the randomness was hiding

    Moniguard trains on 60,000 synthetic Nigerian transactions over a 180-day period. I already passed a seed to the generator, so I expected identical data each time. Three things broke that.

    IDs from uuid4. Transaction, order, session and device IDs came from Python’s uuid4, which reads from the operating system’s random source and ignores any seed. The IDs themselves are not features, but the generator uses them to build shared-device and shared-entity links, so different IDs meant slightly different data. I replaced them with a helper that draws the hex digits from the seeded random module.

    “Now” was today. Users’ registration dates and transaction times were measured back from the current date. Run the same seed on Monday and on Friday and you get different account ages, which is a real feature for fraud. The generator now takes an anchor date (the simulated period ends on 1 October 2026) and everything is measured from that.

    Faker had its own seed. Names and addresses came from Faker, which keeps its own random state. Seeding random alone was not enough; Faker is now seeded with the same value.

    None of these is exotic. Each one quietly made the data a little different on every run, so the metrics moved, and no single run was obviously wrong.

    The check that keeps it fixed

    Fixing it once is not enough, because the next change to the generator can break it again. So the retraining workflow in GitHub Actions now:

    1. trains the model with seed 42 and writes the model and reports/metrics.json;
    2. trains it again from scratch into a separate folder;
    3. compares the two metrics files field by field and fails the run if any value differs, with a relative tolerance of one in a billion for floating-point values;
    4. runs the test suite against the new model;
    5. only then commits the new model and metrics.

    Three fields are left out of the comparison on purpose: the timestamp, the elapsed time and the p95 scoring latency. They depend on how busy the machine is, not on the data or the model, so including them would make the check fail for the wrong reason.

    What changed in the numbers

    After the fix, the retrained model scores PR-AUC 0.875 (95% bootstrap interval 0.841 to 0.898) and ROC-AUC 0.945, catches 34.1% of fraud at a 1% review rate and 89.5% at 5%, and scores a transaction in 13.0 ms at the 95th percentile. Fraud is 2.84% of the data. Some of these numbers differ slightly from what the README said before, and that is the point: the old numbers came from one run that nobody could repeat. The new ones come back exactly whenever anyone runs make demo.

    Why a business should care

    If a vendor or an internal team shows you a model’s performance, ask them to retrain it in front of you, or in CI, and show that the number comes back. If it does not, you cannot tell an improvement from noise, you cannot audit a past decision, and you cannot tell whether a drop in performance after a change came from the change or from luck. Reproducibility is not academic polish; it is what makes the number worth reading.

    The same caveat as always applies: these results are on synthetic data with fraud patterns I wrote myself, and the most important next step is a public benchmark and then a pilot on real transactions. The workflow file and the seeded generator are in the Moniguard repository on my GitHub.