The 5 worst bugs I've seen on production - #3 the €300,000 double refund
Published about 1 year ago
This one reached us through friends at another company and made everyone sit up. Refunds were going out twice per request. It ran for about 15 minutes before someone noticed the totals looked wrong and pulled the plug. The cost was roughly €300,000.
What is it?
In payment systems, operations like “refund” should be idempotent, which means running them more than once does not change the result beyond the first time. When idempotency breaks, the same request can be applied twice. See: Idempotence (Wikipedia).
Problem
There was a bug in the refund logic that caused a double payment. I don’t know the exact code path, only that it resulted in every refund going out twice for a short window.
Impact
Real money was lost. The issue ran for about 15 minutes. It was caught by an engineering manager doing a quick sanity check after noticing an anomaly in the refund numbers on the monitoring dashboard.
Solution
I wasn’t involved in the fix. My takeaway was simple: add more double‑checks and tests around payment code, set clear alerts for unusual refund volume or totals, and do a short manual sanity check after any change that touches refunds.
Lesson learned: treat payment code like a loaded tool. Assume duplicates, add idempotency and invariants, and set hard alerts on rate/volume.
Read previous
Read next
Related articles
The 5 worst bugs I've seen on production - #4 the $60 otp ddos
An OTP endpoint was abused across many IPs; budget burned twice after no action items.
The 5 worst bugs I've seen on production - #5 the animation memory leak
Long sessions leaked ~2MB per animation; snapshots revealed unreleased lottie instances.
Microlearning is the new way to learn
How I use microlearning to keep up with fast-moving tech trends through short, daily doses of content across social media and YouTube.