The release went out on a Thursday afternoon. CI was green, all 214 API tests passed, and by Friday morning support had a dozen tickets from people who couldn’t apply a discount code at checkout.

Nothing was down. The endpoint returned 200. The response had every field our tests looked for. The discount just wasn’t being applied to the total.

I’ve thought about that release a lot since, mostly because the tests weren’t wrong. They checked exactly what we told them to check. That was the problem.

What the tests were actually asserting

Here’s roughly what our test for the discount endpoint looked like:

def test_apply_discount(client):
    resp = client.post("/cart/42/discount", json={"code": "SAVE10"})
    assert resp.status_code == 200
    body = resp.json()
    assert "total" in body
    assert "discount" in body
    assert body["discount"]["code"] == "SAVE10"

Read it again. It checks that the endpoint answers, that the shape is right, and that the code we sent comes back. It never checks that the total went down.

During a refactor, someone moved the price calculation into a separate pricing service and the discount step quietly stopped running. The response still echoed the code. The total was the original price. Every assertion passed.

We had written schema tests and called them functional tests. If you’d asked anyone on the team, they would have said our coverage was good. It was, for the wrong thing.

The difference I care about now

When I explain this to newer folks on the team I keep it simple. A schema test asks whether the response looks right. A functional test asks whether the thing the user wanted actually happened. Both are useful. Only one would have caught our bug.

If you want the longer version, this guide on functional testing walks through the types and where each one fits, better than I can in a blog post. What I want to share here is the practical stuff we changed.

What we changed

The first fix was boring. We rewrote the assertion to check the math:

def test_apply_discount_reduces_total(client):
    cart = client.get("/cart/42").json()
    before = cart["total"]

    resp = client.post("/cart/42/discount", json={"code": "SAVE10"})
    after = resp.json()["total"]

    assert after == round(before * 0.9, 2)

Then we went through the rest of the suite and asked the same question of every test: if the feature silently did nothing, would this fail? A depressing number wouldn’t. Maybe a third of them.

A few patterns kept showing up:

  • Asserting a field exists instead of asserting its value
  • Checking the status code and nothing else
  • Comparing the response to a snapshot that had been regenerated so many times nobody knew if it was correct
  • Testing an endpoint in isolation when the real behaviour depended on an earlier call

That last one bit us twice. Applying a discount only means something if the cart has items in it. Our fixture created an empty cart, and the endpoint happily returned a total of 0.00 with or without the discount. Zero times 0.9 is still zero. Test passes.

Snapshot tests deserve a special mention

I used to like snapshot testing for APIs. Record the response once, compare future responses against it, done.

The trouble starts when a snapshot fails. Nine times out of ten, someone looks at the diff, decides the change was intentional, and regenerates the file. A few months of that and the snapshot just records whatever the code did last. At that point it’s a changelog with assertions attached.

I haven’t thrown them out. They’re decent at catching accidental changes to the response shape, like a renamed field that would break a mobile client. But I don’t let a snapshot be the only check on anything involving money, permissions or state changes.

Where the bugs were hiding

Looking back at the bugs that got past us over the last year, almost none were “the endpoint crashed.” Those get caught. The ones that reached production were all some version of “it returned successfully but did the wrong thing.”

A refund endpoint that marked the order refunded but never called the payment provider. A search filter that ignored one parameter after a query builder upgrade. A permissions check that passed because the test user was an admin.

That last one is embarrassing and also really common. If your fixtures run as a superuser, your access control is basically untested.

Tools didn’t fix this

We tried a few functional testing tools during the cleanup, and some of them helped with the grunt work. Recording real traffic to generate tests (Keploy does this, and so do a couple of others) is handy for getting realistic payloads and multi-step flows instead of hand-built fixtures with empty carts. Contract testing helped between services.

None of them can tell you what “correct” means for your business, though. A tool can notice that the total changed. It can’t know the total was supposed to change. That part is on you, and mostly it comes down to writing assertions about outcomes instead of shapes.

One habit worth stealing

Before merging a new API test, break the feature on purpose. Comment out the line that does the real work, run the test, and make sure it goes red. If it stays green, you’ve written a test that checks the endpoint exists.

It takes thirty seconds. It would have saved us a very long Friday.