The Error With 500 Users Might Matter More Than the Error With 50,000

We had this problem recently while looking at production issues.

One error was happening constantly. Thousands of times. It looked terrible on the dashboard and naturally got everyone's attention.

Another error was much smaller.

A few hundred users.

The difference was that the smaller one happened during a critical flow.

People were getting all the way to the final step and then something failed. They tried again, got the same result, and left.

If you sort an error dashboard by occurrence count, you probably fix the first issue.

If you actually look at what users were doing when the errors happened, you fix the second one.

Occurrence count is the wrong signal#

That's something I've started thinking about a lot with monitoring tools.

We measure technical severity really well.

Occurrences.

Affected users.

Stack traces.

Versions.

But an error doesn't exist in a vacuum.

An exception on a profile avatar and an exception on the Pay button shouldn't necessarily have the same priority just because they happened the same number of times.

What I'd rather see#

PaymentInitializationError412 users affectedMost occurred inside checkout79% did not complete the flow afterwardBegan after the latest deployment180 sessions available to review

At that point the issue is no longer just an exception with a stack trace. I understand what it did to the product.

This isn't platform-specific#

This applies to a browser just as much as it does to a mobile app.

On web it might be a JavaScript exception or a request that keeps returning 500.

On mobile it might be a crash.

I don't really care which runtime produced the signal when I'm deciding what the team should fix Monday morning.

I care about which broken thing is hurting people.

Why this matters#

That's one of the reasons we're interested in keeping replay, errors and product events connected inside ScaleBun instead of making them three completely separate datasets.

Error monitoring is good at answering what broke.

I think it becomes much more useful when it can also answer:

Did anyone actually care that it broke?