Problem: noise drowns the signal
The most common configuration mistake is the lack of a limit. A single repeating event then generates hundreds of identical notifications, and the on-duty engineer stops reacting to them. The opposite mistake is filtering too aggressively, which also cuts the first signal of a real failure.
Good settings are a compromise based on measurement. Before you set a limit, measure how many events of each type the system generates on the worst day without incidents. Only such data lets you set a limit that does not cut normal work and stops an avalanche of repetitions.
Practice 1: a limit per error type set from measurement
Set the limit of events per hour per error type based on measurement, not guessing. For example, a limit of 21 per hour per error type sits above normal noise and clearly below the level of an avalanche of repetitions. As a result, the on-duty engineer sees every error type, but no single type uses up the whole budget.
Remember that the monthly budget is often shared by several instances. For example, it may be 5 000 errors, a shared resource, so the limit protects not only the on-duty engineer but also the other instances.
Example event limits per hour
Practice 2: a limit per instance for log entries
Log entries have a separate difficulty. A message with a variable glued in, for example an order identifier, looks to the monitoring like a new error type at every occurrence. A limit per error type will not stop it, because there are hundreds of types.
The solution is an additional shared limit per instance, for example 89 events per hour for log entries. The second limit closes the gap that the first one leaves open.
Practices 3 and 4: exclude non-failures, keep the alarm
Exceptions describing invalid input data, such as no access, a missing page or a malformed request, are not system failures and should be excluded from reporting. The user entered a wrong address; the system did not fail. Excluding such exceptions markedly reduces noise without losing signal.
Strictly separate occupational safety alarms from the limit mechanism. Safety alarms are never muted: a person in danger always raises an alarm, and the log records the event. The rule is: error monitoring may be limited, but an alarm for a person in danger may not.
Practice 5: release version in the report and a test after the update
Require every error report to carry the release version. The release version is the commit hash of the deployed image, the same as the image tag in the registry, so every error can be linked to a specific change. For the administrator this means one question instead of an investigation: from which version does this error appear?
After each update, check whether the new events do not look like previously known ones. If the release version in the report matches the deployed one, comparing before and after the update takes a few minutes.
- limit per error type set from measurement (example: 21 per hour)
- limit per instance for log entries (example: 89 per hour)
- exclusion of exceptions describing invalid input data
- safety alarms independent of the limit mechanism
- release version in every error report
How to implement these practices step by step
Start with a week of measurement without limits: record how many events per hour each error type generates on a normal day. Choose the worst day without incidents and set the limit with a margin above that level. Then check that the limit is clearly lower than the level that would mean an avalanche of repetitions.
Next, set the exclusions of non-failures, add a limit per instance for logs and check the behavior under a large number of repetitions in a test environment. Finally, write the thresholds into the on-duty procedure, so that the person receiving an alarm knows what a limited number of events for a given error means: that the error occurs more often than the counter shows.
Checklist
Video
Frequently asked questions
Can a limit hide a real failure?
A limit restricts the number of repetitions of the same error type, not its first occurrence. Occupational safety alarms are outside the limit and are not muted.
Why is an error with an identifier in its text a problem?
Monitoring treats every different text as a different error type. That is why an additional limit per instance works for log entries: 89 events per hour.
What is the release version in an error report for?
It is the commit hash of the deployed image, the same as the image tag in the registry. This lets you link an error to a specific change.
Let us set up monitoring for your plant
We will go through limits, exclusions and the on-duty procedure with your IT team. Book a short conversation with InnerWeb.
Let's talk