InnerWeb
Skip to main content

Error monitoring at the plant: best practices

Error monitoring has two opposing tasks: show every real failure and do not flood the on-duty engineer with hundreds of repetitions of the same event. Below are practices for the system administrator at the plant: limits per error type, exclusion of non-failures, linking an error to the release version and separating safety alarms.

21
events per hour, limit per error type
Source: function documentation, error reporting
89
events per hour, limit per instance for logs
Source: function documentation, error reporting
5 000
errors per month, shared budget
Source: function documentation, error reporting

Problem: noise drowns the signal

The most common configuration mistake is the lack of a limit. A single repeating event then generates hundreds of identical notifications, and the on-duty engineer stops reacting to them. The opposite mistake is filtering too aggressively, which also cuts the first signal of a real failure.

Good settings are a compromise based on measurement. Before you set a limit, measure how many events of each type the system generates on the worst day without incidents. Only such data lets you set a limit that does not cut normal work and stops an avalanche of repetitions.

Practice 1: a limit per error type set from measurement

Set the limit of events per hour per error type based on measurement, not guessing. For example, a limit of 21 per hour per error type sits above normal noise and clearly below the level of an avalanche of repetitions. As a result, the on-duty engineer sees every error type, but no single type uses up the whole budget.

Remember that the monthly budget is often shared by several instances. For example, it may be 5 000 errors, a shared resource, so the limit protects not only the on-duty engineer but also the other instances.

Example event limits per hour

Example event limits per hourHorizontal bar chart: example limits of 21 events per hour per error type and 89 per instance.020406080100events/hLimit per error typeLimit per error type: 21 events/h21Limit per instance (logs)Limit per instance (logs): 89 events/h89
Source: Function documentation: error reporting; example values

Practice 2: a limit per instance for log entries

Log entries have a separate difficulty. A message with a variable glued in, for example an order identifier, looks to the monitoring like a new error type at every occurrence. A limit per error type will not stop it, because there are hundreds of types.

The solution is an additional shared limit per instance, for example 89 events per hour for log entries. The second limit closes the gap that the first one leaves open.

Practices 3 and 4: exclude non-failures, keep the alarm

Exceptions describing invalid input data, such as no access, a missing page or a malformed request, are not system failures and should be excluded from reporting. The user entered a wrong address; the system did not fail. Excluding such exceptions markedly reduces noise without losing signal.

Strictly separate occupational safety alarms from the limit mechanism. Safety alarms are never muted: a person in danger always raises an alarm, and the log records the event. The rule is: error monitoring may be limited, but an alarm for a person in danger may not.

Practice 5: release version in the report and a test after the update

Require every error report to carry the release version. The release version is the commit hash of the deployed image, the same as the image tag in the registry, so every error can be linked to a specific change. For the administrator this means one question instead of an investigation: from which version does this error appear?

After each update, check whether the new events do not look like previously known ones. If the release version in the report matches the deployed one, comparing before and after the update takes a few minutes.

  • limit per error type set from measurement (example: 21 per hour)
  • limit per instance for log entries (example: 89 per hour)
  • exclusion of exceptions describing invalid input data
  • safety alarms independent of the limit mechanism
  • release version in every error report

How to implement these practices step by step

Start with a week of measurement without limits: record how many events per hour each error type generates on a normal day. Choose the worst day without incidents and set the limit with a margin above that level. Then check that the limit is clearly lower than the level that would mean an avalanche of repetitions.

Next, set the exclusions of non-failures, add a limit per instance for logs and check the behavior under a large number of repetitions in a test environment. Finally, write the thresholds into the on-duty procedure, so that the person receiving an alarm knows what a limited number of events for a given error means: that the error occurs more often than the counter shows.

Checklist

Video

Frequently asked questions

Can a limit hide a real failure?

A limit restricts the number of repetitions of the same error type, not its first occurrence. Occupational safety alarms are outside the limit and are not muted.

Why is an error with an identifier in its text a problem?

Monitoring treats every different text as a different error type. That is why an additional limit per instance works for log entries: 89 events per hour.

What is the release version in an error report for?

It is the commit hash of the deployed image, the same as the image tag in the registry. This lets you link an error to a specific change.

Let us set up monitoring for your plant

We will go through limits, exclusions and the on-duty procedure with your IT team. Book a short conversation with InnerWeb.

Let's talk