InnerWeb
Skip to main content

Safety system error monitoring: implementation notes for IT

Error reporting should help and not flood the mailbox or slow users down when the recipient is unavailable. The system sets event limits in code and short connection timeouts, while security alarms are never muted. Below are the stages, checkpoints and conditions for deploying monitoring.

21
events per hour per error type
Source: card
89
log events per hour per instance
Source: card
3 s
longest delay with an unavailable recipient (1 s + 2 s)
Source: card

Stage 1: one shared reporting project

Error reports go to a single shared Sentry project. For the plant this means events from the instances are reviewed in one place by the vendor's team, and the plant's IT department does not have to maintain its own recipient. If security policy forbids sending data outside, this must be settled at the start.

Ask the IT department about the outbound traffic policy. If the network blocks all outgoing traffic from the application host, reporting will not get out, and the same applies to webhooks. In such a network, alarm webhooks are saved persistently and an external poller picks them up over an inbound channel.

Stage 2: event limits so the mailbox is not flooded

The code-side limit is 21 events per hour per error type, and 89 per hour per instance for log entries in total. This prevents a single fault repeated thousands of times from flooding the recipient and the team.

For the plant's IT the conclusion is practical: the number of notifications depends on the number of different error types, not on the number of repeats of one. If you get many alerts, it means many different things are happening, not that the system repeats one.

Reporting limits per hour

Reporting limits per hourBar chart: 21 events per hour per error type, 89 per instance020406080100eventsPer error type: 21 events21 eventsPer error typePer instance (log entries): 89 events89 eventsPer instance (log entries)
Source: Cell card: error reporting description

Checkpoint 1: security alarms are not muted

Event limits apply to repeated application errors. Security alarms are not muted: they reach the recipient regardless of the limit, because missing them would be more dangerous than too many notifications.

For the plant's IT the conclusion is practical: set up a separate mailbox or channel for security alarms and name the person responsible for the response. A security alarm requires a decision within the working day, not archiving.

Checkpoint 2: synchronous transport and an unavailable recipient

Sentry transport is synchronous, so a limit of 1 second for connection and 2 seconds for response is set. This way an unavailable recipient does not extend the handling of every error. Without such a limit the user would wait for the server response as long as the report recipient does not answer.

In the worst case, a user's error is handled 3 seconds longer, which is the sum of the connection and response limits. This is an order of magnitude the user will notice, but which will not block the system.

Transport time limits

Transport time limitsChart: connection limit 1 second, response limit 2 seconds012secondsConnectionConnection: 1 seconds1ResponseResponse: 2 seconds2
Source: Cell card: synchronous transport, 1 s and 2 s limits

Stage 3: release version as a commit hash

The release version in error reports is the commit hash of the deployed image, the same as the image tag in the registry. This lets the administrator link a report to the exact code version on the instance and to the specific container image.

When analysing an error after an update, first check whether the release version in the report matches the version running on the instance. A difference means part of the traffic is still served by the old image.

When accepting the monitoring, record three decisions in the protocol: who receives ordinary notifications, who receives security alarms, and how cover works during holidays. The vendor's team and the plant's IT should know the same rules, so that after the first alarm no time is lost establishing who it goes to. It is also worth reviewing the list of recipients and error types once a quarter, and removing those that nobody needed.

  • Agree the outbound traffic policy to the report recipient.
  • Decide who in IT receives notifications and in which mailbox.
  • After an update, compare the release version with the running image.
  • Do not overload the mailbox: choose the error types that require a response.

Video

Frequently asked questions

Will a failure of the report recipient stop the system?

No. The limit is 1 second for connection and 2 seconds for response, so at most it extends error handling by a few seconds.

Does the event limit mute security alarms?

No. The limit applies to repeated application errors, and security alarms are not muted.

How do I know which code version reported an error?

The release version in the report is the commit hash of the deployed image, the same as the image tag in the registry.

Let's set up monitoring without an alert flood

We will agree with your IT the outbound traffic policy and notification recipients, so that alerts go only where they need a response.

Let's talk