InnerWeb
Skip to main content

Alarms in a network without outbound traffic: notes

When the plant firewall blocks all outbound connections from the application server, alarms cannot leave for the team messenger or for an external address. Alarm webhooks are then stored durably in a queue, and an external poller collects them over an inbound channel. Below are the implementation notes for the IT/OT team: architecture, order of steps, pitfalls and tests.

0
firewall exceptions for outbound traffic from the application server in the described architecture
55 min
aggregation window of the missing-training alert per company

The situation: a server without the right to outbound connections

In plants with a high security regime the application server often stands in a segment from which external connections are not allowed. The rule is justified: it limits the risk of data leakage and communication with control servers of compromised machines. The side effect is that a classic webhook, which the system sends to the messenger, reaches nobody.

The IT/OT team faces a choice: open an exception in the firewall or change the direction of the flow. The architecture described below makes it possible not to open outbound traffic from the application server.

The solution: a durable queue and a poller on the external side

When the client's network blocks outbound traffic from the application host, alarm webhooks are stored durably in a queue, and an external poller collects them over an inbound channel. The alarm is not lost, it waits in the queue to be collected. The direction of the connection is reversed: it is the poller, placed in a segment with access to the outside, that asks for new alarms and passes them on.

For the IT team this has two advantages. The plant firewall remains without exceptions for outbound traffic from the application server, and the list of events is not lost during a brief unavailability of the recipient.

What the poller must provide

The poller should run in a segment that has legitimate access to the target resource, for example the team messenger. At each polling it collects alarms from the queue, passes them on and confirms receipt. If delivery fails, the alarm stays in the queue and is retried.

Sample age of the oldest queue entry during a poller failure test

Sample age of the oldest queue entry during a poller failure testBar chart: the age of the oldest entry grows to 5 minutes during the test and returns to zero after the poller resumes.012345minStart of test: 0 min0 minStart of testAfter 3 min: 3 min3 minAfter 3 minAfter 5 min: 5 min5 minAfter 5 minAfter resumption: 0 min0 minAfter resumption
Source: Sample calculation: poller interrupted for 5 minutes, illustrative values

Order of implementation steps

The recommended order avoids a situation in which a critical alarm learns about a configuration error only during an actual event.

  • Determine which outbound connections from the application server are blocked and which exceptions are allowed.
  • Identify the segment in which the poller can run with the right to go outside.
  • Configure the poller with authentication and a list of alarm recipients.
  • Trigger a test alarm and check that it reaches the queue and then the recipient.
  • Interrupt the poller's connectivity for a few minutes and check that alarms wait in the queue and arrive after it resumes.

Pitfalls for IT/OT

The first pitfall is assuming that no error means delivery. An alarm saved in the queue is not yet a delivered alarm, so an indicator of the age of the oldest queue entry is needed. The second pitfall is a poller that runs but has no right to exit to the messenger: the queue grows and nobody sees it. The third pitfall is polling intervals that are too long. For critical alarms the plant should set the acceptable delay and choose the interval accordingly.

It is also worth remembering channels that do not depend on outbound traffic from the application server. Phone calls from the PBX and SMS messages go through their own infrastructure and have a different failure model, so the notification matrix should not rely on a single channel.

Tests and the audit trail

Sent SMS messages and PBX calls are recorded in the registers, which gives an audit trail. For alarms passed on by the poller, the plant should provide an equivalent trail: the time of entry into the queue, the time of collection and the confirmation of handover.

A regular connectivity test, for example once a month, makes it possible to notice early a change of firewall rules that broke the poller's path. The test result is worth attaching to the plant's connectivity control documentation.

Video

Frequently asked questions

Does an alarm get lost when the firewall blocks outbound traffic?

No. Alarm webhooks are stored durably in a queue, and an external poller collects them over an inbound channel.

Do we need to open a firewall exception for the application server?

No. The direction of the connection is reversed: the poller asks for new alarms and passes them on.

How do we check that the queue works?

Interrupt the poller for a few minutes and check that alarms wait in the queue and reach the recipient after it resumes.

Plan alarms for a network with a firewall

We will go through the network segments, the poller placement and the connectivity test with your IT/OT team before alarms are launched.

Let's talk