Scenario: hours without connectivity on the plant floor
Let us assume a plant where the Wi-Fi is down and GSM coverage is weak. Employees and visitors have the application on their phones, and events keep happening in the background: entries into zones, confirmations of permits, reports, attendance check-ins. This is a model scenario, not a report from a specific plant.
The mobile application works offline and queues the data, which reaches the server once connectivity is restored. On the server side this means that events do not arrive evenly but in bursts: a few dozen to a few hundred phones may send their backlog within the same quarter of an hour.
What the phone does when there is no network
The phone does not wait for the server. It stores the event locally, together with the time it happened, and moves on. For the employee this means work continues uninterrupted. For the administrator there is one important assumption: the event time is the time from the phone, and the reception time is the server time. Both have to be stored separately.
This way the post-incident report shows both the moment the event happened and the delay in delivery. That is the basis for a conversation with an auditor who asks exactly when an employee confirmed that they had read a permit.
Example inflow of backlogged events after connectivity returns
What happens on the server when connectivity returns
The server must accept the batches of delayed events after the outage. This means a short but sharp traffic peak: many phones regain the network at the same time and each one sends its queue. The application layer cannot assume that events arrive in order and only once.
A good practice is to separate reception from processing. Reception should be fast and resistant to repeats, while heavier calculations can go to a queue on the server. For this reason, when sizing, it is worth checking the supporting services of the stack, including Redis, which is covered on a separate page about the supporting services.
- Accepting batches without blocking the interface for other users.
- Resistance to the same batch being sent again by a phone.
- Keeping the order of events based on the event time, not the reception time.
- A clear message to the user about the synchronisation status.
Example peak: a calculation with stated assumptions
The calculation below is an example and serves to scale your own values. Assumptions: 300 phones in the hall, a 3-hour outage, an average of 20 events per phone per hour, and the return of connectivity spread over 15 minutes. These numbers are assumptions, not a measurement from a specific plant.
For these assumptions the backlog is 300 times 3 times 20, which is 18 000 events. Spread over 15 minutes this gives an average of 20 events per second, although the momentary peak may be several times higher. This is a value that a typical installation on a single server can handle, provided that the reception of events is lightweight and does not run heavy queries in the same transaction.
What to check on the infrastructure side
Before go-live the administrator should run an outage test on preproduction: disconnect a group of phones from the network for several hours, collect events, and then restore connectivity at the same time. The result of the test is the time needed to fully empty the queues and the server load at the peak.
- Limits on connections and response times at the entry point with the application firewall.
- Free disk space for the database in case of a larger inflow.
- Memory headroom for the supporting services at peak time.
- Monitoring of queue length and of the delivery delay of events.
Conclusions for the administrator
A connectivity outage is not a system failure but a design situation. The phone queues, the server accepts the batches, and the administrator measures the peak on preproduction before it really happens. It is worth reviewing this case together with the description of the entry point and the network zones, because this is where the first connections tend to choke.
Video
Frequently asked questions
Will events from the offline period be lost?
No, the mobile application queues the data and sends it to the server once connectivity returns. The condition is that the phone is not wiped or uninstalled in the meantime.
How do you tell the event time from the reporting time?
The phone records the moment of the event locally, and the server records the moment of reception. Both times are needed in the report after the outage.
Can the server handle a wave of delayed events?
It depends on the size of the fleet and the length of the outage. The best way to find out is a test on preproduction, not an assumption made in advance.
Do we need a separate server for the queues?
Usually not. It is enough to check the spare capacity of one server and its supporting services at peak time.
Sources
Let's test the peak on your preproduction
Together with your IT team we will prepare a connectivity outage test and measure how fast the queues empty before the system goes live.
Let's talk