What the system stack consists of
The local stack of the system consists of a TLS-terminating reverse proxy, a PHP 8.4 application with Apache, a MySQL 8.0 database, an S3-compatible object store, Redis, a mail catcher, a PDF generation service and a .NET helper service. The data sits in named volumes. The production installation at the plant has the same split, with the caveat that the mail catcher is a tool of the test environment.
The recommendations below organise what to do with each element during the first deployment and what to add to the plant's monitoring.
Recommendation 1: back up and monitor the database as a separate service
The MySQL 8.0 database stores people, permits, qualifications and events. It is the most valuable element of the stack, so it gets its own backup schedule, its own restore test and its own alerts: disk space, number of connections, response latency. Whether the backup is logical or physical depends on the plant's standard, but the restore must be tested regularly on preproduction.
When updating the database or the application, remember the migrations, which we describe on a separate page. Their duration depends on the history of the installation.
Example split of backup and alert priority
Recommendation 2: treat Redis as a cache with consequences
Redis 8.6 in the stack serves as a fast support layer. The administrator should establish which data in it is purely temporary and which requires persistence. Monitoring covers memory usage, the number of connections and the key eviction policy.
Running Redis out of memory produces symptoms that are easy to mistake for an application error. That is why it is worth having a separate alert threshold for this service, instead of relying on the general state of the host.
Recommendation 3: back up the S3-compatible store independently of the database
Files such as photos, attachments, printouts and training materials are kept in the S3-compatible store, not in the database. A backup of the database therefore does not cover the files. Restoring the system requires both backups, kept consistent in time: the database points to objects that must exist in the store.
We recommend that the backup schedules for the database and the store are set in the same window, and that the restore order is recorded in the installation description.
Recommendation 4: monitor the PDF generator for response time
The PDF generation service is responsible for printing permits, passes and reports. Its failure does not stop the whole system, but it stops the issuing of documents, which can be just as painful for the OHS department. Measure the generation time of a typical document and set an alert when it grows above the agreed norm.
During the first deployment, make a test printout of each document type on preproduction and keep the sample as a reference point.
Recommendation 5: the .NET helper service and mail have their own cases
The .NET helper service performs specialised tasks and should have its own entry in monitoring, even if its load is small. Mail in the test environment goes to the catcher, so nobody receives messages from preproduction. On production the correct mail server of the plant must be configured and the delivery of notifications checked.
Recommendation 6: memory limits and start order
On one host the services compete for memory. Set limits for the containers and the start order: the database and Redis before the application. After a server restart the application should not start before the database is ready. Record in the installation description the order in which the administrator starts the services after a power failure.
The data is stored in named volumes, so restarting the containers does not destroy their contents. The data and the backups stay on the customer's disks.
Video
Frequently asked questions
Is a database backup enough to restore the system?
No. The files are in the S3-compatible store, so a backup of both elements and of the configuration is needed. Restoring is practised on preproduction.
Does Redis need a backup?
It depends on which data we keep in it. The administrator should establish what is temporary and what requires persistence, and record the decision.
What to do when PDF generation stops working?
Check the PDF generation service separately from the application. Its failure stops printing, not the whole system.
Does the mail catcher work in production?
No. It is an element of the test environment that collects messages so they do not reach real recipients. Production uses the plant's mail server.
Sources
Let's make a list of services to monitor
Together with the plant's administrator we will prepare a backup and alert plan for each stack service before the first start.
Let's talk