InnerWeb
Skip to main content

Stack services: database, Redis, S3 store and PDF generator

The application is just one container in the stack. The database, Redis, an S3-compatible file store and the PDF generation service have different requirements, different failure symptoms and different backup methods. An administrator who treats them separately during the first deployment will find the cause of a problem faster and will restore the system more reliably after a failure.

8
services in the local system stack
Source: karta
8.0
MySQL database version in the stack
Source: karta

What the system stack consists of

The local stack of the system consists of a TLS-terminating reverse proxy, a PHP 8.4 application with Apache, a MySQL 8.0 database, an S3-compatible object store, Redis, a mail catcher, a PDF generation service and a .NET helper service. The data sits in named volumes. The production installation at the plant has the same split, with the caveat that the mail catcher is a tool of the test environment.

The recommendations below organise what to do with each element during the first deployment and what to add to the plant's monitoring.

Recommendation 1: back up and monitor the database as a separate service

The MySQL 8.0 database stores people, permits, qualifications and events. It is the most valuable element of the stack, so it gets its own backup schedule, its own restore test and its own alerts: disk space, number of connections, response latency. Whether the backup is logical or physical depends on the plant's standard, but the restore must be tested regularly on preproduction.

When updating the database or the application, remember the migrations, which we describe on a separate page. Their duration depends on the history of the installation.

Example split of backup and alert priority

Example split of backup and alert priorityHorizontal bar chart: the database and the S3 store have the highest backup priority, Redis and PDF the lowest.012345priority points (1-5)DatabaseDatabase: 5 priority points (1-5)5S3 storeS3 store: 4 priority points (1-5)4ConfigurationConfiguration: 3 priority points (1-5)3PDFPDF: 2 priority points (1-5)2RedisRedis: 2 priority points (1-5)2
Source: Example rating by the author: the 1-5 scale depends on the plant's policy

Recommendation 2: treat Redis as a cache with consequences

Redis 8.6 in the stack serves as a fast support layer. The administrator should establish which data in it is purely temporary and which requires persistence. Monitoring covers memory usage, the number of connections and the key eviction policy.

Running Redis out of memory produces symptoms that are easy to mistake for an application error. That is why it is worth having a separate alert threshold for this service, instead of relying on the general state of the host.

Recommendation 3: back up the S3-compatible store independently of the database

Files such as photos, attachments, printouts and training materials are kept in the S3-compatible store, not in the database. A backup of the database therefore does not cover the files. Restoring the system requires both backups, kept consistent in time: the database points to objects that must exist in the store.

We recommend that the backup schedules for the database and the store are set in the same window, and that the restore order is recorded in the installation description.

Recommendation 4: monitor the PDF generator for response time

The PDF generation service is responsible for printing permits, passes and reports. Its failure does not stop the whole system, but it stops the issuing of documents, which can be just as painful for the OHS department. Measure the generation time of a typical document and set an alert when it grows above the agreed norm.

During the first deployment, make a test printout of each document type on preproduction and keep the sample as a reference point.

Recommendation 5: the .NET helper service and mail have their own cases

The .NET helper service performs specialised tasks and should have its own entry in monitoring, even if its load is small. Mail in the test environment goes to the catcher, so nobody receives messages from preproduction. On production the correct mail server of the plant must be configured and the delivery of notifications checked.

Recommendation 6: memory limits and start order

On one host the services compete for memory. Set limits for the containers and the start order: the database and Redis before the application. After a server restart the application should not start before the database is ready. Record in the installation description the order in which the administrator starts the services after a power failure.

The data is stored in named volumes, so restarting the containers does not destroy their contents. The data and the backups stay on the customer's disks.

Video

Frequently asked questions

Is a database backup enough to restore the system?

No. The files are in the S3-compatible store, so a backup of both elements and of the configuration is needed. Restoring is practised on preproduction.

Does Redis need a backup?

It depends on which data we keep in it. The administrator should establish what is temporary and what requires persistence, and record the decision.

What to do when PDF generation stops working?

Check the PDF generation service separately from the application. Its failure stops printing, not the whole system.

Does the mail catcher work in production?

No. It is an element of the test environment that collects messages so they do not reach real recipients. Production uses the plant's mail server.

Sources

Let's make a list of services to monitor

Together with the plant's administrator we will prepare a backup and alert plan for each stack service before the first start.

Let's talk