Who these recommendations are for
The recommendations are addressed to the plant's IT manager, who approves the update window, and to the health and safety manager, who is responsible for the continuity of permits and alarms. They apply when the vendor announces a new version of the system and the plant has to decide whether to accept it now or after checking.
The starting point is a simple observation: the fact that the vendor tested the version is not enough. It is worth knowing which tests were run, on what, and with what result. The recommendations below make it possible to ask these questions specifically.
Recommendations 1 and 2: a package in a real browser, on a real backend
Ask for tests that run a real browser against a real backend. Unit tests run against a mocked API and will not detect a user-visible error that appears only at the boundary between modules. The example InnerWeb package includes 242 tests in 31 files.
Check whether the scope matches work at the plant: logging in, the admin panel, work planning, work permits, reports, documentation, maps and analytics. A package that skips permits or maps does not protect the functions on which the plant's safety depends.
Duration of a full E2E test run
Recommendations 3 and 4: hard errors and a clean database
Require a rule that a test fails when the browser shows an uncaught JavaScript exception or the server responds with a 5xx status. This rule turns silent errors into visible ones. Without it a test can pass even though something broke on the screen.
Ask how the data is prepared. The test database should be rebuilt from migrations once per run, so that the result does not depend on data from the previous run. A test that is green yesterday and red today, without any code change, usually points to a dirty database rather than to an application error.
Recommendations 5 and 6: a register of known bugs and a test after a fix
Ask for a list of known, unfixed bugs. Good practice is to mark them in the test as expected failures with a ticket number, not to skip the test. The package then keeps guarding the area, and after a fix the test itself reports that the annotation has to be removed.
When the vendor delivers a fix for a bug you reported, require that browser tests are run after the fix. Such a test catches a user-visible error that unit tests do not detect, because they run against a mocked API. A fixed defect should leave a regression test behind.
Recommendation 7: a fresh result from the last run
The full set of browser tests takes about 2 to 2.5 hours, so it is not run for every small change. It runs on request and automatically at midnight on weekends. Therefore, before consenting to an update, ask for the result of the last full run and the date when it was performed.
The result is evidence for internal audit: it shows that critical scenarios were checked before the new version was deployed. InnerWeb is a software producer and integrates with the client's existing systems, so an update on our side does not replace the readers or cards at the plant. The tests concern the software, not the hardware.
- a package on a real browser and a real backend, not a mocked API
- a scope covering work permits, maps and alarms
- a hard failure on a JavaScript exception or a 5xx response
- a database rebuilt from migrations before each run
- known bugs as expected failures with a ticket number
- a test after the fix of a reported bug
- the date and result of the last full run
How to put this into the acceptance procedure
The recommendations are easiest to introduce as a standing annex to the plant's change procedure. The annex is one page: seven questions, a field for the vendor's answer and a field for the date of the result. The IT manager ticks the points before signing the consent for the update window.
If the vendor cannot answer one of the questions, this does not automatically mean refusal. It means it is worth postponing the update until the evidence is supplied, or limiting it to a window in which the plant's team can quickly check the critical scenarios. The plant sets the decision rule; the vendor supplies the evidence.
Video
Frequently asked questions
Can the plant run the vendor's tests on its own?
It depends on the contract. Usually it is enough to ask the vendor to run the full set on request and to provide the result with its date. A full run takes about 2 to 2.5 hours.
Are unit tests not enough?
No. They run against a mocked API and do not detect errors visible to the user. For a BHP system it is worth having both kinds of tests.
What should be done about a bug the vendor marked as known?
Record the ticket number and assess its impact on the plant's critical scenarios. An expected failure with a ticket number is evidence that the bug is registered, not hidden.
Let's agree on evidence before the update
Share your procedure for accepting changes and we will show which regression test results we can deliver before each update window.
Let's talk