Software testing before release on live business systems

When I change something in a system that is already in use, somebody is on the other end of that change. A teacher entering marks, a parent checking a statement, a receptionist taking a booking. Software testing before release is the set of automated checks I run before new code reaches those people, so that a small change does not quietly break something nobody looks at every day.

This post describes what actually gets tested in the systems I build and operate from my own Linux servers in Belize. It is written for owners and managers rather than programmers, so I explain each term the first time it appears.

Why a live system changes the rules

A new system with no users is forgiving. If something breaks on day one, you fix it and nobody has lost anything.

Once a school has a term of attendance and marks in the database, or a hotel has reservations for next month, the arithmetic changes. The expensive failure is not the one that throws an error on screen. It is the one that works for most people and fails silently for a few, and is only noticed two weeks later when the numbers do not add up.

Manual checking does not survive contact with a long-lived system. For the first few changes you click through everything. By the tenth, you check only the part you touched, because checking the rest takes an hour you do not have. Automated tests do not get bored, and they check the parts I am confident about as well as the parts I am worried about.

What does software testing before release actually mean?

It is not one thing. It is several layers of checking, each catching a different kind of mistake.

• Module tests, which exercise one piece of code on its own, such as the function that calculates a nightly rate. Fast to run, good at catching arithmetic and logic errors.
• End-to-end tests, where a programme drives a real web browser the way a person would, logging in, filling forms, clicking buttons, and checking what appears on screen.
• Database invariant checks, which assert things that must always be true of the stored data. No invoice without a customer. No enrolment pointing at a deleted class. No booking overlapping another booking for the same room.
• Authorisation checks, which send requests to the server as one kind of user and confirm they are refused. A parent account asking for an administrator page should be turned away, not shown the page.
• Page-render checks, which load every important page and confirm it comes back without an error, with the content it is supposed to contain.
• Mobile audits, which load the same pages at phone width and look for text that overflows, buttons that cannot be reached and layouts that collapse.

None of these is clever. Together they catch most of what goes wrong in ordinary business software.

How do you test safely on a system with real users?

This is the question that matters to an administrator, and it has a practical answer.

For Scholar, the school management platform, I built a reusable end-to-end test sandbox. It is a marked set of test records that live on the real tenant, the real school's own instance of the system, and can be removed cleanly afterwards. A test student, a test class, a test parent, all tagged so the system knows they are not real and so I can delete them without touching anything else.

Against that sandbox runs a harness using a headless browser, a real web browser with no visible window, controlled by a script. It logs in as an Administrator, then as a Teacher, then as a Parent, and works through actual pages the way each of those people would. Alongside it the database invariant checks run, the authorisation checks confirm that a parent cannot reach administrative routes, and the page-render checks confirm nothing returns an error.

The reason for testing on the live tenant rather than a copy is simple. A copy drifts. It has fewer records, older configuration, none of the odd data that accumulates in real use. Tests that pass on a tidy copy and fail on the real thing are not worth much. A marked, removable sandbox gives me the realism without putting a single real student's record at risk.

What gets tested on a hotel and booking platform?

Direct Belize, the hotel website and direct-booking platform I run, has a module test suite with dozens of tests covering the pieces that calculate and store reservations. On top of that, the hotel sites get booking-flow tests, where the harness walks the full path a guest takes, from choosing dates to confirming, and checks that what lands in the system matches what was requested.

The platform also gets mobile audits at phone width, because most people booking a room in Belize are doing it on a phone, often on a slow connection. A layout problem on a booking button is not cosmetic. It is a lost reservation.

What about moving an old website to a new one?

Migrations have their own failure mode: links that used to work and no longer do. Search engines have indexed the old pages. Customers have bookmarked them. Other sites link to them.

Before a migration I crawl the old site, which means walking it automatically and recording every address it publishes. After the migration, a test runs through that list and confirms each old address returns a redirect, an instruction sent by the server telling the browser where the page has moved to, and that it points at the correct new page. It is a boring test. It protects traffic that took years to accumulate.

Checked by hand versus checked by a test suite

| | Checked by hand | Checked by a test suite | |---|---|---| | Coverage after ten releases | Shrinks to the part just changed | Same every time | | Time per release | Grows | A few minutes, unattended | | Catches silent data errors | Rarely | Yes, through invariant checks | | Catches permission mistakes | Almost never | Yes, directly | | Cost | Paid every release, forever | Paid once, maintained cheaply |

What testing does not promise

I would rather be straight about the limits. Tests cover the failures somebody thought of. They do not cover the one nobody imagined, and they are not a substitute for backups, for deploying at a quiet hour, or for being reachable when something does go wrong.

What they do is make change affordable. A system that can be changed safely keeps getting better. A system that is frightening to touch slowly rots, because every improvement is postponed. That is the real argument for testing, and it is why it is built into the way I deliver work across the systems and services I offer, along with source code, documentation and in-app help that you keep.

Where to start

If you are deciding whether to commission software, keep a hosted product or carry on by hand, these are the practical first steps.

1. Write down the three things that must never break. For a school it might be marks, attendance and fee balances. For a hotel, availability, rates and confirmations. That list becomes the test plan.
2. Ask any developer you are considering what runs before a release, and ask them to show you. The answer should be specific, not reassuring.
3. Ask who can see what. Permission mistakes are the most common serious bug in small-business software, and the easiest to test for.
4. If you are replacing a website, ask for the old addresses to be preserved and checked after the move.
5. Agree where test records live and how they are removed, so nothing invented for testing ends up in a real report.

If you would like to talk through what this would look like for your organisation, you can reach me through the contact page.