Every rate change needs a test suite: regression testing for insurance pricing
How to regression test insurance pricing models: building test cases from worked examples, covering declines and referrals, and checking every rate change before release.
Ask a pricing team how they know a rate change is correct and the answer is often that they checked a few quotes. Spot checks catch obvious errors. They tend to miss subtle ones, such as a factor applied in the wrong order, a band boundary that is off by one, or a referral rule that has stopped working.
Software teams solved this problem with regression tests many years ago. Insurance pricing can use the same approach, and now that AI makes changes faster, it needs to.
What a pricing test is
A pricing test is a complete set of inputs and the result you expect. For example:
- Persian cat, aged 3, Greater London, £4,000 limit, £100 excess, neutered: £255.34, valid.
- Cat aged 15: declined.
- French Bulldog on £7,000 cover: valid, referred to an underwriter.
Run every test after every change. If a result moves, you find out straight away and you know which case moved.
Where test cases come from
- The rating guide's worked examples. These are the reference point. If the model disagrees with the actuary's example, the model needs fixing.
- Boundary cases. Ages, sums insured and postcodes on either side of each band boundary.
- Every rule. At least one case that should be declined and one that should be referred for each underwriting rule, plus cases just inside each rule that should not trigger it.
- Real quotes. A sample of production quotes, saved with their results, guards against unintended changes when the model is restructured.
- Past bugs. Each pricing bug becomes a test so that it cannot happen again.
A model with a failing test should be treated as broken. Results should match to the penny.
Testing a rate change
When a rate change is intended, some tests are expected to change. The process is:
- Run the test suite before the change to record the starting results.
- Make the change in a draft version.
- Run the suite again and review every difference.
- Update expected results only for the cases the change was meant to affect, and be able to explain each one.
- Approve and publish only when there are no unexplained differences.
This turns the question "did anything else change?" into a report you can read.
Why this matters more with AI
AI makes pricing changes quickly, so testing becomes more important. An AI agent editing a model should run the test suite after every edit and report which tests passed and which failed. A person then reviews the report rather than the raw edit. The tests act as the agreement between the AI doing the work and the person approving it.
Tests also prevent a serious AI error: changing a test's expected value to make a failing test pass. Expected results should only change when a person decides they should, for a stated reason.
Tests as regulatory evidence
A versioned model with a test suite is strong evidence of control. It shows what the model was expected to do, shows that it did it, and records who approved each change. Regulators increasingly expect this kind of record for automated pricing.
Start when the model is built
The cheapest time to create a test suite is when the model is first built, using the same worked examples used to design it. In Ptally, tests are stored in every model and run on every change, so you can see the whole suite pass before a version goes live. You can see them on our model building page, and our quote debugging page shows how to investigate a failing case.