Tests
Tests checks that what an application calls still answers the same: after a change of configuration, an update of the system behind an API, a new version. A test is written without code — a case is a flow of calls to the endpoints of an API connector, each with what is expected of its answer — and compared from run to run with a reference.
Access: Liberty Next → Logic → Tests. The results over time are in Operate → Test results.
At a glance
Suites, cases, steps
| Suite | Names an API connector and holds cases. New suite asks for its name. |
| Case | One thing checked — The customer list still has its columns. New case, under the suite. |
| Step | One call to an endpoint of the suite's connector, with what is expected. |
The left column lists the suites and their cases, each with the icon of its last result. The middle draws the picked case: a start, one card per step — its endpoint, its name, a summary of what it expects, coloured by its last result — and an end. The right panel edits what is picked.
The case
| Field | Meaning |
|---|---|
| Id | The case's identifier in its suite. |
| What it checks | The sentence shown in the lists and the results. |
| This case changes data: played only when a run allows it | Tick it when a step writes. |
| Last runs | The last eight runs of the case; a recorded run is tagged reference. |
A step
The + on a link adds a step at that place; a click on a card opens it.
| Field | Meaning |
|---|---|
| Name | Unique in the case: the results and the reference are kept by step. |
| Endpoint called | Picked in the connector's list. |
| Parameters | The endpoint's own form: one labelled field per parameter, its help under it, a dropdown when it has a list of values. |
| The call | must work or must be refused — with The refusal says to check the message. |
| Rows, at least / exactly / at most | The number of rows, counted where the endpoint says its rows are. |
| The first row holds | Values of the first row, by the endpoint's column names. |
| Played even after a failure (a cleanup) | The step runs even when an earlier one failed: how a case that writes removes what it created. |
Fill from the answer plays the call now and proposes what to expect — the number of rows and the first row's values: keep the lines that must stay true, delete the others. An endpoint that changes data is not played from there: save the step and run the case.
Advanced — values kept, paths in the raw answer holds what the fields above do not cover: a value kept for the next steps (used there as {{name}} in a parameter), a value of the raw answer that equals or holds a text, paths that must hold something or nothing, a warning, and the paths compared with the reference run. It is needed for an endpoint that does not say where its rows are.
Run and compare
| Button | Does |
|---|---|
| Run this case | Plays the picked case. |
| Run the suite | Plays every case, one after the other. |
| Record the reference | Plays the suite and keeps, for the cases that passed, what their steps keep: the reference the next runs are compared with. |
| also the cases that write | Allows the cases marked as changing data on this run. |
A run saves the unsaved changes first, then says how it went: N passed · N failed · N changed · N skipped.
| Result | Colour | Meaning |
|---|---|---|
| Passed | Green | Every check holds, and nothing differs from the reference. |
| Failed | Red | A check does not hold, the call failed, or a value expected from an earlier step was never kept. |
| Changed | Orange | The checks hold, but something kept differs from the reference — gone, new, another count, another value. |
| Skipped | Grey | The case writes and the run did not allow it, an earlier step failed, or the case has no step. |
A case takes the worst result of its steps. The twenty last runs of each case are kept.
Where the tests are kept
A suite is a file of the configuration (tests/<suite>.toml): it is packaged, versioned and delivered with the application, and each save keeps the former version in Deploy → History. The runs and the reference are kept beside it and stay on the instance: a reference recorded on the test environment is not the production's.
From a job, from a menu
- A job's run form can offer the list of suites, so that an application ships a job that plays a suite on a schedule and fails when a case fails.
- An application's menu can carry the two pages — Tests and Test results — for its own users: see Item types.
Access
The tests have their own permissions, in a role's Beyond the menus rules, type Tests — each includes the one before, and a superuser holds them all:
| Permission | Label in the role editor | Allows |
|---|---|---|
tests:view | See the test suites and their results | Read the suites, the cases and the results. |
tests:run | Play the tests (and see them) | Run a case or a suite, record the reference, fill from the answer. |
tests:edit | Write the tests: suites, cases, steps (and play them) | Create, change and delete suites, cases and steps. |
A functional team can therefore own its tests without being administrators.
Tips & best practices
- One case, one thing checked: a failure then names what broke.
- Expect a refusal where one is due — an unknown identifier, a missing required value: a system that starts accepting anything is a regression too.
- Record the reference just before a change, run the suite just after.
- Keep the cases that write apart, each ending with its cleanup step.