Skip to main content

Tests

Tests checks that what an application calls still answers the same: after a change of configuration, an update of the system behind an API, a new version. A test is written without code — a case is a flow of calls to the endpoints of an API connector, each with what is expected of its answer — and compared from run to run with a reference.

Access: Liberty Next → Logic → Tests. The results over time are in Operate → Test results.


At a glance​

TestsSaveRun this caseRun the suiteRecord the referencealso the cases that write+ New suiteorderslist-has-columnscreate-then-cancelunknown-id-refusedprice-is-computed+ New casecreate-then-cancelORDER_CREATEcreateworks · 1 row(s)+ORDER_CANCELcleanupworksEndpassedchanged: differsfrom the referenceTHE STEPENDPOINT CALLEDorder_create ▾PARAMETERSCUSTOMER 1001WHAT IS EXPECTEDFill from the answerThe callmust workRows, at least1The first row holdsSTATUS = OPENSuites and caseswith their last resultA case is a flowone card per callThe + adds a stepat that placeRecord the referencebefore the changeFill from the answerproposes what to expect

Suites, cases, steps​

SuiteNames an API connector and holds cases. New suite asks for its name.
CaseOne thing checked — The customer list still has its columns. New case, under the suite.
StepOne call to an endpoint of the suite's connector, with what is expected.

The left column lists the suites and their cases, each with the icon of its last result. The middle draws the picked case: a start, one card per step — its endpoint, its name, a summary of what it expects, coloured by its last result — and an end. The right panel edits what is picked.

The case​

FieldMeaning
IdThe case's identifier in its suite.
What it checksThe sentence shown in the lists and the results.
This case changes data: played only when a run allows itTick it when a step writes.
Last runsThe last eight runs of the case; a recorded run is tagged reference.

A step​

The + on a link adds a step at that place; a click on a card opens it.

FieldMeaning
NameUnique in the case: the results and the reference are kept by step.
Endpoint calledPicked in the connector's list.
ParametersThe endpoint's own form: one labelled field per parameter, its help under it, a dropdown when it has a list of values.
The callmust work or must be refused — with The refusal says to check the message.
Rows, at least / exactly / at mostThe number of rows, counted where the endpoint says its rows are.
The first row holdsValues of the first row, by the endpoint's column names.
Played even after a failure (a cleanup)The step runs even when an earlier one failed: how a case that writes removes what it created.

Fill from the answer plays the call now and proposes what to expect — the number of rows and the first row's values: keep the lines that must stay true, delete the others. An endpoint that changes data is not played from there: save the step and run the case.

Advanced — values kept, paths in the raw answer holds what the fields above do not cover: a value kept for the next steps (used there as {{name}} in a parameter), a value of the raw answer that equals or holds a text, paths that must hold something or nothing, a warning, and the paths compared with the reference run. It is needed for an endpoint that does not say where its rows are.


Run and compare​

ButtonDoes
Run this casePlays the picked case.
Run the suitePlays every case, one after the other.
Record the referencePlays the suite and keeps, for the cases that passed, what their steps keep: the reference the next runs are compared with.
also the cases that writeAllows the cases marked as changing data on this run.

A run saves the unsaved changes first, then says how it went: N passed · N failed · N changed · N skipped.

ResultColourMeaning
PassedGreenEvery check holds, and nothing differs from the reference.
FailedRedA check does not hold, the call failed, or a value expected from an earlier step was never kept.
ChangedOrangeThe checks hold, but something kept differs from the reference — gone, new, another count, another value.
SkippedGreyThe case writes and the run did not allow it, an earlier step failed, or the case has no step.

A case takes the worst result of its steps. The twenty last runs of each case are kept.


Where the tests are kept​

A suite is a file of the configuration (tests/<suite>.toml): it is packaged, versioned and delivered with the application, and each save keeps the former version in Deploy → History. The runs and the reference are kept beside it and stay on the instance: a reference recorded on the test environment is not the production's.


From a job, from a menu​

  • A job's run form can offer the list of suites, so that an application ships a job that plays a suite on a schedule and fails when a case fails.
  • An application's menu can carry the two pages — Tests and Test results — for its own users: see Item types.

Access​

The tests have their own permissions, in a role's Beyond the menus rules, type Tests — each includes the one before, and a superuser holds them all:

PermissionLabel in the role editorAllows
tests:viewSee the test suites and their resultsRead the suites, the cases and the results.
tests:runPlay the tests (and see them)Run a case or a suite, record the reference, fill from the answer.
tests:editWrite the tests: suites, cases, steps (and play them)Create, change and delete suites, cases and steps.

A functional team can therefore own its tests without being administrators.


Tips & best practices​

  • One case, one thing checked: a failure then names what broke.
  • Expect a refusal where one is due — an unknown identifier, a missing required value: a system that starts accepting anything is a regression too.
  • Record the reference just before a change, run the suite just after.
  • Keep the cases that write apart, each ending with its cleanup step.