Skip to content

Automated runs

GxPValidate can run a system's pytest-gxp suite itself, including browser tests written with pytest-playwright, so manual and automated tests are managed in one place. One Docker image is built per commit and reused for every environment, so a pass in Test and a pass in Prod ran byte-identical test code, dependencies and browsers. The image ID is recorded on each run and execution.

Repository

Set the system's Test code repository (Edit system). Test repositories are uv projects (with uv.lock). Run reviews, executions and Repository tests link to the code at the exact commit for GitHub-style https or git@ URLs.

Environments

Environments (Test, Prod, …) are set up by a platform administrator, because the runner host is remote-execution configuration. Each environment has:

  • the runner host (user@host), SSH port and a pinned known_hosts line;
  • a work directory for builds;
  • an env file on the runner host, passed to docker run --env-file. Credentials, server URLs and other settings live only there, never in GxPValidate, the image or the log;
  • the variable names the tests require (checked before every run; *_URL variables are also checked for reachability);
  • the pytest command and the pytest-gxp output folder.

An environment without a runner host is for manual testing only.

A test case can list the environments it must pass in. The RTM shows the latest result in each, and the gap not-passed-in-environment stays open until the latest execution there passes, its failure has a closed deviation, or a closed deviation records that the test cannot pass in that environment.

Runner host

The runner host needs Docker and git with read access to the repository. The GxPValidate worker's public SSH key goes in the runner account's authorized_keys.

Preflight

Preflight checks the env file and the required variables, printing variable names only, never values. It runs before every run and can also be queued on its own.

Scan

Scan runs pytest --collect-only in the image, without the env file, and lists each test's node ID (parametrized IDs included), requirement IDs, file, line and source. Requirement IDs come from the closest requirements marker: on a pytest.param, the test, its class, or the module's pytestmark. Repository tests shows them next to matching test cases and can add unmatched ones to a draft Test Protocol.

Running

A run builds vms-tests:<sha> on the runner host if it does not exist, starts a container with the selected node IDs and the environment's env file, and copies the output folder back. Every evidence file is checked against the SHA-256 in the evidence manifest; any mismatch fails the run.

Trial runs

Authors and testers can start trial runs with the same image, env file and command. Trial runs are never recordable and never reach the RTM. They need no approved protocol; with no tests selected they run the whole suite.

Recording results

With the Tester role and an approved Test Protocol, review the run's log and evidence, add notes and extra evidence files per test if needed, then Sign and record results. This records one signed execution per test in that environment, with the evidence stored in GxPValidate. Failed tests get a Raise deviation link.