Verification

Overview

Verification answers one question for every requirement: what evidence shows it is met? Holarch keeps the plan (verification method, level, success criteria), the test cases that carry it out, the results of each run and the resulting coverage in the same model as the requirements, so the coverage report is always current.

The Verification group in the navigation rail holds:

The rail badge on Verification counts failing tests and model check errors.

Concepts

Verification planning attributes

Every Requirement has three planning attributes:

The Verification Method attribute is mirrored by the Verification Method labels (Analysis, Demonstration, Inspection, Test) under the requirement's labels. Setting either one updates the other in the same undo step, so imported projects that use labels and projects that use the attribute behave the same.

Test cases and test suites

A Test Case verifies requirements through the verifies relationship (seen from the requirement: verified by). A test case has:

• Description: the procedure.
• Set Up, Expected Result, Actual Result.
• Status: Not Run, In Progress, Blocked, Failed or Passed.
• Steps: an optional numbered procedure, each step with Action, Expected Result, Actual Result and Pass/Fail.
• Run evidence, written by each recorded run: Tested By, Test Date, Configuration (build or version under test), Test Article (unit or serial number) and Test Evidence (a file).
• A run log: every recorded run with date, status, tester, configuration, article, actual result, evidence file name and step results.

A test case can be decomposed into child test cases. A parent's status is rolled up from its leaves: Failed if any leaf failed, else Blocked if any is blocked, else Passed if all passed, else Not Run if none ran, else In Progress.

A Test Suite is an Artifact with the Test Suite label. It holds its test cases through source of (or referenced by in imported data). Test cases in no suite appear under Unassigned Test Cases.

Requirement verification status

Each requirement gets an own status from its directly verifying test cases:

The rolled-up status follows decomposition (decomposed by): a parent requirement is Passed when it has its own passing test or all its child requirements are Passed; a failure anywhere below makes it Failed; otherwise it shows the further-along of its own status and its least-covered child (Unverified → Planned-only → Not Run). Verified always means Passed.

Test cycles

A test cycle is a saved snapshot of every leaf test case's status and actual result in a suite. Save one at the end of each campaign, then reset the suite to Not Run for the next. Cycles can be compared with each other or with the current results, and reverted.

How to use it

When and why

Plan verification while requirements are written, not after the design is built. A requirement whose method and success criteria cannot be stated is usually not verifiable as written. The usual flow:

1. Requirements analysis: set Verification Method, Level and Success Criteria on each requirement.
2. Design: write test cases (or analysis, demonstration and inspection records, which are test cases too) and link them to the requirements they verify.
3. Integration and test: run the test cases, record results and evidence, raise Issues for failures, save a test cycle per campaign.
4. Reviews and acceptance: report coverage with the RVTM.

Plan verification for each requirement

1. Open Verification → Test Center and scroll to Requirement Verification Coverage. It lists every requirement that is not yet verified.
2. On each row, click the method buttons A, D, I, T (Analysis, Demonstration, Inspection, Test) to toggle the methods.
3. Choose the level in Level… (Component, Subsystem, System).
4. Click + Criteria, type the measurable outcome, and click Save. The button then reads Criteria ✓.

Result: the requirement moves from Unverified to Planned-only. The same attributes can be edited in the inspector of any requirement.

Create test suites and test cases

1. Click + New ▾ → Test Suite… in the top bar. Enter a Number (e.g. TS-3), a Name and a Description, then Create.
2. On the suite card, click + Test Case. In Create Entity, name the test case. It is added to the suite with status Not Run.
3. In the test case row, click + in the Verifies column and pick the requirements it verifies.
4. Type the Expected Result in the row. Write the procedure in the test case's description (inspector or Entity View).
5. Click + Steps to write a step-by-step procedure: + Add Step for each step, fill Action and Expected Result, reorder with ↑, remove with ×, then Apply.

Shortcuts:

• In Requirement Verification Coverage, + Test Case on a requirement row creates Verify <requirement name> already linked to it (under Unassigned Test Cases).
• With a requirement selected in a document or in its Entity View, ✦ AI ▾ → Generate Test Case… drafts the name, ordered steps, set-up, expected result and method. Review it in Verify Test Case — <requirement> and click Create Test Case. See AI.

Run tests and record results

1. Click Run Tests in the top bar. To run only some tests, first click a status filter such as Not Run (12); only leaf test cases with that status are run.
2. In Run Tests, fill the session fields once: Tester (defaults to your name), Configuration / Build and Test Article.
3. For each test, read the Procedure, Set Up, Steps and Expected Result. Fill each step's actual value and Pass/Fail, describe the Actual Result, check Date and Time, and click Attach Evidence… to add a log, data file or photo.
4. Click ✔ Pass, ✘ Fail or Blocked. Skip moves on without recording; ‹ Previous goes back.
5. After ✘ Fail, the dialog says Recorded. Create an Issue to track the failure; it is linked to this test case and the requirements it verifies. Click + Create Issue, then Next ›.
6. At the end, Run complete shows a status bar with the counts. Review from start goes back to the first test.

Result: each recorded result updates the test case's Status, Actual Result, Tested By, Test Date, Configuration, Test Article and Test Evidence, saves the steps, and adds an entry to the test's run log, all as one undo step. The run starts at the first test that is Not Run.

You can also set a status directly with the five status buttons in a test case row, and type the actual result in the row.

Save a test cycle

1. When a campaign ends, click New Test Cycle on the suite card.
2. In Reset Test Cycle, name the cycle (default Test Cycle N).
3. Click Save and Reset to save the results and set every Status in the suite to Not Run with empty actual results, or Save Only to keep the current results.
4. Later, open ⋯ → Test Cycles… (N) to view, compare, revert or delete cycles.

Report coverage

1. Open Verification → RVTM. The first card shows verified / total with a progress bar; the others count Failed, Not Run, Planned-only and Unverified.
2. Click a card to list only those requirements.
3. Click Export ▾ and choose Excel (.xlsx), CSV or Word (.docx).

Worked example: the CubeSat EO-1 demo

Create the demo with Manage Projects → Project Files ▾ → Create Demo Project, then:

1. Open Verification → RVTM. The first card reads 11 / 32 Verified (Passed); the others read 3 Failed, 12 Not Run, 3 Planned-only and 3 Unverified.
2. Look at 1.6 Revisit Time: Own is Planned-only (method Analysis, no test of its own) but Status is Not Run. Its children 1.6.1 Ground Track Repeat (Passed by TC-13) and 1.6.2 Off-Nadir Imaging (TC-14, Not Run) roll up: the least-covered child is Not Run, which is further along than Planned-only.
3. Click the Failed card. 2.1 Pointing Accuracy and 2.2 Pointing Stability fail through TC-04 Pointing Accuracy Test; 4.3 fails through TC-09 Safe Mode Entry Test.
4. Open Verification → Test Center. The cards read 19 test cases in 2 suites. In TS-1 Spacecraft Qualification Tests, the TC-04 row shows J. Alvarez · 2026-10-02 10:15 · 📎 tc04-pointing-errors.csv and Issue: Test failure: Pointing Accuracy Test. Click Steps (3): step 3 failed with 0.14 degrees on targets 7, 12 and 18.
5. Open ⋯ → Test Cycles… (2) on TS-1. Compare Flatsat Cycle 1 → Flatsat Cycle 2: TC-03 and TC-06 changed from Not Run to Passed (changed rows are highlighted).
6. Rerun TC-09 after the watchdog fix: click the Failed (2) filter, then Run Tests. The run opens on TC-04; click Skip to reach TC-09. Enter Flight software 2.3.2 as Configuration / Build, type Safe mode in 8 seconds as the Actual Result, and click ✔ Pass. Back on the RVTM, 4.3 turns Passed and the first card reads 12 / 32.
7. In Requirement Verification Coverage, 2.5 Safe Mode Recovery is Unverified. Click T, choose System, add criteria, and click + Test Case to start its test.

Tips and good practice

• Write success criteria as measurable numbers with conditions (Range ≥ 25 km in 3 of 3 flights at 20 °C), not restatements of the requirement.
• Verify leaf requirements; let parents roll up. Model check Requirement.10 flags leaf requirements without a verifying test case; parents are covered through their children.
• Use one test case per verification event, even for Analysis, Inspection and Demonstration. The RVTM then shows evidence for every method.
• Record the configuration on every run. When a result changes between cycles, the run log shows which build caused it.
• Raise an Issue for every failure. Model check TestCase.1 flags failed test cases with no linked Issue.
• Common mistake: setting Status to Passed on a parent test case. Parents show the rolled-up status of their children and have no status buttons; record results on the leaves.
• Common mistake: a requirement linked to its test with traced to instead of verifies. It stays Planned-only or Unverified. Use + in the test case row, which creates verifies.
• Common mistake: a test case that verifies nothing. It counts in the Test Center but never in the RVTM.

How it connects to other features

• Fed by: requirements and their decomposition (Requirements); interface requirements on Conduits (Interfaces); simulation results, which can verify a requirement through a Measure (see Simulation).
• Feeds: the RVTM and its exports; Issues for failures, which can be planned as tasks on a Kanban board (Plan & Deliver); Model Checks (Requirement.10, TestCase.1); Home health cards and the Verification badge; baselines and review packages.
• Traceability Matrix: Open in Matrix on the coverage card opens requirements × test cases by verified by; a suite's ⋯ → Open in Traceability Matrix opens its hierarchy.

Test Center

Address: test-center; test-center?root=<id> expands, scrolls to and selects a test case or suite (used by Open ▾ → Test Center on a test case).

Top bar

Health cards and filters

• Test cases: number of leaf test cases, and the number of suites.
• Executed: percentage and count of leaves that are not Not Run.
• Passed: percentage and count of passed leaves.
• Failed: number of failed leaves, with the number blocked.

Below them a status bar and the filters All (N), Not Run, In Progress, Blocked, Failed and Passed (each with its count; click again to clear). On the right: the suite sort (Sort suites by Number, Sort suites by Name, Sort suites by Modified) and Search test cases (number, name or description; a suite whose name matches shows all its test cases).

Suite card

The header shows ▾/▸ (collapse), the suite link, the Test Suite chip, the rolled-up status, the test count, the cycle count, a mini status bar and:

• + Test Case — create a test case in this suite.
• New Test Cycle — save a cycle (see above).
• ⋯ (suite options):

Test case table

Click a row outside its fields to show the test case in the inspector. Empty suite: No test cases yet. No match: No test cases match the filter.

Test Steps dialog

Test Steps — <test case>: a table of #, Action (placeholder Do this), Expected Result (Expect this), Actual Result (Observed), Pass/Fail (—, Pass, Fail), ↑ (Move up) and × (Remove step); + Add Step; Cancel and Apply. Empty steps are dropped on Apply. With none: No steps yet. A test case without steps is run as a whole.

Run Tests dialog

Test Cases Output (DOCX)

File Name (default <suite> Test Cases) and Select Attributes: Number, Description, Set Up, Expected Result, Actual Result, Status, Status Roll-Up, Verifies, plus every other Test Case attribute (Tested By, Test Date, Configuration, Test Article, Test Evidence and custom ones). Number, Description, Expected Result, Actual Result and Status are ticked by default. Download writes a title section (test case count and generation date) and one section per test case.

Test Cycles dialog

Test Cycles — <suite>: a table of Test Cycle, Saved, Results (mini status bar and N/M passed) with View, Revert and Delete. Revert asks Write the statuses and actual results saved in "<name>" back to the test cases? Compare has two selectors (any cycle or Current Test Suite) and a table of Test Case, Before, After; changed rows are highlighted. New Test Cycle… and Close at the bottom. With none: No test cycles yet. Use "New Test Cycle" to save the current results and reset the suite.

Requirement Verification Coverage

The last card on the page. Its header reads N of M requirements verified by a passing test (P%) with Open in Matrix. Below: a status bar, the filters Not verified (N) (default), All (N) and one per status, and the note Verified = a verifying test case Passed. A parent requirement is verified when it has its own passing test or all of its child requirements are verified. Method buttons: Analysis, Demonstration, Inspection, Test.

Each row: the status pill (pointing at it shows the own status when it differs), the requirement link, the number of tests, the method buttons A D I T, Level…, + Criteria / Criteria ✓ and + Test Case. Up to 300 rows are shown, then …and N more. When all are verified: Every requirement is verified by a passing test.

Create Issue

+ Create Issue (in the row or in run mode) creates an Issue named Test failure: <test case> with status Open and a description holding the expected and actual results. The Issue is linked to the test case (causes, or related to) and to each requirement the test verifies. Toast: Created issue <name>.

RVTM

Address: rvtm, with rvtm?status=<status> and ?q=<text>.

Top bar

All three exports hold Requirement Number, Requirement, Verification Method, Verification Level, Success Criteria, Verified By (each test case with its rolled-up status), Own Status and Status (rolled up).

Cards

Verified (Passed) shows verified / total with a progress bar. Failed, Not Run, Planned-only and Unverified show counts; click one to filter (Showing only these. Click to show all.). The chip Status: <status> × clears the filter.

Table

Filter by number or name narrows the rows. Click a row (or press Enter) to show the requirement in the inspector.

Shortcuts

Messages

Limits

• Test evidence files: 25 MB (26,214,400 bytes) per file. Larger files are recorded by name only. Evidence counts toward the project's storage (see Storage and limits).
• The coverage card shows the first 300 requirements of the current filter.
• One evidence file per test case is kept on the test case (the latest); the run log keeps each run's evidence file name.

Related

• Interfaces — interface requirements and ICDs.
• Budgets & TPMs — Measures that track performance against thresholds.
• Plan & Deliver — plan fixes for failed tests.
• Help: Set Up Verification, Run Tests and Produce the RVTM, RVTM, Test Center, Traceability Matrix, Model Checks, Simulate and Check Against a Requirement.