Reference
Recipe: Test Results Through Claude
Bring results from a test system into Holarch test cases, with a REST script for CI or with Claude, which runs the tests, proposes the results for review and reports what the RVTM shows; setup, a script, a worked example and troubleshooting.
On this page
Overview
This recipe records results from an automated test system on the matching Holarch test cases, so the RVTM shows which requirements pass. It has two parts:
- Results from CI (REST API): a script reads the JUnit XML report that most test runners write and records each result on the matching test case: Status, Actual Result, Tested By, Test Date and a run-log entry.
- Results through Claude (MCP): Claude Code runs the tests, matches the results to test cases, and proposes the results for you to review in Holarch. Then it reads the RVTM and tells you which requirements still fail.
To import a results file by hand, use Verification → Test Center → Import Results… instead (Import results from another test system). The REST API and MCP record results the same way: Tested By, Test Date and a run-log entry on each test case.
Prerequisites
- A plan with the API and MCP: Individual or Team, or the Individual trial. On the Free plan, an account with a verified academic email address or in an active class organization also has them (Plans and limits).
- A Holarch project with test cases whose numbers the tests can name, such as TC-14. You need the editor or owner role.
- For the script: a Read and write token limited to the project (Account Settings → API tokens → New token, Create a token), and Python 3.9 or later with the
requestspackage (python3 -m pip install requests). - For the Claude part: Claude Code connected to Holarch with Read and propose changes access (Connect Holarch to Claude), in the repository that holds the tests.
- A test runner that writes JUnit XML, such as
pytest --junitxml=results.xml, or most CI test steps.
How to use it
When and why
Verification closes the loop from requirement to evidence. When tests run automatically, typing their results into Holarch is slow and error-prone. Use the script when results should land without review, such as nightly regression runs. Use Claude when a person should see the results before they are recorded, such as a qualification run.
Name the test case in each test
The script and Claude match a test to a Holarch test case by the test case number in the test's name.
- Put the number in the test's name:
test_TC_14_off_nadir_imagingin pytest, orTC-14 off-nadir imagingin other runners. - Keep one Holarch test case per test. A test case with child test cases takes its status from its children; record results on the children.
Record results from CI
The script records Passed or Failed on each matched test case, in one request. For a failure it also records the failure message as Actual Result. Skipped tests are left unchanged.
# junit_to_holarch.py: record JUnit XML results on Holarch test cases.
# Usage: python3 junit_to_holarch.py results.xml
import os
import re
import sys
import xml.etree.ElementTree as ET
import requests
HOLARCH = os.environ.get("HOLARCH_URL", "https://app.holarch.app") + "/api/v1"
PROJECT = os.environ["HOLARCH_PROJECT"]
HEADERS = {"Authorization": "Bearer " + os.environ["HOLARCH_TOKEN"]}
NUMBER = re.compile(r"TC[-_ ](\d+)") # TC-14 or test_TC_14_…; change it to match your numbering
def holarch(method, path, **kw):
r = requests.request(method, HOLARCH + path, headers=HEADERS, timeout=60, **kw)
if r.status_code >= 400:
d = r.json()
raise SystemExit(f"Holarch answered {r.status_code} {d.get('code')}: {d.get('error')}")
return r.json()
results = []
for test in ET.parse(sys.argv[1]).getroot().iter("testcase"):
m = NUMBER.search(test.get("name", ""))
if not m or test.find("skipped") is not None:
continue
failure = test.find("failure")
if failure is None:
failure = test.find("error")
result = {"testCase": "TC-" + m.group(1), "status": "Passed" if failure is None else "Failed"}
if failure is not None:
result["actualResult"] = (failure.get("message") or failure.text or "Failed").strip()[:4000]
results.append(result)
answer = holarch("POST", f"/projects/{PROJECT}/test-results",
json={"source": os.path.basename(sys.argv[1]), "testedBy": "CI", "results": results})
for r in answer["recorded"]:
print(f"{r['number']} {r['name']}: {r['status']}")
for r in answer["skipped"]:
if r["code"] == "not-found":
print(f"{r['testCase']}: no test case with this number in Holarch")
else:
print(f"{r['testCase']}: not recorded ({r['reason']})")
- Set the environment variables
HOLARCH_TOKEN(the token) andHOLARCH_PROJECT(the project id fromGET /api/v1/projects). - Run your tests with a JUnit report, such as
pytest --junitxml=results.xml. - Run
python3 junit_to_holarch.py results.xml.You see one line per recorded result, such as
TC-14 Off-Nadir Imaging Test: Passed. - In Holarch, open Verification → RVTM.
The requirements those test cases verify show the new status.
How it works:
POST /api/v1/projects/<project>/test-resultsrecords all results of the report as one change, as Import Results… does: Status, Actual Result for failures, Tested By (CI here; your account name when left out), Test Date (the time of the request) and a run-log entry withsourceset to the report's file name (Test results).testCasenames the test case by its number, such asTC-14; an id or a name works too. Status takes Passed or Failed here; the endpoint also takes Blocked, In Progress and Not Run, and test tools' words such as pass and fail.- A result that cannot be recorded, such as a number Holarch does not have, is listed in
skippedwith its reason; the others are still recorded. - The change is recorded under your name and seen at once by everyone with the project open.
In a CI job, run the script as a step after the tests, with the token as a secret. In GitHub Actions, for example:
- name: Record results in Holarch
if: always()
env:
HOLARCH_TOKEN: ${{ secrets.HOLARCH_TOKEN }}
HOLARCH_PROJECT: ${{ vars.HOLARCH_PROJECT }}
run: |
python3 -m pip install requests
python3 junit_to_holarch.py results.xml
if: always() records the results also when tests fail, which is when they matter most.
Run the tests and propose the results with Claude
- In the repository that holds the tests, ask Claude Code, for example: "Run the pointing tests with a JUnit report. Then propose their results in the Holarch CubeSat project: set Status and, for failures, Actual Result on the matching test cases."
- Claude Code runs the tests. In Holarch it adds the results to a proposal with
record_test_results(one change per result, with Status, Actual Result, Tested By and Test Date). Then it sends the proposal withpropose_changes.Claude Code answers with the proposal's title, the results it proposes and a link to review it.
- In Holarch, open ✦ AI ▾ → Review “<title>”…, or the notification under the bell.
The dialog shows each test case with its current and proposed values.
- Untick any result you do not accept, then click Apply Selected.
The results are recorded with your name in the history, each with a run-log entry as from Import Results…, and a single Undo takes it all back.
- Ask Claude Code: "Which requirements in the CubeSat project still fail verification?"
Claude Code reads the RVTM with
get_matrix(matrixrvtm,statusFailed) and lists each requirement with its failing test cases.
Claude Code can also work from a report your CI already wrote: attach or point it to results.xml instead of asking it to run the tests.
Worked example: the CubeSat EO-1 demo
- Create the demo with Manage Projects → Project Files ▾ → New Demo Project.
In Verification → Test Center, TC-14 Off-Nadir Imaging Test is Not Run and TC-04 Pointing Accuracy Test is Failed.
- Write a JUnit report with two tests:
test_TC_14_off_nadir_imagingpassed, andtest_TC_04_pointing_accuracyfailed with the message 0.12 degrees on 2 of 20 targets. - Run
python3 junit_to_holarch.py results.xml.You see
TC-14 Off-Nadir Imaging Test: PassedandTC-04 Pointing Accuracy Test: Failed. - Open Verification → RVTM.
1.6.2 Off-Nadir Imaging now shows Passed. 2.1 Pointing Accuracy still fails, and TC-04's Actual Result reads 0.12 degrees on 2 of 20 targets.
- With Claude, the same results arrive as a proposal first. After Apply Selected, the RVTM shows the same.
Tips
- Record results on the lowest test cases. Parents roll up from their children.
- Keep the script's token limited to the one project, and revoke it when the pipeline is retired.
- A test that names a number Holarch does not have is listed as "no test case with this number". Fix the test's name or create the test case.
- Save a test cycle in the Test Center when a campaign ends, so its results are kept apart from the next run (Save a test cycle).
Troubleshooting
| Message or symptom | Cause and fix |
|---|---|
Holarch answered 401 token-invalid | The token is wrong, revoked or expired. Create a new one under Account Settings → API tokens. |
Holarch answered 403 plan-api | The account's plan does not include the API and MCP. Start the Individual trial or choose a plan; the token then works again. |
Holarch answered 403 read-only-token | The script writes results. Create a Read and write token. |
Holarch answered 403 forbidden | Your role in the project is viewer or reviewer. An owner can make you an editor. |
TC-14: not recorded (…) | Holarch could not record that result, for example because the test case is locked or has child test cases. The reason says which; the other results are recorded. |
TC-04: not recorded (… configuration control …) | The test case is under configuration control. Record its result in the app through a change request. |
Holarch answered 429 rate-limited | Too many requests a minute. Wait the seconds in Retry-After, then run the script again. |
| A test case keeps its old status | A test case with children shows the rolled-up status of its children. Record the result on a child. |
| Claude Code does not offer the Holarch write tools | The connection is read only. Run /mcp, reconnect holarch with Read and propose changes, then start a new conversation. |