How we test Holarch

Every release

Each release is built in a clean copy of the source, never in a working copy. The build then goes through the release gate: a fixed set of automated checks. A failed check stops the release. The cause is fixed, and the failed checks run again before the release goes on.

• Release build: the app is built for release, and the built code is checked against the page’s content security policy.
• Language engine build: the compiled language engine is rebuilt from its source, and the result must match the build that ships.
• Engine guard tests: deep, long, wide, cyclic and over-limit inputs on every engine operation, with limits on response time and memory, and a check that time grows in line with the size of the input.
• Dependency audit: the engine’s third-party dependencies are checked for known security advisories, licenses and sources.
• Injection lint: a lint fails the release on any place in the code that inserts HTML or runs text as code, unless that place has been reviewed and recorded as safe.
• Third-party files: every third-party file shipped with the app must match its recorded checksum.
• Performance: the built app is timed on a large generated project, in the browser and on the server, against fixed limits.
• Smoke tests: the built app is opened and used in a real browser, Safari, both as the desktop app and in server mode, the way the hosted service runs it.
• Tests of what changed: the browser test suites for every area the release changes, in both modes where the area has both, and the Python tests of each changed server module.

Browser tests drive the app the way a person does: they click, drag and type in the page, then check the model and the screen.

From release to the hosted service

• Staging first: every release runs on a separate staging service before production. Production accepts only a build that has run on staging.
• Backup before database changes: before a release changes the database, a copy of the database is saved.
• Health checks: after the switch, the service must answer as healthy and report the released version.
• Rollback: the previous version is kept. If the new version fails its health check, the service goes back to the previous version.

Backups of the hosted service are restore-tested every week; see Service targets.

The Language Engine

The Holarch Language Engine parses, resolves and checks SysML v2 and KerML text. Besides the release checks, it is measured under a written acceptance plan. The acceptance criteria were written down before any results were measured. A published result is never edited; a correction is a new run that names the one it replaces.

• Example models: the engine is tested against the SysML v2 example models in the Object Management Group’s official release repository (the training, examples and validation sets), pinned to a specific release.
• Differential testing: the engine’s results on those models are compared with OMG’s pilot implementation: whether each model is accepted, and which element each name binds to. Each disagreement is investigated on its own and classified as an engine bug, an unsupported language feature, an error in the example, or a difference in the pilot.
• Robustness: large sets of damaged inputs, such as missing braces, cut-off declarations, broken imports, cycles, invalid characters and common typing mistakes, check that the engine never crashes, gives the same answer after the damage is repaired, and keeps checking the rest of the model.
• Scale: very large generated models measure the response time of edits, the time to analyze a whole model, memory use, and how time grows with model size.
• Fuzzing: the engine’s request, lexer and parser entry points are fuzz-tested.
• Test cases: the engine’s own tests are written from the specification text, and each is tagged with the clause it checks.

Results will be published on the Language Engine page after an official run completes, failures included, with the engine version, the version of the method and any known defects. What the language support covers today, and its known gaps: SysML v2 language coverage.

Public evidence: the EO-1 reference model

The EO-1 reference model is a complete model of a CubeSat built in Holarch, from needs to verified evidence. Its project file and exports are free to download, so anyone can open the model and check its traceability for themselves.

A formal verification cycle on the EO-1 model follows one requirement change through a formal program: a failed re-test, a nonconformance and its waiver, hazards and failure modes, sign-off, review and the new baseline.

Project size

The item limits of each plan rest on a load test of the hosted service. Tested limits describes the test, its pass bar, its results and what it did not measure.

What this testing does not show

• This is Holarch’s own testing. It is not a certification, and the hosted service holds no security certifications; see Certifications.
• Engine results report what Holarch’s tests found on the models they ran. The Object Management Group has not reviewed or endorsed them.
• An independent review of the testing method is planned.
• Testing lowers the chance of a defect; it does not rule one out. Report a problem with Send Feedback in the app or at [email protected].