Language Engine test results

Jump to: This run · Summary · Example models · Damaged text · Speed and memory · Not supported yet · Reproducing the results · Earlier runs · What these results do not show

This run

• Engine: version 1.0.0, build c184e57, the build that ships in Holarch 2.11.0.
• Method: measured under version 3 of the written acceptance plan and shown here under version 4. The criteria were set down before the first results were measured. Each later change of method is recorded with its reason, and earlier results keep the method they were measured under.
• Change in method version 4: cold analysis time is no longer a target. It is still measured and reported at every size. The criterion is now that Holarch shows a progress bar for any analysis that can take over 3 seconds. Version 4 was decided after this run’s results were known; under version 3 this run did not meet the cold analysis targets.
• Example models: the training, examples and validation sets of OMG’s SysML v2 release repository, release 2026-08 (commit fb97b754): 252 files, 14,957 lines. The models are fetched from OMG’s repository for each run and are not copied into Holarch.
• Reference: OMG’s pilot implementation, version 0.62.0, run as a black box.
• Large models: generated models of 10,000 to 250,000 lines with deep imports, references across packages and long specialization chains.

Every figure on this page comes from one official run of each measure on this build. A published run is never edited. A correction is a new run that names the one it replaces.

Summary

Example models

Each set of example models is analyzed together, as one project, because the examples import packages from each other. The pilot implementation gets the same files in one session. For each file, the two answers are compared: does the file have errors or not, and which element does each name bind to.

Acceptance agreement

• Raw: the engine and the pilot agree on whether the file has errors for 220 of 252 files.
• Classified: 228 of 228 files agree. Classified acceptance counts only the files that use supported constructs (252 files, less the 24 that use a construct on the published unsupported list), and counts a disagreement as agreement only where the specification shows the engine is right: an error in the example or in the pilot, each with the clause that shows it. Every class is defined in the rules below.
• False acceptances: 0. No file that the pilot rejects is accepted by the engine.
• Crashes and time-outs: none.

Classification rules. Each disagreement with the pilot is investigated on its own and recorded with one of these classes before a run, never during it:

• Example error: the engine is right and the example is invalid, with the specification clause that shows it. Example: arithmetic that adds a length to a mass. Counted as agreement.
• Pilot error: the engine is right and the pilot’s answer contradicts a cited specification clause. Counted as agreement.
• Unsupported construct: the disagreement comes from a construct on the engine’s published list of unsupported constructs (below). The file leaves the count. When the construct ships, it leaves the list and the file is measured again.
• Engine defect: a known bug in the engine. Never counted as agreement.

In this run: 24 files with unsupported constructs left the count, 7 disagreements were errors in the example and 1 was an error of the pilot. No known engine defect and no unclassified disagreement remains.

Name binding

For each name in a model, both tools report the element it binds to. Bindings that both tools report for the same name are compared.

• 4,753 of 4,768 paired bindings agree.
• Bindings that only one tool reports are not compared: 1,640 from the engine and 741 from the pilot. The run lists them, and the 15 that differ, for review.

Damaged text

Models are damaged on purpose, one mutation at a time: removed or extra braces, cut-off declarations, broken imports, circular references, invalid characters, random token changes and common typing mistakes such as a missing semicolon or a misspelled keyword. The inputs are the engine’s own test models, a generated model and the example models.

Speed and memory

Times are for the engine’s native build. In Holarch the engine runs as WebAssembly in the browser, which is slower; the criteria allow for that and for slower computers by a factor of 8 for edits and analysis.

Edits

Each edit type is applied 200 times, after 20 warm-up edits, in one editing session, the way Holarch’s editor sends them. The table shows the 95th percentile: 95 of 100 edits were this fast or faster.

Criteria: a value change within 12 ms (25 ms at 250,000 lines); a rename, import or supertype change within 60 ms (150 ms at 250,000 lines).

Analyzing a whole model

Cold analysis: a fresh process with the standard library loaded analyzes the whole model from nothing, the median of 5 runs. Memory is the engine’s peak heap during that analysis, the standard library included. The WebAssembly memory of the same analysis is shown for reference.

• Cold analysis: reported, not a target since method version 4. Holarch shows a progress bar for any analysis that can take over 3 seconds. The earlier target of 0.6 s at 100,000 lines (1.2 s at 250,000 lines) is not met: a first analysis of a very large model takes 1.31 s and 3.45 s. Edits after that first analysis are within their criteria at every size.
• Met: the progress bar appears one second into a long analysis, names the stage (starting the engine, loading the standard library, analyzing the model) and stays until the analysis finishes.
• Met: peak engine heap within 512 MiB at 100,000 lines and within 1 GiB at 250,000 lines.
• Met: time grows in line with model size. Cold analysis at 250,000 lines takes 23.3 times as long as at 10,000 lines, for a model 25 times as large (criterion: at most 40).

Not supported yet

These constructs are on the engine’s published list of unsupported constructs. Files of the example models that disagree with the pilot because of them leave the classified count; 24 files in this run.

• Individuals, snapshots and time slices beyond the base occurrence.
• Variability: variation and variant are kept as written; configurations are not computed.
• User-defined keywords and other language extensions.
• Textual representations in other languages are kept as text and not checked.
• Transition effects written as accept or action are not checked yet.

Every grammar production of the textual notation has at least one test that parses it and checks the result. The full list of what works in Holarch, with its limits: SysML v2 language coverage.

Reproducing the results

The results are independently reproducible: the evidence package holds everything needed to repeat each run on the same engine build.

• The measuring scripts and driver, with one command per measure.
• The pinned external inputs: the OMG release commit and the pilot package with its checksum. Both are fetched from their public sources; nothing from them is copied into the package.
• The generator of the large models, deterministic per seed.
• The classification table, with the class and specification clause of each resolved disagreement, and the list of unsupported constructs.
• Each run’s machine-readable results, its summary, its known defects and the checksum of every file.

The evidence package is available on request with an engine evaluation.

Earlier runs

Earlier official runs stay as they were measured, each under its own method version. Under methods 1 to 3, cold analysis within 0.6 s at 100,000 lines was a target; no run met it. Method 4 reports it without a target. Peak memory before method version 3 was the measuring process’s peak resident memory, which also counts memory the allocator keeps for reuse; it is not comparable with the engine heap figure above.

What these results do not show

• These are Holarch’s own measurements of the models they ran on. They are not a certification. The Object Management Group has not reviewed or endorsed them.
• The pilot implementation and the example models are not the specification. Where they and the engine differ, the classification above says which side the specification supports, and why.
• An outside review of a sample of the engine’s test cases against the specification is planned.
• The engine checks models; it does not execute actions or state machines.