Holarch Language Engine: evidence package
=========================================

This package holds what is needed to repeat the published test runs of the Holarch Language Engine
(https://holarch.app/language-engine/evidence/) on the same engine build: the method, the classification tables,
the run records, the model generator and the measuring and comparison scripts.

Start with REPRODUCE.md.

These are Holarch's own measurements. They are not a certification, and the Object Management Group has not
reviewed or endorsed them. Every classification and review in this package is an internal review (METHOD.md,
"Reviews").

Contents
--------
  REPRODUCE.md                  one page: exact commands, dependencies, input checksums, expected results, run times
  METHOD.md                     the test method, the criteria and the method versions
  LICENSE.txt                   terms for this package
  classification/
    unsupported.tsv             constructs on the published unsupported list (one pattern each)
    resolutions.tsv             every classified disagreement with the pilot: class, basis (clause), note
    register.tsv                the disagreements the specification settles, with clause and review status
    diagnostics-<build>.tsv     every engine error on the evaluated files, reviewed: rule, clause, primary or
                                cascade, disposition
    bindings-<build>.tsv        the paired name-binding disagreements, each explained
    bindings-summary-<build>.json  bindings only one tool reports, by category
  runs/<run-id>/<measure>/      summary.md, measurements.jsonl and known-defects.md of each official run
  runs/wasm-safari-<build>.jsonl   the WebAssembly timings measured in Safari
  runs/wasm-safari-session-<build>.jsonl   the WebAssembly session memory measured in Safari
  scripts/                      the measuring and comparison scripts (Python 3.9+, standard library only),
                                the driver source (Rust), the model generator, the Safari timing script
  SHA256SUMS                    checksum of every file in this package

What is not included
--------------------
  - The engine itself. The driver builds against the engine's source or SDK, available with an engine
    evaluation (https://holarch.app/contact/?topic=Licensing).
  - OMG's example models and the pilot implementation. scripts/fetch.py fetches both from their public sources at
    the commit and package checksum pinned in scripts/pins.json.
  - The engine's own test corpus (used by the damaged-text and same-answer measures).

Reproducing a run
-----------------
  1. Fetch the pinned inputs:          python3 scripts/fetch.py omg pilot docs
  2. Build the driver (needs the engine source from the evaluation, placed at crates/sysml in this folder):
                                       cd scripts/driver && cargo build --release
  3. Example models and name binding:  python3 scripts/run.py omg --pilot --bindings
  4. Large models (speed and memory):  python3 scripts/run.py perf
                                       python3 scripts/run.py session-memory
  5. Damaged text:                     python3 scripts/run.py robust --gen 10000 --omg
  6. Classified errors and bindings:   python3 scripts/review_diagnostics.py <run-id>
                                       python3 scripts/bindings_breakdown.py <run-id>
Runs outside Holarch are dry runs: they land in the cache folder (scripts/results.py), not in runs/. Compare their
summary with runs/<run-id>/. Timings depend on the computer; counts do not.

Requirements: Python 3.9+, Rust (the toolchain pinned with the engine), git, zstd, pdftotext (poppler), Java 21+ for
the pilot.
