AI
How Holarch Chooses AI Models
Why Holarch's AI uses Claude, and GLM 5.3 in mainland China, Hong Kong, Macau and Taiwan, with Kimi K3 and GLM 5.3 as own-key options: how models from Anthropic, OpenAI, Google and xAI are tested on Holarch's own AI tasks, the quality bar, the latest result, and how another model can be added.
On this page
Holarch's AI features use Claude (Anthropic). On the hosted service, AI requests from mainland China, Hong Kong, Macau and Taiwan use GLM 5.3 (Z.ai) instead (see GLM 5.3 for mainland China, Hong Kong, Macau and Taiwan). This page explains how these choices were made and when they change.
How models are tested
Holarch tests each model on its own AI tasks, with real requests from Holarch projects:
- Requirement reviews of specifications that contain seeded defects and near-duplicates.
- Rewrites: improving, splitting and making requirements measurable.
- Decompositions of requirements and functions.
- Flow diagrams written as text, checked by Holarch's own parser.
- Trade-study criteria and risk candidates.
- Ask AI questions with known answers, and "not yet" questions about features Holarch does not have.
Each request is sent three times, so one lucky or unlucky answer does not decide the result.
The quality bar
A model is offered when it meets all of these:
- At least 90% of the tasks are fully usable.
- Every "not yet" question is answered truthfully.
- Every seeded defect and duplicate is caught.
- Every answer is valid for Holarch to read (correct structure and flow syntax).
Latest result
Models that met the bar:
- Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5. Sonnet had the highest share of fully usable tasks (98%).
- Google Gemini 3.8 Flash, just over the 90% line (91%). It answered every "not yet" question truthfully, caught every seeded defect and duplicate, and gave valid answers in every run.
- Kimi K3 (Moonshot AI), tested later on the same review set and the live AI checks: it flagged every seeded defect in every run, answered every "not yet" question truthfully and gave valid answers. It is offered with your own Moonshot key (below).
- GLM 5.3 (Z.ai), tested on the same review set and the qualification requests: it flagged every seeded defect and duplicate, answered the "not yet" questions truthfully, gave valid answers in every run and answered about as fast as Claude. Its decompositions were shallower than Claude's. It is offered with your own Z.ai key, and on the hosted service it is the AI for requests from mainland China, Hong Kong, Macau and Taiwan (below).
Models that did not meet the bar yet:
- Google Gemini 3.1 Pro: some requests ran past Holarch's 300-second limit per AI request, and 76% of tasks were fully usable.
- OpenAI (GPT-5, GPT-6.1 Sol, GPT-6 Luna): 72–81% of tasks fully usable. Requirement reviews sometimes rated a seeded vague requirement as correct while naming the vague words in the comment.
- xAI (Grok 4.7, Grok 4.3): 85% and 62% of tasks fully usable. Grok 4.7 missed one seeded defect; Grok 4.3 gave two "not yet" answers that did not say the feature is missing, and one flow Holarch could not read.
- Google Gemma 4 (open model), in four sizes on an earlier, smaller set of 18 requests:
- E2B and E4B, small enough to run in a browser, failed most tasks: rewrites that stayed vague, decompositions that restated the parent, and replies Holarch could not read.
- 31B and 26B-A4B gave fully usable results on 9 of 18 requests, and results that needed real editing on 8. They caught seeded defects and wrote valid flows, but new requirements, decompositions and test cases stayed vague. The 26B-A4B model also answered a "not yet" question as if the feature existed.
Holarch's own AI is Claude, except for requests from mainland China, Hong Kong, Macau and Taiwan (below): Claude scored higher on these tasks than Gemini 3.8 Flash, Kimi K3 and GLM 5.3, the other models that met the bar.
Kimi K3 (Moonshot AI) is qualified too, and offered as an option with your own Moonshot key (requests with it send your prompts and the project content they include to Moonshot AI), for teams where Claude is not available (see Kimi K3 with your own key). On Holarch's requirement reviews it flagged every seeded defect in every run and found the seeded duplicate in nearly every run, answered every "not yet" question truthfully, gave valid answers, and wrote fluent Chinese. It is 2 to 4 times slower than Claude per request.
GLM 5.3 (Z.ai) is qualified as well, and offered as an option with your own Z.ai key; your requests go to Z.ai (Singapore) under your agreement with Z.ai (see GLM 5.3 with your own key). It flagged every seeded defect and duplicate, gave valid answers in every run and wrote fluent Chinese, about as fast as Claude. When it decomposes a requirement, its child requirements are shallower than Claude's.
Kimi K3 and GLM 5.3 are options you can turn on; in an organization they stay off until an organization administrator turns them on (see Kimi K3 and GLM 5.3 in organizations).
The included AI uses these models: Claude Sonnet on Individual and Team, Claude Opus on Team for heavy tasks, and Claude Haiku on the Free plan for accounts with an academic email address and members of an active class organization (see Included AI). The PDF specification import uses Claude Sonnet on every plan: in our comparison it found every requirement. All three met the bar; Claude Haiku costs the least per request, which is why academic Free accounts can include AI at all.
Note: Whatever the model, every AI change is still a preview: you review it, edit it if needed, and apply it; a single Undo takes it all back.
GLM 5.3 for mainland China, Hong Kong, Macau and Taiwan
On the hosted service, Holarch does not use Claude for AI requests from mainland China, Hong Kong, Macau and Taiwan. The AI included in the plans answers these requests with GLM 5.3, through Holarch's own Z.ai key, with the same number of requests as everywhere else. Support for customers there also uses GLM 5.3: the translation and summaries of their support requests, and the translation in live chats and voice calls.
GLM 5.3 was chosen because it met the same bar on the same tests as the other qualified models: it flagged every seeded defect and duplicate, answered the "not yet" questions truthfully, gave valid answers in every run, and answers about as fast as Claude, faster than Kimi K3. Its decompositions of a requirement are shallower than Claude's. PDF specification import is not available there yet, because it needs Claude.
What people there see, where their requests go and how travel works: AI in Mainland China.
When this changes
- New models are tested as they appear.
- A provider can be offered again when its models meet the bar. Models are tested again when Holarch's AI tasks change.
- An API key you added earlier for another provider is kept and not used. If you no longer need it, click Remove next to it in Account Settings → AI keys or in AI Settings….
Want another model?
Use Contact Us… in the account menu, or the contact form. Name the model and the tasks you use AI for.