# 130 — Real-model experiments

Date: **2026-10-13**. Standa confirmed his Matrix demo test and authorized real LLM optimization (D41). This record separates completed measurements, manual content review and rejected experiments. All recorded API jobs have completed.

## Protocol

- Primary exact models: `gpt-6-luna` and `claude-sonnet-5-5`, from the existing catalog/clients. No substitute model, TTS or plot search. After their repeated factual failures, two clearly labeled control cases use existing `gpt-6.1-sol` / `claude-opus-5-5` at medium effort; no model catalog or persistent selection is changed.
- Matrix (1999), initially Hard 20 scenes to expose length, order and whole-story problems. One fresh production preparation per ordinary case; no automatic repair inference.
- Provider/effort overrides are local to the headless process. Existing Kodi settings are not changed.
- Real requests pass through the existing `AIProvider`, accounting and parser. Optional benchmark budget/timeout overrides are explicitly recorded; do not present an overridden case as proof of unchanged production defaults.
- `--capture-content` saves only this authorized diagnostic prompt/response/pack under `scripts/game_harness/out/story-ai-review/`. No headers, credentials, hidden reasoning transcript, production cache or later pack reuse.
- Automatic PASS means complete structural validation and local engine completion only. Review all setup/question/options/correct/reveal/ending fields separately for facts, presentation order, language, answer leaks and ambiguity. A valid JSON game is not evidence of canonical accuracy.
- Manual Matrix review uses the known film and the authored demo as a reference. No online grounding or paid second-model judge is used. These qualitative checks are not universal truth proofs; uncertain details must be identified as such.
- Costs use returned token usage and current catalog rates. Timeout without returned usage has **unknown cost**, never zero. Include failed calls in latency/reliability results.

## Reusable command

```bash
python3 scripts/game_harness/game.py story-review --meta matrix --difficulty hard \
  --provider openai --model gpt-6-luna --set openai_reasoning_effort=low \
  --capture-content --out scripts/game_harness/out/story-ai-review/example
```

Use `--provider anthropic --model claude-sonnet-5-5 --set anthropic_effort=high` for Sonnet. Experimental high-budget cases use `--output-tokens 32000 --request-timeout 300`. Both clients retain their previous default timeout when the override is absent.

## Completed stages

| Stage | Configuration | Structural result | Content/result finding |
|---|---|---|---|
| Baseline | Both models, low/medium, 16K output | 0/4 playable | Overlong options; Luna low also had unequal chapters. Manual review found invented bridges, confused rescues/deaths and order errors. |
| v2 | Explicit chapter sequence, 12–22 character option target, stronger factual constraints | 4/4 playable | All four still had significant plot/order/consistency defects. Low was faster; Luna medium almost doubled latency without fixing the observed content problems. |
| v2 high | Both models, original budget/timeouts | 0/2 playable | Luna read timeout at 121 s, charge unknown. Sonnet exhausted 16K output tokens at 127 s with no usable text. This measures the configuration ceiling, not high-effort factual quality. |
| v3 | More internal planning/shorter facts, medium | 0/2 playable | Luna timed out; Sonnet exceeded option limits and still confused events. Rejected as a production candidate. |
| Outline experiment | English factual extraction, then Czech pack, low | 0/2 attempts playable; 3 inferences total | Luna returned fewer than 20 events and stopped before step two. Sonnet's outline had factual/order defects; the second call preserved them and exceeded option lengths. Rejected; no two-call production architecture. Experimental harness snapshot retained in the output folder only. |
| v4 | Facts first in JSON; A canonical, B–D distractors; local letter shuffle preserved | 4/4 playable | Low still had major defects. Extended-budget high completed: Luna 103 s / $0.005865 but English output; Sonnet 158 s / $0.204152 with better core actions but remaining order/details to review. |
| v5 | Explicit Czech field language; chapter numbers purely positional, without thematic grouping cues | 8/10 playable | Luna high returned 21 scenes; max exhausted 32K entirely in reasoning. Sonnet high produced a strong 20-scene sample, but repeated 15/20 packs still reversed Oracle/Cypher or vase/cookie order. Low/medium retained major errors or multiple true options. |
| v6 | Shorter neutral setup and one-event reveal; concrete actions prioritized | 4/7 playable | Sonnet high improved measured latency to 117 s / $0.154202 versus v5's 190–224 s / $0.248412–0.290312. It still invented a doctor at the pod and moved cookie before vase. xhigh exhausted 32K; max exhausted 64K with no full JSON. Fresh 10/15 high packs are structurally valid but fail manual order/ambiguity checks. |

Each case contains `summary.json`, optional `diagnostic.json` / `pack.json`, and `report.md`. Existing `manual-review.json` records explain failures rather than hiding them behind a structural PASS. Prompt snapshots `baseline-prompts.py`, `v2-prompts.py`, … make stage comparisons reproducible.

## Official parameter sources

Luna supports the tested low/medium/high/max efforts and structured output. [OpenAI Luna model](https://developers.openai.com/api/docs/models/gpt-6-luna).

Sonnet's effort and hidden-thinking token behavior are documented separately; thinking consumes the output limit. [Anthropic effort](https://platform.claude.com/docs/en/build-with-claude/effort).

Native structured output can enforce shape but cannot prove film facts. Provider schema subsets differ, including string-length restrictions, so do not add a copied schema/client or assume identical constraint support. [OpenAI structured outputs](https://developers.openai.com/api/docs/guides/structured-outputs), [Anthropic structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs).

## Completion gate

Select the smallest supported preparation/configuration with demonstrated improvements. Confirm the chosen prompt/configuration on fresh 10/15/20-scene packs, inspect actual content, preserve one-shot/no-cache behavior and record practical latency/cost. Update production code only for justified fixes and rerun affected offline checks. Do not claim guaranteed accuracy or an optimal setting for every title from Matrix-only samples. This measured-comparison gate is now completed with the Sol medium samples below.

At this experiment checkpoint additional authored demos were deferred under D40. The later D42/D43 follow-up completed them in `140-demo-catalog.md`. A failed experiment is evidence to reject that variant, not a reason to keep adding speculative runtime layers.

## Implementation during comparison

The existing Story source now requests 32K/300 s for OpenAI/Anthropic high/xhigh/max, otherwise 16K/client defaults. This is a call-specific ceiling; selected provider/model/effort and persistent Kodi settings are unchanged. Both clients support an optional per-call timeout with their original defaults preserved. One-shot generation, strict full-pack rejection, local option shuffling and content privacy remain intact.

The final max benchmark explicitly requests 64K/600 s to distinguish the tested production ceiling from a model's behavior with more room. Do not recommend that result as supported by unchanged production configuration.

## Current practical result

Within the primary Luna/Sonnet comparison, **Sonnet 5.5 / high** produced the strongest core-action samples. With the compact v6 prompt and production 32K/300 s ceiling, fresh Easy/Medium cases ran without benchmark overrides; Hard used equivalent explicit overrides. All passed schema, requested counts, lengths and local completion:

| Questions | Seconds | Returned output tokens, including thinking | USD | Manual content gate |
|---|---:|---:|---:|---|
| 10 | 81.573 | 10061 | 0.103860 | FAIL: vase/cookie order, two true escape options |
| 15 | 94.641 | 11827 | 0.121544 | FAIL: vase/cookie order, final-phone order, important narrative jumps |
| 20 | 116.738 | 15089 | 0.154202 | FAIL: invented doctor at pod, vase/cookie order |

These are individual measurements, not stable averages or an accuracy guarantee. Low/medium are cheaper/faster but repeatedly confused major events or allowed multiple true answers. Luna is substantially cheaper, yet neither increased effort nor the tested prompt variants established a sufficiently faithful pack; some cases also returned 21 scenes or no usable text.

Sonnet xhigh exhausted 32K at 259.161 s / $0.323312, with no playable pack. Max exhausted 64K at 489.640 s / $0.643312, also without a playable pack. More reasoning was not a reliable accuracy cure in these cases.

**Neither Luna nor Sonnet passed the factual/chronological gate consistently.** Keep this visible in any recommendation. The implemented prompt/transport changes improve compactness, structural playability and budget handling; they do not prove universal knowledge-only accuracy. Further model suites or future grounding must remain separate scoped work, never hidden automatic fallbacks.

### Two stronger-model control cases

The autonomous LLM optimization task was extended with one fresh Hard case each for Sol 6.1 and Opus 5.5 after Luna/Sonnet failed the factual gate. These explicitly announced experiments used the same v6 prompt and production boundary, without a new provider/client, plot search or runtime fallback.

- Sol medium: **72.103 s / $0.037888**, 20 valid scenes. All queried actions/order matched the known film, but one reveal added unsupported no-hand-contact to spoon bending. Manual verdict FAIL for that added fact, rather than ignoring it.
- Opus medium: **63.317 s / $0.120904**, 20 valid scenes. Manual PASS_WITH_NOTES: strong core actions/order, minor compression of Trinity/Neo helicopter cooperation. One sample only, no general reliability conclusion.

### Final recommendation: Sol 6.1 / medium

Three further fresh Sol medium cases use the **ordinary production configuration**, without budget/timeout overrides, saved content or plot grounding. Actual requested and returned model both equal `gpt-6.1-sol`. Budget 16K; default OpenAI timeout 120 s.

| Questions | Seconds | Output / reasoning tokens | Computed USD | Manual content result |
|---|---:|---:|---:|---|
| 10 | 44.459 | 2019 / 516 | 0.022506 | PASS_WITH_NOTES: broad jumps; Smith destruction omitted before flight ending |
| 15 | 66.803 | 3174 / 917 | 0.034088 | PASS_WITH_NOTES: capture conveyed as current state; correct actions/order |
| 20 | 86.542 | 4038 / 1034 | 0.040505 | PASS_WITH_NOTES: rope/jump setup compressed; correct actions/order |

Read the per-case `manual-review.json` as well as `summary.json`; all visible fields/options/ending were reviewed. These samples showed markedly better core fidelity and lower cost than Sonnet high. They do not prove 100% accuracy for every work, and the initial Sol sample's extra spoon claim remains a recorded failure. Fresh requests may still fail strict structure or factual quality.

Recommend Sol medium **for this Story trial**, while preserving Standa's selected provider/model/effort rather than silently changing global defaults. Sonnet high is the best observed Luna/Sonnet-only option, with material remaining accuracy/latency limits. Higher effort is not recommended simply because it is higher.

## Final accounting and code state

- **41 inferences across 40 attempts**, including the rejected two-call outline experiment. Known computed cost **$3.305965**, plus **two unknown timeout charges** (`v2-openai-high`, `v3-openai-medium`). Failed calls are included. This is returned-usage pricing, not a billing receipt.
- Machine-readable complete ledger: `scripts/game_harness/out/story-ai-review/comparison.json`. Prompt snapshots and all failures remain available; no running review jobs remain.
- Final source uses compact generic v6; no Matrix plot injected into the prompt, cached generated pack, second-model judge, automatic repair or provider fallback was introduced.
- Targeted offline verification: Story 16 tests, transport 1 test / five providers plus two overrides, review privacy/accounting 2 tests, Millionaire AI-content 5 tests. Seven related Python files compile. Three updated skills validate. Plan links/fences are checked in the final handoff.
- Source/diagnostic changes are **local and unpublished**; latest published addon remains 2.1.65 / skin 1.3.52. No commit. Further authored demos were deferred at this checkpoint and subsequently completed under D42/D43; see `140-demo-catalog.md`.

Exact model/capability references: [Sol 6.1](https://developers.openai.com/api/docs/models/gpt-6.1-sol), [Opus 5.5](https://platform.claude.com/docs/en/models/opus-5-5/whats-new-opus-5-5). Opus effort names need calibration within each model; their computational amount is not identical across model families. [Opus prompting guidance](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5).