LLM providers

LLM provider selection and parity. For integrators choosing a provider and setting models.

Select a provider

from prettyplay.config import Config
from prettyplay.llm import create_provider

config = Config(provider="anthropic", model="claude-sonnet-4-5")
provider = create_provider(config)

The provider is a project setting: openai or anthropic; env override PRETTYPLAY_PROVIDER. API keys come only from environment variables: OPENAI_API_KEY for openai, ANTHROPIC_API_KEY for anthropic — read lazily on the first request.

Models

Setting	Purpose	Fallback
model	the main model for both operations	—
generation_model	code generation only	model
classification_model	failure classification, the group diagnosis and the compliance gate	model

The compliance gate runs on the effective classification model (classification_model or model) — never on the model that wrote the candidate.

base_url overrides the provider endpoint when set.

Parity

Both providers expose the same four operations — generate_step_code, classify_step_failure, classify_group_failure and check_instruction_compliance — with identical inputs, identical output shapes and the identical failure taxonomy: a provider service failure raises LLMUnavailableError; cached step code never depends on the provider. One request per attempt; attempt budgets belong to the calling engine.

User instructions parity: each operation carries its own instructions — generation requests render the generation_prompt setting, classification requests render the classification_prompt setting — as a verbatim USER INSTRUCTIONS block with identical placement semantics in both providers. A parity requirement, not a capability difference: classification requests never carry the generation instructions and generation requests never carry the classification instructions.

Regeneration block parity: every generation request renders a STEP TYPE line (action or assertion) immediately before the STEP line and may carry three extra blocks after the CHEAT SHEET and the optional USER INSTRUCTIONS blocks — HISTORY (the verbatim per-step attempt records, when present: each record carries the attempt outcome, the URL before -> after line, the complete candidate code and the complete error, with the original cached code anchored as record 0 on a healing path),RECOMMENDATION (the diagnosis of the classification that preceded the regeneration, when present) and USER GUIDANCE (the engineer guidance message of the interactive steering, when present) — rendered in this fixed order, identically in both providers. An unset input renders no block. Unrecognized classification labels fall back to incurable in both providers alike.

Group framing parity: a generation, regeneration or guided request of a step inside a group carries the group framing — a GROUP PROMPT block with the group prompt verbatim, rendered immediately before the PREVIOUS STEPS block, whose group entries are marked (- {sentence} [group step — {group prompt}]; ordinary entries render as the plain sentence). None renders no block and no marking, identically in both providers. The framing is an input of the request, never a capability difference.

Page-URL parity: a generation request may carry the current page URL — a non-empty page_url renders as its own PAGE URL line immediately after the PAGE SNAPSHOT block, identically in both providers; None renders no line. Supplied by every engine generation and healing request and by the guided requests of the interactive steering. A parity requirement, not a capability difference.

Cheat-sheet parity: every generation request renders the CHEAT SHEET block after the scenario inputs and immediately before the USER INSTRUCTIONS block — the compact standard Playwright sync API reference supplied by the calling engine; guidance, not an allowlist. Both providers render it identically at the same position. A parity requirement, not a capability difference.

The one transport-level asymmetry: the anthropic Messages API requires max_tokens, so anthropic requests carry a fixed completion cap (4096 tokens, sized so a full step-code response never truncates); the openai side sends no cap and the model maximum applies.

Answer shape

generate_step_code returns step code of the fixed form (see Driver facade). Models often answer with a fenced python block (```python … ```); the provider unwraps the first fenced block before returning, so the engine receives clean code either way — an answer with no closed fence is returned verbatim and, if unparsable, keeps failing downstream in execution.

Classification

classification = provider.classify_step_failure(
    prompt=system_prompt,  # the system prompt text comes from the calling engine
    user_instructions="",  # the classification instructions from the classification_prompt setting; empty — no block
    step_text="click the «Sign in» button",
    code=step_code,
    error="element not found: button «Sign in»",
    snapshot=snapshot_text,
    screenshot=None,
)
print(classification.category, classification.explanation, classification.recommendation)

The classification categories — rot, product_defect, fixable, incurable — and their consequences are covered in Self-healing.

A non-empty user_instructions renders as a separate USER INSTRUCTIONS block in the request — the final block of the user content, after all classification inputs.

The group diagnosis operation

classify_group_failure is the diagnosis request of the group recovery (see Groups): one request per recovery cycle, carrying the group prompt verbatim, the composed traces of the group's steps, the failed step's sentence and attempt history, the page snapshot and the optional screenshot — sent through the effective classification model (classification_model or model) with the project's classification_prompt instructions, full parity between the providers. The answer parses strictly; a degraded answer maps to the conservative incurable with the raw answer logged — never a granted regeneration.

The compliance operation

check_instruction_compliance is the verdict request of the compliance gate (see Configuration): the engine calls it once per successfully executed candidate before caching — judging both the instruction compliance and the step adequacy in one request — never for replayed cached code, never when generation_approve is off or generation_prompt is empty (zero calls).

findings = provider.check_instruction_compliance(
    prompt=system_prompt,  # the gate system prompt text comes from the calling engine
    user_instructions=instructions,  # the project's user instructions (the generation_prompt setting)
    step_text="click the «Sign in» button",
    step_type="action",  # action | assertion — the adequacy dimension judges by it
    code=step_code,
    attempt_history=[r.render() for r in history],  # the verbatim per-step attempt records, when present
)

• full parity between the providers: the user content carries four blocks in the fixed order — INSTRUCTIONS, STEP (with its STEP TYPE line), ATTEMPT HISTORY (omitted when the step has no attempt records) and CODE — built identically by both through one shared builder; no screenshot input on this operation
• the gate model is the effective classification model (classification_model or model)
• the answer parses strictly: a JSON list of findings, each with instruction, priority (high|medium|low), explanation and dimension (instruction|adequacy) — the instruction dimension quotes the violated instruction, the adequacy dimension names the fragment of the step sentence the code fails to accomplish, judged from the step type and the attempt history; an empty list [] means compliant; a JSON syntax glitch of the answer is salvaged once (the json-repair library) before the validation — a model dropping a quote, a comma or a bracket does not fail the run — and a malformed verdict (an answer of the old shape — a finding without a dimension — included) raises ComplianceVerdictError — never a silent pass. Only a high finding — in either dimension — blocks the candidate, and that decision belongs to the calling engine, not the provider
• SDK errors map to LLMUnavailableError exactly like the other operations — llm unavailable: {provider} request failed
• one request per call, no retry inside the provider