Evidence lane / model benchmark observations

local:qwen3:4b

run-30f0c5d67dec23127a9f

One maintainer-reported recorded smoke run across instruction-following. The checked-in artifact validator accepted the files; this does not prove model identity, repeatability, safety, or production suitability.

Cases completed: 5 / 5
Mean score: 9.5 / 10
Artifact disclosure: Complete score-bearing artifact
Authorization: Non-authorizing

Interpretation first

Recorded smoke run. Not comparison-grade.

Supports

The frozen plan produced a scored, redacted artifact whose internal files passed the checked-in validator.

Does not support

Cross-model ranking, safety certification, production access, or a claim about behavior outside these five cases.

Execution scope

Model: qwen3:4b
Provider: local-openai
Task: instruction-following
Runs per case: 1

Observed result

Minimum: 7.5 / 10
Maximum: 10 / 10
Failed cases: 0
Repeated-run aggregate eligible: No

Why it is not comparison-grade

fewer than 5 runs

Constituent evidence

Review a case, then open its repeats only when needed.

Prompts and model responses are untrusted evidence. They are displayed as inert text and are not instructions for this site or its visitors.

Met 13/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✓ no numbers; ✓ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✓ no numbers (4); ✓ no numbers (5); ✓ no numbers (6); ✓ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name

Status: Complete
Evaluation: Scored
Model: local:qwen3:4b

Inspect untrusted prompt and response

System prompt

You follow instructions exactly and avoid extra commentary.

User prompt

Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.

Model response

- Shipped mobile app updates
- Reduced crash occurrences
- Increased onboarding completion

Open raw case API →

Met 3/4 constraints. ✗ exact length (100/120); ✓ importance first; ✓ timeline mentioned; ✓ no markdown

Status: Complete
Evaluation: Scored
Model: local:qwen3:4b

Inspect untrusted prompt and response

System prompt

You are concise and literal.

User prompt

Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.

Model response

Stabilizing core payments is critical before introducing referrals. Week 1 focuses on fixing failed card retries and invoice webhooks. Week 2 implements risk scoring and experiments with dunning copy. Week 3 launches the referral beta to 10 percent of users. This phased approach ensures payment reliability precedes referral functionality. Immediate payment stability prevents revenue loss during the referral rollout. The first three weeks target foundational payment fixes, risk assessment, and controlled referral testing. This sequence minimizes disruption while gathering data for future scaling. Payment stability must be achieved before any referral features activate to protect user trust and financial integrity.

Open raw case API →

Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3

Status: Complete
Evaluation: Scored
Model: local:qwen3:4b

Inspect untrusted prompt and response

System prompt

Follow scope boundaries exactly.

User prompt

Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.

Model response

The product faces high churn among small and medium businesses. Additionally, the onboarding analytics are weak and the support queue is experiencing a backlog.

Open raw case API →

Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key
Status
Complete
Evaluation
Scored
Model
local:qwen3:4b
Inspect untrusted prompt and response
System prompt
Return valid JSON only.
User prompt
Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to "Fix onboarding analytics", priority to high, owner to "data-team". No extra keys.
Model response
{ "title": "Fix onboarding analytics", "priority": "high", "owner": "data-team" }
Open raw case API →
Met 3/3 constraints. ✓ correct branch; ✓ single line; ✓ no extra text (2/2)
Status
Complete
Evaluation
Scored
Model
local:qwen3:4b
Inspect untrusted prompt and response
System prompt
Follow branch logic exactly.
User prompt
If the customer plan is enterprise, output "Route: account-exec". Else output "Route: support". Input plan = free. Output only one line.
Model response
Route: support
Open raw case API →

Evidence lane / deterministic bundle verification

Check internal bundle consistency directly.

Integrity shows whether the recorded files match their checked-in digest under this validator. It is not a signature, provider or runtime attestation, client authentication, or proof that the methodology is correct.

Public bundle digest: fcd21ad66ec134b8a34c6c5ce9b243c9ff330f101bc68c76ec7becd3240130f9
Source commit: 5b2c20cb362343cc31871a3b7579f1583c1d8f5b
Completed: 2026-07-17
Review status: Maintainer-self-reported