{"case_id":"if-02","case_sha256":"e9bb583b60c583f5dd69a7563ba439234342207240acf453b860874df57b613b","category":"instruction-following","completed_at":"2026-07-17T07:01:45.533Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/4 constraints. ✗ exact length (78/120); ✓ importance first; ✓ timeline mentioned; ✓ no markdown","model":"local:qwen3:4b","plan_ordinal":2,"prompt":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","raw_score":7.5,"record_id":"record-a611e3a69708ef7c18e4f895","response":"Stabilizing core payments is the critical prerequisite for referral integration. Timeline: Week 1 resolves failed card retries and invoice webhook failures. Week 2 introduces risk scoring and experiments with dunning copy variations. Week 3 executes referral beta for 10% of users. This phased approach ensures payment stability before referral features, preventing revenue leakage from unstable transactions. Core payment stability must be achieved prior to referral rollout to avoid financial risks and maintain user trust during the beta phase.","run_index":1,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:01:17.274Z","status":"complete","system_prompt":"You are concise and literal.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:01:45.533Z","latencyMs":28259,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-940","rawResponse":"Stabilizing core payments is the critical prerequisite for referral integration. Timeline: Week 1 resolves failed card retries and invoice webhook failures. Week 2 introduces risk scoring and experiments with dunning copy variations. Week 3 executes referral beta for 10% of users. This phased approach ensures payment stability before referral features, preventing revenue leakage from unstable transactions. Core payment stability must be achieved prior to referral rollout to avoid financial risks and maintain user trust during the beta phase.","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You are concise and literal.","role":"system"},{"content":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":547,"responseJsonBytes":549,"startedAt":"2026-07-17T07:01:17.274Z","synthetic":false,"usage":{"inputTokens":103,"outputTokens":892,"totalTokens":995}}]}
{"case_id":"if-01","case_sha256":"fb32d183379053efee74c73dc7c5cc70d5c4d2364ba838f137aff5c9bcf564d7","category":"instruction-following","completed_at":"2026-07-17T07:02:16.358Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 13/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✓ no numbers; ✓ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✓ no numbers (4); ✓ no numbers (5); ✓ no numbers (6); ✓ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name","model":"local:qwen3:4b","plan_ordinal":1,"prompt":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","raw_score":10,"record_id":"record-7714abb05506fb6c51debc7b","response":"- Deployed updates to the mobile application\n- Reduced crash incidents significantly\n- Increased onboarding completion rate through streamlined user flow","run_index":1,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:01:17.274Z","status":"complete","system_prompt":"You follow instructions exactly and avoid extra commentary.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:02:16.357Z","latencyMs":59083,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-517","rawResponse":"- Deployed updates to the mobile application\n- Reduced crash incidents significantly\n- Increased onboarding completion rate through streamlined user flow","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You follow instructions exactly and avoid extra commentary.","role":"system"},{"content":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":153,"responseJsonBytes":157,"startedAt":"2026-07-17T07:01:17.274Z","synthetic":false,"usage":{"inputTokens":108,"outputTokens":869,"totalTokens":977}}]}
{"case_id":"if-03","case_sha256":"6983acf5aeabf690a436232c07b86a56d3f4041a596072f98128182d7ae1d28d","category":"instruction-following","completed_at":"2026-07-17T07:02:48.412Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3","model":"local:qwen3:4b","plan_ordinal":3,"prompt":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","raw_score":10,"record_id":"record-c98f948f0f4ca27058569b70","response":"The product risks include high churn in small and medium businesses and weak onboarding analytics. Additionally, the support queue backlog represents a critical operational challenge.","run_index":1,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:01:17.274Z","status":"complete","system_prompt":"Follow scope boundaries exactly.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:02:48.412Z","latencyMs":91138,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-718","rawResponse":"The product risks include high churn in small and medium businesses and weak onboarding analytics. Additionally, the support queue backlog represents a critical operational challenge.","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow scope boundaries exactly.","role":"system"},{"content":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":183,"responseJsonBytes":185,"startedAt":"2026-07-17T07:01:17.274Z","synthetic":false,"usage":{"inputTokens":82,"outputTokens":845,"totalTokens":927}}]}
{"case_id":"if-04","case_sha256":"ee0e92615cbb9c51799f7dd8b4ca49d8cf8b3c7466d7ee9ed78979990838b780","category":"instruction-following","completed_at":"2026-07-17T07:04:11.726Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key","model":"local:qwen3:4b","plan_ordinal":4,"prompt":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","raw_score":10,"record_id":"record-e9b7aab679176f1f8daf3899","response":"{\"title\":\"Fix onboarding analytics\",\"priority\":\"high\",\"owner\":\"data-team\"}","run_index":1,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:01:45.539Z","status":"complete","system_prompt":"Return valid JSON only.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:04:11.726Z","latencyMs":146187,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-422","rawResponse":"{\"title\":\"Fix onboarding analytics\",\"priority\":\"high\",\"owner\":\"data-team\"}","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Return valid JSON only.","role":"system"},{"content":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":74,"responseJsonBytes":88,"startedAt":"2026-07-17T07:01:45.539Z","synthetic":false,"usage":{"inputTokens":67,"outputTokens":1850,"totalTokens":1917}}]}
{"case_id":"if-05","case_sha256":"10ac97b006bc810f1848f42ffe9d407b39bd8f684ed974fa487aa34409963510","category":"instruction-following","completed_at":"2026-07-17T07:04:14.596Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/3 constraints. ✓ correct branch; ✓ single line; ✓ no extra text (2/2)","model":"local:qwen3:4b","plan_ordinal":5,"prompt":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","raw_score":10,"record_id":"record-d0b4a18ead91e3c07c621aa4","response":"Route: support","run_index":1,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:02:16.389Z","status":"complete","system_prompt":"Follow branch logic exactly.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:04:14.596Z","latencyMs":118207,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-129","rawResponse":"Route: support","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow branch logic exactly.","role":"system"},{"content":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":14,"responseJsonBytes":16,"startedAt":"2026-07-17T07:02:16.389Z","synthetic":false,"usage":{"inputTokens":51,"outputTokens":66,"totalTokens":117}}]}
{"case_id":"if-02","case_sha256":"e9bb583b60c583f5dd69a7563ba439234342207240acf453b860874df57b613b","category":"instruction-following","completed_at":"2026-07-17T07:04:40.855Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/4 constraints. ✗ exact length (112/120); ✓ importance first; ✓ timeline mentioned; ✓ no markdown","model":"local:qwen3:4b","plan_ordinal":7,"prompt":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","raw_score":7.5,"record_id":"record-76269411357bc6d8a9893949","response":"Stabilizing core payments is the critical priority before introducing referrals. Timeline: Week 1 resolves payment failure retries and invoice webhook failures; Week 2 implements risk scoring and experiments with dunning copy; Week 3 launches referral program beta to 10 percent of users. This sequence ensures payment reliability precedes referral functionality. Payment stability fixes prevent revenue loss during early adoption. Risk scoring mitigates fraud while dunning copy tests user retention messaging. The referral beta targets a small cohort for controlled impact measurement. All tasks directly support the core payment stabilization objective. This phased approach minimizes risk and maximizes learning before full referral rollout. Payment reliability must be confirmed before any referral features activate.","run_index":2,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:04:14.612Z","status":"complete","system_prompt":"You are concise and literal.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:04:40.855Z","latencyMs":26243,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-592","rawResponse":"Stabilizing core payments is the critical priority before introducing referrals. Timeline: Week 1 resolves payment failure retries and invoice webhook failures; Week 2 implements risk scoring and experiments with dunning copy; Week 3 launches referral program beta to 10 percent of users. This sequence ensures payment reliability precedes referral functionality. Payment stability fixes prevent revenue loss during early adoption. Risk scoring mitigates fraud while dunning copy tests user retention messaging. The referral beta targets a small cohort for controlled impact measurement. All tasks directly support the core payment stabilization objective. This phased approach minimizes risk and maximizes learning before full referral rollout. Payment reliability must be confirmed before any referral features activate.","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You are concise and literal.","role":"system"},{"content":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":822,"responseJsonBytes":824,"startedAt":"2026-07-17T07:04:14.612Z","synthetic":false,"usage":{"inputTokens":103,"outputTokens":560,"totalTokens":663}}]}
{"case_id":"if-01","case_sha256":"fb32d183379053efee74c73dc7c5cc70d5c4d2364ba838f137aff5c9bcf564d7","category":"instruction-following","completed_at":"2026-07-17T07:06:03.271Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 13/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✓ no numbers; ✓ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✓ no numbers (4); ✓ no numbers (5); ✓ no numbers (6); ✓ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name","model":"local:qwen3:4b","plan_ordinal":6,"prompt":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","raw_score":10,"record_id":"record-da7317aba30addc67d0a80a2","response":"- Simplified initial user flow\n- Rewrote onboarding copy\n- Increased onboarding completion","run_index":2,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:04:14.612Z","status":"complete","system_prompt":"You follow instructions exactly and avoid extra commentary.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:06:03.271Z","latencyMs":108659,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-517","rawResponse":"- Simplified initial user flow\n- Rewrote onboarding copy\n- Increased onboarding completion","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You follow instructions exactly and avoid extra commentary.","role":"system"},{"content":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":90,"responseJsonBytes":94,"startedAt":"2026-07-17T07:04:14.612Z","synthetic":false,"usage":{"inputTokens":108,"outputTokens":2352,"totalTokens":2460}}]}
{"case_id":"if-03","case_sha256":"6983acf5aeabf690a436232c07b86a56d3f4041a596072f98128182d7ae1d28d","category":"instruction-following","completed_at":"2026-07-17T07:07:20.634Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3","model":"local:qwen3:4b","plan_ordinal":8,"prompt":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","raw_score":10,"record_id":"record-e9f4b365f9c6299221019912","response":"The product experiences high SMB churn. Additionally, it has weak onboarding analytics and a support queue backlog.","run_index":2,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:04:14.612Z","status":"complete","system_prompt":"Follow scope boundaries exactly.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:07:20.634Z","latencyMs":186022,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-568","rawResponse":"The product experiences high SMB churn. Additionally, it has weak onboarding analytics and a support queue backlog.","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow scope boundaries exactly.","role":"system"},{"content":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":115,"responseJsonBytes":117,"startedAt":"2026-07-17T07:04:14.612Z","synthetic":false,"usage":{"inputTokens":82,"outputTokens":2172,"totalTokens":2254}}]}
{"case_id":"if-04","case_sha256":"ee0e92615cbb9c51799f7dd8b4ca49d8cf8b3c7466d7ee9ed78979990838b780","category":"instruction-following","completed_at":"2026-07-17T07:07:23.425Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key","model":"local:qwen3:4b","plan_ordinal":9,"prompt":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","raw_score":10,"record_id":"record-cfb541def660f59ba34eb718","response":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","run_index":2,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:04:40.873Z","status":"complete","system_prompt":"Return valid JSON only.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:07:23.425Z","latencyMs":162552,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-755","rawResponse":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Return valid JSON only.","role":"system"},{"content":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":87,"responseJsonBytes":105,"startedAt":"2026-07-17T07:04:40.873Z","synthetic":false,"usage":{"inputTokens":67,"outputTokens":88,"totalTokens":155}}]}
{"case_id":"if-05","case_sha256":"10ac97b006bc810f1848f42ffe9d407b39bd8f684ed974fa487aa34409963510","category":"instruction-following","completed_at":"2026-07-17T07:07:25.822Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/3 constraints. ✓ correct branch; ✓ single line; ✓ no extra text (2/2)","model":"local:qwen3:4b","plan_ordinal":10,"prompt":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","raw_score":10,"record_id":"record-e8f1db7afeba6aba8657a870","response":"Route: support","run_index":2,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:06:03.277Z","status":"complete","system_prompt":"Follow branch logic exactly.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:07:25.822Z","latencyMs":82545,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-218","rawResponse":"Route: support","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow branch logic exactly.","role":"system"},{"content":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":14,"responseJsonBytes":16,"startedAt":"2026-07-17T07:06:03.277Z","synthetic":false,"usage":{"inputTokens":51,"outputTokens":66,"totalTokens":117}}]}
{"case_id":"if-03","case_sha256":"6983acf5aeabf690a436232c07b86a56d3f4041a596072f98128182d7ae1d28d","category":"instruction-following","completed_at":"2026-07-17T07:07:58.476Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3","model":"local:qwen3:4b","plan_ordinal":13,"prompt":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","raw_score":10,"record_id":"record-3d324ab67dd0c24256dab43b","response":"The product faces significant risks including high churn rates among small and medium businesses and inadequate onboarding analytics. Additionally, the support queue backlog is causing operational strain.","run_index":3,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:07:25.828Z","status":"complete","system_prompt":"Follow scope boundaries exactly.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:07:58.475Z","latencyMs":32648,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-265","rawResponse":"The product faces significant risks including high churn rates among small and medium businesses and inadequate onboarding analytics. Additionally, the support queue backlog is causing operational strain.","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow scope boundaries exactly.","role":"system"},{"content":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":204,"responseJsonBytes":206,"startedAt":"2026-07-17T07:07:25.828Z","synthetic":false,"usage":{"inputTokens":82,"outputTokens":911,"totalTokens":993}}]}
{"case_id":"if-01","case_sha256":"fb32d183379053efee74c73dc7c5cc70d5c4d2364ba838f137aff5c9bcf564d7","category":"instruction-following","completed_at":"2026-07-17T07:08:51.090Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 13/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✓ no numbers; ✓ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✓ no numbers (4); ✓ no numbers (5); ✓ no numbers (6); ✓ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name","model":"local:qwen3:4b","plan_ordinal":11,"prompt":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","raw_score":10,"record_id":"record-0686f5df8facc6988b89e47d","response":"- Ship updates to the mobile app\n- Reduce crash rate significantly\n- Increase onboarding completion rate","run_index":3,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:07:25.828Z","status":"complete","system_prompt":"You follow instructions exactly and avoid extra commentary.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:08:51.089Z","latencyMs":85261,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-880","rawResponse":"- Ship updates to the mobile app\n- Reduce crash rate significantly\n- Increase onboarding completion rate","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You follow instructions exactly and avoid extra commentary.","role":"system"},{"content":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":104,"responseJsonBytes":108,"startedAt":"2026-07-17T07:07:25.828Z","synthetic":false,"usage":{"inputTokens":108,"outputTokens":1556,"totalTokens":1664}}]}
{"case_id":"if-02","case_sha256":"e9bb583b60c583f5dd69a7563ba439234342207240acf453b860874df57b613b","category":"instruction-following","completed_at":"2026-07-17T07:09:06.622Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/4 constraints. ✗ exact length (104/120); ✓ importance first; ✓ timeline mentioned; ✓ no markdown","model":"local:qwen3:4b","plan_ordinal":12,"prompt":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","raw_score":7.5,"record_id":"record-0be5bb4444557f2d9a2998da","response":"Stabilizing core payments is the critical priority before introducing referrals. Week 1 focuses on fixing failed card retries and invoice webhooks. Week 2 adds risk scoring and tests dunning copy. Week 3 launches the referral beta to 10 percent of users. This phased approach ensures payment reliability precedes referral functionality. Immediate fixes for payment failures prevent revenue loss during the beta. Risk scoring protects against fraud while dunning copy experiments optimize user retention. The referral beta targets a small segment to minimize impact and gather actionable feedback before full rollout. This sequence guarantees payment stability supports the referral feature launch without compromising financial integrity.","run_index":3,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:07:25.828Z","status":"complete","system_prompt":"You are concise and literal.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:09:06.622Z","latencyMs":100794,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-679","rawResponse":"Stabilizing core payments is the critical priority before introducing referrals. Week 1 focuses on fixing failed card retries and invoice webhooks. Week 2 adds risk scoring and tests dunning copy. Week 3 launches the referral beta to 10 percent of users. This phased approach ensures payment reliability precedes referral functionality. Immediate fixes for payment failures prevent revenue loss during the beta. Risk scoring protects against fraud while dunning copy experiments optimize user retention. The referral beta targets a small segment to minimize impact and gather actionable feedback before full rollout. This sequence guarantees payment stability supports the referral feature launch without compromising financial integrity.","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You are concise and literal.","role":"system"},{"content":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":738,"responseJsonBytes":740,"startedAt":"2026-07-17T07:07:25.828Z","synthetic":false,"usage":{"inputTokens":103,"outputTokens":449,"totalTokens":552}}]}
{"case_id":"if-04","case_sha256":"ee0e92615cbb9c51799f7dd8b4ca49d8cf8b3c7466d7ee9ed78979990838b780","category":"instruction-following","completed_at":"2026-07-17T07:09:10.024Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key","model":"local:qwen3:4b","plan_ordinal":14,"prompt":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","raw_score":10,"record_id":"record-6b4bb22b2cf4ae92e10e6957","response":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","run_index":3,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:07:58.481Z","status":"complete","system_prompt":"Return valid JSON only.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:09:10.024Z","latencyMs":71542,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-571","rawResponse":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Return valid JSON only.","role":"system"},{"content":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":87,"responseJsonBytes":105,"startedAt":"2026-07-17T07:07:58.482Z","synthetic":false,"usage":{"inputTokens":67,"outputTokens":88,"totalTokens":155}}]}
{"case_id":"if-05","case_sha256":"10ac97b006bc810f1848f42ffe9d407b39bd8f684ed974fa487aa34409963510","category":"instruction-following","completed_at":"2026-07-17T07:09:12.772Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/3 constraints. ✓ correct branch; ✓ single line; ✓ no extra text (2/2)","model":"local:qwen3:4b","plan_ordinal":15,"prompt":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","raw_score":10,"record_id":"record-c6e18954acfc1cc6bcc03c2a","response":"Route: support","run_index":3,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:08:51.095Z","status":"complete","system_prompt":"Follow branch logic exactly.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:09:12.772Z","latencyMs":21677,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-631","rawResponse":"Route: support","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow branch logic exactly.","role":"system"},{"content":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":14,"responseJsonBytes":16,"startedAt":"2026-07-17T07:08:51.095Z","synthetic":false,"usage":{"inputTokens":51,"outputTokens":71,"totalTokens":122}}]}
{"case_id":"if-03","case_sha256":"6983acf5aeabf690a436232c07b86a56d3f4041a596072f98128182d7ae1d28d","category":"instruction-following","completed_at":"2026-07-17T07:10:25.613Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3","model":"local:qwen3:4b","plan_ordinal":18,"prompt":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","raw_score":10,"record_id":"record-2a32953c756fb41244a87b71","response":"The product risks include high churn among small and medium businesses. The product also has weak onboarding analytics and a support queue backlog.","run_index":4,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:09:12.776Z","status":"complete","system_prompt":"Follow scope boundaries exactly.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:10:25.613Z","latencyMs":72837,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-716","rawResponse":"The product risks include high churn among small and medium businesses. The product also has weak onboarding analytics and a support queue backlog.","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow scope boundaries exactly.","role":"system"},{"content":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":147,"responseJsonBytes":149,"startedAt":"2026-07-17T07:09:12.776Z","synthetic":false,"usage":{"inputTokens":82,"outputTokens":1877,"totalTokens":1959}}]}
{"case_id":"if-02","case_sha256":"e9bb583b60c583f5dd69a7563ba439234342207240acf453b860874df57b613b","category":"instruction-following","completed_at":"2026-07-17T07:10:43.164Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ exact length (116/120); ✓ importance first; ✓ timeline mentioned; ✓ no markdown","model":"local:qwen3:4b","plan_ordinal":17,"prompt":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","raw_score":10,"record_id":"record-fecef25166aa8166e8b5d5e8","response":"Stabilizing core payments is critical before introducing referrals. Week 1: Fix failed card retries and invoice webhooks. Week 2: Implement risk scoring and test dunning copy. Week 3: Launch referral beta to 10 percent of users. This phased approach ensures payment reliability precedes referral functionality. Immediate focus on resolving payment failure mechanisms prevents revenue loss during the initial rollout phase. Risk scoring integration in week two provides necessary fraud protection for potential referral traffic. Dunning copy experimentation allows optimization of user communication before full referral activation. The referral beta targets a small user segment to minimize risk while gathering essential feedback for scaling. This sequence prioritizes payment stability as the non-negotiable foundation for all subsequent features.","run_index":4,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:09:12.776Z","status":"complete","system_prompt":"You are concise and literal.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:10:43.163Z","latencyMs":90387,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-189","rawResponse":"Stabilizing core payments is critical before introducing referrals. Week 1: Fix failed card retries and invoice webhooks. Week 2: Implement risk scoring and test dunning copy. Week 3: Launch referral beta to 10 percent of users. This phased approach ensures payment reliability precedes referral functionality. Immediate focus on resolving payment failure mechanisms prevents revenue loss during the initial rollout phase. Risk scoring integration in week two provides necessary fraud protection for potential referral traffic. Dunning copy experimentation allows optimization of user communication before full referral activation. The referral beta targets a small user segment to minimize risk while gathering essential feedback for scaling. This sequence prioritizes payment stability as the non-negotiable foundation for all subsequent features.","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You are concise and literal.","role":"system"},{"content":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":849,"responseJsonBytes":851,"startedAt":"2026-07-17T07:09:12.776Z","synthetic":false,"usage":{"inputTokens":103,"outputTokens":496,"totalTokens":599}}]}
{"case_id":"if-01","case_sha256":"fb32d183379053efee74c73dc7c5cc70d5c4d2364ba838f137aff5c9bcf564d7","category":"instruction-following","completed_at":"2026-07-17T07:11:53.426Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 13/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✓ no numbers; ✓ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✓ no numbers (4); ✓ no numbers (5); ✓ no numbers (6); ✓ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name","model":"local:qwen3:4b","plan_ordinal":16,"prompt":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","raw_score":10,"record_id":"record-16d749a3bab33bad11782172","response":"- Shipped updates to the mobile app\n- Reduced crash rates\n- Increased onboarding completion","run_index":4,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:09:12.776Z","status":"complete","system_prompt":"You follow instructions exactly and avoid extra commentary.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:11:53.426Z","latencyMs":160650,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-921","rawResponse":"- Shipped updates to the mobile app\n- Reduced crash rates\n- Increased onboarding completion","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You follow instructions exactly and avoid extra commentary.","role":"system"},{"content":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":91,"responseJsonBytes":95,"startedAt":"2026-07-17T07:09:12.776Z","synthetic":false,"usage":{"inputTokens":108,"outputTokens":1883,"totalTokens":1991}}]}
{"case_id":"if-04","case_sha256":"ee0e92615cbb9c51799f7dd8b4ca49d8cf8b3c7466d7ee9ed78979990838b780","category":"instruction-following","completed_at":"2026-07-17T07:11:56.466Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key","model":"local:qwen3:4b","plan_ordinal":19,"prompt":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","raw_score":10,"record_id":"record-2df2af7c21d8d48033a13579","response":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","run_index":4,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:10:25.618Z","status":"complete","system_prompt":"Return valid JSON only.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:11:56.466Z","latencyMs":90848,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-934","rawResponse":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Return valid JSON only.","role":"system"},{"content":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":87,"responseJsonBytes":105,"startedAt":"2026-07-17T07:10:25.618Z","synthetic":false,"usage":{"inputTokens":67,"outputTokens":88,"totalTokens":155}}]}
{"case_id":"if-05","case_sha256":"10ac97b006bc810f1848f42ffe9d407b39bd8f684ed974fa487aa34409963510","category":"instruction-following","completed_at":"2026-07-17T07:11:58.981Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/3 constraints. ✓ correct branch; ✓ single line; ✓ no extra text (2/2)","model":"local:qwen3:4b","plan_ordinal":20,"prompt":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","raw_score":10,"record_id":"record-5bfc9364275647cba0e21e7b","response":"Route: support","run_index":4,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:10:43.171Z","status":"complete","system_prompt":"Follow branch logic exactly.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:11:58.981Z","latencyMs":75810,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-135","rawResponse":"Route: support","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow branch logic exactly.","role":"system"},{"content":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":14,"responseJsonBytes":16,"startedAt":"2026-07-17T07:10:43.171Z","synthetic":false,"usage":{"inputTokens":51,"outputTokens":66,"totalTokens":117}}]}
{"case_id":"if-03","case_sha256":"6983acf5aeabf690a436232c07b86a56d3f4041a596072f98128182d7ae1d28d","category":"instruction-following","completed_at":"2026-07-17T07:13:35.606Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3","model":"local:qwen3:4b","plan_ordinal":23,"prompt":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","raw_score":10,"record_id":"record-d2af2ce1f5ec018ce486ff2c","response":"The product risks include churn in SMB and a support queue backlog. Onboarding analytics are weak.","run_index":5,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:11:58.987Z","status":"complete","system_prompt":"Follow scope boundaries exactly.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:13:35.606Z","latencyMs":96619,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-584","rawResponse":"The product risks include churn in SMB and a support queue backlog. Onboarding analytics are weak.","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow scope boundaries exactly.","role":"system"},{"content":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":98,"responseJsonBytes":100,"startedAt":"2026-07-17T07:11:58.987Z","synthetic":false,"usage":{"inputTokens":82,"outputTokens":2594,"totalTokens":2676}}]}
{"case_id":"if-02","case_sha256":"e9bb583b60c583f5dd69a7563ba439234342207240acf453b860874df57b613b","category":"instruction-following","completed_at":"2026-07-17T07:13:48.740Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/4 constraints. ✗ exact length (106/120); ✓ importance first; ✓ timeline mentioned; ✓ no markdown","model":"local:qwen3:4b","plan_ordinal":22,"prompt":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","raw_score":7.5,"record_id":"record-e19a6acd802495800753b32c","response":"Stabilizing core payments is critical before introducing referrals. The launch plan prioritizes fixing payment system failures. Week 1 focuses on resolving failed card retries and invoice webhook issues. Week 2 implements risk scoring functionality and experiments with dunning copy variations. Week 3 executes a referral program beta for 10 percent of users. This sequence ensures payment reliability precedes referral features. Core payment stability must be achieved before any referral functionality can safely launch. The three-week timeline provides clear milestones for payment fixes, risk scoring integration, and the referral beta rollout. This phased approach minimizes risk while building necessary infrastructure. Payment stability is non-negotiable for referral success.","run_index":5,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:11:58.987Z","status":"complete","system_prompt":"You are concise and literal.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:13:48.740Z","latencyMs":109753,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-745","rawResponse":"Stabilizing core payments is critical before introducing referrals. The launch plan prioritizes fixing payment system failures. Week 1 focuses on resolving failed card retries and invoice webhook issues. Week 2 implements risk scoring functionality and experiments with dunning copy variations. Week 3 executes a referral program beta for 10 percent of users. This sequence ensures payment reliability precedes referral features. Core payment stability must be achieved before any referral functionality can safely launch. The three-week timeline provides clear milestones for payment fixes, risk scoring integration, and the referral beta rollout. This phased approach minimizes risk while building necessary infrastructure. Payment stability is non-negotiable for referral success.","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You are concise and literal.","role":"system"},{"content":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":783,"responseJsonBytes":785,"startedAt":"2026-07-17T07:11:58.987Z","synthetic":false,"usage":{"inputTokens":103,"outputTokens":414,"totalTokens":517}}]}
{"case_id":"if-01","case_sha256":"fb32d183379053efee74c73dc7c5cc70d5c4d2364ba838f137aff5c9bcf564d7","category":"instruction-following","completed_at":"2026-07-17T07:15:03.636Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 13/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✓ no numbers; ✓ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✓ no numbers (4); ✓ no numbers (5); ✓ no numbers (6); ✓ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name","model":"local:qwen3:4b","plan_ordinal":21,"prompt":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","raw_score":10,"record_id":"record-6aeb9771d31ca1aa892c7799","response":"- Shipped mobile app updates\n- Reduced crash rate\n- Increased onboarding completion","run_index":5,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:11:58.986Z","status":"complete","system_prompt":"You follow instructions exactly and avoid extra commentary.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:15:03.636Z","latencyMs":184649,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-769","rawResponse":"- Shipped mobile app updates\n- Reduced crash rate\n- Increased onboarding completion","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You follow instructions exactly and avoid extra commentary.","role":"system"},{"content":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":83,"responseJsonBytes":87,"startedAt":"2026-07-17T07:11:58.987Z","synthetic":false,"usage":{"inputTokens":108,"outputTokens":2031,"totalTokens":2139}}]}
{"case_id":"if-04","case_sha256":"ee0e92615cbb9c51799f7dd8b4ca49d8cf8b3c7466d7ee9ed78979990838b780","category":"instruction-following","completed_at":"2026-07-17T07:15:06.902Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key","model":"local:qwen3:4b","plan_ordinal":24,"prompt":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","raw_score":10,"record_id":"record-8d567f4d758fd1c86f65ff48","response":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","run_index":5,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:13:35.614Z","status":"complete","system_prompt":"Return valid JSON only.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:15:06.902Z","latencyMs":91288,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-760","rawResponse":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Return valid JSON only.","role":"system"},{"content":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":87,"responseJsonBytes":105,"startedAt":"2026-07-17T07:13:35.614Z","synthetic":false,"usage":{"inputTokens":67,"outputTokens":92,"totalTokens":159}}]}
{"case_id":"if-05","case_sha256":"10ac97b006bc810f1848f42ffe9d407b39bd8f684ed974fa487aa34409963510","category":"instruction-following","completed_at":"2026-07-17T07:15:09.205Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/3 constraints. ✓ correct branch; ✓ single line; ✓ no extra text (2/2)","model":"local:qwen3:4b","plan_ordinal":25,"prompt":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","raw_score":10,"record_id":"record-f53bca0dce0342a553ca7659","response":"Route: support","run_index":5,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T07:13:48.746Z","status":"complete","system_prompt":"Follow branch logic exactly.","target_model":{"alias":"local:qwen3:4b","provider":"local-openai","requestedModelId":"qwen3:4b"},"target_requests":[{"completedAt":"2026-07-17T07:15:09.205Z","latencyMs":80459,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-502","rawResponse":"Route: support","reportedModelId":"qwen3:4b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow branch logic exactly.","role":"system"},{"content":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","role":"user"}],"model":"qwen3:4b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:4b","responseBytes":14,"responseJsonBytes":16,"startedAt":"2026-07-17T07:13:48.746Z","synthetic":false,"usage":{"inputTokens":51,"outputTokens":66,"totalTokens":117}}]}