{"case_id":"if-03","case_sha256":"6983acf5aeabf690a436232c07b86a56d3f4041a596072f98128182d7ae1d28d","category":"instruction-following","completed_at":"2026-07-17T08:09:02.839Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3","model":"local:qwen3:1.7b","plan_ordinal":3,"prompt":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","raw_score":10,"record_id":"record-60d700e612f01b50f2d76a38","response":"Churn in small and medium businesses is a critical risk, threatening customer retention and revenue growth. Weak onboarding analytics and a backlog in the support queue further complicate customer experience and operational efficiency.","run_index":1,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:08:58.226Z","status":"complete","system_prompt":"Follow scope boundaries exactly.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:02.838Z","latencyMs":4612,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-266","rawResponse":"Churn in small and medium businesses is a critical risk, threatening customer retention and revenue growth. Weak onboarding analytics and a backlog in the support queue further complicate customer experience and operational efficiency.","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow scope boundaries exactly.","role":"system"},{"content":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":235,"responseJsonBytes":237,"startedAt":"2026-07-17T08:08:58.226Z","synthetic":false,"usage":{"inputTokens":82,"outputTokens":206,"totalTokens":288}}]}
{"case_id":"if-02","case_sha256":"e9bb583b60c583f5dd69a7563ba439234342207240acf453b860874df57b613b","category":"instruction-following","completed_at":"2026-07-17T08:09:08.708Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/4 constraints. ✗ exact length (51/120); ✓ importance first; ✓ timeline mentioned; ✓ no markdown","model":"local:qwen3:1.7b","plan_ordinal":2,"prompt":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","raw_score":7.5,"record_id":"record-240b3ba30e6a7642915e44f7","response":"Stabilizing core payments is the priority before adding referrals. Week 1 focuses on fixing failed card retries and invoice webhooks. Week 2 introduces risk scoring and tests dunning copy. Week 3 launches a referral beta to 10% of users. The plan ensures reliability before expansion, with phased implementation to minimize risk.","run_index":1,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:08:58.226Z","status":"complete","system_prompt":"You are concise and literal.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:08.708Z","latencyMs":10482,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-950","rawResponse":"Stabilizing core payments is the priority before adding referrals. Week 1 focuses on fixing failed card retries and invoice webhooks. Week 2 introduces risk scoring and tests dunning copy. Week 3 launches a referral beta to 10% of users. The plan ensures reliability before expansion, with phased implementation to minimize risk.","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You are concise and literal.","role":"system"},{"content":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":329,"responseJsonBytes":331,"startedAt":"2026-07-17T08:08:58.226Z","synthetic":false,"usage":{"inputTokens":103,"outputTokens":366,"totalTokens":469}}]}
{"case_id":"if-01","case_sha256":"fb32d183379053efee74c73dc7c5cc70d5c4d2364ba838f137aff5c9bcf564d7","category":"instruction-following","completed_at":"2026-07-17T08:09:13.138Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 13/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✓ no numbers; ✓ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✓ no numbers (4); ✓ no numbers (5); ✓ no numbers (6); ✓ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name","model":"local:qwen3:1.7b","plan_ordinal":1,"prompt":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","raw_score":10,"record_id":"record-e2db3894ed86a784f27bce47","response":"- Shipped  \n- Reduced  \n- Moved","run_index":1,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:08:58.226Z","status":"complete","system_prompt":"You follow instructions exactly and avoid extra commentary.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:13.138Z","latencyMs":14912,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-254","rawResponse":"- Shipped  \n- Reduced  \n- Moved","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You follow instructions exactly and avoid extra commentary.","role":"system"},{"content":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":31,"responseJsonBytes":35,"startedAt":"2026-07-17T08:08:58.226Z","synthetic":false,"usage":{"inputTokens":108,"outputTokens":231,"totalTokens":339}}]}
{"case_id":"if-04","case_sha256":"ee0e92615cbb9c51799f7dd8b4ca49d8cf8b3c7466d7ee9ed78979990838b780","category":"instruction-following","completed_at":"2026-07-17T08:09:18.092Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key","model":"local:qwen3:1.7b","plan_ordinal":4,"prompt":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","raw_score":10,"record_id":"record-0720e39e7c0b83dc20ceb408","response":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","run_index":1,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:02.845Z","status":"complete","system_prompt":"Return valid JSON only.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:18.092Z","latencyMs":15247,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-407","rawResponse":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Return valid JSON only.","role":"system"},{"content":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":87,"responseJsonBytes":105,"startedAt":"2026-07-17T08:09:02.845Z","synthetic":false,"usage":{"inputTokens":67,"outputTokens":341,"totalTokens":408}}]}
{"case_id":"if-05","case_sha256":"10ac97b006bc810f1848f42ffe9d407b39bd8f684ed974fa487aa34409963510","category":"instruction-following","completed_at":"2026-07-17T08:09:20.385Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/3 constraints. ✓ correct branch; ✓ single line; ✓ no extra text (2/2)","model":"local:qwen3:1.7b","plan_ordinal":5,"prompt":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","raw_score":10,"record_id":"record-dd02ba46d378031a39100097","response":"Route: support","run_index":1,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:08.714Z","status":"complete","system_prompt":"Follow branch logic exactly.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:20.384Z","latencyMs":11669,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-832","rawResponse":"Route: support","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow branch logic exactly.","role":"system"},{"content":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":14,"responseJsonBytes":16,"startedAt":"2026-07-17T08:09:08.715Z","synthetic":false,"usage":{"inputTokens":51,"outputTokens":153,"totalTokens":204}}]}
{"case_id":"if-02","case_sha256":"e9bb583b60c583f5dd69a7563ba439234342207240acf453b860874df57b613b","category":"instruction-following","completed_at":"2026-07-17T08:09:25.586Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/4 constraints. ✗ exact length (53/120); ✓ importance first; ✓ timeline mentioned; ✓ no markdown","model":"local:qwen3:1.7b","plan_ordinal":7,"prompt":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","raw_score":7.5,"record_id":"record-e5a6d14a091afebd08754bcc","response":"Stabilizing core payments is the priority before introducing referrals. Week 1 focuses on fixing failed card retries and invoice webhooks. Week 2 adds risk scoring and experiments with dunning copy. Week 3 launches referral beta to 10% of users. The plan ensures reliability before expansion, with phased implementation to test and refine processes.","run_index":2,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:20.389Z","status":"complete","system_prompt":"You are concise and literal.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:25.586Z","latencyMs":5197,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-24","rawResponse":"Stabilizing core payments is the priority before introducing referrals. Week 1 focuses on fixing failed card retries and invoice webhooks. Week 2 adds risk scoring and experiments with dunning copy. Week 3 launches referral beta to 10% of users. The plan ensures reliability before expansion, with phased implementation to test and refine processes.","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You are concise and literal.","role":"system"},{"content":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":349,"responseJsonBytes":351,"startedAt":"2026-07-17T08:09:20.389Z","synthetic":false,"usage":{"inputTokens":103,"outputTokens":351,"totalTokens":454}}]}
{"case_id":"if-01","case_sha256":"fb32d183379053efee74c73dc7c5cc70d5c4d2364ba838f137aff5c9bcf564d7","category":"instruction-following","completed_at":"2026-07-17T08:09:28.918Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✗ no numbers; ✗ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✗ no numbers (4); ✗ no numbers (5); ✓ no numbers (6); ✗ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name","model":"local:qwen3:1.7b","plan_ordinal":6,"prompt":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","raw_score":6.15,"record_id":"record-a98c9088b222d8fdc2c9fb9f","response":"- Shipped updates to improve app functionality  \n- Reduced crash rate by 40 percent through redesign  \n- Boosted onboarding completion from 55 to 71 percent","run_index":2,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:20.389Z","status":"complete","system_prompt":"You follow instructions exactly and avoid extra commentary.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:28.918Z","latencyMs":8529,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-850","rawResponse":"- Shipped updates to improve app functionality  \n- Reduced crash rate by 40 percent through redesign  \n- Boosted onboarding completion from 55 to 71 percent","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You follow instructions exactly and avoid extra commentary.","role":"system"},{"content":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":156,"responseJsonBytes":160,"startedAt":"2026-07-17T08:09:20.389Z","synthetic":false,"usage":{"inputTokens":108,"outputTokens":222,"totalTokens":330}}]}
{"case_id":"if-03","case_sha256":"6983acf5aeabf690a436232c07b86a56d3f4041a596072f98128182d7ae1d28d","category":"instruction-following","completed_at":"2026-07-17T08:09:32.176Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3","model":"local:qwen3:1.7b","plan_ordinal":8,"prompt":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","raw_score":10,"record_id":"record-976582ba8e13574540901bd8","response":"Churn in small and medium businesses is a critical risk, threatening customer retention and revenue growth. Weak onboarding analytics and a backlog in the support queue further complicate user experience and operational efficiency.","run_index":2,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:20.389Z","status":"complete","system_prompt":"Follow scope boundaries exactly.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:32.176Z","latencyMs":11787,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-377","rawResponse":"Churn in small and medium businesses is a critical risk, threatening customer retention and revenue growth. Weak onboarding analytics and a backlog in the support queue further complicate user experience and operational efficiency.","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow scope boundaries exactly.","role":"system"},{"content":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":231,"responseJsonBytes":233,"startedAt":"2026-07-17T08:09:20.389Z","synthetic":false,"usage":{"inputTokens":82,"outputTokens":219,"totalTokens":301}}]}
{"case_id":"if-04","case_sha256":"ee0e92615cbb9c51799f7dd8b4ca49d8cf8b3c7466d7ee9ed78979990838b780","category":"instruction-following","completed_at":"2026-07-17T08:09:36.468Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key","model":"local:qwen3:1.7b","plan_ordinal":9,"prompt":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","raw_score":10,"record_id":"record-b834dfecae887d37154eba25","response":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","run_index":2,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:25.591Z","status":"complete","system_prompt":"Return valid JSON only.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:36.468Z","latencyMs":10876,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-463","rawResponse":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Return valid JSON only.","role":"system"},{"content":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":87,"responseJsonBytes":105,"startedAt":"2026-07-17T08:09:25.592Z","synthetic":false,"usage":{"inputTokens":67,"outputTokens":280,"totalTokens":347}}]}
{"case_id":"if-05","case_sha256":"10ac97b006bc810f1848f42ffe9d407b39bd8f684ed974fa487aa34409963510","category":"instruction-following","completed_at":"2026-07-17T08:09:39.364Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/3 constraints. ✓ correct branch; ✓ single line; ✓ no extra text (2/2)","model":"local:qwen3:1.7b","plan_ordinal":10,"prompt":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","raw_score":10,"record_id":"record-2a1ec132d6b7d3e32b397328","response":"Route: support","run_index":2,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:28.927Z","status":"complete","system_prompt":"Follow branch logic exactly.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:39.364Z","latencyMs":10437,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-629","rawResponse":"Route: support","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow branch logic exactly.","role":"system"},{"content":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":14,"responseJsonBytes":16,"startedAt":"2026-07-17T08:09:28.927Z","synthetic":false,"usage":{"inputTokens":51,"outputTokens":153,"totalTokens":204}}]}
{"case_id":"if-02","case_sha256":"e9bb583b60c583f5dd69a7563ba439234342207240acf453b860874df57b613b","category":"instruction-following","completed_at":"2026-07-17T08:09:44.616Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/4 constraints. ✗ exact length (52/120); ✓ importance first; ✓ timeline mentioned; ✓ no markdown","model":"local:qwen3:1.7b","plan_ordinal":12,"prompt":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","raw_score":7.5,"record_id":"record-1476d4139c241bdd8e371a05","response":"Stabilizing core payments is the priority before adding referrals. Week 1 focuses on fixing failed card retries and invoice webhooks. Week 2 introduces risk scoring and tests dunning copy. Week 3 launches a beta referral program for 10% of users. The timeline ensures stability before expanding features, balancing risk and user growth.","run_index":3,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:39.391Z","status":"complete","system_prompt":"You are concise and literal.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:44.616Z","latencyMs":5225,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-413","rawResponse":"Stabilizing core payments is the priority before adding referrals. Week 1 focuses on fixing failed card retries and invoice webhooks. Week 2 introduces risk scoring and tests dunning copy. Week 3 launches a beta referral program for 10% of users. The timeline ensures stability before expanding features, balancing risk and user growth.","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You are concise and literal.","role":"system"},{"content":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":336,"responseJsonBytes":338,"startedAt":"2026-07-17T08:09:39.391Z","synthetic":false,"usage":{"inputTokens":103,"outputTokens":354,"totalTokens":457}}]}
{"case_id":"if-01","case_sha256":"fb32d183379053efee74c73dc7c5cc70d5c4d2364ba838f137aff5c9bcf564d7","category":"instruction-following","completed_at":"2026-07-17T08:09:50.166Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 13/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✓ no numbers; ✓ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✓ no numbers (4); ✓ no numbers (5); ✓ no numbers (6); ✓ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name","model":"local:qwen3:1.7b","plan_ordinal":11,"prompt":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","raw_score":10,"record_id":"record-4686ac8d574a427e5b579ce1","response":"- Ship updates  \n- Reduce crash rate  \n- Improve onboarding completion","run_index":3,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:39.391Z","status":"complete","system_prompt":"You follow instructions exactly and avoid extra commentary.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:50.166Z","latencyMs":10775,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-902","rawResponse":"- Ship updates  \n- Reduce crash rate  \n- Improve onboarding completion","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You follow instructions exactly and avoid extra commentary.","role":"system"},{"content":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":70,"responseJsonBytes":74,"startedAt":"2026-07-17T08:09:39.391Z","synthetic":false,"usage":{"inputTokens":108,"outputTokens":374,"totalTokens":482}}]}
{"case_id":"if-03","case_sha256":"6983acf5aeabf690a436232c07b86a56d3f4041a596072f98128182d7ae1d28d","category":"instruction-following","completed_at":"2026-07-17T08:09:54.717Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3","model":"local:qwen3:1.7b","plan_ordinal":13,"prompt":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","raw_score":10,"record_id":"record-f2b0f56f19fa3260db917106","response":"Churn in small and medium businesses is a significant risk, impacting customer retention and growth. Weak onboarding analytics and a backlog in the support queue further complicate customer experience and operational efficiency.","run_index":3,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:39.391Z","status":"complete","system_prompt":"Follow scope boundaries exactly.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:09:54.716Z","latencyMs":15325,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-916","rawResponse":"Churn in small and medium businesses is a significant risk, impacting customer retention and growth. Weak onboarding analytics and a backlog in the support queue further complicate customer experience and operational efficiency.","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow scope boundaries exactly.","role":"system"},{"content":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":228,"responseJsonBytes":230,"startedAt":"2026-07-17T08:09:39.391Z","synthetic":false,"usage":{"inputTokens":82,"outputTokens":231,"totalTokens":313}}]}
{"case_id":"if-04","case_sha256":"ee0e92615cbb9c51799f7dd8b4ca49d8cf8b3c7466d7ee9ed78979990838b780","category":"instruction-following","completed_at":"2026-07-17T08:10:01.330Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key","model":"local:qwen3:1.7b","plan_ordinal":14,"prompt":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","raw_score":10,"record_id":"record-d260c337353e557b66bb0965","response":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","run_index":3,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:44.620Z","status":"complete","system_prompt":"Return valid JSON only.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:01.329Z","latencyMs":16710,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-189","rawResponse":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Return valid JSON only.","role":"system"},{"content":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":87,"responseJsonBytes":105,"startedAt":"2026-07-17T08:09:44.620Z","synthetic":false,"usage":{"inputTokens":67,"outputTokens":362,"totalTokens":429}}]}
{"case_id":"if-05","case_sha256":"10ac97b006bc810f1848f42ffe9d407b39bd8f684ed974fa487aa34409963510","category":"instruction-following","completed_at":"2026-07-17T08:10:04.076Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/3 constraints. ✓ correct branch; ✓ single line; ✓ no extra text (2/2)","model":"local:qwen3:1.7b","plan_ordinal":15,"prompt":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","raw_score":10,"record_id":"record-6643b2fc9c91683027798cac","response":"Route: support","run_index":3,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:09:50.173Z","status":"complete","system_prompt":"Follow branch logic exactly.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:04.076Z","latencyMs":13903,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-178","rawResponse":"Route: support","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow branch logic exactly.","role":"system"},{"content":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":14,"responseJsonBytes":16,"startedAt":"2026-07-17T08:09:50.173Z","synthetic":false,"usage":{"inputTokens":51,"outputTokens":186,"totalTokens":237}}]}
{"case_id":"if-02","case_sha256":"e9bb583b60c583f5dd69a7563ba439234342207240acf453b860874df57b613b","category":"instruction-following","completed_at":"2026-07-17T08:10:10.625Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 2/4 constraints. ✗ exact length (87/120); ✗ importance first; ✓ timeline mentioned; ✓ no markdown","model":"local:qwen3:1.7b","plan_ordinal":17,"prompt":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","raw_score":5,"record_id":"record-880f8f777ba60bcb1809646e","response":"The plan prioritizes stabilizing core payments before introducing referrals, with Week 1 focused on fixing failed card retries and invoice webhooks, Week 2 aiming to implement risk scoring and test dunning copy, and Week 3 launching a beta referral program for 10% of users. The timeline ensures stability is addressed before expansion, with Week 1 fixes addressing technical issues, Week 2 enhancements improving risk assessment, and Week 3 pilot testing refining referral integration. This phased approach balances immediate stability with gradual expansion, ensuring reliability before broader adoption.","run_index":4,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:10:04.084Z","status":"complete","system_prompt":"You are concise and literal.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:10.624Z","latencyMs":6540,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-192","rawResponse":"The plan prioritizes stabilizing core payments before introducing referrals, with Week 1 focused on fixing failed card retries and invoice webhooks, Week 2 aiming to implement risk scoring and test dunning copy, and Week 3 launching a beta referral program for 10% of users. The timeline ensures stability is addressed before expansion, with Week 1 fixes addressing technical issues, Week 2 enhancements improving risk assessment, and Week 3 pilot testing refining referral integration. This phased approach balances immediate stability with gradual expansion, ensuring reliability before broader adoption.","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You are concise and literal.","role":"system"},{"content":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":606,"responseJsonBytes":608,"startedAt":"2026-07-17T08:10:04.084Z","synthetic":false,"usage":{"inputTokens":103,"outputTokens":267,"totalTokens":370}}]}
{"case_id":"if-03","case_sha256":"6983acf5aeabf690a436232c07b86a56d3f4041a596072f98128182d7ae1d28d","category":"instruction-following","completed_at":"2026-07-17T08:10:14.695Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3","model":"local:qwen3:1.7b","plan_ordinal":18,"prompt":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","raw_score":10,"record_id":"record-11970f98d5850f3eb60c1dbe","response":"Section 2 highlights three product risks: churn in small and medium businesses, due to inadequate retention strategies, and weak onboarding analytics, which may hinder user engagement. Additionally, a backlog in the support queue could lead to delayed resolution of customer issues, impacting overall satisfaction.","run_index":4,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:10:04.084Z","status":"complete","system_prompt":"Follow scope boundaries exactly.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:14.695Z","latencyMs":10611,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-547","rawResponse":"Section 2 highlights three product risks: churn in small and medium businesses, due to inadequate retention strategies, and weak onboarding analytics, which may hinder user engagement. Additionally, a backlog in the support queue could lead to delayed resolution of customer issues, impacting overall satisfaction.","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow scope boundaries exactly.","role":"system"},{"content":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":314,"responseJsonBytes":316,"startedAt":"2026-07-17T08:10:04.084Z","synthetic":false,"usage":{"inputTokens":82,"outputTokens":269,"totalTokens":351}}]}
{"case_id":"if-01","case_sha256":"fb32d183379053efee74c73dc7c5cc70d5c4d2364ba838f137aff5c9bcf564d7","category":"instruction-following","completed_at":"2026-07-17T08:10:33.997Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✗ no numbers; ✗ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✗ no numbers (4); ✗ no numbers (5); ✓ no numbers (6); ✗ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name","model":"local:qwen3:1.7b","plan_ordinal":16,"prompt":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","raw_score":6.15,"record_id":"record-b0332935ac326f77ec4bc830","response":"- Shipped updates  \n- Reduced crash rate by 40%  \n- Moved onboarding completion from 55 to 71 percent","run_index":4,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:10:04.084Z","status":"complete","system_prompt":"You follow instructions exactly and avoid extra commentary.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:33.997Z","latencyMs":29913,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-970","rawResponse":"- Shipped updates  \n- Reduced crash rate by 40%  \n- Moved onboarding completion from 55 to 71 percent","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You follow instructions exactly and avoid extra commentary.","role":"system"},{"content":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":101,"responseJsonBytes":105,"startedAt":"2026-07-17T08:10:04.084Z","synthetic":false,"usage":{"inputTokens":108,"outputTokens":1304,"totalTokens":1412}}]}
{"case_id":"if-04","case_sha256":"ee0e92615cbb9c51799f7dd8b4ca49d8cf8b3c7466d7ee9ed78979990838b780","category":"instruction-following","completed_at":"2026-07-17T08:10:38.122Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key","model":"local:qwen3:1.7b","plan_ordinal":19,"prompt":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","raw_score":10,"record_id":"record-175cd35d88b57b7a6bd46cc0","response":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","run_index":4,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:10:10.631Z","status":"complete","system_prompt":"Return valid JSON only.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:38.122Z","latencyMs":27490,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-525","rawResponse":"{\n  \"title\": \"Fix onboarding analytics\",\n  \"priority\": \"high\",\n  \"owner\": \"data-team\"\n}","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Return valid JSON only.","role":"system"},{"content":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":87,"responseJsonBytes":105,"startedAt":"2026-07-17T08:10:10.632Z","synthetic":false,"usage":{"inputTokens":67,"outputTokens":285,"totalTokens":352}}]}
{"case_id":"if-05","case_sha256":"10ac97b006bc810f1848f42ffe9d407b39bd8f684ed974fa487aa34409963510","category":"instruction-following","completed_at":"2026-07-17T08:10:42.027Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 1/3 constraints. ✓ correct branch; ✗ single line; ✗ no extra text (20/2)","model":"local:qwen3:1.7b","plan_ordinal":20,"prompt":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","raw_score":3.33,"record_id":"record-32d1ee42a19dd62b83fc0ed3","response":"The input plan is `free`, which is not equal to `enterprise`. Therefore, the else clause is executed, outputting:\n\n```\nRoute: support\n```","run_index":4,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:10:14.701Z","status":"complete","system_prompt":"Follow branch logic exactly.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:42.027Z","latencyMs":27326,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-685","rawResponse":"The input plan is `free`, which is not equal to `enterprise`. Therefore, the else clause is executed, outputting:\n\n```\nRoute: support\n```","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow branch logic exactly.","role":"system"},{"content":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":137,"responseJsonBytes":143,"startedAt":"2026-07-17T08:10:14.701Z","synthetic":false,"usage":{"inputTokens":51,"outputTokens":267,"totalTokens":318}}]}
{"case_id":"if-03","case_sha256":"6983acf5aeabf690a436232c07b86a56d3f4041a596072f98128182d7ae1d28d","category":"instruction-following","completed_at":"2026-07-17T08:10:45.687Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 4/4 constraints. ✓ two sentences (2/2); ✓ section2 content; ✓ exclude section1; ✓ exclude section3","model":"local:qwen3:1.7b","plan_ordinal":23,"prompt":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","raw_score":10,"record_id":"record-6c4d46d61c381acdf19e8f66","response":"Section 2 highlights that churn in small and medium businesses (SMBs) poses a significant risk, while weak onboarding analytics and a backlog in the support queue further complicate user retention and satisfaction. Addressing these challenges requires targeted improvements to onboarding processes and support efficiency.","run_index":5,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:10:42.033Z","status":"complete","system_prompt":"Follow scope boundaries exactly.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:45.687Z","latencyMs":3654,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-856","rawResponse":"Section 2 highlights that churn in small and medium businesses (SMBs) poses a significant risk, while weak onboarding analytics and a backlog in the support queue further complicate user retention and satisfaction. Addressing these challenges requires targeted improvements to onboarding processes and support efficiency.","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow scope boundaries exactly.","role":"system"},{"content":"Document with 3 sections: Section 1: market context. Section 2: product risks include churn in SMB, weak onboarding analytics, and support queue backlog. Section 3: hiring plan. Summarize ONLY section 2 in two sentences. Do not mention sections 1 or 3.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":321,"responseJsonBytes":323,"startedAt":"2026-07-17T08:10:42.033Z","synthetic":false,"usage":{"inputTokens":82,"outputTokens":238,"totalTokens":320}}]}
{"case_id":"if-02","case_sha256":"e9bb583b60c583f5dd69a7563ba439234342207240acf453b860874df57b613b","category":"instruction-following","completed_at":"2026-07-17T08:10:49.435Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/4 constraints. ✗ exact length (63/120); ✓ importance first; ✓ timeline mentioned; ✓ no markdown","model":"local:qwen3:1.7b","plan_ordinal":22,"prompt":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","raw_score":7.5,"record_id":"record-e9182322d3751f12a4ea0b18","response":"Stabilizing core payments is the priority before adding referrals, with Week 1 focusing on fixing card retries and invoice webhooks, and Week 2 introducing risk scoring and testing dunning copy. Week 3 will launch a referral beta to 10% of users, ensuring gradual rollout and monitoring performance metrics. The plan balances technical fixes with strategic testing to build trust and optimize user experience.","run_index":5,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:10:42.033Z","status":"complete","system_prompt":"You are concise and literal.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:49.435Z","latencyMs":7402,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-821","rawResponse":"Stabilizing core payments is the priority before adding referrals, with Week 1 focusing on fixing card retries and invoice webhooks, and Week 2 introducing risk scoring and testing dunning copy. Week 3 will launch a referral beta to 10% of users, ensuring gradual rollout and monitoring performance metrics. The plan balances technical fixes with strategic testing to build trust and optimize user experience.","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You are concise and literal.","role":"system"},{"content":"Write exactly 120 words summarizing this launch plan. Put the most important point in the first sentence. Mention timeline second. Do not use markdown. Plan: We must stabilize core payments before adding referrals. Week 1 fix failed card retries and invoice webhooks. Week 2 add risk scoring and experiment with dunning copy. Week 3 launch referral beta to 10 percent of users.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":409,"responseJsonBytes":411,"startedAt":"2026-07-17T08:10:42.033Z","synthetic":false,"usage":{"inputTokens":103,"outputTokens":245,"totalTokens":348}}]}
{"case_id":"if-01","case_sha256":"fb32d183379053efee74c73dc7c5cc70d5c4d2364ba838f137aff5c9bcf564d7","category":"instruction-following","completed_at":"2026-07-17T08:10:52.282Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 13/13 constraints. ✓ three bullets (3/3); ✓ each bullet starts with a verb-like word; ✓ no numbers; ✓ no numbers (1); ✓ no numbers (2); ✓ no numbers (3); ✓ no numbers (4); ✓ no numbers (5); ✓ no numbers (6); ✓ no numbers (7); ✓ no numbers (8); ✓ no numbers (9); ✓ no author name","model":"local:qwen3:1.7b","plan_ordinal":21,"prompt":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","raw_score":10,"record_id":"record-ab4186872448e5789ddf5153","response":"- Shipped  \n- Reduced  \n- Moved","run_index":5,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:10:42.033Z","status":"complete","system_prompt":"You follow instructions exactly and avoid extra commentary.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:52.282Z","latencyMs":10249,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-324","rawResponse":"- Shipped  \n- Reduced  \n- Moved","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"You follow instructions exactly and avoid extra commentary.","role":"system"},{"content":"Read this paragraph and summarize it in exactly 3 bullet points. Each bullet must start with a verb. Do not include any numbers. Do not mention the author name, Jamie Rivera. Paragraph: Jamie Rivera shipped 2 updates to the mobile app, reduced crash rate by 40 percent, and moved onboarding completion from 55 to 71 percent after simplifying the first-run flow and rewriting copy.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":31,"responseJsonBytes":35,"startedAt":"2026-07-17T08:10:42.033Z","synthetic":false,"usage":{"inputTokens":108,"outputTokens":187,"totalTokens":295}}]}
{"case_id":"if-04","case_sha256":"ee0e92615cbb9c51799f7dd8b4ca49d8cf8b3c7466d7ee9ed78979990838b780","category":"instruction-following","completed_at":"2026-07-17T08:10:55.387Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 8/8 constraints. ✓ valid json; ✓ has title; ✓ has priority; ✓ has owner; ✓ correct title value; ✓ correct priority value; ✓ correct owner value; ✓ no extra key","model":"local:qwen3:1.7b","plan_ordinal":24,"prompt":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","raw_score":10,"record_id":"record-22c0c3926b77460d0162900c","response":"{\"title\": \"Fix onboarding analytics\", \"priority\": \"high\", \"owner\": \"data-team\"}","run_index":5,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:10:45.693Z","status":"complete","system_prompt":"Return valid JSON only.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:55.387Z","latencyMs":9694,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-693","rawResponse":"{\"title\": \"Fix onboarding analytics\", \"priority\": \"high\", \"owner\": \"data-team\"}","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Return valid JSON only.","role":"system"},{"content":"Return a JSON object with keys exactly: title (string), priority (high|medium|low), owner (string). Set title to \"Fix onboarding analytics\", priority to high, owner to \"data-team\". No extra keys.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":79,"responseJsonBytes":93,"startedAt":"2026-07-17T08:10:45.693Z","synthetic":false,"usage":{"inputTokens":67,"outputTokens":215,"totalTokens":282}}]}
{"case_id":"if-05","case_sha256":"10ac97b006bc810f1848f42ffe9d407b39bd8f684ed974fa487aa34409963510","category":"instruction-following","completed_at":"2026-07-17T08:10:58.280Z","error":null,"evaluation_status":"scored","interaction":null,"judge_model":"deterministic-rubric","judge_trace":null,"justification":"Met 3/3 constraints. ✓ correct branch; ✓ single line; ✓ no extra text (2/2)","model":"local:qwen3:1.7b","plan_ordinal":25,"prompt":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","raw_score":10,"record_id":"record-27aa0b5000b4ac82d73eb4c2","response":"Route: support","run_index":5,"schemaId":"clawbotomy.case-record/v1","started_at":"2026-07-17T08:10:49.440Z","status":"complete","system_prompt":"Follow branch logic exactly.","target_model":{"alias":"local:qwen3:1.7b","provider":"local-openai","requestedModelId":"qwen3:1.7b"},"target_requests":[{"completedAt":"2026-07-17T08:10:58.279Z","latencyMs":8839,"modelIdentityStatus":"exact-match","outcome":"received","provider":"local-openai","providerRequestId":"chatcmpl-51","rawResponse":"Route: support","reportedModelId":"qwen3:1.7b","request":{"body":{"max_completion_tokens":1600,"messages":[{"content":"Follow branch logic exactly.","role":"system"},{"content":"If the customer plan is enterprise, output \"Route: account-exec\". Else output \"Route: support\". Input plan = free. Output only one line.","role":"user"}],"model":"qwen3:1.7b","temperature":0.2},"url":"http://127.0.0.1:11434/v1/chat/completions"},"requestIndex":1,"requestedModelId":"qwen3:1.7b","responseBytes":14,"responseJsonBytes":16,"startedAt":"2026-07-17T08:10:49.440Z","synthetic":false,"usage":{"inputTokens":51,"outputTokens":200,"totalTokens":251}}]}