{ "generated_at": "2026-08-07T04:27:39.026558Z", "harness": { "model": "qwen/qwen3.6-27b", "model_info": { "id": "qwen/qwen3.6-27b", "local": true, "host": "192.168.1.197", "quantization": "Q4_K_M", "arch": "qwen35", "context_length": 262144, "loaded_context_length": 131072, "served_by": "lm-studio", "note": "Model pinned locally for reproducibility: the same weights, quantization and context length can be loaded by anyone re-running this benchmark.", "model_type": "vlm", "publisher": "qwen", "format": "gguf" }, "models_observed": [ "qwen/qwen3.6-27b" ], "runs_per_task": 5, "agents": [ "claude-code-cli", "codex-cli" ], "measurement": "mcp-transport-proxy", "tokenizer": "tiktoken/cl100k_base", "image_token_formula": "ceil(width * height / 750)", "image_token_basis": "estimated from measured pixel dimensions", "prompt_hash_algorithm": "sha256", "prompt_identity_verified": true, "prompt_identity_note": "The harness prompt is byte-identical across every agent and every stack: built once per task, hashed, and asserted equal across all arms. What legitimately differs per stack is the server's own tool surface and the instructions it ships — that is the subject of the comparison, and both are dumped in full under stacks[].tools_catalog and stacks[].server_instructions so the difference is auditable.", "prompt_mismatches": [], "run_config": { "model": "qwen/qwen3.6-27b", "model_endpoint_host": "192.168.1.197", "runs_per_task": 5, "task_timeout_s": 300, "turn_limit": null, "decoding": "provider defaults — the harness sets no temperature, top_p or seed, so every arm gets identical decoding settings", "machine": "single host, all arms interleaved in time", "retries": "none; no arm is helped past a failure" }, "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b", "task_timeout_s": 300, "fixture_base_url": "http://10.0.0.3:8899", "skipped_tasks": [], "task_filter": null, "prompt_template": "Complete the task below using only the tools you have been given.\n\nStart URL: {url}\n\nTask: {task}\n\nRules:\n- Use only the tools provided. Do not answer from prior knowledge, memory or guesswork.\n- Base your answer on what the page actually says.\n- If the requested information is genuinely not available, answer exactly NOT_AVAILABLE. Never invent a plausible-looking value.\n- Work efficiently and use as few tool calls as you can.\n- Your reply must end with a final line in exactly this form:\nANSWER: \n", "baseline_prompt": "Do not use any tools at all. Reply with exactly this single line and nothing else:\nANSWER: READY\n", "notes": "Every agent x stack pair ran the identical prompt, task list and run count (5 runs per task) on the same machine against the same model; prompts are never tuned per stack or per agent. Tool calls and payload sizes are measured by a counting proxy between the agent and every MCP server, so all stacks are measured identically instead of trusting each agent CLI's own reporting. 'transport' is what the server actually executed; 'client' is what the agent attempted. Divergence means calls were denied or aborted before reaching the server and is a harness-health signal. 'payload_tokens' counts only the TEXT the servers returned, through one fixed tokenizer. Image blocks are counted separately with byte size and pixel dimensions, because base64 image bytes are priced by dimensions and would make any screenshot-returning tool look far worse than it is. Raw bytes are reported but never headlined. 'success' means the agent produced an answer; 'correct' means that answer matched ground truth; 'wrong_answer' means it produced a confident answer that was wrong (an explicit admission that the value was unavailable is scored as a failure, not a wrong answer). 'tool_count' is a fairness factor, not a neutral fact: a larger advertised tool surface inflates input tokens on every task before any work is done. The 'baseline' section measures exactly that with a no-op prompt per agent x stack. Known agent deviations. (1) Codex sends MCP tools wrapped in an OpenAI-Responses 'namespace' container that the local model ignores, so a compatibility shim flattens them and re-annotates the resulting call with the namespace its router requires; this changes the encoding of the tool list, not its contents. Established by a controlled A/B: namespaced tools produced no tool call, the identical tool flattened produced one. (2) Claude Code runs with every built-in tool disabled (--tools \"\"), so the MCP server is its only capability, while Codex cannot disable its built-in shell, so it runs under the read-only sandbox, which has no network access — the MCP server is still its only route to the web, but the shell tool remains visible to the model. Codex also warns that model metadata for a local model is unknown and falls back to default metadata; this is harmless. Both agents drive identical local weights (qwen/qwen3.6-27b Q4_K_M, context 131072) served from 192.168.1.197, so anyone can re-run against the same model. Cost in USD is therefore meaningless and reported as null: the agent CLI applies hosted pricing to a local model, so its own figure is wrong and is kept only as 'cost_usd_reported' in the raw run details. Side-task models: Claude Code normally calls a small fast model for ancillary work. Against a custom endpoint that model does not exist, so the harness pins it to the benchmark model (ANTHROPIC_SMALL_FAST_MODEL / ANTHROPIC_DEFAULT_HAIKU_MODEL). 'models_observed' lists every model the agent actually billed, so any leak to a second model would be visible; any side-task usage is included in the token totals. Targets are books.toscrape.com (a live but purpose-built scraping sandbox whose catalogue is frozen; those tasks are marked live_site=true so they can be reported separately) and pages served by the harness itself. MCP server versions are pinned, never '@latest'." }, "owl_version": "1.3.2", "agents": [ { "id": "claude-code-cli", "label": "Claude Code", "version": "2.1.223 (Claude Code)", "skipped": false, "skip_reason": null, "verified": true }, { "id": "codex-cli", "label": "Codex CLI", "version": "codex-cli 0.146.0", "skipped": false, "skip_reason": null, "verified": true, "llm_shim": { "mode": "namespace", "namespaces_expanded": 828, "tools_flattened": 34 }, "llm_shim_note": "This agent sends MCP tools to the model inside an OpenAI-Responses 'namespace' wrapper, which the local model ignores — without help it reports having no browser tools. A local shim flattens those tools and re-annotates the returned call with the namespace the agent's router expects. Same tools, same schemas, same servers; only the encoding differs." } ], "stacks": [ { "id": "owl", "label": "Owl Browser MCP", "version": "1.3.2", "tool_count": 20, "skipped": false, "transport": "http", "launch": "POST http://localhost/mcp", "notes": "Tool surface follows OWL_MCP_PROFILE=agent. Over the HTTP transport the profile is fixed by the Owl instance at startup; this harness records the tool count the server actually advertised rather than assuming it.", "server_reported_name": "owl-browser", "server_reported_version": "1.3.2", "server_instructions": "Owl Browser is agent-native: you read pages as TEXT, not pixels.\nREADING one page or searching -> browser_go / browser_search, no context needed (content only - you cannot click or type from the result).\nACTING on a page (forms, logins, clicking) -> browser_create_context, then browser_observe and act on its handles, then browser_close_context; action tools already return the updated page, so do not call browser_observe straight after acting.", "tools_catalog": [ { "name": "browser_create_context", "description": "Open an isolated session and return its context_id. Use for anything stateful: logins, forms, moving between pages. To READ one page you need no context - use browser_go, or browser_rows for a list. Close with browser_close_context.", "input_schema_keys": [ "locale", "timezone" ] }, { "name": "browser_close_context", "description": "Close a context and free its session, cookies and memory. Contexts are a limited pool, so close each one when you are finished with it.", "input_schema_keys": [ "context_id" ] }, { "name": "browser_go", "description": "ONE-SHOT page READ, no context needed. Markdown by default. READ ONLY - the result has no handles, so you CANNOT click, type or submit from it; for that use browser_create_context then browser_observe. For a list, table or grid use browser_rows.", "input_schema_keys": [ "output", "url", "wait_until" ] }, { "name": "browser_search", "description": "ONE-SHOT web search: ranked results with titles, URLs and snippets, one call, no context. Do not drive a search engine's own page yourself.", "input_schema_keys": [ "page", "provider", "query" ] }, { "name": "browser_navigate", "description": "Point an existing context at a URL. Returns as soon as navigation STARTS, so pass wait_until='load' ('networkidle' for a JS-rendered page) whenever you will read the result. The updated page view is attached. To read one page without a context use browser_go.", "input_schema_keys": [ "context_id", "url", "wait_until" ] }, { "name": "browser_go_back", "description": "Go back one entry in this context's history. Does nothing if there is no previous page.", "input_schema_keys": [ "context_id", "wait_until" ] }, { "name": "browser_observe", "description": "PRIMARY way to read a page: compact text render plus a handle table of every interactive element ('b3', 'l5', 'x2') to pass as the 'selector' of browser_click or browser_type. 20-100x cheaper than a screenshot. Call it after navigating - action tools already attach the updated view, so do NOT observe straight after acting. detail='outline' maps a long page cheaply. For a list or table use browser_rows.", "input_schema_keys": [ "context_id", "detail", "max_tokens", "region" ] }, { "name": "browser_expand", "description": "Re-render ONE collapsed region or repeated template from the last browser_observe in more detail. Gives you structure and handles; for a node's raw text use browser_read_node.", "input_schema_keys": [ "context_id", "detail", "selector" ] }, { "name": "browser_read_node", "description": "Return one node's complete untruncated text, value and attributes, by its handle from the last browser_observe. Use when the render showed an ellipsis. Text, not structure - for structure use browser_expand.", "input_schema_keys": [ "context_id", "selector" ] }, { "name": "browser_click", "description": "Click one element. The updated page view is attached, so do NOT observe afterward. If you know the whole sequence, batch it with browser_do.", "input_schema_keys": [ "context_id", "hold_ms", "selector" ] }, { "name": "browser_type", "description": "Type text into an input. Does NOT clear it first - use browser_clear_input to replace. For a dropdown use browser_pick. The updated page view is attached, so do not observe afterward.", "input_schema_keys": [ "context_id", "selector", "text" ] }, { "name": "browser_pick", "description": "Choose an option in a dropdown or element.", "input_schema_keys": [ "includeSnapshot", "uid", "value" ] }, { "name": "fill_form", "description": "Fill out multiple form elements (inputs, selects, checkboxes, radios) at once. ALWAYS prefer this tool over multiple individual 'fill' or 'click' calls when interacting with forms. It is significantly faster, more reliable, and reduces turn count. Example: Fill username, password, and check \"Remember Me\" in one call.", "input_schema_keys": [ "elements", "includeSnapshot" ] }, { "name": "get_console_message", "description": "Gets a console message by its ID. You can get all messages by calling list_console_messages.", "input_schema_keys": [ "msgid" ] }, { "name": "get_network_request", "description": "Gets a network request by an optional reqid, if omitted returns the currently selected request in the DevTools Network panel.", "input_schema_keys": [ "reqid", "requestFilePath", "responseFilePath" ] }, { "name": "handle_dialog", "description": "If a browser dialog was opened, use this command to handle it", "input_schema_keys": [ "action", "promptText" ] }, { "name": "hover", "description": "Hover over the provided element", "input_schema_keys": [ "includeSnapshot", "uid" ] }, { "name": "lighthouse_audit", "description": "Get Lighthouse score and reports for accessibility, SEO, best practices, and agentic browsing. This excludes performance. For performance audits, run performance_start_trace", "input_schema_keys": [ "device", "mode", "outputDirPath" ] }, { "name": "list_console_messages", "description": "List all console messages for the currently selected page since the last navigation.", "input_schema_keys": [ "includePreservedMessages", "pageIdx", "pageSize", "serviceWorkerId", "types" ] }, { "name": "list_network_requests", "description": "List all requests for the currently selected page since the last navigation.", "input_schema_keys": [ "includePreservedRequests", "pageIdx", "pageSize", "resourceTypes" ] }, { "name": "list_pages", "description": "Get a list of pages open in the browser.", "input_schema_keys": [] }, { "name": "navigate_page", "description": "Go to a URL, or back, forward, or reload. Use project URL if not specified otherwise.", "input_schema_keys": [ "handleBeforeUnload", "ignoreCache", "initScript", "timeout", "type", "url" ] }, { "name": "new_page", "description": "Open a new tab and load a URL. Use project URL if not specified otherwise.", "input_schema_keys": [ "background", "isolatedContext", "timeout", "url" ] }, { "name": "performance_analyze_insight", "description": "Provides more detailed information on a specific Performance Insight of an insight set that was highlighted in the results of a trace recording.", "input_schema_keys": [ "insightName", "insightSetId" ] }, { "name": "performance_start_trace", "description": "Start a performance trace on the selected webpage. Use to find frontend performance issues, Core Web Vitals (LCP, INP, CLS), and improve page load speed.", "input_schema_keys": [ "autoStop", "filePath", "reload" ] }, { "name": "performance_stop_trace", "description": "Stop the active performance trace recording on the selected webpage.", "input_schema_keys": [ "filePath" ] }, { "name": "press_key", "description": "Press a key or key combination. Use this when other input methods like fill() cannot be used (e.g., keyboard shortcuts, navigation keys, or special key combinations).", "input_schema_keys": [ "includeSnapshot", "key" ] }, { "name": "resize_page", "description": "Resizes the selected page's window so that the page has specified dimension", "input_schema_keys": [ "height", "width" ] }, { "name": "select_page", "description": "Select a page as a context for future tool calls.", "input_schema_keys": [ "bringToFront", "pageId" ] }, { "name": "take_heapsnapshot", "description": "Capture a heap snapshot of the currently selected page. Use to analyze the memory distribution of JavaScript objects and debug memory leaks.", "input_schema_keys": [ "filePath" ] }, { "name": "take_screenshot", "description": "Take a screenshot of the page or element.", "input_schema_keys": [ "filePath", "format", "fullPage", "quality", "uid" ] }, { "name": "take_snapshot", "description": "Take a text snapshot of the currently selected page based on the a11y tree. The snapshot lists page elements along with a unique\nidentifier (uid). Always use the latest snapshot. Prefer taking a snapshot over taking a screenshot. The snapshot indicates the element selected\nin the DevTools Elements panel (if any).", "input_schema_keys": [ "filePath", "verbose" ] }, { "name": "type_text", "description": "Type text using keyboard into a previously focused input", "input_schema_keys": [ "submitKey", "text" ] }, { "name": "upload_file", "description": "Upload a file through a provided element.", "input_schema_keys": [ "filePath", "includeSnapshot", "uid" ] }, { "name": "wait_for", "description": "Wait for the specified text to appear on the selected page.", "input_schema_keys": [ "text", "timeout" ] } ] } ], "tasks": [ { "id": "cheapest-mystery-book", "label": "Find the cheapest book in the Mystery category", "shape": "list-extraction", "target": "live", "live_site": true, "source": "books.toscrape.com (static scraping sandbox, content frozen)", "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "notes": "32 books across 2 pages. Cheapest is GBP 10.69.", "drift_risk": "low — live site, but a purpose-built scraping sandbox whose catalogue has not changed since publication. Report live-target tasks separately from fixture tasks if the site ever changes.", "tolerances": [], "prompt_text": "Complete the task below using only the tools you have been given.\n\nStart URL: https://books.toscrape.com/catalogue/category/books/mystery_3/index.html\n\nTask: This is the Mystery category of an online bookstore. Find the book with the LOWEST price in this category. The category spans more than one page of results, so check every page before answering. Report the exact title of the cheapest book.\n\nRules:\n- Use only the tools provided. Do not answer from prior knowledge, memory or guesswork.\n- Base your answer on what the page actually says.\n- If the requested information is genuinely not available, answer exactly NOT_AVAILABLE. Never invent a plausible-looking value.\n- Work efficiently and use as few tool calls as you can.\n- Your reply must end with a final line in exactly this form:\nANSWER: \n", "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "any_stack_correct": true, "solved_by_any_stack": true, "solved_by": [ "claude-code-cli/chrome-devtools", "claude-code-cli/owl", "claude-code-cli/playwright", "codex-cli/chrome-devtools", "codex-cli/owl", "codex-cli/playwright" ] }, { "id": "catalogue-last-page", "label": "Read the last item on the last page of a 1000-item catalogue", "shape": "deep-pagination", "target": "live", "live_site": true, "source": "books.toscrape.com (static scraping sandbox, content frozen)", "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "notes": "Requires reaching /catalogue/page-50.html; the answer is the 20th entry.", "drift_risk": "low — live site, but a purpose-built scraping sandbox whose catalogue has not changed since publication. Report live-target tasks separately from fixture tasks if the site ever changes.", "tolerances": [], "prompt_text": "Complete the task below using only the tools you have been given.\n\nStart URL: https://books.toscrape.com/catalogue/page-1.html\n\nTask: This catalogue contains 1000 books spread over 50 pages. Go to the LAST page of the catalogue and report the exact title of the LAST book listed on that page.\n\nRules:\n- Use only the tools provided. Do not answer from prior knowledge, memory or guesswork.\n- Base your answer on what the page actually says.\n- If the requested information is genuinely not available, answer exactly NOT_AVAILABLE. Never invent a plausible-looking value.\n- Work efficiently and use as few tool calls as you can.\n- Your reply must end with a final line in exactly this form:\nANSWER: \n", "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "any_stack_correct": true, "solved_by_any_stack": true, "solved_by": [ "claude-code-cli/chrome-devtools", "claude-code-cli/owl", "claude-code-cli/playwright", "codex-cli/chrome-devtools", "codex-cli/owl", "codex-cli/playwright" ] }, { "id": "book-upc-table", "label": "Read a value out of an HTML product table", "shape": "table-read", "target": "live", "live_site": true, "source": "books.toscrape.com (static scraping sandbox, content frozen)", "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "notes": "Exact 16-character token; no partial credit.", "drift_risk": "low — live site, but a purpose-built scraping sandbox whose catalogue has not changed since publication. Report live-target tasks separately from fixture tasks if the site ever changes.", "tolerances": [], "prompt_text": "Complete the task below using only the tools you have been given.\n\nStart URL: https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html\n\nTask: This product page contains a 'Product Information' table. Report the UPC value listed in that table.\n\nRules:\n- Use only the tools provided. Do not answer from prior knowledge, memory or guesswork.\n- Base your answer on what the page actually says.\n- If the requested information is genuinely not available, answer exactly NOT_AVAILABLE. Never invent a plausible-looking value.\n- Work efficiently and use as few tool calls as you can.\n- Your reply must end with a final line in exactly this form:\nANSWER: \n", "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "any_stack_correct": true, "solved_by_any_stack": true, "solved_by": [ "claude-code-cli/chrome-devtools", "claude-code-cli/owl", "claude-code-cli/playwright", "codex-cli/chrome-devtools", "codex-cli/owl", "codex-cli/playwright" ] }, { "id": "travel-stock-drilldown", "label": "Open a detail page from a listing and read its stock count", "shape": "detail-drilldown", "target": "live", "live_site": true, "source": "books.toscrape.com (static scraping sandbox, content frozen)", "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "notes": "The value is only visible on the detail page, not on the listing.", "drift_risk": "low — live site, but a purpose-built scraping sandbox whose catalogue has not changed since publication. Report live-target tasks separately from fixture tasks if the site ever changes.", "tolerances": [ "Scored correct if the integer 6 appears anywhere in the answer, so an answer that also quotes the price could in principle match on a coincidental 6." ], "prompt_text": "Complete the task below using only the tools you have been given.\n\nStart URL: https://books.toscrape.com/catalogue/category/books/travel_2/index.html\n\nTask: This is the Travel category listing. Open the product page for the book titled 'The Great Railway Bazaar' and report how many copies are in stock (the number shown in the Availability field). Answer with just the number.\n\nRules:\n- Use only the tools provided. Do not answer from prior knowledge, memory or guesswork.\n- Base your answer on what the page actually says.\n- If the requested information is genuinely not available, answer exactly NOT_AVAILABLE. Never invent a plausible-looking value.\n- Work efficiently and use as few tool calls as you can.\n- Your reply must end with a final line in exactly this form:\nANSWER: \n", "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "any_stack_correct": true, "solved_by_any_stack": true, "solved_by": [ "claude-code-cli/chrome-devtools", "claude-code-cli/owl", "claude-code-cli/playwright", "codex-cli/chrome-devtools", "codex-cli/owl", "codex-cli/playwright" ] }, { "id": "quote-form-submit", "label": "Fill and submit a six-field form, report the confirmation code", "shape": "form-fill", "target": "fixture", "live_site": false, "source": "local fixture server (this repository)", "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "notes": "The confirmation code is only rendered after a submission where all six fields match exactly, so it cannot be guessed or read from the form page.", "drift_risk": "none", "tolerances": [], "prompt_text": "Complete the task below using only the tools you have been given.\n\nStart URL: http://10.0.0.3:8899/form.html\n\nTask: Fill in this enterprise quote request form exactly as follows and submit it:\n Full name: Dana Whitfield\n Work email: dana.whitfield@example.com\n Company: Northwind Analytics\n Number of seats: 24\n Plan: Growth\n Tick the 'I agree to the terms of service' checkbox\nThen report the confirmation code shown on the page that appears after submitting. The server rejects the form if any field is wrong or missing.\n\nRules:\n- Use only the tools provided. Do not answer from prior knowledge, memory or guesswork.\n- Base your answer on what the page actually says.\n- If the requested information is genuinely not available, answer exactly NOT_AVAILABLE. Never invent a plausible-looking value.\n- Work efficiently and use as few tool calls as you can.\n- Your reply must end with a final line in exactly this form:\nANSWER: \n", "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "any_stack_correct": true, "solved_by_any_stack": true, "solved_by": [ "claude-code-cli/chrome-devtools", "claude-code-cli/owl", "claude-code-cli/playwright", "codex-cli/chrome-devtools", "codex-cli/owl", "codex-cli/playwright" ] }, { "id": "restricted-figure", "label": "Negative control: report that a figure is not available", "shape": "negative-control", "target": "fixture", "live_site": false, "source": "local fixture server (this repository)", "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "notes": "The Q3 figure exists nowhere on the page or in its HTML. Q1 (18.4) and Q2 (21.7) are visible decoys. Any confident number is a hallucination; quoting a decoy is scored wrong even if the answer also says 'restricted'.", "drift_risk": "none", "tolerances": [], "prompt_text": "Complete the task below using only the tools you have been given.\n\nStart URL: http://10.0.0.3:8899/reports.html\n\nTask: Report the Q3 2025 revenue figure for Northwind Analytics from this page. If the figure genuinely is not available, answer exactly NOT_AVAILABLE.\n\nRules:\n- Use only the tools provided. Do not answer from prior knowledge, memory or guesswork.\n- Base your answer on what the page actually says.\n- If the requested information is genuinely not available, answer exactly NOT_AVAILABLE. Never invent a plausible-looking value.\n- Work efficiently and use as few tool calls as you can.\n- Your reply must end with a final line in exactly this form:\nANSWER: \n", "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "any_stack_correct": true, "solved_by_any_stack": true, "solved_by": [ "claude-code-cli/chrome-devtools", "claude-code-cli/owl", "claude-code-cli/playwright", "codex-cli/chrome-devtools", "codex-cli/owl", "codex-cli/playwright" ] } ], "baseline": { "claude-code-cli/owl": { "agent": "claude-code-cli", "stack": "owl", "tool_count": 20, "tools_list_tokens": 3958, "client_tokens": { "input": 5991, "cached": 0, "output": 26, "total": 6017 }, "transport_tool_calls": 0, "duration_ms": 2932.75, "ok": true, "error": null, "note": "Cost of merely connecting the server: one no-op prompt that forbids tool use. Dominated by the tool schemas the model must carry, so a larger tool surface taxes every task. Doubles as a smoke check that the pair works at all before the matrix runs." }, "claude-code-cli/playwright": { "agent": "claude-code-cli", "stack": "playwright", "tool_count": 24, "tools_list_tokens": 3928, "client_tokens": { "input": 5774, "cached": 0, "output": 32, "total": 5806 }, "transport_tool_calls": 0, "duration_ms": 3339.9, "ok": true, "error": null, "note": "Cost of merely connecting the server: one no-op prompt that forbids tool use. Dominated by the tool schemas the model must carry, so a larger tool surface taxes every task. Doubles as a smoke check that the pair works at all before the matrix runs." }, "claude-code-cli/chrome-devtools": { "agent": "claude-code-cli", "stack": "chrome-devtools", "tool_count": 29, "tools_list_tokens": 4968, "client_tokens": { "input": 7133, "cached": 0, "output": 26, "total": 7159 }, "transport_tool_calls": 0, "duration_ms": 3694.15, "ok": true, "error": null, "note": "Cost of merely connecting the server: one no-op prompt that forbids tool use. Dominated by the tool schemas the model must carry, so a larger tool surface taxes every task. Doubles as a smoke check that the pair works at all before the matrix runs." }, "codex-cli/owl": { "agent": "codex-cli", "stack": "owl", "tool_count": 20, "tools_list_tokens": 3958, "client_tokens": { "input": 13869, "cached": 0, "output": 20, "total": 13889 }, "transport_tool_calls": 0, "duration_ms": 5178.64, "ok": true, "error": null, "note": "Cost of merely connecting the server: one no-op prompt that forbids tool use. Dominated by the tool schemas the model must carry, so a larger tool surface taxes every task. Doubles as a smoke check that the pair works at all before the matrix runs." }, "codex-cli/playwright": { "agent": "codex-cli", "stack": "playwright", "tool_count": 24, "tools_list_tokens": 3928, "client_tokens": { "input": 13784, "cached": 0, "output": 20, "total": 13804 }, "transport_tool_calls": 0, "duration_ms": 5658.89, "ok": true, "error": null, "note": "Cost of merely connecting the server: one no-op prompt that forbids tool use. Dominated by the tool schemas the model must carry, so a larger tool surface taxes every task. Doubles as a smoke check that the pair works at all before the matrix runs." }, "codex-cli/chrome-devtools": { "agent": "codex-cli", "stack": "chrome-devtools", "tool_count": 29, "tools_list_tokens": 4968, "client_tokens": { "input": 14968, "cached": 0, "output": 20, "total": 14988 }, "transport_tool_calls": 0, "duration_ms": 6057.57, "ok": true, "error": null, "note": "Cost of merely connecting the server: one no-op prompt that forbids tool use. Dominated by the tool schemas the model must carry, so a larger tool surface taxes every task. Doubles as a smoke check that the pair works at all before the matrix runs." } }, "cost_note": "Monetary cost is NOT reported. The model runs locally, so there is no price to measure, and any cost figure an agent CLI emits is fabricated by applying a hosted pricing table to local token counts (Claude Code reported USD 0.39 for a local run). The raw CLI figure is kept per run as 'cost_usd_reported' for auditing only. To illustrate cost, multiply the token counts by a named reference model's published price and label the result derived, not measured.", "ranking": { "basis": "correct_rate", "tie_breakers": [ "wrong_answer_rate (lower is better)", "tokens_median_correct_runs", "calls_median_correct_runs", "duration_median_correct_runs" ], "note": "Stacks are ranked on how often they finished the job correctly. Token, call and time figures are context and tie-breakers, measured over CORRECT runs only — a failed run is usually short and cheap, so averaging failures in would flatter a stack that gets things wrong. No composite score is computed on purpose.", "stacks": [ { "stack": "owl", "correct_rate": 1.0, "wrong_answer_rate": 0.0, "tokens_median_correct_runs": 38590.0, "calls_median_correct_runs": 2.0, "duration_median_correct_runs": 14437.72 }, { "stack": "chrome-devtools", "correct_rate": 1.0, "wrong_answer_rate": 0.0, "tokens_median_correct_runs": 71437.0, "calls_median_correct_runs": 3.0, "duration_median_correct_runs": 16177.62 }, { "stack": "playwright", "correct_rate": 1.0, "wrong_answer_rate": 0.0, "tokens_median_correct_runs": 73420.0, "calls_median_correct_runs": 4.0, "duration_median_correct_runs": 15610.8 } ] }, "vision": { "note": "Whether finishing the job required the model to look at pixels. This is a capability claim, not an efficiency one: a stack that can be driven correctly with zero images can be driven by a text-only model. Measured on these tasks with this model — never asserted as a general property of a stack. 'forced' images came back from a tool that was not a screenshot tool, so the stack's normal output gave the model no choice; 'chosen' images came from the model calling a screenshot tool when a text path existed.", "by_stack": { "owl": { "vision_required": false, "vision_mode": "none", "total_runs": 60, "correct_runs": 60, "text_only_runs": 60, "text_only_correct_runs": 60, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "playwright": { "vision_required": false, "vision_mode": "none", "total_runs": 60, "correct_runs": 60, "text_only_runs": 60, "text_only_correct_runs": 60, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "chrome-devtools": { "vision_required": false, "vision_mode": "none", "total_runs": 60, "correct_runs": 60, "text_only_runs": 60, "text_only_correct_runs": 60, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." } }, "by_agent_stack": { "claude-code-cli/owl": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "claude-code-cli/playwright": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "claude-code-cli/chrome-devtools": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "codex-cli/owl": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "codex-cli/playwright": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "codex-cli/chrome-devtools": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." } } }, "results": [ { "agent": "claude-code-cli", "stack": "owl", "task": "cheapest-mystery-book", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 4.2, "min": 3.0, "max": 5.0, "stdev": 0.84 }, "client": { "median": 4.0, "mean": 4.2, "min": 3.0, "max": 5.0, "stdev": 0.84 } }, "payload_tokens": { "median": 5630.0, "mean": 4767.4, "min": 1313.0, "max": 5633.0, "stdev": 1931.07 }, "client_tokens": { "input": { "median": 45711.0, "mean": 48307.4, "min": 31119.0, "max": 59919.0, "stdev": 11894.71 }, "cached": { "median": 32680.0, "mean": 36179.8, "min": 22949.0, "max": 46579.0, "stdev": 10136.56 }, "output": { "median": 1730.0, "mean": 1671.8, "min": 1435.0, "max": 1905.0, "stdev": 191.27 } }, "tokens": { "median": 80121.0, "mean": 86159.0, "min": 55588.0, "max": 108403.0, "stdev": 22138.04 }, "duration_ms": { "median": 44177.51, "mean": 42329.35, "min": 27017.83, "max": 57148.96, "stdev": 12998.89 }, "time_to_first_action_ms": { "median": 6165.16, "mean": 6516.55, "min": 4797.39, "max": 8783.24, "stdev": 1565.4 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.2, "min": 3.0, "max": 5.0, "stdev": 0.84 }, "client": { "median": 4.0, "mean": 4.2, "min": 3.0, "max": 5.0, "stdev": 0.84 } }, "payload_tokens": { "median": 5630.0, "mean": 4767.4, "min": 1313.0, "max": 5633.0, "stdev": 1931.07 }, "text_tokens": { "median": 5630.0, "mean": 4767.4, "min": 1313.0, "max": 5633.0, "stdev": 1931.07 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 5630.0, "mean": 4767.4, "min": 1313.0, "max": 5633.0, "stdev": 1931.07 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 17848.0, "mean": 15130.2, "min": 4259.0, "max": 17852.0, "stdev": 6077.19 }, "client_tokens": { "input": { "median": 45711.0, "mean": 48307.4, "min": 31119.0, "max": 59919.0, "stdev": 11894.71 }, "cached": { "median": 32680.0, "mean": 36179.8, "min": 22949.0, "max": 46579.0, "stdev": 10136.56 }, "output": { "median": 1730.0, "mean": 1671.8, "min": 1435.0, "max": 1905.0, "stdev": 191.27 } }, "tokens": { "median": 80121.0, "mean": 86159.0, "min": 55588.0, "max": 108403.0, "stdev": 22138.04 }, "duration_ms": { "median": 44177.51, "mean": 42329.35, "min": 27017.83, "max": 57148.96, "stdev": 12998.89 }, "time_to_first_action_ms": { "median": 6165.16, "mean": 6516.55, "min": 4797.39, "max": 8783.24, "stdev": 1565.4 }, "agent_turns": { "median": 5.0, "mean": 5.2, "min": 4.0, "max": 6.0, "stdev": 0.84 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.2, "min": 3.0, "max": 5.0, "stdev": 0.84 }, "client": { "median": 4.0, "mean": 4.2, "min": 3.0, "max": 5.0, "stdev": 0.84 } }, "payload_tokens": { "median": 5630.0, "mean": 4767.4, "min": 1313.0, "max": 5633.0, "stdev": 1931.07 }, "text_tokens": { "median": 5630.0, "mean": 4767.4, "min": 1313.0, "max": 5633.0, "stdev": 1931.07 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 5630.0, "mean": 4767.4, "min": 1313.0, "max": 5633.0, "stdev": 1931.07 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 17848.0, "mean": 15130.2, "min": 4259.0, "max": 17852.0, "stdev": 6077.19 }, "client_tokens": { "input": { "median": 45711.0, "mean": 48307.4, "min": 31119.0, "max": 59919.0, "stdev": 11894.71 }, "cached": { "median": 32680.0, "mean": 36179.8, "min": 22949.0, "max": 46579.0, "stdev": 10136.56 }, "output": { "median": 1730.0, "mean": 1671.8, "min": 1435.0, "max": 1905.0, "stdev": 191.27 } }, "tokens": { "median": 80121.0, "mean": 86159.0, "min": 55588.0, "max": 108403.0, "stdev": 22138.04 }, "duration_ms": { "median": 44177.51, "mean": 42329.35, "min": 27017.83, "max": 57148.96, "stdev": 12998.89 }, "time_to_first_action_ms": { "median": 6165.16, "mean": 6516.55, "min": 4797.39, "max": 8783.24, "stdev": 1565.4 }, "agent_turns": { "median": 5.0, "mean": 5.2, "min": 4.0, "max": 6.0, "stdev": 0.84 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 4797.39, "payload_tokens": 5632, "text_tokens": 5632, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 5632, "image_token_share": 0.0, "payload_bytes": 17848, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 45483, "tokens_cached": 32435, "tokens_out": 1435, "tokens": 79353, "duration_ms": 27017.83, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.2795075, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5533.66, "payload_tokens": 5629, "text_tokens": 5629, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 5629, "image_token_share": 0.0, "payload_bytes": 17844, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 45711, "tokens_cached": 32680, "tokens_out": 1730, "tokens": 80121, "duration_ms": 31223.23, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.288145, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 6, "time_to_first_action_ms": 8783.24, "payload_tokens": 5633, "text_tokens": 5633, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 5633, "image_token_share": 0.0, "payload_bytes": 17852, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 59919, "tokens_cached": 46579, "tokens_out": 1905, "tokens": 108403, "duration_ms": 57148.96, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.3705095, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 3, "client_tool_calls": 3, "client_tool_calls_total": 3, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 4, "time_to_first_action_ms": 7303.32, "payload_tokens": 1313, "text_tokens": 1313, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1313, "image_token_share": 0.0, "payload_bytes": 4259, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 31119, "tokens_cached": 22949, "tokens_out": 1520, "tokens": 55588, "duration_ms": 52079.2, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.2050695, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 6, "time_to_first_action_ms": 6165.16, "payload_tokens": 5630, "text_tokens": 5630, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 5630, "image_token_share": 0.0, "payload_bytes": 17848, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 59305, "tokens_cached": 46256, "tokens_out": 1769, "tokens": 107330, "duration_ms": 44177.51, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.36387800000000003, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "owl", "task": "catalogue-last-page", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 1.0, "mean": 1.4, "min": 1.0, "max": 2.0, "stdev": 0.55 }, "client": { "median": 1.0, "mean": 1.4, "min": 1.0, "max": 2.0, "stdev": 0.55 } }, "payload_tokens": { "median": 1216.0, "mean": 2793.2, "min": 1216.0, "max": 5159.0, "stdev": 2159.67 }, "client_tokens": { "input": { "median": 14210.0, "mean": 19096.0, "min": 14203.0, "max": 26515.0, "stdev": 6695.27 }, "cached": { "median": 6220.0, "mean": 9337.6, "min": 6217.0, "max": 14102.0, "stdev": 4271.21 }, "output": { "median": 438.0, "mean": 469.8, "min": 318.0, "max": 719.0, "stdev": 169.38 } }, "tokens": { "median": 20858.0, "mean": 28903.4, "min": 20744.0, "max": 41336.0, "stdev": 11118.58 }, "duration_ms": { "median": 11870.97, "mean": 12797.57, "min": 8163.68, "max": 16921.99, "stdev": 3396.21 }, "time_to_first_action_ms": { "median": 5626.98, "mean": 5627.36, "min": 4642.02, "max": 7438.73, "stdev": 1120.38 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 1.0, "mean": 1.4, "min": 1.0, "max": 2.0, "stdev": 0.55 }, "client": { "median": 1.0, "mean": 1.4, "min": 1.0, "max": 2.0, "stdev": 0.55 } }, "payload_tokens": { "median": 1216.0, "mean": 2793.2, "min": 1216.0, "max": 5159.0, "stdev": 2159.67 }, "text_tokens": { "median": 1216.0, "mean": 2793.2, "min": 1216.0, "max": 5159.0, "stdev": 2159.67 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1216.0, "mean": 2793.2, "min": 1216.0, "max": 5159.0, "stdev": 2159.67 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 3899.0, "mean": 9036.6, "min": 3899.0, "max": 16743.0, "stdev": 7034.95 }, "client_tokens": { "input": { "median": 14210.0, "mean": 19096.0, "min": 14203.0, "max": 26515.0, "stdev": 6695.27 }, "cached": { "median": 6220.0, "mean": 9337.6, "min": 6217.0, "max": 14102.0, "stdev": 4271.21 }, "output": { "median": 438.0, "mean": 469.8, "min": 318.0, "max": 719.0, "stdev": 169.38 } }, "tokens": { "median": 20858.0, "mean": 28903.4, "min": 20744.0, "max": 41336.0, "stdev": 11118.58 }, "duration_ms": { "median": 11870.97, "mean": 12797.57, "min": 8163.68, "max": 16921.99, "stdev": 3396.21 }, "time_to_first_action_ms": { "median": 5626.98, "mean": 5627.36, "min": 4642.02, "max": 7438.73, "stdev": 1120.38 }, "agent_turns": { "median": 2.0, "mean": 2.4, "min": 2.0, "max": 3.0, "stdev": 0.55 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 1.0, "mean": 1.4, "min": 1.0, "max": 2.0, "stdev": 0.55 }, "client": { "median": 1.0, "mean": 1.4, "min": 1.0, "max": 2.0, "stdev": 0.55 } }, "payload_tokens": { "median": 1216.0, "mean": 2793.2, "min": 1216.0, "max": 5159.0, "stdev": 2159.67 }, "text_tokens": { "median": 1216.0, "mean": 2793.2, "min": 1216.0, "max": 5159.0, "stdev": 2159.67 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1216.0, "mean": 2793.2, "min": 1216.0, "max": 5159.0, "stdev": 2159.67 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 3899.0, "mean": 9036.6, "min": 3899.0, "max": 16743.0, "stdev": 7034.95 }, "client_tokens": { "input": { "median": 14210.0, "mean": 19096.0, "min": 14203.0, "max": 26515.0, "stdev": 6695.27 }, "cached": { "median": 6220.0, "mean": 9337.6, "min": 6217.0, "max": 14102.0, "stdev": 4271.21 }, "output": { "median": 438.0, "mean": 469.8, "min": 318.0, "max": 719.0, "stdev": 169.38 } }, "tokens": { "median": 20858.0, "mean": 28903.4, "min": 20744.0, "max": 41336.0, "stdev": 11118.58 }, "duration_ms": { "median": 11870.97, "mean": 12797.57, "min": 8163.68, "max": 16921.99, "stdev": 3396.21 }, "time_to_first_action_ms": { "median": 5626.98, "mean": 5627.36, "min": 4642.02, "max": 7438.73, "stdev": 1120.38 }, "agent_turns": { "median": 2.0, "mean": 2.4, "min": 2.0, "max": 3.0, "stdev": 0.55 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5677.26, "payload_tokens": 5159, "text_tokens": 5159, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 5159, "image_token_share": 0.0, "payload_bytes": 16743, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 26345, "tokens_cached": 13930, "tokens_out": 552, "tokens": 40827, "duration_ms": 16921.99, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.15249000000000001, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 5626.98, "payload_tokens": 1216, "text_tokens": 1216, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1216, "image_token_share": 0.0, "payload_bytes": 3899, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14203, "tokens_cached": 6217, "tokens_out": 438, "tokens": 20858, "duration_ms": 11870.97, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.0850735, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 7438.73, "payload_tokens": 1216, "text_tokens": 1216, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1216, "image_token_share": 0.0, "payload_bytes": 3899, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14207, "tokens_cached": 6219, "tokens_out": 318, "tokens": 20744, "duration_ms": 11818.04, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.0820945, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 4751.82, "payload_tokens": 1216, "text_tokens": 1216, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1216, "image_token_share": 0.0, "payload_bytes": 3899, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14210, "tokens_cached": 6220, "tokens_out": 322, "tokens": 20752, "duration_ms": 8163.68, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.08221, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 4642.02, "payload_tokens": 5159, "text_tokens": 5159, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 5159, "image_token_share": 0.0, "payload_bytes": 16743, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 26515, "tokens_cached": 14102, "tokens_out": 719, "tokens": 41336, "duration_ms": 15213.17, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.157601, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "owl", "task": "book-upc-table", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 13453.0, "mean": 13454.0, "min": 13452.0, "max": 13459.0, "stdev": 2.92 }, "cached": { "median": 6221.0, "mean": 6218.8, "min": 6208.0, "max": 6225.0, "stdev": 6.53 }, "output": { "median": 287.0, "mean": 293.0, "min": 282.0, "max": 313.0, "stdev": 12.71 } }, "tokens": { "median": 19961.0, "mean": 19965.8, "min": 19952.0, "max": 19982.0, "stdev": 11.73 }, "duration_ms": { "median": 7659.05, "mean": 7621.26, "min": 6939.59, "max": 8824.71, "stdev": 759.88 }, "time_to_first_action_ms": { "median": 5352.21, "mean": 5259.13, "min": 4824.0, "max": 5850.37, "stdev": 421.12 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "text_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2200.0, "mean": 2200.0, "min": 2200.0, "max": 2200.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 13453.0, "mean": 13454.0, "min": 13452.0, "max": 13459.0, "stdev": 2.92 }, "cached": { "median": 6221.0, "mean": 6218.8, "min": 6208.0, "max": 6225.0, "stdev": 6.53 }, "output": { "median": 287.0, "mean": 293.0, "min": 282.0, "max": 313.0, "stdev": 12.71 } }, "tokens": { "median": 19961.0, "mean": 19965.8, "min": 19952.0, "max": 19982.0, "stdev": 11.73 }, "duration_ms": { "median": 7659.05, "mean": 7621.26, "min": 6939.59, "max": 8824.71, "stdev": 759.88 }, "time_to_first_action_ms": { "median": 5352.21, "mean": 5259.13, "min": 4824.0, "max": 5850.37, "stdev": 421.12 }, "agent_turns": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "text_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2200.0, "mean": 2200.0, "min": 2200.0, "max": 2200.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 13453.0, "mean": 13454.0, "min": 13452.0, "max": 13459.0, "stdev": 2.92 }, "cached": { "median": 6221.0, "mean": 6218.8, "min": 6208.0, "max": 6225.0, "stdev": 6.53 }, "output": { "median": 287.0, "mean": 293.0, "min": 282.0, "max": 313.0, "stdev": 12.71 } }, "tokens": { "median": 19961.0, "mean": 19965.8, "min": 19952.0, "max": 19982.0, "stdev": 11.73 }, "duration_ms": { "median": 7659.05, "mean": 7621.26, "min": 6939.59, "max": 8824.71, "stdev": 759.88 }, "time_to_first_action_ms": { "median": 5352.21, "mean": 5259.13, "min": 4824.0, "max": 5850.37, "stdev": 421.12 }, "agent_turns": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 5352.21, "payload_tokens": 532, "text_tokens": 532, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 532, "image_token_share": 0.0, "payload_bytes": 2200, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 13459, "tokens_cached": 6225, "tokens_out": 298, "tokens": 19982, "duration_ms": 7686.94, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.0778575, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 5390.22, "payload_tokens": 532, "text_tokens": 532, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 532, "image_token_share": 0.0, "payload_bytes": 2200, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 13454, "tokens_cached": 6222, "tokens_out": 285, "tokens": 19961, "duration_ms": 7659.05, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.077506, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 5850.37, "payload_tokens": 532, "text_tokens": 532, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 532, "image_token_share": 0.0, "payload_bytes": 2200, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 13452, "tokens_cached": 6208, "tokens_out": 313, "tokens": 19973, "duration_ms": 8824.71, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.07818900000000001, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 4878.86, "payload_tokens": 532, "text_tokens": 532, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 532, "image_token_share": 0.0, "payload_bytes": 2200, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 13453, "tokens_cached": 6221, "tokens_out": 287, "tokens": 19961, "duration_ms": 6996.0, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.0775505, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 4824.0, "payload_tokens": 532, "text_tokens": 532, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 532, "image_token_share": 0.0, "payload_bytes": 2200, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 13452, "tokens_cached": 6218, "tokens_out": 282, "tokens": 19952, "duration_ms": 6939.59, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.07741899999999999, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "owl", "task": "travel-stock-drilldown", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 2057.0, "mean": 2057.0, "min": 2057.0, "max": 2057.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 22376.0, "mean": 22396.0, "min": 22373.0, "max": 22449.0, "stdev": 33.04 }, "cached": { "median": 13382.0, "mean": 13401.2, "min": 13379.0, "max": 13457.0, "stdev": 33.43 }, "output": { "median": 408.0, "mean": 413.6, "min": 373.0, "max": 461.0, "stdev": 39.49 } }, "tokens": { "median": 36166.0, "mean": 36210.8, "min": 36125.0, "max": 36353.0, "stdev": 100.03 }, "duration_ms": { "median": 11239.82, "mean": 11140.47, "min": 9703.78, "max": 12742.13, "stdev": 1161.14 }, "time_to_first_action_ms": { "median": 5292.03, "mean": 5361.92, "min": 4710.77, "max": 6225.59, "stdev": 548.34 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 2057.0, "mean": 2057.0, "min": 2057.0, "max": 2057.0, "stdev": 0.0 }, "text_tokens": { "median": 2057.0, "mean": 2057.0, "min": 2057.0, "max": 2057.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 2057.0, "mean": 2057.0, "min": 2057.0, "max": 2057.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 7267.0, "mean": 7267.0, "min": 7267.0, "max": 7267.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 22376.0, "mean": 22396.0, "min": 22373.0, "max": 22449.0, "stdev": 33.04 }, "cached": { "median": 13382.0, "mean": 13401.2, "min": 13379.0, "max": 13457.0, "stdev": 33.43 }, "output": { "median": 408.0, "mean": 413.6, "min": 373.0, "max": 461.0, "stdev": 39.49 } }, "tokens": { "median": 36166.0, "mean": 36210.8, "min": 36125.0, "max": 36353.0, "stdev": 100.03 }, "duration_ms": { "median": 11239.82, "mean": 11140.47, "min": 9703.78, "max": 12742.13, "stdev": 1161.14 }, "time_to_first_action_ms": { "median": 5292.03, "mean": 5361.92, "min": 4710.77, "max": 6225.59, "stdev": 548.34 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 2057.0, "mean": 2057.0, "min": 2057.0, "max": 2057.0, "stdev": 0.0 }, "text_tokens": { "median": 2057.0, "mean": 2057.0, "min": 2057.0, "max": 2057.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 2057.0, "mean": 2057.0, "min": 2057.0, "max": 2057.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 7267.0, "mean": 7267.0, "min": 7267.0, "max": 7267.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 22376.0, "mean": 22396.0, "min": 22373.0, "max": 22449.0, "stdev": 33.04 }, "cached": { "median": 13382.0, "mean": 13401.2, "min": 13379.0, "max": 13457.0, "stdev": 33.43 }, "output": { "median": 408.0, "mean": 413.6, "min": 373.0, "max": 461.0, "stdev": 39.49 } }, "tokens": { "median": 36166.0, "mean": 36210.8, "min": 36125.0, "max": 36353.0, "stdev": 100.03 }, "duration_ms": { "median": 11239.82, "mean": 11140.47, "min": 9703.78, "max": 12742.13, "stdev": 1161.14 }, "time_to_first_action_ms": { "median": 5292.03, "mean": 5361.92, "min": 4710.77, "max": 6225.59, "stdev": 548.34 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5196.43, "payload_tokens": 2057, "text_tokens": 2057, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 2057, "image_token_share": 0.0, "payload_bytes": 7267, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 22374, "tokens_cached": 13380, "tokens_out": 379, "tokens": 36133, "duration_ms": 11239.82, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.128035, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5292.03, "payload_tokens": 2057, "text_tokens": 2057, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 2057, "image_token_share": 0.0, "payload_bytes": 7267, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 22376, "tokens_cached": 13382, "tokens_out": 408, "tokens": 36166, "duration_ms": 11607.78, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.128771, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 6225.59, "payload_tokens": 2057, "text_tokens": 2057, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 2057, "image_token_share": 0.0, "payload_bytes": 7267, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 22408, "tokens_cached": 13408, "tokens_out": 461, "tokens": 36277, "duration_ms": 12742.13, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.13026900000000002, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5384.79, "payload_tokens": 2057, "text_tokens": 2057, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 2057, "image_token_share": 0.0, "payload_bytes": 7267, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 22449, "tokens_cached": 13457, "tokens_out": 447, "tokens": 36353, "duration_ms": 10408.82, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1301485, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 4710.77, "payload_tokens": 2057, "text_tokens": 2057, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 2057, "image_token_share": 0.0, "payload_bytes": 7267, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 22373, "tokens_cached": 13379, "tokens_out": 373, "tokens": 36125, "duration_ms": 9703.78, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.12787949999999998, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "owl", "task": "quote-form-submit", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 682.0, "mean": 683.4, "min": 680.0, "max": 687.0, "stdev": 3.36 }, "client_tokens": { "input": { "median": 34977.0, "mean": 35024.4, "min": 34939.0, "max": 35139.0, "stdev": 89.0 }, "cached": { "median": 27400.0, "mean": 27452.0, "min": 27360.0, "max": 27569.0, "stdev": 92.65 }, "output": { "median": 783.0, "mean": 788.8, "min": 751.0, "max": 839.0, "stdev": 35.1 } }, "tokens": { "median": 63148.0, "mean": 63265.2, "min": 63063.0, "max": 63547.0, "stdev": 214.13 }, "duration_ms": { "median": 16049.96, "mean": 16856.93, "min": 14963.18, "max": 19310.78, "stdev": 1913.95 }, "time_to_first_action_ms": { "median": 5183.73, "mean": 5151.73, "min": 4505.7, "max": 5803.05, "stdev": 469.79 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 682.0, "mean": 683.4, "min": 680.0, "max": 687.0, "stdev": 3.36 }, "text_tokens": { "median": 682.0, "mean": 683.4, "min": 680.0, "max": 687.0, "stdev": 3.36 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 682.0, "mean": 683.4, "min": 680.0, "max": 687.0, "stdev": 3.36 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2303.0, "mean": 2305.4, "min": 2303.0, "max": 2309.0, "stdev": 3.29 }, "client_tokens": { "input": { "median": 34977.0, "mean": 35024.4, "min": 34939.0, "max": 35139.0, "stdev": 89.0 }, "cached": { "median": 27400.0, "mean": 27452.0, "min": 27360.0, "max": 27569.0, "stdev": 92.65 }, "output": { "median": 783.0, "mean": 788.8, "min": 751.0, "max": 839.0, "stdev": 35.1 } }, "tokens": { "median": 63148.0, "mean": 63265.2, "min": 63063.0, "max": 63547.0, "stdev": 214.13 }, "duration_ms": { "median": 16049.96, "mean": 16856.93, "min": 14963.18, "max": 19310.78, "stdev": 1913.95 }, "time_to_first_action_ms": { "median": 5183.73, "mean": 5151.73, "min": 4505.7, "max": 5803.05, "stdev": 469.79 }, "agent_turns": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 682.0, "mean": 683.4, "min": 680.0, "max": 687.0, "stdev": 3.36 }, "text_tokens": { "median": 682.0, "mean": 683.4, "min": 680.0, "max": 687.0, "stdev": 3.36 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 682.0, "mean": 683.4, "min": 680.0, "max": 687.0, "stdev": 3.36 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2303.0, "mean": 2305.4, "min": 2303.0, "max": 2309.0, "stdev": 3.29 }, "client_tokens": { "input": { "median": 34977.0, "mean": 35024.4, "min": 34939.0, "max": 35139.0, "stdev": 89.0 }, "cached": { "median": 27400.0, "mean": 27452.0, "min": 27360.0, "max": 27569.0, "stdev": 92.65 }, "output": { "median": 783.0, "mean": 788.8, "min": 751.0, "max": 839.0, "stdev": 35.1 } }, "tokens": { "median": 63148.0, "mean": 63265.2, "min": 63063.0, "max": 63547.0, "stdev": 214.13 }, "duration_ms": { "median": 16049.96, "mean": 16856.93, "min": 14963.18, "max": 19310.78, "stdev": 1913.95 }, "time_to_first_action_ms": { "median": 5183.73, "mean": 5151.73, "min": 4505.7, "max": 5803.05, "stdev": 469.79 }, "agent_turns": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5273.63, "payload_tokens": 682, "text_tokens": 682, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 682, "image_token_share": 0.0, "payload_bytes": 2303, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 35100, "tokens_cached": 27533, "tokens_out": 807, "tokens": 63440, "duration_ms": 15503.28, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.2094415, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 4505.7, "payload_tokens": 687, "text_tokens": 687, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 687, "image_token_share": 0.0, "payload_bytes": 2309, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 34967, "tokens_cached": 27398, "tokens_out": 783, "tokens": 63148, "duration_ms": 14963.18, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.20810900000000002, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5803.05, "payload_tokens": 681, "text_tokens": 681, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 681, "image_token_share": 0.0, "payload_bytes": 2303, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 34977, "tokens_cached": 27400, "tokens_out": 751, "tokens": 63128, "duration_ms": 19310.78, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.20736, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5183.73, "payload_tokens": 687, "text_tokens": 687, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 687, "image_token_share": 0.0, "payload_bytes": 2309, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 35139, "tokens_cached": 27569, "tokens_out": 839, "tokens": 63547, "duration_ms": 16049.96, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.2104545, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 4992.54, "payload_tokens": 680, "text_tokens": 680, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 680, "image_token_share": 0.0, "payload_bytes": 2303, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 34939, "tokens_cached": 27360, "tokens_out": 764, "tokens": 63063, "duration_ms": 18457.44, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.207475, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "owl", "task": "restricted-figure", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 13063.0, "mean": 13064.0, "min": 13055.0, "max": 13078.0, "stdev": 8.57 }, "cached": { "median": 6241.0, "mean": 6248.4, "min": 6225.0, "max": 6304.0, "stdev": 32.13 }, "output": { "median": 481.0, "mean": 485.6, "min": 460.0, "max": 512.0, "stdev": 19.32 } }, "tokens": { "median": 19816.0, "mean": 19798.0, "min": 19760.0, "max": 19828.0, "stdev": 31.1 }, "duration_ms": { "median": 9951.23, "mean": 10215.99, "min": 8832.64, "max": 12634.88, "stdev": 1499.21 }, "time_to_first_action_ms": { "median": 5818.38, "mean": 5804.75, "min": 4923.38, "max": 6814.63, "stdev": 716.91 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "text_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 598.0, "mean": 598.0, "min": 598.0, "max": 598.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 13063.0, "mean": 13064.0, "min": 13055.0, "max": 13078.0, "stdev": 8.57 }, "cached": { "median": 6241.0, "mean": 6248.4, "min": 6225.0, "max": 6304.0, "stdev": 32.13 }, "output": { "median": 481.0, "mean": 485.6, "min": 460.0, "max": 512.0, "stdev": 19.32 } }, "tokens": { "median": 19816.0, "mean": 19798.0, "min": 19760.0, "max": 19828.0, "stdev": 31.1 }, "duration_ms": { "median": 9951.23, "mean": 10215.99, "min": 8832.64, "max": 12634.88, "stdev": 1499.21 }, "time_to_first_action_ms": { "median": 5818.38, "mean": 5804.75, "min": 4923.38, "max": 6814.63, "stdev": 716.91 }, "agent_turns": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "text_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 598.0, "mean": 598.0, "min": 598.0, "max": 598.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 13063.0, "mean": 13064.0, "min": 13055.0, "max": 13078.0, "stdev": 8.57 }, "cached": { "median": 6241.0, "mean": 6248.4, "min": 6225.0, "max": 6304.0, "stdev": 32.13 }, "output": { "median": 481.0, "mean": 485.6, "min": 460.0, "max": 512.0, "stdev": 19.32 } }, "tokens": { "median": 19816.0, "mean": 19798.0, "min": 19760.0, "max": 19828.0, "stdev": 31.1 }, "duration_ms": { "median": 9951.23, "mean": 10215.99, "min": 8832.64, "max": 12634.88, "stdev": 1499.21 }, "time_to_first_action_ms": { "median": 5818.38, "mean": 5804.75, "min": 4923.38, "max": 6814.63, "stdev": 716.91 }, "agent_turns": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 4923.38, "payload_tokens": 139, "text_tokens": 139, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 139, "image_token_share": 0.0, "payload_bytes": 598, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 13063, "tokens_cached": 6241, "tokens_out": 512, "tokens": 19816, "duration_ms": 8832.64, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.08123549999999999, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 5380.92, "payload_tokens": 139, "text_tokens": 139, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 139, "image_token_share": 0.0, "payload_bytes": 598, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 13055, "tokens_cached": 6225, "tokens_out": 480, "tokens": 19760, "duration_ms": 9175.33, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.0803875, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 6814.63, "payload_tokens": 139, "text_tokens": 139, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 139, "image_token_share": 0.0, "payload_bytes": 598, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 13060, "tokens_cached": 6228, "tokens_out": 481, "tokens": 19769, "duration_ms": 12634.88, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.08043900000000001, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 5818.38, "payload_tokens": 139, "text_tokens": 139, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 139, "image_token_share": 0.0, "payload_bytes": 598, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 13078, "tokens_cached": 6244, "tokens_out": 495, "tokens": 19817, "duration_ms": 9951.23, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.080887, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 2, "time_to_first_action_ms": 6086.45, "payload_tokens": 139, "text_tokens": 139, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 139, "image_token_share": 0.0, "payload_bytes": 598, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 13064, "tokens_cached": 6304, "tokens_out": 460, "tokens": 19828, "duration_ms": 10485.85, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.07997199999999999, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "playwright", "task": "cheapest-mystery-book", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 4.8, "min": 4.0, "max": 6.0, "stdev": 1.1 }, "client": { "median": 4.0, "mean": 4.8, "min": 4.0, "max": 6.0, "stdev": 1.1 } }, "payload_tokens": { "median": 14301.0, "mean": 14616.6, "min": 14301.0, "max": 15090.0, "stdev": 432.15 }, "client_tokens": { "input": { "median": 67521.0, "mean": 86991.6, "min": 67179.0, "max": 116875.0, "stdev": 26960.74 }, "cached": { "median": 44905.0, "mean": 64020.0, "min": 44557.0, "max": 93374.0, "stdev": 26477.51 }, "output": { "median": 1708.0, "mean": 1779.0, "min": 1508.0, "max": 2049.0, "stdev": 232.34 } }, "tokens": { "median": 114417.0, "mean": 152790.6, "min": 113244.0, "max": 212298.0, "stdev": 53531.49 }, "duration_ms": { "median": 38612.23, "mean": 48037.5, "min": 33696.47, "max": 72802.16, "stdev": 16749.03 }, "time_to_first_action_ms": { "median": 5675.71, "mean": 6484.0, "min": 4882.4, "max": 10850.9, "stdev": 2472.29 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.8, "min": 4.0, "max": 6.0, "stdev": 1.1 }, "client": { "median": 4.0, "mean": 4.8, "min": 4.0, "max": 6.0, "stdev": 1.1 } }, "payload_tokens": { "median": 14301.0, "mean": 14616.6, "min": 14301.0, "max": 15090.0, "stdev": 432.15 }, "text_tokens": { "median": 14301.0, "mean": 14616.6, "min": 14301.0, "max": 15090.0, "stdev": 432.15 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 14301.0, "mean": 14616.6, "min": 14301.0, "max": 15090.0, "stdev": 432.15 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 55008.0, "mean": 56065.6, "min": 55008.0, "max": 57652.0, "stdev": 1448.18 }, "client_tokens": { "input": { "median": 67521.0, "mean": 86991.6, "min": 67179.0, "max": 116875.0, "stdev": 26960.74 }, "cached": { "median": 44905.0, "mean": 64020.0, "min": 44557.0, "max": 93374.0, "stdev": 26477.51 }, "output": { "median": 1708.0, "mean": 1779.0, "min": 1508.0, "max": 2049.0, "stdev": 232.34 } }, "tokens": { "median": 114417.0, "mean": 152790.6, "min": 113244.0, "max": 212298.0, "stdev": 53531.49 }, "duration_ms": { "median": 38612.23, "mean": 48037.5, "min": 33696.47, "max": 72802.16, "stdev": 16749.03 }, "time_to_first_action_ms": { "median": 5675.71, "mean": 6484.0, "min": 4882.4, "max": 10850.9, "stdev": 2472.29 }, "agent_turns": { "median": 5.0, "mean": 5.8, "min": 5.0, "max": 7.0, "stdev": 1.1 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.8, "min": 4.0, "max": 6.0, "stdev": 1.1 }, "client": { "median": 4.0, "mean": 4.8, "min": 4.0, "max": 6.0, "stdev": 1.1 } }, "payload_tokens": { "median": 14301.0, "mean": 14616.6, "min": 14301.0, "max": 15090.0, "stdev": 432.15 }, "text_tokens": { "median": 14301.0, "mean": 14616.6, "min": 14301.0, "max": 15090.0, "stdev": 432.15 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 14301.0, "mean": 14616.6, "min": 14301.0, "max": 15090.0, "stdev": 432.15 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 55008.0, "mean": 56065.6, "min": 55008.0, "max": 57652.0, "stdev": 1448.18 }, "client_tokens": { "input": { "median": 67521.0, "mean": 86991.6, "min": 67179.0, "max": 116875.0, "stdev": 26960.74 }, "cached": { "median": 44905.0, "mean": 64020.0, "min": 44557.0, "max": 93374.0, "stdev": 26477.51 }, "output": { "median": 1708.0, "mean": 1779.0, "min": 1508.0, "max": 2049.0, "stdev": 232.34 } }, "tokens": { "median": 114417.0, "mean": 152790.6, "min": 113244.0, "max": 212298.0, "stdev": 53531.49 }, "duration_ms": { "median": 38612.23, "mean": 48037.5, "min": 33696.47, "max": 72802.16, "stdev": 16749.03 }, "time_to_first_action_ms": { "median": 5675.71, "mean": 6484.0, "min": 4882.4, "max": 10850.9, "stdev": 2472.29 }, "agent_turns": { "median": 5.0, "mean": 5.8, "min": 5.0, "max": 7.0, "stdev": 1.1 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 6, "client_tool_calls": 6, "client_tool_calls_total": 6, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 7, "time_to_first_action_ms": 4882.4, "payload_tokens": 15090, "text_tokens": 15090, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 15090, "image_token_share": 0.0, "payload_bytes": 57652, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 116173, "tokens_cached": 92672, "tokens_out": 1708, "tokens": 210553, "duration_ms": 33696.47, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.669901, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 6, "client_tool_calls": 6, "client_tool_calls_total": 6, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 7, "time_to_first_action_ms": 5155.04, "payload_tokens": 15090, "text_tokens": 15090, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 15090, "image_token_share": 0.0, "payload_bytes": 57652, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 116875, "tokens_cached": 93374, "tokens_out": 2049, "tokens": 212298, "duration_ms": 38612.23, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.682287, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5675.71, "payload_tokens": 14301, "text_tokens": 14301, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 14301, "image_token_share": 0.0, "payload_bytes": 55008, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 67179, "tokens_cached": 44557, "tokens_out": 1508, "tokens": 113244, "duration_ms": 57878.34, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.3958735, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 10850.9, "payload_tokens": 14301, "text_tokens": 14301, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 14301, "image_token_share": 0.0, "payload_bytes": 55008, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 67521, "tokens_cached": 44905, "tokens_out": 1991, "tokens": 114417, "duration_ms": 72802.16, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.40983250000000004, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5855.94, "payload_tokens": 14301, "text_tokens": 14301, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 14301, "image_token_share": 0.0, "payload_bytes": 55008, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 67210, "tokens_cached": 44592, "tokens_out": 1639, "tokens": 113441, "duration_ms": 37198.3, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.399321, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "playwright", "task": "catalogue-last-page", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 3.6, "min": 2.0, "max": 4.0, "stdev": 0.89 }, "client": { "median": 4.0, "mean": 3.6, "min": 2.0, "max": 4.0, "stdev": 0.89 } }, "payload_tokens": { "median": 8720.0, "mean": 8910.8, "min": 7830.0, "max": 11153.0, "stdev": 1311.15 }, "client_tokens": { "input": { "median": 60092.0, "mean": 54268.6, "min": 27869.0, "max": 63210.0, "stdev": 14821.94 }, "cached": { "median": 44251.0, "mean": 37947.2, "min": 12775.0, "max": 44373.0, "stdev": 14072.41 }, "output": { "median": 830.0, "mean": 744.6, "min": 512.0, "max": 859.0, "stdev": 147.12 } }, "tokens": { "median": 104803.0, "mean": 92960.4, "min": 41156.0, "max": 108400.0, "stdev": 28997.32 }, "duration_ms": { "median": 21087.37, "mean": 19837.67, "min": 12099.9, "max": 23812.58, "stdev": 4640.12 }, "time_to_first_action_ms": { "median": 4947.27, "mean": 5176.38, "min": 4537.25, "max": 6361.69, "stdev": 698.32 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 3.6, "min": 2.0, "max": 4.0, "stdev": 0.89 }, "client": { "median": 4.0, "mean": 3.6, "min": 2.0, "max": 4.0, "stdev": 0.89 } }, "payload_tokens": { "median": 8720.0, "mean": 8910.8, "min": 7830.0, "max": 11153.0, "stdev": 1311.15 }, "text_tokens": { "median": 8720.0, "mean": 8910.8, "min": 7830.0, "max": 11153.0, "stdev": 1311.15 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8720.0, "mean": 8910.8, "min": 7830.0, "max": 11153.0, "stdev": 1311.15 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 34757.0, "mean": 35238.2, "min": 31526.0, "max": 42737.0, "stdev": 4428.65 }, "client_tokens": { "input": { "median": 60092.0, "mean": 54268.6, "min": 27869.0, "max": 63210.0, "stdev": 14821.94 }, "cached": { "median": 44251.0, "mean": 37947.2, "min": 12775.0, "max": 44373.0, "stdev": 14072.41 }, "output": { "median": 830.0, "mean": 744.6, "min": 512.0, "max": 859.0, "stdev": 147.12 } }, "tokens": { "median": 104803.0, "mean": 92960.4, "min": 41156.0, "max": 108400.0, "stdev": 28997.32 }, "duration_ms": { "median": 21087.37, "mean": 19837.67, "min": 12099.9, "max": 23812.58, "stdev": 4640.12 }, "time_to_first_action_ms": { "median": 4947.27, "mean": 5176.38, "min": 4537.25, "max": 6361.69, "stdev": 698.32 }, "agent_turns": { "median": 5.0, "mean": 4.6, "min": 3.0, "max": 5.0, "stdev": 0.89 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 3.6, "min": 2.0, "max": 4.0, "stdev": 0.89 }, "client": { "median": 4.0, "mean": 3.6, "min": 2.0, "max": 4.0, "stdev": 0.89 } }, "payload_tokens": { "median": 8720.0, "mean": 8910.8, "min": 7830.0, "max": 11153.0, "stdev": 1311.15 }, "text_tokens": { "median": 8720.0, "mean": 8910.8, "min": 7830.0, "max": 11153.0, "stdev": 1311.15 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8720.0, "mean": 8910.8, "min": 7830.0, "max": 11153.0, "stdev": 1311.15 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 34757.0, "mean": 35238.2, "min": 31526.0, "max": 42737.0, "stdev": 4428.65 }, "client_tokens": { "input": { "median": 60092.0, "mean": 54268.6, "min": 27869.0, "max": 63210.0, "stdev": 14821.94 }, "cached": { "median": 44251.0, "mean": 37947.2, "min": 12775.0, "max": 44373.0, "stdev": 14072.41 }, "output": { "median": 830.0, "mean": 744.6, "min": 512.0, "max": 859.0, "stdev": 147.12 } }, "tokens": { "median": 104803.0, "mean": 92960.4, "min": 41156.0, "max": 108400.0, "stdev": 28997.32 }, "duration_ms": { "median": 21087.37, "mean": 19837.67, "min": 12099.9, "max": 23812.58, "stdev": 4640.12 }, "time_to_first_action_ms": { "median": 4947.27, "mean": 5176.38, "min": 4537.25, "max": 6361.69, "stdev": 698.32 }, "agent_turns": { "median": 5.0, "mean": 4.6, "min": 3.0, "max": 5.0, "stdev": 0.89 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "The page title confirms the exact title. The last book listed on the last page (page 50) of the catalogue is:\n\n**ANSWER: 1,000 Places to See Before You Die**", "answer_format_ok": false, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5148.98, "payload_tokens": 8720, "text_tokens": 8720, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8720, "image_token_share": 0.0, "payload_bytes": 34757, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 60457, "tokens_cached": 44373, "tokens_out": 830, "tokens": 105660, "duration_ms": 22786.9, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.3452215, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "The page title confirms the full title. The last book listed on page 50 (the last page of the catalogue) is:\n\n**ANSWER: 1,000 Places to See Before You Die**", "answer_format_ok": false, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 4886.7, "payload_tokens": 8131, "text_tokens": 8131, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8131, "image_token_share": 0.0, "payload_bytes": 32414, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 59715, "tokens_cached": 44251, "tokens_out": 837, "tokens": 104803, "duration_ms": 21087.37, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.3416255, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 6361.69, "payload_tokens": 8720, "text_tokens": 8720, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8720, "image_token_share": 0.0, "payload_bytes": 34757, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 60092, "tokens_cached": 44006, "tokens_out": 685, "tokens": 104783, "duration_ms": 23812.58, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.339588, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 4537.25, "payload_tokens": 7830, "text_tokens": 7830, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 7830, "image_token_share": 0.0, "payload_bytes": 31526, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 27869, "tokens_cached": 12775, "tokens_out": 512, "tokens": 41156, "duration_ms": 12099.9, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1585325, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 4947.27, "payload_tokens": 11153, "text_tokens": 11153, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 11153, "image_token_share": 0.0, "payload_bytes": 42737, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 63210, "tokens_cached": 44331, "tokens_out": 859, "tokens": 108400, "duration_ms": 19401.61, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.3596905, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "playwright", "task": "book-upc-table", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1218.0, "mean": 1029.4, "min": 275.0, "max": 1218.0, "stdev": 421.72 }, "client_tokens": { "input": { "median": 20058.0, "mean": 19856.0, "min": 19032.0, "max": 20079.0, "stdev": 460.74 }, "cached": { "median": 12748.0, "mean": 12751.6, "min": 12741.0, "max": 12764.0, "stdev": 9.24 }, "output": { "median": 327.0, "mean": 337.2, "min": 308.0, "max": 378.0, "stdev": 27.85 } }, "tokens": { "median": 33114.0, "mean": 32944.8, "min": 32142.0, "max": 33185.0, "stdev": 449.95 }, "duration_ms": { "median": 8852.71, "mean": 8610.74, "min": 7242.28, "max": 9779.06, "stdev": 1152.29 }, "time_to_first_action_ms": { "median": 4836.0, "mean": 4756.84, "min": 4205.42, "max": 5258.09, "stdev": 435.08 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1218.0, "mean": 1029.4, "min": 275.0, "max": 1218.0, "stdev": 421.72 }, "text_tokens": { "median": 1218.0, "mean": 1029.4, "min": 275.0, "max": 1218.0, "stdev": 421.72 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1218.0, "mean": 1029.4, "min": 275.0, "max": 1218.0, "stdev": 421.72 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 4269.0, "mean": 3583.6, "min": 842.0, "max": 4269.0, "stdev": 1532.6 }, "client_tokens": { "input": { "median": 20058.0, "mean": 19856.0, "min": 19032.0, "max": 20079.0, "stdev": 460.74 }, "cached": { "median": 12748.0, "mean": 12751.6, "min": 12741.0, "max": 12764.0, "stdev": 9.24 }, "output": { "median": 327.0, "mean": 337.2, "min": 308.0, "max": 378.0, "stdev": 27.85 } }, "tokens": { "median": 33114.0, "mean": 32944.8, "min": 32142.0, "max": 33185.0, "stdev": 449.95 }, "duration_ms": { "median": 8852.71, "mean": 8610.74, "min": 7242.28, "max": 9779.06, "stdev": 1152.29 }, "time_to_first_action_ms": { "median": 4836.0, "mean": 4756.84, "min": 4205.42, "max": 5258.09, "stdev": 435.08 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1218.0, "mean": 1029.4, "min": 275.0, "max": 1218.0, "stdev": 421.72 }, "text_tokens": { "median": 1218.0, "mean": 1029.4, "min": 275.0, "max": 1218.0, "stdev": 421.72 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1218.0, "mean": 1029.4, "min": 275.0, "max": 1218.0, "stdev": 421.72 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 4269.0, "mean": 3583.6, "min": 842.0, "max": 4269.0, "stdev": 1532.6 }, "client_tokens": { "input": { "median": 20058.0, "mean": 19856.0, "min": 19032.0, "max": 20079.0, "stdev": 460.74 }, "cached": { "median": 12748.0, "mean": 12751.6, "min": 12741.0, "max": 12764.0, "stdev": 9.24 }, "output": { "median": 327.0, "mean": 337.2, "min": 308.0, "max": 378.0, "stdev": 27.85 } }, "tokens": { "median": 33114.0, "mean": 32944.8, "min": 32142.0, "max": 33185.0, "stdev": 449.95 }, "duration_ms": { "median": 8852.71, "mean": 8610.74, "min": 7242.28, "max": 9779.06, "stdev": 1152.29 }, "time_to_first_action_ms": { "median": 4836.0, "mean": 4756.84, "min": 4205.42, "max": 5258.09, "stdev": 435.08 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 4836.0, "payload_tokens": 1218, "text_tokens": 1218, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1218, "image_token_share": 0.0, "payload_bytes": 4269, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 20059, "tokens_cached": 12748, "tokens_out": 378, "tokens": 33185, "duration_ms": 9779.06, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.116119, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5054.72, "payload_tokens": 1218, "text_tokens": 1218, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1218, "image_token_share": 0.0, "payload_bytes": 4269, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 20079, "tokens_cached": 12764, "tokens_out": 327, "tokens": 33170, "duration_ms": 8852.71, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.114952, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5258.09, "payload_tokens": 1218, "text_tokens": 1218, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1218, "image_token_share": 0.0, "payload_bytes": 4269, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 20058, "tokens_cached": 12747, "tokens_out": 308, "tokens": 33113, "duration_ms": 9593.31, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1143635, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 4429.95, "payload_tokens": 275, "text_tokens": 275, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 275, "image_token_share": 0.0, "payload_bytes": 842, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 19032, "tokens_cached": 12758, "tokens_out": 352, "tokens": 32142, "duration_ms": 7586.32, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.11033899999999999, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 4205.42, "payload_tokens": 1218, "text_tokens": 1218, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1218, "image_token_share": 0.0, "payload_bytes": 4269, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 20052, "tokens_cached": 12741, "tokens_out": 321, "tokens": 33114, "duration_ms": 7242.28, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1146555, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "playwright", "task": "travel-stock-drilldown", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 4.2, "min": 4.0, "max": 5.0, "stdev": 0.45 }, "client": { "median": 4.0, "mean": 4.2, "min": 4.0, "max": 5.0, "stdev": 0.45 } }, "payload_tokens": { "median": 815.0, "mean": 820.6, "min": 815.0, "max": 841.0, "stdev": 11.44 }, "client_tokens": { "input": { "median": 33522.0, "mean": 34906.2, "min": 33446.0, "max": 40593.0, "stdev": 3179.24 }, "cached": { "median": 26562.0, "mean": 27940.0, "min": 26491.0, "max": 33590.0, "stdev": 3158.64 }, "output": { "median": 486.0, "mean": 486.6, "min": 441.0, "max": 551.0, "stdev": 40.8 } }, "tokens": { "median": 60571.0, "mean": 63332.8, "min": 60378.0, "max": 74734.0, "stdev": 6374.11 }, "duration_ms": { "median": 11170.9, "mean": 11292.55, "min": 9490.87, "max": 13233.43, "stdev": 1603.33 }, "time_to_first_action_ms": { "median": 4460.36, "mean": 4613.66, "min": 4133.27, "max": 5286.5, "stdev": 433.95 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.2, "min": 4.0, "max": 5.0, "stdev": 0.45 }, "client": { "median": 4.0, "mean": 4.2, "min": 4.0, "max": 5.0, "stdev": 0.45 } }, "payload_tokens": { "median": 815.0, "mean": 820.6, "min": 815.0, "max": 841.0, "stdev": 11.44 }, "text_tokens": { "median": 815.0, "mean": 820.6, "min": 815.0, "max": 841.0, "stdev": 11.44 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 815.0, "mean": 820.6, "min": 815.0, "max": 841.0, "stdev": 11.44 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2793.0, "mean": 2813.2, "min": 2793.0, "max": 2891.0, "stdev": 43.51 }, "client_tokens": { "input": { "median": 33522.0, "mean": 34906.2, "min": 33446.0, "max": 40593.0, "stdev": 3179.24 }, "cached": { "median": 26562.0, "mean": 27940.0, "min": 26491.0, "max": 33590.0, "stdev": 3158.64 }, "output": { "median": 486.0, "mean": 486.6, "min": 441.0, "max": 551.0, "stdev": 40.8 } }, "tokens": { "median": 60571.0, "mean": 63332.8, "min": 60378.0, "max": 74734.0, "stdev": 6374.11 }, "duration_ms": { "median": 11170.9, "mean": 11292.55, "min": 9490.87, "max": 13233.43, "stdev": 1603.33 }, "time_to_first_action_ms": { "median": 4460.36, "mean": 4613.66, "min": 4133.27, "max": 5286.5, "stdev": 433.95 }, "agent_turns": { "median": 5.0, "mean": 5.2, "min": 5.0, "max": 6.0, "stdev": 0.45 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.2, "min": 4.0, "max": 5.0, "stdev": 0.45 }, "client": { "median": 4.0, "mean": 4.2, "min": 4.0, "max": 5.0, "stdev": 0.45 } }, "payload_tokens": { "median": 815.0, "mean": 820.6, "min": 815.0, "max": 841.0, "stdev": 11.44 }, "text_tokens": { "median": 815.0, "mean": 820.6, "min": 815.0, "max": 841.0, "stdev": 11.44 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 815.0, "mean": 820.6, "min": 815.0, "max": 841.0, "stdev": 11.44 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2793.0, "mean": 2813.2, "min": 2793.0, "max": 2891.0, "stdev": 43.51 }, "client_tokens": { "input": { "median": 33522.0, "mean": 34906.2, "min": 33446.0, "max": 40593.0, "stdev": 3179.24 }, "cached": { "median": 26562.0, "mean": 27940.0, "min": 26491.0, "max": 33590.0, "stdev": 3158.64 }, "output": { "median": 486.0, "mean": 486.6, "min": 441.0, "max": 551.0, "stdev": 40.8 } }, "tokens": { "median": 60571.0, "mean": 63332.8, "min": 60378.0, "max": 74734.0, "stdev": 6374.11 }, "duration_ms": { "median": 11170.9, "mean": 11292.55, "min": 9490.87, "max": 13233.43, "stdev": 1603.33 }, "time_to_first_action_ms": { "median": 4460.36, "mean": 4613.66, "min": 4133.27, "max": 5286.5, "stdev": 433.95 }, "agent_turns": { "median": 5.0, "mean": 5.2, "min": 5.0, "max": 6.0, "stdev": 0.45 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 4443.28, "payload_tokens": 815, "text_tokens": 815, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 815, "image_token_share": 0.0, "payload_bytes": 2793, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 33447, "tokens_cached": 26494, "tokens_out": 466, "tokens": 60407, "duration_ms": 11170.9, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.192132, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 6, "time_to_first_action_ms": 4744.88, "payload_tokens": 841, "text_tokens": 841, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 841, "image_token_share": 0.0, "payload_bytes": 2891, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 40593, "tokens_cached": 33590, "tokens_out": 551, "tokens": 74734, "duration_ms": 13233.43, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.23353500000000002, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5286.5, "payload_tokens": 817, "text_tokens": 817, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 817, "image_token_share": 0.0, "payload_bytes": 2796, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 33523, "tokens_cached": 26562, "tokens_out": 486, "tokens": 60571, "duration_ms": 12559.43, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.19304599999999997, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 4133.27, "payload_tokens": 815, "text_tokens": 815, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 815, "image_token_share": 0.0, "payload_bytes": 2793, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 33446, "tokens_cached": 26491, "tokens_out": 441, "tokens": 60378, "duration_ms": 9490.87, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.19150050000000002, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 4460.36, "payload_tokens": 815, "text_tokens": 815, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 815, "image_token_share": 0.0, "payload_bytes": 2793, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 33522, "tokens_cached": 26563, "tokens_out": 489, "tokens": 60574, "duration_ms": 10008.11, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.19311649999999997, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "playwright", "task": "quote-form-submit", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 8.0, "mean": 8.8, "min": 6.0, "max": 14.0, "stdev": 3.03 }, "client": { "median": 8.0, "mean": 8.8, "min": 6.0, "max": 14.0, "stdev": 3.03 } }, "payload_tokens": { "median": 904.0, "mean": 1075.8, "min": 862.0, "max": 1805.0, "stdev": 408.04 }, "client_tokens": { "input": { "median": 49432.0, "mean": 57524.2, "min": 41839.0, "max": 97491.0, "stdev": 22582.65 }, "cached": { "median": 42340.0, "mean": 50242.6, "min": 34817.0, "max": 89379.0, "stdev": 22119.05 }, "output": { "median": 991.0, "mean": 1172.4, "min": 822.0, "max": 2065.0, "stdev": 504.54 } }, "tokens": { "median": 92745.0, "mean": 108939.2, "min": 77478.0, "max": 188935.0, "stdev": 45206.05 }, "duration_ms": { "median": 18829.2, "mean": 24457.3, "min": 17107.29, "max": 46196.88, "stdev": 12366.63 }, "time_to_first_action_ms": { "median": 4477.39, "mean": 4781.44, "min": 4422.82, "max": 5662.13, "stdev": 530.8 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 8.0, "mean": 8.8, "min": 6.0, "max": 14.0, "stdev": 3.03 }, "client": { "median": 8.0, "mean": 8.8, "min": 6.0, "max": 14.0, "stdev": 3.03 } }, "payload_tokens": { "median": 904.0, "mean": 1075.8, "min": 862.0, "max": 1805.0, "stdev": 408.04 }, "text_tokens": { "median": 904.0, "mean": 1075.8, "min": 862.0, "max": 1805.0, "stdev": 408.04 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 904.0, "mean": 1075.8, "min": 862.0, "max": 1805.0, "stdev": 408.04 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2958.0, "mean": 3588.8, "min": 2773.0, "max": 6297.0, "stdev": 1516.05 }, "client_tokens": { "input": { "median": 49432.0, "mean": 57524.2, "min": 41839.0, "max": 97491.0, "stdev": 22582.65 }, "cached": { "median": 42340.0, "mean": 50242.6, "min": 34817.0, "max": 89379.0, "stdev": 22119.05 }, "output": { "median": 991.0, "mean": 1172.4, "min": 822.0, "max": 2065.0, "stdev": 504.54 } }, "tokens": { "median": 92745.0, "mean": 108939.2, "min": 77478.0, "max": 188935.0, "stdev": 45206.05 }, "duration_ms": { "median": 18829.2, "mean": 24457.3, "min": 17107.29, "max": 46196.88, "stdev": 12366.63 }, "time_to_first_action_ms": { "median": 4477.39, "mean": 4781.44, "min": 4422.82, "max": 5662.13, "stdev": 530.8 }, "agent_turns": { "median": 9.0, "mean": 9.8, "min": 7.0, "max": 15.0, "stdev": 3.03 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 8.0, "mean": 8.8, "min": 6.0, "max": 14.0, "stdev": 3.03 }, "client": { "median": 8.0, "mean": 8.8, "min": 6.0, "max": 14.0, "stdev": 3.03 } }, "payload_tokens": { "median": 904.0, "mean": 1075.8, "min": 862.0, "max": 1805.0, "stdev": 408.04 }, "text_tokens": { "median": 904.0, "mean": 1075.8, "min": 862.0, "max": 1805.0, "stdev": 408.04 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 904.0, "mean": 1075.8, "min": 862.0, "max": 1805.0, "stdev": 408.04 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2958.0, "mean": 3588.8, "min": 2773.0, "max": 6297.0, "stdev": 1516.05 }, "client_tokens": { "input": { "median": 49432.0, "mean": 57524.2, "min": 41839.0, "max": 97491.0, "stdev": 22582.65 }, "cached": { "median": 42340.0, "mean": 50242.6, "min": 34817.0, "max": 89379.0, "stdev": 22119.05 }, "output": { "median": 991.0, "mean": 1172.4, "min": 822.0, "max": 2065.0, "stdev": 504.54 } }, "tokens": { "median": 92745.0, "mean": 108939.2, "min": 77478.0, "max": 188935.0, "stdev": 45206.05 }, "duration_ms": { "median": 18829.2, "mean": 24457.3, "min": 17107.29, "max": 46196.88, "stdev": 12366.63 }, "time_to_first_action_ms": { "median": 4477.39, "mean": 4781.44, "min": 4422.82, "max": 5662.13, "stdev": 530.8 }, "agent_turns": { "median": 9.0, "mean": 9.8, "min": 7.0, "max": 15.0, "stdev": 3.03 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 8, "client_tool_calls": 8, "client_tool_calls_total": 8, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 9, "time_to_first_action_ms": 4422.82, "payload_tokens": 904, "text_tokens": 904, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 904, "image_token_share": 0.0, "payload_bytes": 2958, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 49398, "tokens_cached": 42306, "tokens_out": 991, "tokens": 92695, "duration_ms": 17107.29, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.292918, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 8, "client_tool_calls": 8, "client_tool_calls_total": 8, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 9, "time_to_first_action_ms": 4441.63, "payload_tokens": 904, "text_tokens": 904, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 904, "image_token_share": 0.0, "payload_bytes": 2958, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 49461, "tokens_cached": 42371, "tokens_out": 1011, "tokens": 92843, "duration_ms": 17333.87, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.2937655, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "The form was filled and submitted successfully. The confirmation code shown on the page is:\n\n**ANSWER: OWL-7Q42-XK**", "answer_format_ok": false, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 8, "client_tool_calls": 8, "client_tool_calls_total": 8, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 9, "time_to_first_action_ms": 5662.13, "payload_tokens": 904, "text_tokens": 904, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 904, "image_token_share": 0.0, "payload_bytes": 2958, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 49432, "tokens_cached": 42340, "tokens_out": 973, "tokens": 92745, "duration_ms": 22819.28, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.292655, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 14, "client_tool_calls": 14, "client_tool_calls_total": 14, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 15, "time_to_first_action_ms": 4477.39, "payload_tokens": 1805, "text_tokens": 1805, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1805, "image_token_share": 0.0, "payload_bytes": 6297, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 97491, "tokens_cached": 89379, "tokens_out": 2065, "tokens": 188935, "duration_ms": 46196.88, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.5837695000000002, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 6, "client_tool_calls": 6, "client_tool_calls_total": 6, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 7, "time_to_first_action_ms": 4903.24, "payload_tokens": 862, "text_tokens": 862, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 862, "image_token_share": 0.0, "payload_bytes": 2773, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 41839, "tokens_cached": 34817, "tokens_out": 822, "tokens": 77478, "duration_ms": 18829.2, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.2471535, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "playwright", "task": "restricted-figure", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 19397.0, "mean": 19396.6, "min": 19388.0, "max": 19405.0, "stdev": 6.27 }, "cached": { "median": 12789.0, "mean": 12789.4, "min": 12782.0, "max": 12797.0, "stdev": 5.86 }, "output": { "median": 543.0, "mean": 550.6, "min": 509.0, "max": 635.0, "stdev": 49.55 } }, "tokens": { "median": 32729.0, "mean": 32736.6, "min": 32691.0, "max": 32815.0, "stdev": 49.14 }, "duration_ms": { "median": 11042.4, "mean": 11324.62, "min": 9516.77, "max": 14683.89, "stdev": 2004.63 }, "time_to_first_action_ms": { "median": 5149.35, "mean": 5412.44, "min": 4593.19, "max": 7195.33, "stdev": 1027.45 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "text_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1747.0, "mean": 1747.0, "min": 1747.0, "max": 1747.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 19397.0, "mean": 19396.6, "min": 19388.0, "max": 19405.0, "stdev": 6.27 }, "cached": { "median": 12789.0, "mean": 12789.4, "min": 12782.0, "max": 12797.0, "stdev": 5.86 }, "output": { "median": 543.0, "mean": 550.6, "min": 509.0, "max": 635.0, "stdev": 49.55 } }, "tokens": { "median": 32729.0, "mean": 32736.6, "min": 32691.0, "max": 32815.0, "stdev": 49.14 }, "duration_ms": { "median": 11042.4, "mean": 11324.62, "min": 9516.77, "max": 14683.89, "stdev": 2004.63 }, "time_to_first_action_ms": { "median": 5149.35, "mean": 5412.44, "min": 4593.19, "max": 7195.33, "stdev": 1027.45 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "text_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1747.0, "mean": 1747.0, "min": 1747.0, "max": 1747.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 19397.0, "mean": 19396.6, "min": 19388.0, "max": 19405.0, "stdev": 6.27 }, "cached": { "median": 12789.0, "mean": 12789.4, "min": 12782.0, "max": 12797.0, "stdev": 5.86 }, "output": { "median": 543.0, "mean": 550.6, "min": 509.0, "max": 635.0, "stdev": 49.55 } }, "tokens": { "median": 32729.0, "mean": 32736.6, "min": 32691.0, "max": 32815.0, "stdev": 49.14 }, "duration_ms": { "median": 11042.4, "mean": 11324.62, "min": 9516.77, "max": 14683.89, "stdev": 2004.63 }, "time_to_first_action_ms": { "median": 5149.35, "mean": 5412.44, "min": 4593.19, "max": 7195.33, "stdev": 1027.45 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 4891.67, "payload_tokens": 562, "text_tokens": 562, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 562, "image_token_share": 0.0, "payload_bytes": 1747, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 19394, "tokens_cached": 12786, "tokens_out": 635, "tokens": 32815, "duration_ms": 10126.09, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.11923799999999998, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 4593.19, "payload_tokens": 562, "text_tokens": 562, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 562, "image_token_share": 0.0, "payload_bytes": 1747, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 19397, "tokens_cached": 12789, "tokens_out": 543, "tokens": 32729, "duration_ms": 9516.77, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1169545, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 7195.33, "payload_tokens": 562, "text_tokens": 562, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 562, "image_token_share": 0.0, "payload_bytes": 1747, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 19405, "tokens_cached": 12797, "tokens_out": 545, "tokens": 32747, "duration_ms": 14683.89, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1170485, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5149.35, "payload_tokens": 562, "text_tokens": 562, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 562, "image_token_share": 0.0, "payload_bytes": 1747, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 19388, "tokens_cached": 12782, "tokens_out": 521, "tokens": 32691, "duration_ms": 11253.94, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.11635599999999999, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5232.67, "payload_tokens": 562, "text_tokens": 562, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 562, "image_token_share": 0.0, "payload_bytes": 1747, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 19399, "tokens_cached": 12793, "tokens_out": 509, "tokens": 32701, "duration_ms": 11042.4, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1161165, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "chrome-devtools", "task": "cheapest-mystery-book", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 3.0, "max": 5.0, "stdev": 0.71 }, "client": { "median": 4.0, "mean": 4.0, "min": 3.0, "max": 5.0, "stdev": 0.71 } }, "payload_tokens": { "median": 14917.0, "mean": 13824.4, "min": 9485.0, "max": 14917.0, "stdev": 2425.84 }, "client_tokens": { "input": { "median": 74887.0, "mean": 74400.0, "min": 57546.0, "max": 89108.0, "stdev": 11205.71 }, "cached": { "median": 50503.0, "mean": 51253.0, "min": 33209.0, "max": 70858.0, "stdev": 13333.34 }, "output": { "median": 1334.0, "mean": 1420.8, "min": 1212.0, "max": 1786.0, "stdev": 244.32 } }, "tokens": { "median": 126937.0, "mean": 127073.8, "min": 91967.0, "max": 161300.0, "stdev": 24528.35 }, "duration_ms": { "median": 31752.86, "mean": 35839.34, "min": 26695.43, "max": 52223.08, "stdev": 10917.83 }, "time_to_first_action_ms": { "median": 6392.65, "mean": 7006.52, "min": 5958.01, "max": 8373.14, "stdev": 1196.67 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 3.0, "max": 5.0, "stdev": 0.71 }, "client": { "median": 4.0, "mean": 4.0, "min": 3.0, "max": 5.0, "stdev": 0.71 } }, "payload_tokens": { "median": 14917.0, "mean": 13824.4, "min": 9485.0, "max": 14917.0, "stdev": 2425.84 }, "text_tokens": { "median": 14917.0, "mean": 13824.4, "min": 9485.0, "max": 14917.0, "stdev": 2425.84 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 14917.0, "mean": 13824.4, "min": 9485.0, "max": 14917.0, "stdev": 2425.84 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 44340.0, "mean": 41075.8, "min": 28125.0, "max": 44340.0, "stdev": 7239.86 }, "client_tokens": { "input": { "median": 74887.0, "mean": 74400.0, "min": 57546.0, "max": 89108.0, "stdev": 11205.71 }, "cached": { "median": 50503.0, "mean": 51253.0, "min": 33209.0, "max": 70858.0, "stdev": 13333.34 }, "output": { "median": 1334.0, "mean": 1420.8, "min": 1212.0, "max": 1786.0, "stdev": 244.32 } }, "tokens": { "median": 126937.0, "mean": 127073.8, "min": 91967.0, "max": 161300.0, "stdev": 24528.35 }, "duration_ms": { "median": 31752.86, "mean": 35839.34, "min": 26695.43, "max": 52223.08, "stdev": 10917.83 }, "time_to_first_action_ms": { "median": 6392.65, "mean": 7006.52, "min": 5958.01, "max": 8373.14, "stdev": 1196.67 }, "agent_turns": { "median": 5.0, "mean": 5.0, "min": 4.0, "max": 6.0, "stdev": 0.71 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 3.0, "max": 5.0, "stdev": 0.71 }, "client": { "median": 4.0, "mean": 4.0, "min": 3.0, "max": 5.0, "stdev": 0.71 } }, "payload_tokens": { "median": 14917.0, "mean": 13824.4, "min": 9485.0, "max": 14917.0, "stdev": 2425.84 }, "text_tokens": { "median": 14917.0, "mean": 13824.4, "min": 9485.0, "max": 14917.0, "stdev": 2425.84 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 14917.0, "mean": 13824.4, "min": 9485.0, "max": 14917.0, "stdev": 2425.84 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 44340.0, "mean": 41075.8, "min": 28125.0, "max": 44340.0, "stdev": 7239.86 }, "client_tokens": { "input": { "median": 74887.0, "mean": 74400.0, "min": 57546.0, "max": 89108.0, "stdev": 11205.71 }, "cached": { "median": 50503.0, "mean": 51253.0, "min": 33209.0, "max": 70858.0, "stdev": 13333.34 }, "output": { "median": 1334.0, "mean": 1420.8, "min": 1212.0, "max": 1786.0, "stdev": 244.32 } }, "tokens": { "median": 126937.0, "mean": 127073.8, "min": 91967.0, "max": 161300.0, "stdev": 24528.35 }, "duration_ms": { "median": 31752.86, "mean": 35839.34, "min": 26695.43, "max": 52223.08, "stdev": 10917.83 }, "time_to_first_action_ms": { "median": 6392.65, "mean": 7006.52, "min": 5958.01, "max": 8373.14, "stdev": 1196.67 }, "agent_turns": { "median": 5.0, "mean": 5.0, "min": 4.0, "max": 6.0, "stdev": 0.71 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5958.01, "payload_tokens": 14917, "text_tokens": 14917, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 14917, "image_token_share": 0.0, "payload_bytes": 44340, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 74858, "tokens_cached": 50474, "tokens_out": 1225, "tokens": 126557, "duration_ms": 27083.29, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.430152, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 3, "client_tool_calls": 3, "client_tool_calls_total": 3, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 4, "time_to_first_action_ms": 6072.61, "payload_tokens": 14886, "text_tokens": 14886, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 14886, "image_token_share": 0.0, "payload_bytes": 44234, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 57546, "tokens_cached": 33209, "tokens_out": 1212, "tokens": 91967, "duration_ms": 26695.43, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.33463449999999995, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 6, "time_to_first_action_ms": 8373.14, "payload_tokens": 9485, "text_tokens": 9485, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 9485, "image_token_share": 0.0, "payload_bytes": 28125, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 89108, "tokens_cached": 70858, "tokens_out": 1334, "tokens": 161300, "duration_ms": 41442.02, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.514319, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 8236.18, "payload_tokens": 14917, "text_tokens": 14917, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 14917, "image_token_share": 0.0, "payload_bytes": 44340, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 75601, "tokens_cached": 51221, "tokens_out": 1786, "tokens": 128608, "duration_ms": 52223.08, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.4482655, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 6392.65, "payload_tokens": 14917, "text_tokens": 14917, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 14917, "image_token_share": 0.0, "payload_bytes": 44340, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 74887, "tokens_cached": 50503, "tokens_out": 1547, "tokens": 126937, "duration_ms": 31752.86, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.4383615, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "chrome-devtools", "task": "catalogue-last-page", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 8517.0, "mean": 8517.0, "min": 8517.0, "max": 8517.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 32424.0, "mean": 32417.6, "min": 32402.0, "max": 32425.0, "stdev": 10.01 }, "cached": { "median": 15397.0, "mean": 15392.6, "min": 15379.0, "max": 15400.0, "stdev": 8.96 }, "output": { "median": 555.0, "mean": 656.0, "min": 543.0, "max": 858.0, "stdev": 148.36 } }, "tokens": { "median": 48378.0, "mean": 48466.2, "min": 48353.0, "max": 48683.0, "stdev": 146.27 }, "duration_ms": { "median": 17248.86, "mean": 18216.4, "min": 13621.41, "max": 26935.16, "stdev": 5101.65 }, "time_to_first_action_ms": { "median": 5836.08, "mean": 6170.09, "min": 5285.73, "max": 8388.18, "stdev": 1271.87 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 8517.0, "mean": 8517.0, "min": 8517.0, "max": 8517.0, "stdev": 0.0 }, "text_tokens": { "median": 8517.0, "mean": 8517.0, "min": 8517.0, "max": 8517.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8517.0, "mean": 8517.0, "min": 8517.0, "max": 8517.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 25112.0, "mean": 25112.0, "min": 25112.0, "max": 25112.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 32424.0, "mean": 32417.6, "min": 32402.0, "max": 32425.0, "stdev": 10.01 }, "cached": { "median": 15397.0, "mean": 15392.6, "min": 15379.0, "max": 15400.0, "stdev": 8.96 }, "output": { "median": 555.0, "mean": 656.0, "min": 543.0, "max": 858.0, "stdev": 148.36 } }, "tokens": { "median": 48378.0, "mean": 48466.2, "min": 48353.0, "max": 48683.0, "stdev": 146.27 }, "duration_ms": { "median": 17248.86, "mean": 18216.4, "min": 13621.41, "max": 26935.16, "stdev": 5101.65 }, "time_to_first_action_ms": { "median": 5836.08, "mean": 6170.09, "min": 5285.73, "max": 8388.18, "stdev": 1271.87 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 8517.0, "mean": 8517.0, "min": 8517.0, "max": 8517.0, "stdev": 0.0 }, "text_tokens": { "median": 8517.0, "mean": 8517.0, "min": 8517.0, "max": 8517.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8517.0, "mean": 8517.0, "min": 8517.0, "max": 8517.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 25112.0, "mean": 25112.0, "min": 25112.0, "max": 25112.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 32424.0, "mean": 32417.6, "min": 32402.0, "max": 32425.0, "stdev": 10.01 }, "cached": { "median": 15397.0, "mean": 15392.6, "min": 15379.0, "max": 15400.0, "stdev": 8.96 }, "output": { "median": 555.0, "mean": 656.0, "min": 543.0, "max": 858.0, "stdev": 148.36 } }, "tokens": { "median": 48378.0, "mean": 48466.2, "min": 48353.0, "max": 48683.0, "stdev": 146.27 }, "duration_ms": { "median": 17248.86, "mean": 18216.4, "min": 13621.41, "max": 26935.16, "stdev": 5101.65 }, "time_to_first_action_ms": { "median": 5836.08, "mean": 6170.09, "min": 5285.73, "max": 8388.18, "stdev": 1271.87 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5950.62, "payload_tokens": 8517, "text_tokens": 8517, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8517, "image_token_share": 0.0, "payload_bytes": 25112, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 32413, "tokens_cached": 15388, "tokens_out": 552, "tokens": 48353, "duration_ms": 17366.22, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.18355900000000003, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5836.08, "payload_tokens": 8517, "text_tokens": 8517, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8517, "image_token_share": 0.0, "payload_bytes": 25112, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 32424, "tokens_cached": 15399, "tokens_out": 555, "tokens": 48378, "duration_ms": 15910.36, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.18369449999999998, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 8388.18, "payload_tokens": 8517, "text_tokens": 8517, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8517, "image_token_share": 0.0, "payload_bytes": 25112, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 32425, "tokens_cached": 15400, "tokens_out": 858, "tokens": 48683, "duration_ms": 26935.16, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.19127499999999997, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5285.73, "payload_tokens": 8517, "text_tokens": 8517, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8517, "image_token_share": 0.0, "payload_bytes": 25112, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 32424, "tokens_cached": 15397, "tokens_out": 543, "tokens": 48364, "duration_ms": 13621.41, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.18339350000000001, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5389.82, "payload_tokens": 8517, "text_tokens": 8517, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8517, "image_token_share": 0.0, "payload_bytes": 25112, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 32402, "tokens_cached": 15379, "tokens_out": 772, "tokens": 48553, "duration_ms": 17248.86, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1889995, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "chrome-devtools", "task": "book-upc-table", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 23882.0, "mean": 23884.0, "min": 23870.0, "max": 23906.0, "stdev": 13.71 }, "cached": { "median": 15377.0, "mean": 15377.4, "min": 15361.0, "max": 15397.0, "stdev": 13.74 }, "output": { "median": 383.0, "mean": 396.0, "min": 345.0, "max": 441.0, "stdev": 42.49 } }, "tokens": { "median": 39631.0, "mean": 39657.4, "min": 39576.0, "max": 39744.0, "stdev": 67.54 }, "duration_ms": { "median": 9564.73, "mean": 9871.75, "min": 9360.66, "max": 10744.07, "stdev": 593.56 }, "time_to_first_action_ms": { "median": 5285.35, "mean": 5519.08, "min": 5113.04, "max": 6161.08, "stdev": 477.84 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "text_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 3512.0, "mean": 3512.0, "min": 3512.0, "max": 3512.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 23882.0, "mean": 23884.0, "min": 23870.0, "max": 23906.0, "stdev": 13.71 }, "cached": { "median": 15377.0, "mean": 15377.4, "min": 15361.0, "max": 15397.0, "stdev": 13.74 }, "output": { "median": 383.0, "mean": 396.0, "min": 345.0, "max": 441.0, "stdev": 42.49 } }, "tokens": { "median": 39631.0, "mean": 39657.4, "min": 39576.0, "max": 39744.0, "stdev": 67.54 }, "duration_ms": { "median": 9564.73, "mean": 9871.75, "min": 9360.66, "max": 10744.07, "stdev": 593.56 }, "time_to_first_action_ms": { "median": 5285.35, "mean": 5519.08, "min": 5113.04, "max": 6161.08, "stdev": 477.84 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "text_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 3512.0, "mean": 3512.0, "min": 3512.0, "max": 3512.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 23882.0, "mean": 23884.0, "min": 23870.0, "max": 23906.0, "stdev": 13.71 }, "cached": { "median": 15377.0, "mean": 15377.4, "min": 15361.0, "max": 15397.0, "stdev": 13.74 }, "output": { "median": 383.0, "mean": 396.0, "min": 345.0, "max": 441.0, "stdev": 42.49 } }, "tokens": { "median": 39631.0, "mean": 39657.4, "min": 39576.0, "max": 39744.0, "stdev": 67.54 }, "duration_ms": { "median": 9564.73, "mean": 9871.75, "min": 9360.66, "max": 10744.07, "stdev": 593.56 }, "time_to_first_action_ms": { "median": 5285.35, "mean": 5519.08, "min": 5113.04, "max": 6161.08, "stdev": 477.84 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5142.7, "payload_tokens": 1085, "text_tokens": 1085, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1085, "image_token_share": 0.0, "payload_bytes": 3512, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 23870, "tokens_cached": 15361, "tokens_out": 345, "tokens": 39576, "duration_ms": 9462.97, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.13565549999999998, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 6161.08, "payload_tokens": 1085, "text_tokens": 1085, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1085, "image_token_share": 0.0, "payload_bytes": 3512, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 23882, "tokens_cached": 15377, "tokens_out": 372, "tokens": 39631, "duration_ms": 10226.33, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1363985, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5893.24, "payload_tokens": 1085, "text_tokens": 1085, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1085, "image_token_share": 0.0, "payload_bytes": 3512, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 23876, "tokens_cached": 15369, "tokens_out": 383, "tokens": 39628, "duration_ms": 10744.07, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1366395, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5113.04, "payload_tokens": 1085, "text_tokens": 1085, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1085, "image_token_share": 0.0, "payload_bytes": 3512, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 23906, "tokens_cached": 15397, "tokens_out": 441, "tokens": 39744, "duration_ms": 9360.66, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.13825349999999997, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5285.35, "payload_tokens": 1085, "text_tokens": 1085, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1085, "image_token_share": 0.0, "payload_bytes": 3512, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 23886, "tokens_cached": 15383, "tokens_out": 439, "tokens": 39708, "duration_ms": 9564.73, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1380965, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "chrome-devtools", "task": "travel-stock-drilldown", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 61328.0, "mean": 61329.2, "min": 61265.0, "max": 61405.0, "stdev": 53.34 }, "cached": { "median": 44067.0, "mean": 44069.0, "min": 44006.0, "max": 44142.0, "stdev": 51.94 }, "output": { "median": 541.0, "mean": 528.4, "min": 480.0, "max": 573.0, "stdev": 37.07 } }, "tokens": { "median": 105941.0, "mean": 105926.6, "min": 105751.0, "max": 106120.0, "stdev": 141.22 }, "duration_ms": { "median": 15124.71, "mean": 15404.26, "min": 13532.85, "max": 17000.18, "stdev": 1409.32 }, "time_to_first_action_ms": { "median": 5432.01, "mean": 5748.87, "min": 5305.99, "max": 6297.99, "stdev": 501.92 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "text_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 26962.0, "mean": 26962.0, "min": 26962.0, "max": 26962.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 61328.0, "mean": 61329.2, "min": 61265.0, "max": 61405.0, "stdev": 53.34 }, "cached": { "median": 44067.0, "mean": 44069.0, "min": 44006.0, "max": 44142.0, "stdev": 51.94 }, "output": { "median": 541.0, "mean": 528.4, "min": 480.0, "max": 573.0, "stdev": 37.07 } }, "tokens": { "median": 105941.0, "mean": 105926.6, "min": 105751.0, "max": 106120.0, "stdev": 141.22 }, "duration_ms": { "median": 15124.71, "mean": 15404.26, "min": 13532.85, "max": 17000.18, "stdev": 1409.32 }, "time_to_first_action_ms": { "median": 5432.01, "mean": 5748.87, "min": 5305.99, "max": 6297.99, "stdev": 501.92 }, "agent_turns": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "text_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 26962.0, "mean": 26962.0, "min": 26962.0, "max": 26962.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 61328.0, "mean": 61329.2, "min": 61265.0, "max": 61405.0, "stdev": 53.34 }, "cached": { "median": 44067.0, "mean": 44069.0, "min": 44006.0, "max": 44142.0, "stdev": 51.94 }, "output": { "median": 541.0, "mean": 528.4, "min": 480.0, "max": 573.0, "stdev": 37.07 } }, "tokens": { "median": 105941.0, "mean": 105926.6, "min": 105751.0, "max": 106120.0, "stdev": 141.22 }, "duration_ms": { "median": 15124.71, "mean": 15404.26, "min": 13532.85, "max": 17000.18, "stdev": 1409.32 }, "time_to_first_action_ms": { "median": 5432.01, "mean": 5748.87, "min": 5305.99, "max": 6297.99, "stdev": 501.92 }, "agent_turns": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5414.05, "payload_tokens": 8771, "text_tokens": 8771, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8771, "image_token_share": 0.0, "payload_bytes": 26962, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 61265, "tokens_cached": 44006, "tokens_out": 480, "tokens": 105751, "duration_ms": 15124.71, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.340328, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 6297.99, "payload_tokens": 8771, "text_tokens": 8771, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8771, "image_token_share": 0.0, "payload_bytes": 26962, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 61328, "tokens_cached": 44067, "tokens_out": 546, "tokens": 105941, "duration_ms": 16593.74, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.3423235, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 6294.31, "payload_tokens": 8771, "text_tokens": 8771, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8771, "image_token_share": 0.0, "payload_bytes": 26962, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 61351, "tokens_cached": 44092, "tokens_out": 541, "tokens": 105984, "duration_ms": 17000.18, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.342326, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5305.99, "payload_tokens": 8771, "text_tokens": 8771, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8771, "image_token_share": 0.0, "payload_bytes": 26962, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 61405, "tokens_cached": 44142, "tokens_out": 573, "tokens": 106120, "duration_ms": 13532.85, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.34342100000000003, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5432.01, "payload_tokens": 8771, "text_tokens": 8771, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8771, "image_token_share": 0.0, "payload_bytes": 26962, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 61297, "tokens_cached": 44038, "tokens_out": 502, "tokens": 105837, "duration_ms": 14769.8, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.341054, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "chrome-devtools", "task": "quote-form-submit", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 505.0, "mean": 647.4, "min": 505.0, "max": 861.0, "stdev": 194.99 }, "client_tokens": { "input": { "median": 39972.0, "mean": 40241.2, "min": 39804.0, "max": 40806.0, "stdev": 514.59 }, "cached": { "median": 32014.0, "mean": 32122.8, "min": 31842.0, "max": 32452.0, "stdev": 302.81 }, "output": { "median": 680.0, "mean": 663.2, "min": 570.0, "max": 714.0, "stdev": 58.73 } }, "tokens": { "median": 72700.0, "mean": 73027.2, "min": 72216.0, "max": 73943.0, "stdev": 852.49 }, "duration_ms": { "median": 15393.76, "mean": 15986.09, "min": 12580.36, "max": 19549.26, "stdev": 3013.24 }, "time_to_first_action_ms": { "median": 5490.55, "mean": 5600.43, "min": 5049.34, "max": 6417.5, "stdev": 560.46 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 505.0, "mean": 647.4, "min": 505.0, "max": 861.0, "stdev": 194.99 }, "text_tokens": { "median": 505.0, "mean": 647.4, "min": 505.0, "max": 861.0, "stdev": 194.99 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 505.0, "mean": 647.4, "min": 505.0, "max": 861.0, "stdev": 194.99 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1684.0, "mean": 2164.4, "min": 1684.0, "max": 2885.0, "stdev": 657.81 }, "client_tokens": { "input": { "median": 39972.0, "mean": 40241.2, "min": 39804.0, "max": 40806.0, "stdev": 514.59 }, "cached": { "median": 32014.0, "mean": 32122.8, "min": 31842.0, "max": 32452.0, "stdev": 302.81 }, "output": { "median": 680.0, "mean": 663.2, "min": 570.0, "max": 714.0, "stdev": 58.73 } }, "tokens": { "median": 72700.0, "mean": 73027.2, "min": 72216.0, "max": 73943.0, "stdev": 852.49 }, "duration_ms": { "median": 15393.76, "mean": 15986.09, "min": 12580.36, "max": 19549.26, "stdev": 3013.24 }, "time_to_first_action_ms": { "median": 5490.55, "mean": 5600.43, "min": 5049.34, "max": 6417.5, "stdev": 560.46 }, "agent_turns": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 505.0, "mean": 647.4, "min": 505.0, "max": 861.0, "stdev": 194.99 }, "text_tokens": { "median": 505.0, "mean": 647.4, "min": 505.0, "max": 861.0, "stdev": 194.99 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 505.0, "mean": 647.4, "min": 505.0, "max": 861.0, "stdev": 194.99 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1684.0, "mean": 2164.4, "min": 1684.0, "max": 2885.0, "stdev": 657.81 }, "client_tokens": { "input": { "median": 39972.0, "mean": 40241.2, "min": 39804.0, "max": 40806.0, "stdev": 514.59 }, "cached": { "median": 32014.0, "mean": 32122.8, "min": 31842.0, "max": 32452.0, "stdev": 302.81 }, "output": { "median": 680.0, "mean": 663.2, "min": 570.0, "max": 714.0, "stdev": 58.73 } }, "tokens": { "median": 72700.0, "mean": 73027.2, "min": 72216.0, "max": 73943.0, "stdev": 852.49 }, "duration_ms": { "median": 15393.76, "mean": 15986.09, "min": 12580.36, "max": 19549.26, "stdev": 3013.24 }, "time_to_first_action_ms": { "median": 5490.55, "mean": 5600.43, "min": 5049.34, "max": 6417.5, "stdev": 560.46 }, "agent_turns": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5161.19, "payload_tokens": 861, "text_tokens": 861, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 861, "image_token_share": 0.0, "payload_bytes": 2885, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 40806, "tokens_cached": 32452, "tokens_out": 680, "tokens": 73938, "duration_ms": 13797.53, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.237256, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5049.34, "payload_tokens": 505, "text_tokens": 505, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 505, "image_token_share": 0.0, "payload_bytes": 1684, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 39829, "tokens_cached": 31865, "tokens_out": 645, "tokens": 72339, "duration_ms": 12580.36, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.23120249999999998, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 6417.5, "payload_tokens": 505, "text_tokens": 505, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 505, "image_token_share": 0.0, "payload_bytes": 1684, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 39972, "tokens_cached": 32014, "tokens_out": 714, "tokens": 72700, "duration_ms": 18609.56, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.233717, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5883.55, "payload_tokens": 505, "text_tokens": 505, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 505, "image_token_share": 0.0, "payload_bytes": 1684, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 39804, "tokens_cached": 31842, "tokens_out": 570, "tokens": 72216, "duration_ms": 15393.76, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.22919099999999998, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 5, "time_to_first_action_ms": 5490.55, "payload_tokens": 861, "text_tokens": 861, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 861, "image_token_share": 0.0, "payload_bytes": 2885, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 40795, "tokens_cached": 32441, "tokens_out": 707, "tokens": 73943, "duration_ms": 19549.26, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.2378705, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "claude-code-cli", "stack": "chrome-devtools", "task": "restricted-figure", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 23169.0, "mean": 23169.4, "min": 23151.0, "max": 23191.0, "stdev": 14.24 }, "cached": { "median": 15427.0, "mean": 15426.2, "min": 15407.0, "max": 15447.0, "stdev": 14.25 }, "output": { "median": 556.0, "mean": 543.6, "min": 503.0, "max": 566.0, "stdev": 26.46 } }, "tokens": { "median": 39127.0, "mean": 39139.2, "min": 39093.0, "max": 39194.0, "stdev": 39.29 }, "duration_ms": { "median": 11968.96, "mean": 12219.01, "min": 10145.68, "max": 14240.59, "stdev": 1501.75 }, "time_to_first_action_ms": { "median": 6398.1, "mean": 6510.25, "min": 5614.51, "max": 7338.21, "stdev": 804.94 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "text_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1171.0, "mean": 1171.0, "min": 1171.0, "max": 1171.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 23169.0, "mean": 23169.4, "min": 23151.0, "max": 23191.0, "stdev": 14.24 }, "cached": { "median": 15427.0, "mean": 15426.2, "min": 15407.0, "max": 15447.0, "stdev": 14.25 }, "output": { "median": 556.0, "mean": 543.6, "min": 503.0, "max": 566.0, "stdev": 26.46 } }, "tokens": { "median": 39127.0, "mean": 39139.2, "min": 39093.0, "max": 39194.0, "stdev": 39.29 }, "duration_ms": { "median": 11968.96, "mean": 12219.01, "min": 10145.68, "max": 14240.59, "stdev": 1501.75 }, "time_to_first_action_ms": { "median": 6398.1, "mean": 6510.25, "min": 5614.51, "max": 7338.21, "stdev": 804.94 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "text_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1171.0, "mean": 1171.0, "min": 1171.0, "max": 1171.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 23169.0, "mean": 23169.4, "min": 23151.0, "max": 23191.0, "stdev": 14.24 }, "cached": { "median": 15427.0, "mean": 15426.2, "min": 15407.0, "max": 15447.0, "stdev": 14.25 }, "output": { "median": 556.0, "mean": 543.6, "min": 503.0, "max": 566.0, "stdev": 26.46 } }, "tokens": { "median": 39127.0, "mean": 39139.2, "min": 39093.0, "max": 39194.0, "stdev": 39.29 }, "duration_ms": { "median": 11968.96, "mean": 12219.01, "min": 10145.68, "max": 14240.59, "stdev": 1501.75 }, "time_to_first_action_ms": { "median": 6398.1, "mean": 6510.25, "min": 5614.51, "max": 7338.21, "stdev": 804.94 }, "agent_turns": { "median": 3.0, "mean": 3.0, "min": 3.0, "max": 3.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5614.51, "payload_tokens": 379, "text_tokens": 379, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 379, "image_token_share": 0.0, "payload_bytes": 1171, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 23169, "tokens_cached": 15427, "tokens_out": 566, "tokens": 39162, "duration_ms": 10145.68, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1377085, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 7338.21, "payload_tokens": 379, "text_tokens": 379, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 379, "image_token_share": 0.0, "payload_bytes": 1171, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 23167, "tokens_cached": 15423, "tokens_out": 503, "tokens": 39093, "duration_ms": 12883.47, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1361215, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 7333.38, "payload_tokens": 379, "text_tokens": 379, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 379, "image_token_share": 0.0, "payload_bytes": 1171, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 23151, "tokens_cached": 15407, "tokens_out": 562, "tokens": 39120, "duration_ms": 14240.59, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1375085, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 6398.1, "payload_tokens": 379, "text_tokens": 379, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 379, "image_token_share": 0.0, "payload_bytes": 1171, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 23169, "tokens_cached": 15427, "tokens_out": 531, "tokens": 39127, "duration_ms": 11968.96, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1368335, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 3, "time_to_first_action_ms": 5867.05, "payload_tokens": 379, "text_tokens": 379, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 379, "image_token_share": 0.0, "payload_bytes": 1171, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 23191, "tokens_cached": 15447, "tokens_out": 556, "tokens": 39194, "duration_ms": 11856.37, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": 0.1375785, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "owl", "task": "cheapest-mystery-book", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 5.0, "mean": 4.8, "min": 4.0, "max": 5.0, "stdev": 0.45 }, "client": { "median": 5.0, "mean": 4.8, "min": 4.0, "max": 5.0, "stdev": 0.45 } }, "payload_tokens": { "median": 6058.0, "mean": 7119.6, "min": 5632.0, "max": 11792.0, "stdev": 2618.46 }, "client_tokens": { "input": { "median": 20763.0, "mean": 20564.0, "min": 19580.0, "max": 20965.0, "stdev": 557.33 }, "cached": { "median": 82732.0, "mean": 79026.8, "min": 64633.0, "max": 83190.0, "stdev": 8064.44 }, "output": { "median": 1223.0, "mean": 1151.4, "min": 877.0, "max": 1328.0, "stdev": 200.98 } }, "tokens": { "median": 104502.0, "mean": 100742.2, "min": 86475.0, "max": 105261.0, "stdev": 8045.17 }, "duration_ms": { "median": 35566.11, "mean": 37446.43, "min": 31157.9, "max": 43738.51, "stdev": 5609.97 }, "time_to_first_action_ms": { "median": 7305.39, "mean": 8358.75, "min": 6739.48, "max": 10825.44, "stdev": 1804.15 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 5.0, "mean": 4.8, "min": 4.0, "max": 5.0, "stdev": 0.45 }, "client": { "median": 5.0, "mean": 4.8, "min": 4.0, "max": 5.0, "stdev": 0.45 } }, "payload_tokens": { "median": 6058.0, "mean": 7119.6, "min": 5632.0, "max": 11792.0, "stdev": 2618.46 }, "text_tokens": { "median": 6058.0, "mean": 7119.6, "min": 5632.0, "max": 11792.0, "stdev": 2618.46 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 6058.0, "mean": 7119.6, "min": 5632.0, "max": 11792.0, "stdev": 2618.46 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 19619.0, "mean": 26781.4, "min": 17844.0, "max": 57206.0, "stdev": 17025.23 }, "client_tokens": { "input": { "median": 20763.0, "mean": 20564.0, "min": 19580.0, "max": 20965.0, "stdev": 557.33 }, "cached": { "median": 82732.0, "mean": 79026.8, "min": 64633.0, "max": 83190.0, "stdev": 8064.44 }, "output": { "median": 1223.0, "mean": 1151.4, "min": 877.0, "max": 1328.0, "stdev": 200.98 } }, "tokens": { "median": 104502.0, "mean": 100742.2, "min": 86475.0, "max": 105261.0, "stdev": 8045.17 }, "duration_ms": { "median": 35566.11, "mean": 37446.43, "min": 31157.9, "max": 43738.51, "stdev": 5609.97 }, "time_to_first_action_ms": { "median": 7305.39, "mean": 8358.75, "min": 6739.48, "max": 10825.44, "stdev": 1804.15 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 5.0, "mean": 4.8, "min": 4.0, "max": 5.0, "stdev": 0.45 }, "client": { "median": 5.0, "mean": 4.8, "min": 4.0, "max": 5.0, "stdev": 0.45 } }, "payload_tokens": { "median": 6058.0, "mean": 7119.6, "min": 5632.0, "max": 11792.0, "stdev": 2618.46 }, "text_tokens": { "median": 6058.0, "mean": 7119.6, "min": 5632.0, "max": 11792.0, "stdev": 2618.46 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 6058.0, "mean": 7119.6, "min": 5632.0, "max": 11792.0, "stdev": 2618.46 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 19619.0, "mean": 26781.4, "min": 17844.0, "max": 57206.0, "stdev": 17025.23 }, "client_tokens": { "input": { "median": 20763.0, "mean": 20564.0, "min": 19580.0, "max": 20965.0, "stdev": 557.33 }, "cached": { "median": 82732.0, "mean": 79026.8, "min": 64633.0, "max": 83190.0, "stdev": 8064.44 }, "output": { "median": 1223.0, "mean": 1151.4, "min": 877.0, "max": 1328.0, "stdev": 200.98 } }, "tokens": { "median": 104502.0, "mean": 100742.2, "min": 86475.0, "max": 105261.0, "stdev": 8045.17 }, "duration_ms": { "median": 35566.11, "mean": 37446.43, "min": 31157.9, "max": 43738.51, "stdev": 5609.97 }, "time_to_first_action_ms": { "median": 7305.39, "mean": 8358.75, "min": 6739.48, "max": 10825.44, "stdev": 1804.15 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7305.39, "payload_tokens": 11792, "text_tokens": 11792, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 11792, "image_token_share": 0.0, "payload_bytes": 57206, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 19580, "tokens_cached": 81738, "tokens_out": 1223, "tokens": 102541, "duration_ms": 33832.11, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7200.28, "payload_tokens": 6058, "text_tokens": 6058, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 6058, "image_token_share": 0.0, "payload_bytes": 19619, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 20763, "tokens_cached": 82732, "tokens_out": 1007, "tokens": 104502, "duration_ms": 31157.9, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 10825.44, "payload_tokens": 5632, "text_tokens": 5632, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 5632, "image_token_share": 0.0, "payload_bytes": 17844, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 20965, "tokens_cached": 64633, "tokens_out": 877, "tokens": 86475, "duration_ms": 42937.53, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 9723.17, "payload_tokens": 6058, "text_tokens": 6058, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 6058, "image_token_share": 0.0, "payload_bytes": 19619, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 20763, "tokens_cached": 82841, "tokens_out": 1328, "tokens": 104932, "duration_ms": 43738.51, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6739.48, "payload_tokens": 6058, "text_tokens": 6058, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 6058, "image_token_share": 0.0, "payload_bytes": 19619, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 20749, "tokens_cached": 83190, "tokens_out": 1322, "tokens": 105261, "duration_ms": 35566.11, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "owl", "task": "catalogue-last-page", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 1.6, "min": 1.0, "max": 2.0, "stdev": 0.55 }, "client": { "median": 2.0, "mean": 1.6, "min": 1.0, "max": 2.0, "stdev": 0.55 } }, "payload_tokens": { "median": 5159.0, "mean": 5839.0, "min": 1216.0, "max": 13716.0, "stdev": 4688.65 }, "client_tokens": { "input": { "median": 19505.0, "mean": 18728.6, "min": 15425.0, "max": 21089.0, "stdev": 2128.47 }, "cached": { "median": 30261.0, "mean": 24756.0, "min": 14100.0, "max": 32819.0, "stdev": 9776.89 }, "output": { "median": 500.0, "mean": 469.8, "min": 144.0, "max": 781.0, "stdev": 272.25 } }, "tokens": { "median": 50547.0, "mean": 43954.4, "min": 30025.0, "max": 54585.0, "stdev": 11773.99 }, "duration_ms": { "median": 14313.2, "mean": 21431.8, "min": 13973.26, "max": 37411.83, "stdev": 10567.13 }, "time_to_first_action_ms": { "median": 7150.8, "mean": 7715.37, "min": 6050.14, "max": 12072.0, "stdev": 2502.08 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 1.6, "min": 1.0, "max": 2.0, "stdev": 0.55 }, "client": { "median": 2.0, "mean": 1.6, "min": 1.0, "max": 2.0, "stdev": 0.55 } }, "payload_tokens": { "median": 5159.0, "mean": 5839.0, "min": 1216.0, "max": 13716.0, "stdev": 4688.65 }, "text_tokens": { "median": 5159.0, "mean": 5839.0, "min": 1216.0, "max": 13716.0, "stdev": 4688.65 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 5159.0, "mean": 5839.0, "min": 1216.0, "max": 13716.0, "stdev": 4688.65 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 16743.0, "mean": 22694.0, "min": 3899.0, "max": 63241.0, "stdev": 23265.65 }, "client_tokens": { "input": { "median": 19505.0, "mean": 18728.6, "min": 15425.0, "max": 21089.0, "stdev": 2128.47 }, "cached": { "median": 30261.0, "mean": 24756.0, "min": 14100.0, "max": 32819.0, "stdev": 9776.89 }, "output": { "median": 500.0, "mean": 469.8, "min": 144.0, "max": 781.0, "stdev": 272.25 } }, "tokens": { "median": 50547.0, "mean": 43954.4, "min": 30025.0, "max": 54585.0, "stdev": 11773.99 }, "duration_ms": { "median": 14313.2, "mean": 21431.8, "min": 13973.26, "max": 37411.83, "stdev": 10567.13 }, "time_to_first_action_ms": { "median": 7150.8, "mean": 7715.37, "min": 6050.14, "max": 12072.0, "stdev": 2502.08 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 1.6, "min": 1.0, "max": 2.0, "stdev": 0.55 }, "client": { "median": 2.0, "mean": 1.6, "min": 1.0, "max": 2.0, "stdev": 0.55 } }, "payload_tokens": { "median": 5159.0, "mean": 5839.0, "min": 1216.0, "max": 13716.0, "stdev": 4688.65 }, "text_tokens": { "median": 5159.0, "mean": 5839.0, "min": 1216.0, "max": 13716.0, "stdev": 4688.65 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 5159.0, "mean": 5839.0, "min": 1216.0, "max": 13716.0, "stdev": 4688.65 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 16743.0, "mean": 22694.0, "min": 3899.0, "max": 63241.0, "stdev": 23265.65 }, "client_tokens": { "input": { "median": 19505.0, "mean": 18728.6, "min": 15425.0, "max": 21089.0, "stdev": 2128.47 }, "cached": { "median": 30261.0, "mean": 24756.0, "min": 14100.0, "max": 32819.0, "stdev": 9776.89 }, "output": { "median": 500.0, "mean": 469.8, "min": 144.0, "max": 781.0, "stdev": 272.25 } }, "tokens": { "median": 50547.0, "mean": 43954.4, "min": 30025.0, "max": 54585.0, "stdev": 11773.99 }, "duration_ms": { "median": 14313.2, "mean": 21431.8, "min": 13973.26, "max": 37411.83, "stdev": 10567.13 }, "time_to_first_action_ms": { "median": 7150.8, "mean": 7715.37, "min": 6050.14, "max": 12072.0, "stdev": 2502.08 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7246.87, "payload_tokens": 5159, "text_tokens": 5159, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 5159, "image_token_share": 0.0, "payload_bytes": 16743, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 19505, "tokens_cached": 30261, "tokens_out": 781, "tokens": 50547, "duration_ms": 27211.4, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7150.8, "payload_tokens": 3944, "text_tokens": 3944, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 3944, "image_token_share": 0.0, "payload_bytes": 12844, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 18098, "tokens_cached": 14101, "tokens_out": 144, "tokens": 32343, "duration_ms": 13973.26, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 12072.0, "payload_tokens": 13716, "text_tokens": 13716, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 13716, "image_token_share": 0.0, "payload_bytes": 63241, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 21089, "tokens_cached": 32819, "tokens_out": 677, "tokens": 54585, "duration_ms": 37411.83, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6057.03, "payload_tokens": 1216, "text_tokens": 1216, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1216, "image_token_share": 0.0, "payload_bytes": 3899, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 15425, "tokens_cached": 14100, "tokens_out": 500, "tokens": 30025, "duration_ms": 14313.2, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6050.14, "payload_tokens": 5160, "text_tokens": 5160, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 5160, "image_token_share": 0.0, "payload_bytes": 16743, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 19526, "tokens_cached": 32499, "tokens_out": 247, "tokens": 52272, "duration_ms": 14249.33, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "owl", "task": "book-upc-table", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 14638.0, "mean": 14638.0, "min": 14631.0, "max": 14645.0, "stdev": 4.95 }, "cached": { "median": 14077.0, "mean": 14081.6, "min": 14071.0, "max": 14097.0, "stdev": 10.38 }, "output": { "median": 85.0, "mean": 91.0, "min": 67.0, "max": 113.0, "stdev": 19.49 } }, "tokens": { "median": 28815.0, "mean": 28810.6, "min": 28781.0, "max": 28824.0, "stdev": 17.53 }, "duration_ms": { "median": 8932.58, "mean": 9337.0, "min": 7966.57, "max": 11344.53, "stdev": 1474.64 }, "time_to_first_action_ms": { "median": 6666.92, "mean": 6576.9, "min": 5694.64, "max": 7895.38, "stdev": 860.1 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "text_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2200.0, "mean": 2200.0, "min": 2200.0, "max": 2200.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 14638.0, "mean": 14638.0, "min": 14631.0, "max": 14645.0, "stdev": 4.95 }, "cached": { "median": 14077.0, "mean": 14081.6, "min": 14071.0, "max": 14097.0, "stdev": 10.38 }, "output": { "median": 85.0, "mean": 91.0, "min": 67.0, "max": 113.0, "stdev": 19.49 } }, "tokens": { "median": 28815.0, "mean": 28810.6, "min": 28781.0, "max": 28824.0, "stdev": 17.53 }, "duration_ms": { "median": 8932.58, "mean": 9337.0, "min": 7966.57, "max": 11344.53, "stdev": 1474.64 }, "time_to_first_action_ms": { "median": 6666.92, "mean": 6576.9, "min": 5694.64, "max": 7895.38, "stdev": 860.1 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "text_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 532.0, "mean": 532.0, "min": 532.0, "max": 532.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2200.0, "mean": 2200.0, "min": 2200.0, "max": 2200.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 14638.0, "mean": 14638.0, "min": 14631.0, "max": 14645.0, "stdev": 4.95 }, "cached": { "median": 14077.0, "mean": 14081.6, "min": 14071.0, "max": 14097.0, "stdev": 10.38 }, "output": { "median": 85.0, "mean": 91.0, "min": 67.0, "max": 113.0, "stdev": 19.49 } }, "tokens": { "median": 28815.0, "mean": 28810.6, "min": 28781.0, "max": 28824.0, "stdev": 17.53 }, "duration_ms": { "median": 8932.58, "mean": 9337.0, "min": 7966.57, "max": 11344.53, "stdev": 1474.64 }, "time_to_first_action_ms": { "median": 6666.92, "mean": 6576.9, "min": 5694.64, "max": 7895.38, "stdev": 860.1 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6698.32, "payload_tokens": 532, "text_tokens": 532, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 532, "image_token_share": 0.0, "payload_bytes": 2200, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14638, "tokens_cached": 14076, "tokens_out": 67, "tokens": 28781, "duration_ms": 8932.58, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6666.92, "payload_tokens": 532, "text_tokens": 532, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 532, "image_token_share": 0.0, "payload_bytes": 2200, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14631, "tokens_cached": 14071, "tokens_out": 113, "tokens": 28815, "duration_ms": 11344.53, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7895.38, "payload_tokens": 532, "text_tokens": 532, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 532, "image_token_share": 0.0, "payload_bytes": 2200, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14638, "tokens_cached": 14087, "tokens_out": 85, "tokens": 28810, "duration_ms": 10360.69, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 5694.64, "payload_tokens": 532, "text_tokens": 532, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 532, "image_token_share": 0.0, "payload_bytes": 2200, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14638, "tokens_cached": 14077, "tokens_out": 109, "tokens": 28824, "duration_ms": 8080.61, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 5929.26, "payload_tokens": 532, "text_tokens": 532, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 532, "image_token_share": 0.0, "payload_bytes": 2200, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14645, "tokens_cached": 14097, "tokens_out": 81, "tokens": 28823, "duration_ms": 7966.57, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "owl", "task": "travel-stock-drilldown", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 3.2, "min": 2.0, "max": 5.0, "stdev": 1.64 }, "client": { "median": 2.0, "mean": 3.2, "min": 2.0, "max": 5.0, "stdev": 1.64 } }, "payload_tokens": { "median": 4155.0, "mean": 3452.2, "min": 2393.0, "max": 4155.0, "stdev": 962.36 }, "client_tokens": { "input": { "median": 18843.0, "mean": 18163.0, "min": 17132.0, "max": 18843.0, "stdev": 931.16 }, "cached": { "median": 31626.0, "mean": 49566.2, "min": 31585.0, "max": 76554.0, "stdev": 24589.85 }, "output": { "median": 176.0, "mean": 183.4, "min": 149.0, "max": 216.0, "stdev": 30.39 } }, "tokens": { "median": 50645.0, "mean": 67912.6, "min": 50590.0, "max": 93924.0, "stdev": 23687.59 }, "duration_ms": { "median": 16584.65, "mean": 17920.51, "min": 14409.07, "max": 25596.92, "stdev": 4398.53 }, "time_to_first_action_ms": { "median": 7478.96, "mean": 8006.15, "min": 6825.44, "max": 9442.03, "stdev": 1134.5 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 3.2, "min": 2.0, "max": 5.0, "stdev": 1.64 }, "client": { "median": 2.0, "mean": 3.2, "min": 2.0, "max": 5.0, "stdev": 1.64 } }, "payload_tokens": { "median": 4155.0, "mean": 3452.2, "min": 2393.0, "max": 4155.0, "stdev": 962.36 }, "text_tokens": { "median": 4155.0, "mean": 3452.2, "min": 2393.0, "max": 4155.0, "stdev": 962.36 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 4155.0, "mean": 3452.2, "min": 2393.0, "max": 4155.0, "stdev": 962.36 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 14502.0, "mean": 11949.2, "min": 8114.0, "max": 14502.0, "stdev": 3495.57 }, "client_tokens": { "input": { "median": 18843.0, "mean": 18163.0, "min": 17132.0, "max": 18843.0, "stdev": 931.16 }, "cached": { "median": 31626.0, "mean": 49566.2, "min": 31585.0, "max": 76554.0, "stdev": 24589.85 }, "output": { "median": 176.0, "mean": 183.4, "min": 149.0, "max": 216.0, "stdev": 30.39 } }, "tokens": { "median": 50645.0, "mean": 67912.6, "min": 50590.0, "max": 93924.0, "stdev": 23687.59 }, "duration_ms": { "median": 16584.65, "mean": 17920.51, "min": 14409.07, "max": 25596.92, "stdev": 4398.53 }, "time_to_first_action_ms": { "median": 7478.96, "mean": 8006.15, "min": 6825.44, "max": 9442.03, "stdev": 1134.5 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 3.2, "min": 2.0, "max": 5.0, "stdev": 1.64 }, "client": { "median": 2.0, "mean": 3.2, "min": 2.0, "max": 5.0, "stdev": 1.64 } }, "payload_tokens": { "median": 4155.0, "mean": 3452.2, "min": 2393.0, "max": 4155.0, "stdev": 962.36 }, "text_tokens": { "median": 4155.0, "mean": 3452.2, "min": 2393.0, "max": 4155.0, "stdev": 962.36 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 4155.0, "mean": 3452.2, "min": 2393.0, "max": 4155.0, "stdev": 962.36 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 14502.0, "mean": 11949.2, "min": 8114.0, "max": 14502.0, "stdev": 3495.57 }, "client_tokens": { "input": { "median": 18843.0, "mean": 18163.0, "min": 17132.0, "max": 18843.0, "stdev": 931.16 }, "cached": { "median": 31626.0, "mean": 49566.2, "min": 31585.0, "max": 76554.0, "stdev": 24589.85 }, "output": { "median": 176.0, "mean": 183.4, "min": 149.0, "max": 216.0, "stdev": 30.39 } }, "tokens": { "median": 50645.0, "mean": 67912.6, "min": 50590.0, "max": 93924.0, "stdev": 23687.59 }, "duration_ms": { "median": 16584.65, "mean": 17920.51, "min": 14409.07, "max": 25596.92, "stdev": 4398.53 }, "time_to_first_action_ms": { "median": 7478.96, "mean": 8006.15, "min": 6825.44, "max": 9442.03, "stdev": 1134.5 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7478.96, "payload_tokens": 4155, "text_tokens": 4155, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 4155, "image_token_share": 0.0, "payload_bytes": 14502, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 18843, "tokens_cached": 31585, "tokens_out": 162, "tokens": 50590, "duration_ms": 16584.65, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 9442.03, "payload_tokens": 4155, "text_tokens": 4155, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 4155, "image_token_share": 0.0, "payload_bytes": 14502, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 18843, "tokens_cached": 31626, "tokens_out": 176, "tokens": 50645, "duration_ms": 16921.0, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8972.09, "payload_tokens": 2393, "text_tokens": 2393, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 2393, "image_token_share": 0.0, "payload_bytes": 8114, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 17132, "tokens_cached": 76452, "tokens_out": 214, "tokens": 93798, "duration_ms": 25596.92, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6825.44, "payload_tokens": 2403, "text_tokens": 2403, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 2403, "image_token_share": 0.0, "payload_bytes": 8126, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 17154, "tokens_cached": 76554, "tokens_out": 216, "tokens": 93924, "duration_ms": 16090.92, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7312.21, "payload_tokens": 4155, "text_tokens": 4155, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 4155, "image_token_share": 0.0, "payload_bytes": 14502, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 18843, "tokens_cached": 31614, "tokens_out": 149, "tokens": 50606, "duration_ms": 14409.07, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "owl", "task": "quote-form-submit", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 3.8, "min": 3.0, "max": 4.0, "stdev": 0.45 }, "client": { "median": 4.0, "mean": 3.8, "min": 3.0, "max": 4.0, "stdev": 0.45 } }, "payload_tokens": { "median": 680.0, "mean": 682.6, "min": 679.0, "max": 687.0, "stdev": 4.04 }, "client_tokens": { "input": { "median": 15071.0, "mean": 15152.4, "min": 15034.0, "max": 15297.0, "stdev": 129.69 }, "cached": { "median": 58841.0, "mean": 55893.8, "min": 43671.0, "max": 59086.0, "stdev": 6833.89 }, "output": { "median": 175.0, "mean": 184.2, "min": 147.0, "max": 251.0, "stdev": 39.49 } }, "tokens": { "median": 74285.0, "mean": 71230.4, "min": 58956.0, "max": 74338.0, "stdev": 6861.65 }, "duration_ms": { "median": 16847.8, "mean": 18601.64, "min": 14466.38, "max": 29214.68, "stdev": 6109.73 }, "time_to_first_action_ms": { "median": 6018.45, "mean": 6147.55, "min": 5388.12, "max": 7733.56, "stdev": 950.3 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 3.8, "min": 3.0, "max": 4.0, "stdev": 0.45 }, "client": { "median": 4.0, "mean": 3.8, "min": 3.0, "max": 4.0, "stdev": 0.45 } }, "payload_tokens": { "median": 680.0, "mean": 682.6, "min": 679.0, "max": 687.0, "stdev": 4.04 }, "text_tokens": { "median": 680.0, "mean": 682.6, "min": 679.0, "max": 687.0, "stdev": 4.04 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 680.0, "mean": 682.6, "min": 679.0, "max": 687.0, "stdev": 4.04 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2303.0, "mean": 2304.6, "min": 2299.0, "max": 2309.0, "stdev": 4.34 }, "client_tokens": { "input": { "median": 15071.0, "mean": 15152.4, "min": 15034.0, "max": 15297.0, "stdev": 129.69 }, "cached": { "median": 58841.0, "mean": 55893.8, "min": 43671.0, "max": 59086.0, "stdev": 6833.89 }, "output": { "median": 175.0, "mean": 184.2, "min": 147.0, "max": 251.0, "stdev": 39.49 } }, "tokens": { "median": 74285.0, "mean": 71230.4, "min": 58956.0, "max": 74338.0, "stdev": 6861.65 }, "duration_ms": { "median": 16847.8, "mean": 18601.64, "min": 14466.38, "max": 29214.68, "stdev": 6109.73 }, "time_to_first_action_ms": { "median": 6018.45, "mean": 6147.55, "min": 5388.12, "max": 7733.56, "stdev": 950.3 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 3.8, "min": 3.0, "max": 4.0, "stdev": 0.45 }, "client": { "median": 4.0, "mean": 3.8, "min": 3.0, "max": 4.0, "stdev": 0.45 } }, "payload_tokens": { "median": 680.0, "mean": 682.6, "min": 679.0, "max": 687.0, "stdev": 4.04 }, "text_tokens": { "median": 680.0, "mean": 682.6, "min": 679.0, "max": 687.0, "stdev": 4.04 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 680.0, "mean": 682.6, "min": 679.0, "max": 687.0, "stdev": 4.04 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2303.0, "mean": 2304.6, "min": 2299.0, "max": 2309.0, "stdev": 4.34 }, "client_tokens": { "input": { "median": 15071.0, "mean": 15152.4, "min": 15034.0, "max": 15297.0, "stdev": 129.69 }, "cached": { "median": 58841.0, "mean": 55893.8, "min": 43671.0, "max": 59086.0, "stdev": 6833.89 }, "output": { "median": 175.0, "mean": 184.2, "min": 147.0, "max": 251.0, "stdev": 39.49 } }, "tokens": { "median": 74285.0, "mean": 71230.4, "min": 58956.0, "max": 74338.0, "stdev": 6861.65 }, "duration_ms": { "median": 16847.8, "mean": 18601.64, "min": 14466.38, "max": 29214.68, "stdev": 6109.73 }, "time_to_first_action_ms": { "median": 6018.45, "mean": 6147.55, "min": 5388.12, "max": 7733.56, "stdev": 950.3 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 5388.12, "payload_tokens": 687, "text_tokens": 687, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 687, "image_token_share": 0.0, "payload_bytes": 2309, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 15290, "tokens_cached": 58810, "tokens_out": 167, "tokens": 74267, "duration_ms": 14466.38, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 5438.12, "payload_tokens": 679, "text_tokens": 679, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 679, "image_token_share": 0.0, "payload_bytes": 2303, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 15070, "tokens_cached": 59061, "tokens_out": 175, "tokens": 74306, "duration_ms": 14604.94, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7733.56, "payload_tokens": 680, "text_tokens": 680, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 680, "image_token_share": 0.0, "payload_bytes": 2303, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 15071, "tokens_cached": 59086, "tokens_out": 181, "tokens": 74338, "duration_ms": 29214.68, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 3, "client_tool_calls": 3, "client_tool_calls_total": 3, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6159.48, "payload_tokens": 680, "text_tokens": 680, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 680, "image_token_share": 0.0, "payload_bytes": 2299, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 15034, "tokens_cached": 43671, "tokens_out": 251, "tokens": 58956, "duration_ms": 17874.41, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6018.45, "payload_tokens": 687, "text_tokens": 687, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 687, "image_token_share": 0.0, "payload_bytes": 2309, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 15297, "tokens_cached": 58841, "tokens_out": 147, "tokens": 74285, "duration_ms": 16847.8, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "owl", "task": "restricted-figure", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 14207.0, "mean": 14208.4, "min": 14200.0, "max": 14214.0, "stdev": 5.86 }, "cached": { "median": 14099.0, "mean": 14100.2, "min": 14092.0, "max": 14106.0, "stdev": 5.93 }, "output": { "median": 172.0, "mean": 191.8, "min": 168.0, "max": 247.0, "stdev": 33.41 } }, "tokens": { "median": 28488.0, "mean": 28500.4, "min": 28477.0, "max": 28567.0, "stdev": 37.78 }, "duration_ms": { "median": 12679.03, "mean": 13046.3, "min": 10705.69, "max": 16374.37, "stdev": 2473.5 }, "time_to_first_action_ms": { "median": 8674.53, "mean": 8717.84, "min": 7288.61, "max": 10796.8, "stdev": 1314.59 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "text_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 598.0, "mean": 598.0, "min": 598.0, "max": 598.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 14207.0, "mean": 14208.4, "min": 14200.0, "max": 14214.0, "stdev": 5.86 }, "cached": { "median": 14099.0, "mean": 14100.2, "min": 14092.0, "max": 14106.0, "stdev": 5.93 }, "output": { "median": 172.0, "mean": 191.8, "min": 168.0, "max": 247.0, "stdev": 33.41 } }, "tokens": { "median": 28488.0, "mean": 28500.4, "min": 28477.0, "max": 28567.0, "stdev": 37.78 }, "duration_ms": { "median": 12679.03, "mean": 13046.3, "min": 10705.69, "max": 16374.37, "stdev": 2473.5 }, "time_to_first_action_ms": { "median": 8674.53, "mean": 8717.84, "min": 7288.61, "max": 10796.8, "stdev": 1314.59 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "client": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 } }, "payload_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "text_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 139.0, "mean": 139.0, "min": 139.0, "max": 139.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 598.0, "mean": 598.0, "min": 598.0, "max": 598.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 14207.0, "mean": 14208.4, "min": 14200.0, "max": 14214.0, "stdev": 5.86 }, "cached": { "median": 14099.0, "mean": 14100.2, "min": 14092.0, "max": 14106.0, "stdev": 5.93 }, "output": { "median": 172.0, "mean": 191.8, "min": 168.0, "max": 247.0, "stdev": 33.41 } }, "tokens": { "median": 28488.0, "mean": 28500.4, "min": 28477.0, "max": 28567.0, "stdev": 37.78 }, "duration_ms": { "median": 12679.03, "mean": 13046.3, "min": 10705.69, "max": 16374.37, "stdev": 2473.5 }, "time_to_first_action_ms": { "median": 8674.53, "mean": 8717.84, "min": 7288.61, "max": 10796.8, "stdev": 1314.59 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7989.16, "payload_tokens": 139, "text_tokens": 139, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 139, "image_token_share": 0.0, "payload_bytes": 598, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14207, "tokens_cached": 14099, "tokens_out": 172, "tokens": 28478, "duration_ms": 10788.0, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 10796.8, "payload_tokens": 139, "text_tokens": 139, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 139, "image_token_share": 0.0, "payload_bytes": 598, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14207, "tokens_cached": 14098, "tokens_out": 172, "tokens": 28477, "duration_ms": 16374.37, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8840.1, "payload_tokens": 139, "text_tokens": 139, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 139, "image_token_share": 0.0, "payload_bytes": 598, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14214, "tokens_cached": 14106, "tokens_out": 247, "tokens": 28567, "duration_ms": 14684.41, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8674.53, "payload_tokens": 139, "text_tokens": 139, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 139, "image_token_share": 0.0, "payload_bytes": 598, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14200, "tokens_cached": 14092, "tokens_out": 200, "tokens": 28492, "duration_ms": 12679.03, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 1, "client_tool_calls": 1, "client_tool_calls_total": 1, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7288.61, "payload_tokens": 139, "text_tokens": 139, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 139, "image_token_share": 0.0, "payload_bytes": 598, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 20, "tools_list_tokens": 3958, "transport_available": true, "tokens_in": 14214, "tokens_cached": 14106, "tokens_out": 168, "tokens": 28488, "duration_ms": 10705.69, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "playwright", "task": "cheapest-mystery-book", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 8.0, "mean": 7.6, "min": 5.0, "max": 9.0, "stdev": 1.67 }, "client": { "median": 8.0, "mean": 7.6, "min": 5.0, "max": 9.0, "stdev": 1.67 } }, "payload_tokens": { "median": 9470.0, "mean": 9503.8, "min": 8916.0, "max": 10025.0, "stdev": 419.09 }, "client_tokens": { "input": { "median": 20338.0, "mean": 20272.6, "min": 19398.0, "max": 21130.0, "stdev": 622.53 }, "cached": { "median": 146138.0, "mean": 137580.0, "min": 84787.0, "max": 166503.0, "stdev": 33649.7 }, "output": { "median": 581.0, "mean": 883.2, "min": 414.0, "max": 1832.0, "stdev": 580.72 } }, "tokens": { "median": 168309.0, "mean": 158735.8, "min": 104599.0, "max": 187150.0, "stdev": 34025.64 }, "duration_ms": { "median": 35215.78, "mean": 40016.08, "min": 20664.97, "max": 68895.25, "stdev": 18428.73 }, "time_to_first_action_ms": { "median": 7109.4, "mean": 7858.31, "min": 7001.82, "max": 10265.11, "stdev": 1395.43 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 8.0, "mean": 7.6, "min": 5.0, "max": 9.0, "stdev": 1.67 }, "client": { "median": 8.0, "mean": 7.6, "min": 5.0, "max": 9.0, "stdev": 1.67 } }, "payload_tokens": { "median": 9470.0, "mean": 9503.8, "min": 8916.0, "max": 10025.0, "stdev": 419.09 }, "text_tokens": { "median": 9470.0, "mean": 9503.8, "min": 8916.0, "max": 10025.0, "stdev": 419.09 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 9470.0, "mean": 9503.8, "min": 8916.0, "max": 10025.0, "stdev": 419.09 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 36973.0, "mean": 37049.2, "min": 35033.0, "max": 38964.0, "stdev": 1449.0 }, "client_tokens": { "input": { "median": 20338.0, "mean": 20272.6, "min": 19398.0, "max": 21130.0, "stdev": 622.53 }, "cached": { "median": 146138.0, "mean": 137580.0, "min": 84787.0, "max": 166503.0, "stdev": 33649.7 }, "output": { "median": 581.0, "mean": 883.2, "min": 414.0, "max": 1832.0, "stdev": 580.72 } }, "tokens": { "median": 168309.0, "mean": 158735.8, "min": 104599.0, "max": 187150.0, "stdev": 34025.64 }, "duration_ms": { "median": 35215.78, "mean": 40016.08, "min": 20664.97, "max": 68895.25, "stdev": 18428.73 }, "time_to_first_action_ms": { "median": 7109.4, "mean": 7858.31, "min": 7001.82, "max": 10265.11, "stdev": 1395.43 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 8.0, "mean": 7.6, "min": 5.0, "max": 9.0, "stdev": 1.67 }, "client": { "median": 8.0, "mean": 7.6, "min": 5.0, "max": 9.0, "stdev": 1.67 } }, "payload_tokens": { "median": 9470.0, "mean": 9503.8, "min": 8916.0, "max": 10025.0, "stdev": 419.09 }, "text_tokens": { "median": 9470.0, "mean": 9503.8, "min": 8916.0, "max": 10025.0, "stdev": 419.09 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 9470.0, "mean": 9503.8, "min": 8916.0, "max": 10025.0, "stdev": 419.09 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 36973.0, "mean": 37049.2, "min": 35033.0, "max": 38964.0, "stdev": 1449.0 }, "client_tokens": { "input": { "median": 20338.0, "mean": 20272.6, "min": 19398.0, "max": 21130.0, "stdev": 622.53 }, "cached": { "median": 146138.0, "mean": 137580.0, "min": 84787.0, "max": 166503.0, "stdev": 33649.7 }, "output": { "median": 581.0, "mean": 883.2, "min": 414.0, "max": 1832.0, "stdev": 580.72 } }, "tokens": { "median": 168309.0, "mean": 158735.8, "min": 104599.0, "max": 187150.0, "stdev": 34025.64 }, "duration_ms": { "median": 35215.78, "mean": 40016.08, "min": 20664.97, "max": 68895.25, "stdev": 18428.73 }, "time_to_first_action_ms": { "median": 7109.4, "mean": 7858.31, "min": 7001.82, "max": 10265.11, "stdev": 1395.43 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 9, "client_tool_calls": 9, "client_tool_calls_total": 9, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7109.4, "payload_tokens": 9355, "text_tokens": 9355, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 9355, "image_token_share": 0.0, "payload_bytes": 36570, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 20099, "tokens_cached": 166503, "tokens_out": 548, "tokens": 187150, "duration_ms": 35215.78, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 8, "client_tool_calls": 8, "client_tool_calls_total": 8, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7893.84, "payload_tokens": 10025, "text_tokens": 10025, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 10025, "image_token_share": 0.0, "payload_bytes": 38964, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 21130, "tokens_cached": 146138, "tokens_out": 1041, "tokens": 168309, "duration_ms": 45260.81, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 7, "client_tool_calls": 7, "client_tool_calls_total": 7, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 10265.11, "payload_tokens": 9470, "text_tokens": 9470, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 9470, "image_token_share": 0.0, "payload_bytes": 36973, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 20338, "tokens_cached": 126327, "tokens_out": 1832, "tokens": 148497, "duration_ms": 68895.25, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 9, "client_tool_calls": 9, "client_tool_calls_total": 9, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7001.82, "payload_tokens": 9753, "text_tokens": 9753, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 9753, "image_token_share": 0.0, "payload_bytes": 37706, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 20398, "tokens_cached": 164145, "tokens_out": 581, "tokens": 185124, "duration_ms": 30043.59, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7021.36, "payload_tokens": 8916, "text_tokens": 8916, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8916, "image_token_share": 0.0, "payload_bytes": 35033, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 19398, "tokens_cached": 84787, "tokens_out": 414, "tokens": 104599, "duration_ms": 20664.97, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "playwright", "task": "catalogue-last-page", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 3.8, "min": 2.0, "max": 5.0, "stdev": 1.3 }, "client": { "median": 4.0, "mean": 3.8, "min": 2.0, "max": 5.0, "stdev": 1.3 } }, "payload_tokens": { "median": 8141.0, "mean": 8276.0, "min": 7830.0, "max": 8746.0, "stdev": 443.8 }, "client_tokens": { "input": { "median": 18144.0, "mean": 18264.0, "min": 17701.0, "max": 18813.0, "stdev": 525.93 }, "cached": { "median": 65019.0, "mean": 61175.4, "min": 28311.0, "max": 83182.0, "stdev": 23845.48 }, "output": { "median": 451.0, "mean": 424.0, "min": 251.0, "max": 506.0, "stdev": 99.54 } }, "tokens": { "median": 83669.0, "mean": 79863.4, "min": 46463.0, "max": 102461.0, "stdev": 24387.58 }, "duration_ms": { "median": 18015.51, "mean": 21207.24, "min": 15134.68, "max": 31569.0, "stdev": 6802.66 }, "time_to_first_action_ms": { "median": 7997.57, "mean": 8400.78, "min": 6789.53, "max": 12193.52, "stdev": 2207.94 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 3.8, "min": 2.0, "max": 5.0, "stdev": 1.3 }, "client": { "median": 4.0, "mean": 3.8, "min": 2.0, "max": 5.0, "stdev": 1.3 } }, "payload_tokens": { "median": 8141.0, "mean": 8276.0, "min": 7830.0, "max": 8746.0, "stdev": 443.8 }, "text_tokens": { "median": 8141.0, "mean": 8276.0, "min": 7830.0, "max": 8746.0, "stdev": 443.8 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8141.0, "mean": 8276.0, "min": 7830.0, "max": 8746.0, "stdev": 443.8 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 32516.0, "mean": 33132.2, "min": 31526.0, "max": 34880.0, "stdev": 1634.8 }, "client_tokens": { "input": { "median": 18144.0, "mean": 18264.0, "min": 17701.0, "max": 18813.0, "stdev": 525.93 }, "cached": { "median": 65019.0, "mean": 61175.4, "min": 28311.0, "max": 83182.0, "stdev": 23845.48 }, "output": { "median": 451.0, "mean": 424.0, "min": 251.0, "max": 506.0, "stdev": 99.54 } }, "tokens": { "median": 83669.0, "mean": 79863.4, "min": 46463.0, "max": 102461.0, "stdev": 24387.58 }, "duration_ms": { "median": 18015.51, "mean": 21207.24, "min": 15134.68, "max": 31569.0, "stdev": 6802.66 }, "time_to_first_action_ms": { "median": 7997.57, "mean": 8400.78, "min": 6789.53, "max": 12193.52, "stdev": 2207.94 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 3.8, "min": 2.0, "max": 5.0, "stdev": 1.3 }, "client": { "median": 4.0, "mean": 3.8, "min": 2.0, "max": 5.0, "stdev": 1.3 } }, "payload_tokens": { "median": 8141.0, "mean": 8276.0, "min": 7830.0, "max": 8746.0, "stdev": 443.8 }, "text_tokens": { "median": 8141.0, "mean": 8276.0, "min": 7830.0, "max": 8746.0, "stdev": 443.8 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8141.0, "mean": 8276.0, "min": 7830.0, "max": 8746.0, "stdev": 443.8 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 32516.0, "mean": 33132.2, "min": 31526.0, "max": 34880.0, "stdev": 1634.8 }, "client_tokens": { "input": { "median": 18144.0, "mean": 18264.0, "min": 17701.0, "max": 18813.0, "stdev": 525.93 }, "cached": { "median": 65019.0, "mean": 61175.4, "min": 28311.0, "max": 83182.0, "stdev": 23845.48 }, "output": { "median": 451.0, "mean": 424.0, "min": 251.0, "max": 506.0, "stdev": 99.54 } }, "tokens": { "median": 83669.0, "mean": 79863.4, "min": 46463.0, "max": 102461.0, "stdev": 24387.58 }, "duration_ms": { "median": 18015.51, "mean": 21207.24, "min": 15134.68, "max": 31569.0, "stdev": 6802.66 }, "time_to_first_action_ms": { "median": 7997.57, "mean": 8400.78, "min": 6789.53, "max": 12193.52, "stdev": 2207.94 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8134.24, "payload_tokens": 8141, "text_tokens": 8141, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8141, "image_token_share": 0.0, "payload_bytes": 32516, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 18144, "tokens_cached": 65019, "tokens_out": 506, "tokens": 83669, "duration_ms": 24536.24, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 3, "client_tool_calls": 3, "client_tool_calls_total": 3, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7997.57, "payload_tokens": 7917, "text_tokens": 7917, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 7917, "image_token_share": 0.0, "payload_bytes": 31859, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 17849, "tokens_cached": 46370, "tokens_out": 251, "tokens": 64470, "duration_ms": 16780.79, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 12193.52, "payload_tokens": 8746, "text_tokens": 8746, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8746, "image_token_share": 0.0, "payload_bytes": 34880, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 18813, "tokens_cached": 83182, "tokens_out": 466, "tokens": 102461, "duration_ms": 31569.0, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6789.53, "payload_tokens": 7830, "text_tokens": 7830, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 7830, "image_token_share": 0.0, "payload_bytes": 31526, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 17701, "tokens_cached": 28311, "tokens_out": 451, "tokens": 46463, "duration_ms": 15134.68, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6889.05, "payload_tokens": 8746, "text_tokens": 8746, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8746, "image_token_share": 0.0, "payload_bytes": 34880, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 18813, "tokens_cached": 82995, "tokens_out": 446, "tokens": 102254, "duration_ms": 18015.51, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "playwright", "task": "book-upc-table", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1218.0, "mean": 1218.0, "min": 1218.0, "max": 1218.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 15403.0, "mean": 15401.6, "min": 15389.0, "max": 15410.0, "stdev": 9.13 }, "cached": { "median": 28264.0, "mean": 28264.4, "min": 28229.0, "max": 28287.0, "stdev": 23.13 }, "output": { "median": 115.0, "mean": 117.0, "min": 96.0, "max": 142.0, "stdev": 20.74 } }, "tokens": { "median": 43801.0, "mean": 43783.0, "min": 43716.0, "max": 43835.0, "stdev": 48.43 }, "duration_ms": { "median": 10651.12, "mean": 10674.29, "min": 9431.28, "max": 12494.06, "stdev": 1213.88 }, "time_to_first_action_ms": { "median": 7104.03, "mean": 7038.52, "min": 6215.74, "max": 8157.64, "stdev": 759.89 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1218.0, "mean": 1218.0, "min": 1218.0, "max": 1218.0, "stdev": 0.0 }, "text_tokens": { "median": 1218.0, "mean": 1218.0, "min": 1218.0, "max": 1218.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1218.0, "mean": 1218.0, "min": 1218.0, "max": 1218.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 4269.0, "mean": 4269.0, "min": 4269.0, "max": 4269.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 15403.0, "mean": 15401.6, "min": 15389.0, "max": 15410.0, "stdev": 9.13 }, "cached": { "median": 28264.0, "mean": 28264.4, "min": 28229.0, "max": 28287.0, "stdev": 23.13 }, "output": { "median": 115.0, "mean": 117.0, "min": 96.0, "max": 142.0, "stdev": 20.74 } }, "tokens": { "median": 43801.0, "mean": 43783.0, "min": 43716.0, "max": 43835.0, "stdev": 48.43 }, "duration_ms": { "median": 10651.12, "mean": 10674.29, "min": 9431.28, "max": 12494.06, "stdev": 1213.88 }, "time_to_first_action_ms": { "median": 7104.03, "mean": 7038.52, "min": 6215.74, "max": 8157.64, "stdev": 759.89 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1218.0, "mean": 1218.0, "min": 1218.0, "max": 1218.0, "stdev": 0.0 }, "text_tokens": { "median": 1218.0, "mean": 1218.0, "min": 1218.0, "max": 1218.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1218.0, "mean": 1218.0, "min": 1218.0, "max": 1218.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 4269.0, "mean": 4269.0, "min": 4269.0, "max": 4269.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 15403.0, "mean": 15401.6, "min": 15389.0, "max": 15410.0, "stdev": 9.13 }, "cached": { "median": 28264.0, "mean": 28264.4, "min": 28229.0, "max": 28287.0, "stdev": 23.13 }, "output": { "median": 115.0, "mean": 117.0, "min": 96.0, "max": 142.0, "stdev": 20.74 } }, "tokens": { "median": 43801.0, "mean": 43783.0, "min": 43716.0, "max": 43835.0, "stdev": 48.43 }, "duration_ms": { "median": 10651.12, "mean": 10674.29, "min": 9431.28, "max": 12494.06, "stdev": 1213.88 }, "time_to_first_action_ms": { "median": 7104.03, "mean": 7038.52, "min": 6215.74, "max": 8157.64, "stdev": 759.89 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7104.03, "payload_tokens": 1218, "text_tokens": 1218, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1218, "image_token_share": 0.0, "payload_bytes": 4269, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15410, "tokens_cached": 28287, "tokens_out": 115, "tokens": 43812, "duration_ms": 11062.18, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7251.66, "payload_tokens": 1218, "text_tokens": 1218, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1218, "image_token_share": 0.0, "payload_bytes": 4269, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15396, "tokens_cached": 28259, "tokens_out": 96, "tokens": 43751, "duration_ms": 10651.12, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8157.64, "payload_tokens": 1218, "text_tokens": 1218, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1218, "image_token_share": 0.0, "payload_bytes": 4269, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15403, "tokens_cached": 28264, "tokens_out": 134, "tokens": 43801, "duration_ms": 12494.06, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6463.54, "payload_tokens": 1218, "text_tokens": 1218, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1218, "image_token_share": 0.0, "payload_bytes": 4269, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15389, "tokens_cached": 28229, "tokens_out": 98, "tokens": 43716, "duration_ms": 9431.28, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6215.74, "payload_tokens": 1218, "text_tokens": 1218, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1218, "image_token_share": 0.0, "payload_bytes": 4269, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15410, "tokens_cached": 28283, "tokens_out": 142, "tokens": 43835, "duration_ms": 9732.79, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "playwright", "task": "travel-stock-drilldown", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 815.0, "mean": 815.0, "min": 815.0, "max": 815.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 15100.0, "mean": 15098.6, "min": 15093.0, "max": 15100.0, "stdev": 3.13 }, "cached": { "median": 58180.0, "mean": 58188.0, "min": 58157.0, "max": 58249.0, "stdev": 35.76 }, "output": { "median": 136.0, "mean": 137.0, "min": 119.0, "max": 156.0, "stdev": 13.17 } }, "tokens": { "median": 73419.0, "mean": 73423.6, "min": 73376.0, "max": 73505.0, "stdev": 49.06 }, "duration_ms": { "median": 13811.29, "mean": 13936.88, "min": 11851.51, "max": 15503.41, "stdev": 1394.41 }, "time_to_first_action_ms": { "median": 7570.21, "mean": 7626.85, "min": 6488.64, "max": 8676.98, "stdev": 783.25 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 815.0, "mean": 815.0, "min": 815.0, "max": 815.0, "stdev": 0.0 }, "text_tokens": { "median": 815.0, "mean": 815.0, "min": 815.0, "max": 815.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 815.0, "mean": 815.0, "min": 815.0, "max": 815.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2793.0, "mean": 2793.0, "min": 2793.0, "max": 2793.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 15100.0, "mean": 15098.6, "min": 15093.0, "max": 15100.0, "stdev": 3.13 }, "cached": { "median": 58180.0, "mean": 58188.0, "min": 58157.0, "max": 58249.0, "stdev": 35.76 }, "output": { "median": 136.0, "mean": 137.0, "min": 119.0, "max": 156.0, "stdev": 13.17 } }, "tokens": { "median": 73419.0, "mean": 73423.6, "min": 73376.0, "max": 73505.0, "stdev": 49.06 }, "duration_ms": { "median": 13811.29, "mean": 13936.88, "min": 11851.51, "max": 15503.41, "stdev": 1394.41 }, "time_to_first_action_ms": { "median": 7570.21, "mean": 7626.85, "min": 6488.64, "max": 8676.98, "stdev": 783.25 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 815.0, "mean": 815.0, "min": 815.0, "max": 815.0, "stdev": 0.0 }, "text_tokens": { "median": 815.0, "mean": 815.0, "min": 815.0, "max": 815.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 815.0, "mean": 815.0, "min": 815.0, "max": 815.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 2793.0, "mean": 2793.0, "min": 2793.0, "max": 2793.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 15100.0, "mean": 15098.6, "min": 15093.0, "max": 15100.0, "stdev": 3.13 }, "cached": { "median": 58180.0, "mean": 58188.0, "min": 58157.0, "max": 58249.0, "stdev": 35.76 }, "output": { "median": 136.0, "mean": 137.0, "min": 119.0, "max": 156.0, "stdev": 13.17 } }, "tokens": { "median": 73419.0, "mean": 73423.6, "min": 73376.0, "max": 73505.0, "stdev": 49.06 }, "duration_ms": { "median": 13811.29, "mean": 13936.88, "min": 11851.51, "max": 15503.41, "stdev": 1394.41 }, "time_to_first_action_ms": { "median": 7570.21, "mean": 7626.85, "min": 6488.64, "max": 8676.98, "stdev": 783.25 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7570.21, "payload_tokens": 815, "text_tokens": 815, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 815, "image_token_share": 0.0, "payload_bytes": 2793, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15100, "tokens_cached": 58157, "tokens_out": 119, "tokens": 73376, "duration_ms": 13811.29, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7554.35, "payload_tokens": 815, "text_tokens": 815, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 815, "image_token_share": 0.0, "payload_bytes": 2793, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15100, "tokens_cached": 58185, "tokens_out": 136, "tokens": 73421, "duration_ms": 13646.18, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8676.98, "payload_tokens": 815, "text_tokens": 815, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 815, "image_token_share": 0.0, "payload_bytes": 2793, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15100, "tokens_cached": 58180, "tokens_out": 139, "tokens": 73419, "duration_ms": 15503.41, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6488.64, "payload_tokens": 815, "text_tokens": 815, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 815, "image_token_share": 0.0, "payload_bytes": 2793, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15093, "tokens_cached": 58169, "tokens_out": 135, "tokens": 73397, "duration_ms": 11851.51, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7844.09, "payload_tokens": 815, "text_tokens": 815, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 815, "image_token_share": 0.0, "payload_bytes": 2793, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15100, "tokens_cached": 58249, "tokens_out": 156, "tokens": 73505, "duration_ms": 14872.02, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "playwright", "task": "quote-form-submit", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 11.0, "mean": 10.0, "min": 6.0, "max": 14.0, "stdev": 3.39 }, "client": { "median": 11.0, "mean": 10.0, "min": 6.0, "max": 14.0, "stdev": 3.39 } }, "payload_tokens": { "median": 1252.0, "mean": 1172.2, "min": 862.0, "max": 1456.0, "stdev": 255.82 }, "client_tokens": { "input": { "median": 15934.0, "mean": 15697.8, "min": 15195.0, "max": 16088.0, "stdev": 405.27 }, "cached": { "median": 125702.0, "mean": 116867.8, "min": 74128.0, "max": 155061.0, "stdev": 34048.82 }, "output": { "median": 335.0, "mean": 293.8, "min": 179.0, "max": 391.0, "stdev": 87.23 } }, "tokens": { "median": 142125.0, "mean": 132859.4, "min": 89502.0, "max": 171331.0, "stdev": 34494.17 }, "duration_ms": { "median": 35352.75, "mean": 29686.77, "min": 14954.28, "max": 39862.12, "stdev": 11247.46 }, "time_to_first_action_ms": { "median": 6619.37, "mean": 7129.05, "min": 6343.88, "max": 8298.96, "stdev": 937.04 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 11.0, "mean": 10.0, "min": 6.0, "max": 14.0, "stdev": 3.39 }, "client": { "median": 11.0, "mean": 10.0, "min": 6.0, "max": 14.0, "stdev": 3.39 } }, "payload_tokens": { "median": 1252.0, "mean": 1172.2, "min": 862.0, "max": 1456.0, "stdev": 255.82 }, "text_tokens": { "median": 1252.0, "mean": 1172.2, "min": 862.0, "max": 1456.0, "stdev": 255.82 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1252.0, "mean": 1172.2, "min": 862.0, "max": 1456.0, "stdev": 255.82 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 4080.0, "mean": 3868.4, "min": 2773.0, "max": 4787.0, "stdev": 926.98 }, "client_tokens": { "input": { "median": 15934.0, "mean": 15697.8, "min": 15195.0, "max": 16088.0, "stdev": 405.27 }, "cached": { "median": 125702.0, "mean": 116867.8, "min": 74128.0, "max": 155061.0, "stdev": 34048.82 }, "output": { "median": 335.0, "mean": 293.8, "min": 179.0, "max": 391.0, "stdev": 87.23 } }, "tokens": { "median": 142125.0, "mean": 132859.4, "min": 89502.0, "max": 171331.0, "stdev": 34494.17 }, "duration_ms": { "median": 35352.75, "mean": 29686.77, "min": 14954.28, "max": 39862.12, "stdev": 11247.46 }, "time_to_first_action_ms": { "median": 6619.37, "mean": 7129.05, "min": 6343.88, "max": 8298.96, "stdev": 937.04 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 11.0, "mean": 10.0, "min": 6.0, "max": 14.0, "stdev": 3.39 }, "client": { "median": 11.0, "mean": 10.0, "min": 6.0, "max": 14.0, "stdev": 3.39 } }, "payload_tokens": { "median": 1252.0, "mean": 1172.2, "min": 862.0, "max": 1456.0, "stdev": 255.82 }, "text_tokens": { "median": 1252.0, "mean": 1172.2, "min": 862.0, "max": 1456.0, "stdev": 255.82 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1252.0, "mean": 1172.2, "min": 862.0, "max": 1456.0, "stdev": 255.82 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 4080.0, "mean": 3868.4, "min": 2773.0, "max": 4787.0, "stdev": 926.98 }, "client_tokens": { "input": { "median": 15934.0, "mean": 15697.8, "min": 15195.0, "max": 16088.0, "stdev": 405.27 }, "cached": { "median": 125702.0, "mean": 116867.8, "min": 74128.0, "max": 155061.0, "stdev": 34048.82 }, "output": { "median": 335.0, "mean": 293.8, "min": 179.0, "max": 391.0, "stdev": 87.23 } }, "tokens": { "median": 142125.0, "mean": 132859.4, "min": 89502.0, "max": 171331.0, "stdev": 34494.17 }, "duration_ms": { "median": 35352.75, "mean": 29686.77, "min": 14954.28, "max": 39862.12, "stdev": 11247.46 }, "time_to_first_action_ms": { "median": 6619.37, "mean": 7129.05, "min": 6343.88, "max": 8298.96, "stdev": 937.04 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 6, "client_tool_calls": 6, "client_tool_calls_total": 6, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6397.97, "payload_tokens": 862, "text_tokens": 862, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 862, "image_token_share": 0.0, "payload_bytes": 2773, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15195, "tokens_cached": 74128, "tokens_out": 179, "tokens": 89502, "duration_ms": 14954.28, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 11, "client_tool_calls": 11, "client_tool_calls_total": 11, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6343.88, "payload_tokens": 1456, "text_tokens": 1456, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1456, "image_token_share": 0.0, "payload_bytes": 4787, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 16088, "tokens_cached": 125702, "tokens_out": 335, "tokens": 142125, "duration_ms": 35352.75, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 14, "client_tool_calls": 14, "client_tool_calls_total": 14, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8298.96, "payload_tokens": 1252, "text_tokens": 1252, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1252, "image_token_share": 0.0, "payload_bytes": 4080, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15934, "tokens_cached": 155061, "tokens_out": 336, "tokens": 171331, "duration_ms": 39862.12, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 7, "client_tool_calls": 7, "client_tool_calls_total": 7, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7985.06, "payload_tokens": 949, "text_tokens": 949, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 949, "image_token_share": 0.0, "payload_bytes": 3030, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15329, "tokens_cached": 89578, "tokens_out": 228, "tokens": 105135, "duration_ms": 20393.96, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 12, "client_tool_calls": 12, "client_tool_calls_total": 12, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6619.37, "payload_tokens": 1342, "text_tokens": 1342, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1342, "image_token_share": 0.0, "payload_bytes": 4672, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 15943, "tokens_cached": 139870, "tokens_out": 391, "tokens": 156204, "duration_ms": 37870.75, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "playwright", "task": "restricted-figure", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 14666.0, "mean": 14668.8, "min": 14666.0, "max": 14673.0, "stdev": 3.83 }, "cached": { "median": 28271.0, "mean": 28269.8, "min": 28247.0, "max": 28292.0, "stdev": 18.21 }, "output": { "median": 276.0, "mean": 263.6, "min": 167.0, "max": 318.0, "stdev": 57.95 } }, "tokens": { "median": 43224.0, "mean": 43202.2, "min": 43097.0, "max": 43276.0, "stdev": 69.07 }, "duration_ms": { "median": 15718.18, "mean": 15872.84, "min": 9553.81, "max": 21873.33, "stdev": 4659.86 }, "time_to_first_action_ms": { "median": 9421.25, "mean": 9524.03, "min": 6279.31, "max": 12915.25, "stdev": 2845.25 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "text_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1747.0, "mean": 1747.0, "min": 1747.0, "max": 1747.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 14666.0, "mean": 14668.8, "min": 14666.0, "max": 14673.0, "stdev": 3.83 }, "cached": { "median": 28271.0, "mean": 28269.8, "min": 28247.0, "max": 28292.0, "stdev": 18.21 }, "output": { "median": 276.0, "mean": 263.6, "min": 167.0, "max": 318.0, "stdev": 57.95 } }, "tokens": { "median": 43224.0, "mean": 43202.2, "min": 43097.0, "max": 43276.0, "stdev": 69.07 }, "duration_ms": { "median": 15718.18, "mean": 15872.84, "min": 9553.81, "max": 21873.33, "stdev": 4659.86 }, "time_to_first_action_ms": { "median": 9421.25, "mean": 9524.03, "min": 6279.31, "max": 12915.25, "stdev": 2845.25 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "text_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 562.0, "mean": 562.0, "min": 562.0, "max": 562.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1747.0, "mean": 1747.0, "min": 1747.0, "max": 1747.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 14666.0, "mean": 14668.8, "min": 14666.0, "max": 14673.0, "stdev": 3.83 }, "cached": { "median": 28271.0, "mean": 28269.8, "min": 28247.0, "max": 28292.0, "stdev": 18.21 }, "output": { "median": 276.0, "mean": 263.6, "min": 167.0, "max": 318.0, "stdev": 57.95 } }, "tokens": { "median": 43224.0, "mean": 43202.2, "min": 43097.0, "max": 43276.0, "stdev": 69.07 }, "duration_ms": { "median": 15718.18, "mean": 15872.84, "min": 9553.81, "max": 21873.33, "stdev": 4659.86 }, "time_to_first_action_ms": { "median": 9421.25, "mean": 9524.03, "min": 6279.31, "max": 12915.25, "stdev": 2845.25 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6279.31, "payload_tokens": 562, "text_tokens": 562, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 562, "image_token_share": 0.0, "payload_bytes": 1747, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 14673, "tokens_cached": 28257, "tokens_out": 167, "tokens": 43097, "duration_ms": 9553.81, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 12915.25, "payload_tokens": 562, "text_tokens": 562, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 562, "image_token_share": 0.0, "payload_bytes": 1747, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 14666, "tokens_cached": 28292, "tokens_out": 318, "tokens": 43276, "duration_ms": 21873.33, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 11776.12, "payload_tokens": 562, "text_tokens": 562, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 562, "image_token_share": 0.0, "payload_bytes": 1747, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 14666, "tokens_cached": 28282, "tokens_out": 276, "tokens": 43224, "duration_ms": 18431.65, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7228.2, "payload_tokens": 562, "text_tokens": 562, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 562, "image_token_share": 0.0, "payload_bytes": 1747, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 14666, "tokens_cached": 28247, "tokens_out": 262, "tokens": 43175, "duration_ms": 13787.24, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 9421.25, "payload_tokens": 562, "text_tokens": 562, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 562, "image_token_share": 0.0, "payload_bytes": 1747, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 24, "tools_list_tokens": 3928, "transport_available": true, "tokens_in": 14673, "tokens_cached": 28271, "tokens_out": 295, "tokens": 43239, "duration_ms": 15718.18, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "chrome-devtools", "task": "cheapest-mystery-book", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 }, "client": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 } }, "payload_tokens": { "median": 9263.0, "mean": 9254.2, "min": 9229.0, "max": 9263.0, "stdev": 14.74 }, "client_tokens": { "input": { "median": 20482.0, "mean": 20473.8, "min": 20431.0, "max": 20496.0, "stdev": 25.65 }, "cached": { "median": 91354.0, "mean": 91378.4, "min": 91244.0, "max": 91642.0, "stdev": 156.13 }, "output": { "median": 654.0, "mean": 690.4, "min": 617.0, "max": 797.0, "stdev": 78.03 } }, "tokens": { "median": 112530.0, "mean": 112542.6, "min": 112389.0, "max": 112778.0, "stdev": 156.33 }, "duration_ms": { "median": 29781.72, "mean": 30189.7, "min": 23964.07, "max": 39268.83, "stdev": 6489.62 }, "time_to_first_action_ms": { "median": 8726.32, "mean": 8767.84, "min": 7286.44, "max": 11175.35, "stdev": 1489.85 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 }, "client": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 } }, "payload_tokens": { "median": 9263.0, "mean": 9254.2, "min": 9229.0, "max": 9263.0, "stdev": 14.74 }, "text_tokens": { "median": 9263.0, "mean": 9254.2, "min": 9229.0, "max": 9263.0, "stdev": 14.74 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 9263.0, "mean": 9254.2, "min": 9229.0, "max": 9263.0, "stdev": 14.74 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 27589.0, "mean": 27547.8, "min": 27453.0, "max": 27589.0, "stdev": 61.05 }, "client_tokens": { "input": { "median": 20482.0, "mean": 20473.8, "min": 20431.0, "max": 20496.0, "stdev": 25.65 }, "cached": { "median": 91354.0, "mean": 91378.4, "min": 91244.0, "max": 91642.0, "stdev": 156.13 }, "output": { "median": 654.0, "mean": 690.4, "min": 617.0, "max": 797.0, "stdev": 78.03 } }, "tokens": { "median": 112530.0, "mean": 112542.6, "min": 112389.0, "max": 112778.0, "stdev": 156.33 }, "duration_ms": { "median": 29781.72, "mean": 30189.7, "min": 23964.07, "max": 39268.83, "stdev": 6489.62 }, "time_to_first_action_ms": { "median": 8726.32, "mean": 8767.84, "min": 7286.44, "max": 11175.35, "stdev": 1489.85 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 }, "client": { "median": 5.0, "mean": 5.0, "min": 5.0, "max": 5.0, "stdev": 0.0 } }, "payload_tokens": { "median": 9263.0, "mean": 9254.2, "min": 9229.0, "max": 9263.0, "stdev": 14.74 }, "text_tokens": { "median": 9263.0, "mean": 9254.2, "min": 9229.0, "max": 9263.0, "stdev": 14.74 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 9263.0, "mean": 9254.2, "min": 9229.0, "max": 9263.0, "stdev": 14.74 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 27589.0, "mean": 27547.8, "min": 27453.0, "max": 27589.0, "stdev": 61.05 }, "client_tokens": { "input": { "median": 20482.0, "mean": 20473.8, "min": 20431.0, "max": 20496.0, "stdev": 25.65 }, "cached": { "median": 91354.0, "mean": 91378.4, "min": 91244.0, "max": 91642.0, "stdev": 156.13 }, "output": { "median": 654.0, "mean": 690.4, "min": 617.0, "max": 797.0, "stdev": 78.03 } }, "tokens": { "median": 112530.0, "mean": 112542.6, "min": 112389.0, "max": 112778.0, "stdev": 156.33 }, "duration_ms": { "median": 29781.72, "mean": 30189.7, "min": 23964.07, "max": 39268.83, "stdev": 6489.62 }, "time_to_first_action_ms": { "median": 8726.32, "mean": 8767.84, "min": 7286.44, "max": 11175.35, "stdev": 1489.85 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8726.32, "payload_tokens": 9229, "text_tokens": 9229, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 9229, "image_token_share": 0.0, "payload_bytes": 27453, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 20431, "tokens_cached": 91370, "tokens_out": 617, "tokens": 112418, "duration_ms": 29781.72, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8823.01, "payload_tokens": 9263, "text_tokens": 9263, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 9263, "image_token_share": 0.0, "payload_bytes": 27589, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 20489, "tokens_cached": 91244, "tokens_out": 797, "tokens": 112530, "duration_ms": 33663.47, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 11175.35, "payload_tokens": 9263, "text_tokens": 9263, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 9263, "image_token_share": 0.0, "payload_bytes": 27589, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 20496, "tokens_cached": 91354, "tokens_out": 748, "tokens": 112598, "duration_ms": 39268.83, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7286.44, "payload_tokens": 9253, "text_tokens": 9253, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 9253, "image_token_share": 0.0, "payload_bytes": 27519, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 20471, "tokens_cached": 91282, "tokens_out": 636, "tokens": 112389, "duration_ms": 23964.07, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "Tastes Like Fear (DI Marnie Rome #3)", "answer_format_ok": true, "expected": "Tastes Like Fear (DI Marnie Rome #3)", "grading_rule": "Normalised answer must contain the exact ground-truth title (Tastes Like Fear).", "transport_tool_calls": 5, "client_tool_calls": 5, "client_tool_calls_total": 5, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7828.07, "payload_tokens": 9263, "text_tokens": 9263, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 9263, "image_token_share": 0.0, "payload_bytes": 27589, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 20482, "tokens_cached": 91642, "tokens_out": 654, "tokens": 112778, "duration_ms": 24270.41, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3c7553024707064d1ce7c5118f9f6e6c3fd10c0bd538bb4f7377f821ff7c5c5d", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "chrome-devtools", "task": "catalogue-last-page", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.2, "min": 2.0, "max": 3.0, "stdev": 0.45 }, "client": { "median": 2.0, "mean": 2.2, "min": 2.0, "max": 3.0, "stdev": 0.45 } }, "payload_tokens": { "median": 8517.0, "mean": 8533.0, "min": 8517.0, "max": 8597.0, "stdev": 35.78 }, "client_tokens": { "input": { "median": 19469.0, "mean": 19493.2, "min": 19469.0, "max": 19583.0, "stdev": 50.29 }, "cached": { "median": 30582.0, "mean": 34574.0, "min": 30565.0, "max": 50553.0, "stdev": 8932.54 }, "output": { "median": 423.0, "mean": 431.2, "min": 366.0, "max": 522.0, "stdev": 56.96 } }, "tokens": { "median": 50481.0, "mean": 54498.4, "min": 50416.0, "max": 70658.0, "stdev": 9033.54 }, "duration_ms": { "median": 17378.43, "mean": 17812.0, "min": 15109.22, "max": 23235.33, "stdev": 3305.67 }, "time_to_first_action_ms": { "median": 8146.95, "mean": 8063.07, "min": 7086.25, "max": 9393.47, "stdev": 923.68 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.2, "min": 2.0, "max": 3.0, "stdev": 0.45 }, "client": { "median": 2.0, "mean": 2.2, "min": 2.0, "max": 3.0, "stdev": 0.45 } }, "payload_tokens": { "median": 8517.0, "mean": 8533.0, "min": 8517.0, "max": 8597.0, "stdev": 35.78 }, "text_tokens": { "median": 8517.0, "mean": 8533.0, "min": 8517.0, "max": 8597.0, "stdev": 35.78 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8517.0, "mean": 8533.0, "min": 8517.0, "max": 8597.0, "stdev": 35.78 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 25112.0, "mean": 25166.0, "min": 25112.0, "max": 25382.0, "stdev": 120.75 }, "client_tokens": { "input": { "median": 19469.0, "mean": 19493.2, "min": 19469.0, "max": 19583.0, "stdev": 50.29 }, "cached": { "median": 30582.0, "mean": 34574.0, "min": 30565.0, "max": 50553.0, "stdev": 8932.54 }, "output": { "median": 423.0, "mean": 431.2, "min": 366.0, "max": 522.0, "stdev": 56.96 } }, "tokens": { "median": 50481.0, "mean": 54498.4, "min": 50416.0, "max": 70658.0, "stdev": 9033.54 }, "duration_ms": { "median": 17378.43, "mean": 17812.0, "min": 15109.22, "max": 23235.33, "stdev": 3305.67 }, "time_to_first_action_ms": { "median": 8146.95, "mean": 8063.07, "min": 7086.25, "max": 9393.47, "stdev": 923.68 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.2, "min": 2.0, "max": 3.0, "stdev": 0.45 }, "client": { "median": 2.0, "mean": 2.2, "min": 2.0, "max": 3.0, "stdev": 0.45 } }, "payload_tokens": { "median": 8517.0, "mean": 8533.0, "min": 8517.0, "max": 8597.0, "stdev": 35.78 }, "text_tokens": { "median": 8517.0, "mean": 8533.0, "min": 8517.0, "max": 8597.0, "stdev": 35.78 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8517.0, "mean": 8533.0, "min": 8517.0, "max": 8597.0, "stdev": 35.78 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 25112.0, "mean": 25166.0, "min": 25112.0, "max": 25382.0, "stdev": 120.75 }, "client_tokens": { "input": { "median": 19469.0, "mean": 19493.2, "min": 19469.0, "max": 19583.0, "stdev": 50.29 }, "cached": { "median": 30582.0, "mean": 34574.0, "min": 30565.0, "max": 50553.0, "stdev": 8932.54 }, "output": { "median": 423.0, "mean": 431.2, "min": 366.0, "max": 522.0, "stdev": 56.96 } }, "tokens": { "median": 50481.0, "mean": 54498.4, "min": 50416.0, "max": 70658.0, "stdev": 9033.54 }, "duration_ms": { "median": 17378.43, "mean": 17812.0, "min": 15109.22, "max": 23235.33, "stdev": 3305.67 }, "time_to_first_action_ms": { "median": 8146.95, "mean": 8063.07, "min": 7086.25, "max": 9393.47, "stdev": 923.68 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8385.6, "payload_tokens": 8517, "text_tokens": 8517, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8517, "image_token_share": 0.0, "payload_bytes": 25112, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 19469, "tokens_cached": 30581, "tokens_out": 366, "tokens": 50416, "duration_ms": 17378.43, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 8146.95, "payload_tokens": 8517, "text_tokens": 8517, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8517, "image_token_share": 0.0, "payload_bytes": 25112, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 19469, "tokens_cached": 30589, "tokens_out": 434, "tokens": 50492, "duration_ms": 18115.78, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 3, "client_tool_calls": 3, "client_tool_calls_total": 3, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 9393.47, "payload_tokens": 8597, "text_tokens": 8597, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8597, "image_token_share": 0.0, "payload_bytes": 25382, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 19583, "tokens_cached": 50553, "tokens_out": 522, "tokens": 70658, "duration_ms": 23235.33, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7086.25, "payload_tokens": 8517, "text_tokens": 8517, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8517, "image_token_share": 0.0, "payload_bytes": 25112, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 19476, "tokens_cached": 30582, "tokens_out": 423, "tokens": 50481, "duration_ms": 15109.22, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "1,000 Places to See Before You Die", "answer_format_ok": true, "expected": "1,000 Places to See Before You Die", "grading_rule": "Normalised answer must contain the exact ground-truth title (1,000 Places to See Before You Die); commas are ignored by normalisation.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7303.09, "payload_tokens": 8517, "text_tokens": 8517, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8517, "image_token_share": 0.0, "payload_bytes": 25112, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 19469, "tokens_cached": 30565, "tokens_out": 411, "tokens": 50445, "duration_ms": 15221.26, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "fa50e646c667bb08ef81cdcf215109b81482fa37d05ee46e2c74be5cba5d0db7", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "chrome-devtools", "task": "book-upc-table", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 16392.0, "mean": 16390.6, "min": 16385.0, "max": 16399.0, "stdev": 5.86 }, "cached": { "median": 30522.0, "mean": 30523.8, "min": 30504.0, "max": 30560.0, "stdev": 21.94 }, "output": { "median": 127.0, "mean": 129.6, "min": 117.0, "max": 152.0, "stdev": 14.06 } }, "tokens": { "median": 47035.0, "mean": 47044.0, "min": 47027.0, "max": 47086.0, "stdev": 24.04 }, "duration_ms": { "median": 13138.26, "mean": 13237.87, "min": 10509.09, "max": 16518.11, "stdev": 2742.18 }, "time_to_first_action_ms": { "median": 7798.23, "mean": 8773.06, "min": 6960.13, "max": 11529.55, "stdev": 2044.42 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "text_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 3512.0, "mean": 3512.0, "min": 3512.0, "max": 3512.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 16392.0, "mean": 16390.6, "min": 16385.0, "max": 16399.0, "stdev": 5.86 }, "cached": { "median": 30522.0, "mean": 30523.8, "min": 30504.0, "max": 30560.0, "stdev": 21.94 }, "output": { "median": 127.0, "mean": 129.6, "min": 117.0, "max": 152.0, "stdev": 14.06 } }, "tokens": { "median": 47035.0, "mean": 47044.0, "min": 47027.0, "max": 47086.0, "stdev": 24.04 }, "duration_ms": { "median": 13138.26, "mean": 13237.87, "min": 10509.09, "max": 16518.11, "stdev": 2742.18 }, "time_to_first_action_ms": { "median": 7798.23, "mean": 8773.06, "min": 6960.13, "max": 11529.55, "stdev": 2044.42 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "text_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 1085.0, "mean": 1085.0, "min": 1085.0, "max": 1085.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 3512.0, "mean": 3512.0, "min": 3512.0, "max": 3512.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 16392.0, "mean": 16390.6, "min": 16385.0, "max": 16399.0, "stdev": 5.86 }, "cached": { "median": 30522.0, "mean": 30523.8, "min": 30504.0, "max": 30560.0, "stdev": 21.94 }, "output": { "median": 127.0, "mean": 129.6, "min": 117.0, "max": 152.0, "stdev": 14.06 } }, "tokens": { "median": 47035.0, "mean": 47044.0, "min": 47027.0, "max": 47086.0, "stdev": 24.04 }, "duration_ms": { "median": 13138.26, "mean": 13237.87, "min": 10509.09, "max": 16518.11, "stdev": 2742.18 }, "time_to_first_action_ms": { "median": 7798.23, "mean": 8773.06, "min": 6960.13, "max": 11529.55, "stdev": 2044.42 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 10351.19, "payload_tokens": 1085, "text_tokens": 1085, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1085, "image_token_share": 0.0, "payload_bytes": 3512, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 16392, "tokens_cached": 30522, "tokens_out": 117, "tokens": 47031, "duration_ms": 16518.11, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7798.23, "payload_tokens": 1085, "text_tokens": 1085, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1085, "image_token_share": 0.0, "payload_bytes": 3512, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 16385, "tokens_cached": 30504, "tokens_out": 152, "tokens": 47041, "duration_ms": 13138.26, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 11529.55, "payload_tokens": 1085, "text_tokens": 1085, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1085, "image_token_share": 0.0, "payload_bytes": 3512, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 16392, "tokens_cached": 30524, "tokens_out": 119, "tokens": 47035, "duration_ms": 15436.81, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6960.13, "payload_tokens": 1085, "text_tokens": 1085, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1085, "image_token_share": 0.0, "payload_bytes": 3512, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 16385, "tokens_cached": 30509, "tokens_out": 133, "tokens": 47027, "duration_ms": 10509.09, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "a897fe39b1053632", "answer_format_ok": true, "expected": "a897fe39b1053632", "grading_rule": "Answer must contain the exact 16-character UPC as a whole word: regex \\ba897fe39b1053632\\b, case-insensitive.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7226.18, "payload_tokens": 1085, "text_tokens": 1085, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 1085, "image_token_share": 0.0, "payload_bytes": 3512, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 16399, "tokens_cached": 30560, "tokens_out": 127, "tokens": 47086, "duration_ms": 10587.07, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "f800ef524a292009003356772f0ff1708d5000de43c7b404622d2f8e33137c35", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "chrome-devtools", "task": "travel-stock-drilldown", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 22729.0, "mean": 22731.8, "min": 22729.0, "max": 22736.0, "stdev": 3.83 }, "cached": { "median": 69911.0, "mean": 69909.8, "min": 69875.0, "max": 69945.0, "stdev": 27.22 }, "output": { "median": 172.0, "mean": 172.8, "min": 163.0, "max": 180.0, "stdev": 6.46 } }, "tokens": { "median": 92820.0, "mean": 92814.4, "min": 92767.0, "max": 92858.0, "stdev": 35.15 }, "duration_ms": { "median": 14725.46, "mean": 16128.9, "min": 14198.07, "max": 18823.93, "stdev": 2360.54 }, "time_to_first_action_ms": { "median": 7250.57, "mean": 7554.37, "min": 6828.13, "max": 9109.01, "stdev": 945.47 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "text_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 26962.0, "mean": 26962.0, "min": 26962.0, "max": 26962.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 22729.0, "mean": 22731.8, "min": 22729.0, "max": 22736.0, "stdev": 3.83 }, "cached": { "median": 69911.0, "mean": 69909.8, "min": 69875.0, "max": 69945.0, "stdev": 27.22 }, "output": { "median": 172.0, "mean": 172.8, "min": 163.0, "max": 180.0, "stdev": 6.46 } }, "tokens": { "median": 92820.0, "mean": 92814.4, "min": 92767.0, "max": 92858.0, "stdev": 35.15 }, "duration_ms": { "median": 14725.46, "mean": 16128.9, "min": 14198.07, "max": 18823.93, "stdev": 2360.54 }, "time_to_first_action_ms": { "median": 7250.57, "mean": 7554.37, "min": 6828.13, "max": 9109.01, "stdev": 945.47 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 }, "client": { "median": 4.0, "mean": 4.0, "min": 4.0, "max": 4.0, "stdev": 0.0 } }, "payload_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "text_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 8771.0, "mean": 8771.0, "min": 8771.0, "max": 8771.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 26962.0, "mean": 26962.0, "min": 26962.0, "max": 26962.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 22729.0, "mean": 22731.8, "min": 22729.0, "max": 22736.0, "stdev": 3.83 }, "cached": { "median": 69911.0, "mean": 69909.8, "min": 69875.0, "max": 69945.0, "stdev": 27.22 }, "output": { "median": 172.0, "mean": 172.8, "min": 163.0, "max": 180.0, "stdev": 6.46 } }, "tokens": { "median": 92820.0, "mean": 92814.4, "min": 92767.0, "max": 92858.0, "stdev": 35.15 }, "duration_ms": { "median": 14725.46, "mean": 16128.9, "min": 14198.07, "max": 18823.93, "stdev": 2360.54 }, "time_to_first_action_ms": { "median": 7250.57, "mean": 7554.37, "min": 6828.13, "max": 9109.01, "stdev": 945.47 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7250.57, "payload_tokens": 8771, "text_tokens": 8771, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8771, "image_token_share": 0.0, "payload_bytes": 26962, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 22729, "tokens_cached": 69875, "tokens_out": 163, "tokens": 92767, "duration_ms": 14725.46, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6844.07, "payload_tokens": 8771, "text_tokens": 8771, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8771, "image_token_share": 0.0, "payload_bytes": 26962, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 22729, "tokens_cached": 69893, "tokens_out": 172, "tokens": 92794, "duration_ms": 14312.69, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 9109.01, "payload_tokens": 8771, "text_tokens": 8771, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8771, "image_token_share": 0.0, "payload_bytes": 26962, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 22736, "tokens_cached": 69945, "tokens_out": 177, "tokens": 92858, "duration_ms": 18584.33, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6828.13, "payload_tokens": 8771, "text_tokens": 8771, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8771, "image_token_share": 0.0, "payload_bytes": 26962, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 22736, "tokens_cached": 69925, "tokens_out": 172, "tokens": 92833, "duration_ms": 14198.07, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "6", "answer_format_ok": true, "expected": "6", "grading_rule": "The FIRST standalone integer in the answer must be exactly 6. Decimals such as 30.54 do not count as standalone integers.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7740.08, "payload_tokens": 8771, "text_tokens": 8771, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 8771, "image_token_share": 0.0, "payload_bytes": 26962, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 22729, "tokens_cached": 69911, "tokens_out": 180, "tokens": 92820, "duration_ms": 18823.93, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "587fa41b50feb8e68545fcefa1f612639f9ccb82b8e95a8242189e3b61447866", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "chrome-devtools", "task": "quote-form-submit", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 7.0, "mean": 6.6, "min": 4.0, "max": 8.0, "stdev": 1.52 }, "client": { "median": 7.0, "mean": 6.6, "min": 4.0, "max": 8.0, "stdev": 1.52 } }, "payload_tokens": { "median": 515.0, "mean": 653.0, "min": 502.0, "max": 872.0, "stdev": 195.01 }, "client_tokens": { "input": { "median": 16179.0, "mean": 16220.4, "min": 16009.0, "max": 16449.0, "stdev": 218.01 }, "cached": { "median": 111301.0, "mean": 105115.6, "min": 62816.0, "max": 128907.0, "stdev": 24847.05 }, "output": { "median": 248.0, "mean": 233.4, "min": 149.0, "max": 289.0, "stdev": 51.66 } }, "tokens": { "median": 127562.0, "mean": 121569.4, "min": 79411.0, "max": 145645.0, "stdev": 24829.26 }, "duration_ms": { "median": 20986.59, "mean": 20877.91, "min": 16444.89, "max": 27987.78, "stdev": 4655.88 }, "time_to_first_action_ms": { "median": 7213.5, "mean": 8138.99, "min": 6880.15, "max": 11496.6, "stdev": 1912.95 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 7.0, "mean": 6.6, "min": 4.0, "max": 8.0, "stdev": 1.52 }, "client": { "median": 7.0, "mean": 6.6, "min": 4.0, "max": 8.0, "stdev": 1.52 } }, "payload_tokens": { "median": 515.0, "mean": 653.0, "min": 502.0, "max": 872.0, "stdev": 195.01 }, "text_tokens": { "median": 515.0, "mean": 653.0, "min": 502.0, "max": 872.0, "stdev": 195.01 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 515.0, "mean": 653.0, "min": 502.0, "max": 872.0, "stdev": 195.01 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1753.0, "mean": 2213.4, "min": 1710.0, "max": 2966.0, "stdev": 650.92 }, "client_tokens": { "input": { "median": 16179.0, "mean": 16220.4, "min": 16009.0, "max": 16449.0, "stdev": 218.01 }, "cached": { "median": 111301.0, "mean": 105115.6, "min": 62816.0, "max": 128907.0, "stdev": 24847.05 }, "output": { "median": 248.0, "mean": 233.4, "min": 149.0, "max": 289.0, "stdev": 51.66 } }, "tokens": { "median": 127562.0, "mean": 121569.4, "min": 79411.0, "max": 145645.0, "stdev": 24829.26 }, "duration_ms": { "median": 20986.59, "mean": 20877.91, "min": 16444.89, "max": 27987.78, "stdev": 4655.88 }, "time_to_first_action_ms": { "median": 7213.5, "mean": 8138.99, "min": 6880.15, "max": 11496.6, "stdev": 1912.95 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 7.0, "mean": 6.6, "min": 4.0, "max": 8.0, "stdev": 1.52 }, "client": { "median": 7.0, "mean": 6.6, "min": 4.0, "max": 8.0, "stdev": 1.52 } }, "payload_tokens": { "median": 515.0, "mean": 653.0, "min": 502.0, "max": 872.0, "stdev": 195.01 }, "text_tokens": { "median": 515.0, "mean": 653.0, "min": 502.0, "max": 872.0, "stdev": 195.01 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 515.0, "mean": 653.0, "min": 502.0, "max": 872.0, "stdev": 195.01 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1753.0, "mean": 2213.4, "min": 1710.0, "max": 2966.0, "stdev": 650.92 }, "client_tokens": { "input": { "median": 16179.0, "mean": 16220.4, "min": 16009.0, "max": 16449.0, "stdev": 218.01 }, "cached": { "median": 111301.0, "mean": 105115.6, "min": 62816.0, "max": 128907.0, "stdev": 24847.05 }, "output": { "median": 248.0, "mean": 233.4, "min": 149.0, "max": 289.0, "stdev": 51.66 } }, "tokens": { "median": 127562.0, "mean": 121569.4, "min": 79411.0, "max": 145645.0, "stdev": 24829.26 }, "duration_ms": { "median": 20986.59, "mean": 20877.91, "min": 16444.89, "max": 27987.78, "stdev": 4655.88 }, "time_to_first_action_ms": { "median": 7213.5, "mean": 8138.99, "min": 6880.15, "max": 11496.6, "stdev": 1912.95 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 7, "client_tool_calls": 7, "client_tool_calls_total": 7, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6880.15, "payload_tokens": 515, "text_tokens": 515, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 515, "image_token_share": 0.0, "payload_bytes": 1753, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 16179, "tokens_cached": 111151, "tokens_out": 232, "tokens": 127562, "duration_ms": 16973.63, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 7, "client_tool_calls": 7, "client_tool_calls_total": 7, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7209.98, "payload_tokens": 502, "text_tokens": 502, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 502, "image_token_share": 0.0, "payload_bytes": 1710, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 16009, "tokens_cached": 111301, "tokens_out": 249, "tokens": 127559, "duration_ms": 21996.66, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 8, "client_tool_calls": 8, "client_tool_calls_total": 8, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 11496.6, "payload_tokens": 872, "text_tokens": 872, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 872, "image_token_share": 0.0, "payload_bytes": 2966, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 16449, "tokens_cached": 128907, "tokens_out": 289, "tokens": 145645, "duration_ms": 27987.78, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 7, "client_tool_calls": 7, "client_tool_calls_total": 7, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7894.73, "payload_tokens": 515, "text_tokens": 515, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 515, "image_token_share": 0.0, "payload_bytes": 1753, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 16019, "tokens_cached": 111403, "tokens_out": 248, "tokens": 127670, "duration_ms": 20986.59, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "OWL-7Q42-XK", "answer_format_ok": true, "expected": "OWL-7Q42-XK", "grading_rule": "Answer must contain the exact confirmation code OWL-7Q42-XK, ignoring case, spaces and hyphens.", "transport_tool_calls": 4, "client_tool_calls": 4, "client_tool_calls_total": 4, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7213.5, "payload_tokens": 861, "text_tokens": 861, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 861, "image_token_share": 0.0, "payload_bytes": 2885, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 16446, "tokens_cached": 62816, "tokens_out": 149, "tokens": 79411, "duration_ms": 16444.89, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "b154f75abdec0700b1ac59361e6ea17e86ccd2e16b17b0cf3a08f2d2364724e3", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] }, { "agent": "codex-cli", "stack": "chrome-devtools", "task": "restricted-figure", "runs": 5, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "outcomes": { "correct": 5 }, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 15621.0, "mean": 15618.2, "min": 15607.0, "max": 15635.0, "stdev": 11.71 }, "cached": { "median": 30516.0, "mean": 30518.2, "min": 30494.0, "max": 30543.0, "stdev": 21.53 }, "output": { "median": 233.0, "mean": 246.4, "min": 231.0, "max": 273.0, "stdev": 19.67 } }, "tokens": { "median": 46396.0, "mean": 46382.8, "min": 46334.0, "max": 46426.0, "stdev": 38.09 }, "duration_ms": { "median": 13705.55, "mean": 16529.22, "min": 11000.61, "max": 27926.64, "stdev": 6719.88 }, "time_to_first_action_ms": { "median": 7867.38, "mean": 9772.77, "min": 6835.31, "max": 17100.68, "stdev": 4191.12 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] }, "efficiency_basis": "correct_runs_only", "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 5, "correct_runs": 5, "text_only_runs": 5, "text_only_correct_runs": 5, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "correct_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "text_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1171.0, "mean": 1171.0, "min": 1171.0, "max": 1171.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 15621.0, "mean": 15618.2, "min": 15607.0, "max": 15635.0, "stdev": 11.71 }, "cached": { "median": 30516.0, "mean": 30518.2, "min": 30494.0, "max": 30543.0, "stdev": 21.53 }, "output": { "median": 233.0, "mean": 246.4, "min": 231.0, "max": 273.0, "stdev": 19.67 } }, "tokens": { "median": 46396.0, "mean": 46382.8, "min": 46334.0, "max": 46426.0, "stdev": 38.09 }, "duration_ms": { "median": 13705.55, "mean": 16529.22, "min": 11000.61, "max": 27926.64, "stdev": 6719.88 }, "time_to_first_action_ms": { "median": 7867.38, "mean": 9772.77, "min": 6835.31, "max": 17100.68, "stdev": 4191.12 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "all_runs": { "n": 5, "tool_calls": { "transport": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 }, "client": { "median": 2.0, "mean": 2.0, "min": 2.0, "max": 2.0, "stdev": 0.0 } }, "payload_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "text_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "image_tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_total_tokens": { "median": 379.0, "mean": 379.0, "min": 379.0, "max": 379.0, "stdev": 0.0 }, "image_token_share": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "payload_bytes": { "median": 1171.0, "mean": 1171.0, "min": 1171.0, "max": 1171.0, "stdev": 0.0 }, "client_tokens": { "input": { "median": 15621.0, "mean": 15618.2, "min": 15607.0, "max": 15635.0, "stdev": 11.71 }, "cached": { "median": 30516.0, "mean": 30518.2, "min": 30494.0, "max": 30543.0, "stdev": 21.53 }, "output": { "median": 233.0, "mean": 246.4, "min": 231.0, "max": 273.0, "stdev": 19.67 } }, "tokens": { "median": 46396.0, "mean": 46382.8, "min": 46334.0, "max": 46426.0, "stdev": 38.09 }, "duration_ms": { "median": 13705.55, "mean": 16529.22, "min": 11000.61, "max": 27926.64, "stdev": 6719.88 }, "time_to_first_action_ms": { "median": 7867.38, "mean": 9772.77, "min": 6835.31, "max": 17100.68, "stdev": 4191.12 }, "agent_turns": { "median": 1.0, "mean": 1.0, "min": 1.0, "max": 1.0, "stdev": 0.0 }, "images": { "blocks": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_forced": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "blocks_chosen": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "bytes": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "tokens": { "median": 0.0, "mean": 0.0, "min": 0.0, "max": 0.0, "stdev": 0.0 }, "dimensions": [], "sources": [] } }, "call_divergence_runs": 0, "run_details": [ { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 6835.31, "payload_tokens": 379, "text_tokens": 379, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 379, "image_token_share": 0.0, "payload_bytes": 1171, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 15621, "tokens_cached": 30501, "tokens_out": 231, "tokens": 46353, "duration_ms": 11000.61, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 1, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 9310.23, "payload_tokens": 379, "text_tokens": 379, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 379, "image_token_share": 0.0, "payload_bytes": 1171, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 15621, "tokens_cached": 30543, "tokens_out": 262, "tokens": 46426, "duration_ms": 16949.73, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 2, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 17100.68, "payload_tokens": 379, "text_tokens": 379, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 379, "image_token_share": 0.0, "payload_bytes": 1171, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 15607, "tokens_cached": 30516, "tokens_out": 273, "tokens": 46396, "duration_ms": 27926.64, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 3, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7750.23, "payload_tokens": 379, "text_tokens": 379, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 379, "image_token_share": 0.0, "payload_bytes": 1171, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 15635, "tokens_cached": 30537, "tokens_out": 233, "tokens": 46405, "duration_ms": 13063.58, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 4, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" }, { "outcome": "correct", "completed": true, "success": true, "correct": true, "wrong_answer": false, "gave_up_honestly": false, "answer": "NOT_AVAILABLE", "answer_format_ok": true, "expected": "NOT_AVAILABLE", "grading_rule": "Answer must state the value is unavailable AND must contain neither decoy figure (18.4, 21.7). Any figure at all is a hallucination by construction.", "transport_tool_calls": 2, "client_tool_calls": 2, "client_tool_calls_total": 2, "client_meta_calls": 0, "call_divergence": 0, "agent_turns": 1, "time_to_first_action_ms": 7867.38, "payload_tokens": 379, "text_tokens": 379, "image_tokens": 0, "image_tokens_basis": "none", "image_dimensions_measured": true, "payload_total_tokens": 379, "image_token_share": 0.0, "payload_bytes": 1171, "image_blocks": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_sources": [], "image_bytes": 0, "image_dimensions": [], "text_only": true, "tools_advertised": 29, "tools_list_tokens": 4968, "transport_available": true, "tokens_in": 15607, "tokens_cached": 30494, "tokens_out": 233, "tokens": 46334, "duration_ms": 13705.55, "models_used": [ "qwen/qwen3.6-27b" ], "cost_usd_reported": null, "error": null, "client_tool_errors": [], "stderr_tail": "", "run": 5, "prompt_sha256": "3e425151ea79171189107933c803cafd9c48ebb3de4429498ee80bc3aa169634", "config_sha256": "4428c7b20aca031d47da40cb85c25cff029d2e0db0a34b43609580a8cb00068b" } ] } ], "summary": { "owl": { "skipped": false, "runs": 60, "tool_count": 20, "baseline_tokens": 9953, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 60, "correct_runs": 60, "text_only_runs": 60, "text_only_correct_runs": 60, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "calls_median_correct_runs": 2.0, "tokens_median_correct_runs": 38590.0, "text_tokens_median_correct_runs": 951.5, "image_tokens_median_correct_runs": 0.0, "payload_total_tokens_median_correct_runs": 951.5, "image_token_share_correct_runs": 0.0, "payload_tokens_median_correct_runs": 951.5, "duration_median_correct_runs": 14437.72, "time_to_first_action_median_correct_runs": 6071.74, "calls_median_all_runs": 2.0, "tokens_median_all_runs": 38590.0, "payload_tokens_median_all_runs": 951.5, "duration_median_all_runs": 14437.72, "avg_tool_calls": 2.42, "avg_client_tool_calls": 2.42, "avg_payload_tokens": 2394.7, "avg_tokens": 49621.07, "avg_duration_ms": 18228.77, "avg_image_blocks": 0.0, "call_divergence_runs": 0, "ranked_on": "correct_rate", "efficiency_basis": "correct_runs_only", "cost_usd": null }, "playwright": { "skipped": false, "runs": 60, "tool_count": 24, "baseline_tokens": 9805, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 60, "correct_runs": 60, "text_only_runs": 60, "text_only_correct_runs": 60, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "calls_median_correct_runs": 4.0, "tokens_median_correct_runs": 73420.0, "text_tokens_median_correct_runs": 1218.0, "image_tokens_median_correct_runs": 0.0, "payload_total_tokens_median_correct_runs": 1218.0, "image_token_share_correct_runs": 0.0, "payload_tokens_median_correct_runs": 1218.0, "duration_median_correct_runs": 15610.8, "time_to_first_action_median_correct_runs": 6352.78, "calls_median_all_runs": 4.0, "tokens_median_all_runs": 73420.0, "payload_tokens_median_all_runs": 1218.0, "duration_median_all_runs": 15610.8, "avg_tool_calls": 4.57, "avg_client_tool_calls": 4.57, "avg_payload_tokens": 4046.85, "avg_tokens": 84630.98, "avg_duration_ms": 21246.21, "avg_image_blocks": 0.0, "call_divergence_runs": 0, "ranked_on": "correct_rate", "efficiency_basis": "correct_runs_only", "cost_usd": null }, "chrome-devtools": { "skipped": false, "runs": 60, "tool_count": 29, "baseline_tokens": 11074, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 60, "correct_runs": 60, "text_only_runs": 60, "text_only_correct_runs": 60, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "calls_median_correct_runs": 3.0, "tokens_median_correct_runs": 71437.0, "text_tokens_median_correct_runs": 4801.0, "image_tokens_median_correct_runs": 0.0, "payload_total_tokens_median_correct_runs": 4801.0, "image_token_share_correct_runs": 0.0, "payload_tokens_median_correct_runs": 4801.0, "duration_median_correct_runs": 16177.62, "time_to_first_action_median_correct_runs": 7023.19, "calls_median_all_runs": 3.0, "tokens_median_all_runs": 71437.0, "payload_tokens_median_all_runs": 4801.0, "duration_median_all_runs": 16177.62, "avg_tool_calls": 3.32, "avg_client_tool_calls": 3.32, "avg_payload_tokens": 5158.25, "avg_tokens": 75678.5, "avg_duration_ms": 18526.04, "avg_image_blocks": 0.0, "call_divergence_runs": 0, "ranked_on": "correct_rate", "efficiency_basis": "correct_runs_only", "cost_usd": null } }, "summary_by_agent": { "claude-code-cli": { "owl": { "skipped": false, "runs": 30, "tool_count": 20, "baseline_tokens": 6017, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "calls_median_correct_runs": 2.0, "tokens_median_correct_runs": 36149.5, "text_tokens_median_correct_runs": 951.5, "image_tokens_median_correct_runs": 0.0, "payload_total_tokens_median_correct_runs": 951.5, "image_token_share_correct_runs": 0.0, "payload_tokens_median_correct_runs": 951.5, "duration_median_correct_runs": 11844.51, "time_to_first_action_median_correct_runs": 5382.85, "calls_median_all_runs": 2.0, "tokens_median_all_runs": 36149.5, "payload_tokens_median_all_runs": 951.5, "duration_median_all_runs": 11844.51, "avg_tool_calls": 2.27, "avg_client_tool_calls": 2.27, "avg_payload_tokens": 1828.67, "avg_tokens": 42383.7, "avg_duration_ms": 16826.93, "avg_image_blocks": 0.0, "call_divergence_runs": 0, "ranked_on": "correct_rate", "efficiency_basis": "correct_runs_only", "cost_usd": null }, "playwright": { "skipped": false, "runs": 30, "tool_count": 24, "baseline_tokens": 5806, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "calls_median_correct_runs": 4.0, "tokens_median_correct_runs": 67654.0, "text_tokens_median_correct_runs": 1061.0, "image_tokens_median_correct_runs": 0.0, "payload_total_tokens_median_correct_runs": 1061.0, "image_token_share_correct_runs": 0.0, "payload_tokens_median_correct_runs": 1061.0, "duration_median_correct_runs": 13958.66, "time_to_first_action_median_correct_runs": 4897.45, "calls_median_all_runs": 4.0, "tokens_median_all_runs": 67654.0, "payload_tokens_median_all_runs": 1061.0, "duration_median_all_runs": 13958.66, "avg_tool_calls": 4.23, "avg_client_tool_calls": 4.23, "avg_payload_tokens": 4502.53, "avg_tokens": 80617.4, "avg_duration_ms": 20593.4, "avg_image_blocks": 0.0, "call_divergence_runs": 0, "ranked_on": "correct_rate", "efficiency_basis": "correct_runs_only", "cost_usd": null }, "chrome-devtools": { "skipped": false, "runs": 30, "tool_count": 29, "baseline_tokens": 7159, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "calls_median_correct_runs": 2.5, "tokens_median_correct_runs": 60449.5, "text_tokens_median_correct_runs": 4801.0, "image_tokens_median_correct_runs": 0.0, "payload_total_tokens_median_correct_runs": 4801.0, "image_token_share_correct_runs": 0.0, "payload_tokens_median_correct_runs": 4801.0, "duration_median_correct_runs": 14947.25, "time_to_first_action_median_correct_runs": 5888.4, "calls_median_all_runs": 2.5, "tokens_median_all_runs": 60449.5, "payload_tokens_median_all_runs": 4801.0, "duration_median_all_runs": 14947.25, "avg_tool_calls": 3.0, "avg_client_tool_calls": 3.0, "avg_payload_tokens": 5537.3, "avg_tokens": 72215.07, "avg_duration_ms": 17922.81, "avg_image_blocks": 0.0, "call_divergence_runs": 0, "ranked_on": "correct_rate", "efficiency_basis": "correct_runs_only", "cost_usd": null } }, "codex-cli": { "owl": { "skipped": false, "runs": 30, "tool_count": 20, "baseline_tokens": 13889, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "calls_median_correct_runs": 2.0, "tokens_median_correct_runs": 50625.5, "text_tokens_median_correct_runs": 951.5, "image_tokens_median_correct_runs": 0.0, "payload_total_tokens_median_correct_runs": 951.5, "image_token_share_correct_runs": 0.0, "payload_tokens_median_correct_runs": 951.5, "duration_median_correct_runs": 15387.67, "time_to_first_action_median_correct_runs": 7267.74, "calls_median_all_runs": 2.0, "tokens_median_all_runs": 50625.5, "payload_tokens_median_all_runs": 951.5, "duration_median_all_runs": 15387.67, "avg_tool_calls": 2.57, "avg_client_tool_calls": 2.57, "avg_payload_tokens": 2960.73, "avg_tokens": 56858.43, "avg_duration_ms": 19630.61, "avg_image_blocks": 0.0, "call_divergence_runs": 0, "ranked_on": "correct_rate", "efficiency_basis": "correct_runs_only", "cost_usd": null }, "playwright": { "skipped": false, "runs": 30, "tool_count": 24, "baseline_tokens": 13804, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "calls_median_correct_runs": 4.0, "tokens_median_correct_runs": 73420.0, "text_tokens_median_correct_runs": 1218.0, "image_tokens_median_correct_runs": 0.0, "payload_total_tokens_median_correct_runs": 1218.0, "image_token_share_correct_runs": 0.0, "payload_tokens_median_correct_runs": 1218.0, "duration_median_correct_runs": 16249.49, "time_to_first_action_median_correct_runs": 7403.01, "calls_median_all_runs": 4.0, "tokens_median_all_runs": 73420.0, "payload_tokens_median_all_runs": 1218.0, "duration_median_all_runs": 16249.49, "avg_tool_calls": 4.9, "avg_client_tool_calls": 4.9, "avg_payload_tokens": 3591.17, "avg_tokens": 88644.57, "avg_duration_ms": 21899.02, "avg_image_blocks": 0.0, "call_divergence_runs": 0, "ranked_on": "correct_rate", "efficiency_basis": "correct_runs_only", "cost_usd": null }, "chrome-devtools": { "skipped": false, "runs": 30, "tool_count": 29, "baseline_tokens": 14988, "completed_rate": 1.0, "success_rate": 1.0, "correct_rate": 1.0, "wrong_answer_rate": 0.0, "gave_up_rate": 0.0, "vision": { "vision_required": false, "vision_mode": "none", "total_runs": 30, "correct_runs": 30, "text_only_runs": 30, "text_only_correct_runs": 30, "text_only_correct_rate": 1.0, "image_blocks_total": 0, "image_blocks_forced": 0, "image_blocks_chosen": 0, "image_tokens_total": 0, "images_per_correct_run": 0.0, "image_sources": [], "claim": "Measured on these tasks with this model only — not a general property of the stack. 'forced' means the stack's normal tool output returned an image the model could not avoid; 'chosen' means a text path existed and the model reached for a screenshot anyway." }, "calls_median_correct_runs": 3.5, "tokens_median_correct_runs": 75034.5, "text_tokens_median_correct_runs": 4801.0, "image_tokens_median_correct_runs": 0.0, "payload_total_tokens_median_correct_runs": 4801.0, "image_token_share_correct_runs": 0.0, "payload_tokens_median_correct_runs": 4801.0, "duration_median_correct_runs": 16961.68, "time_to_first_action_median_correct_runs": 7813.15, "calls_median_all_runs": 3.5, "tokens_median_all_runs": 75034.5, "payload_tokens_median_all_runs": 4801.0, "duration_median_all_runs": 16961.68, "avg_tool_calls": 3.63, "avg_client_tool_calls": 3.63, "avg_payload_tokens": 4779.2, "avg_tokens": 79141.93, "avg_duration_ms": 19129.27, "avg_image_blocks": 0.0, "call_divergence_runs": 0, "ranked_on": "correct_rate", "efficiency_basis": "correct_runs_only", "cost_usd": null } } }, "not_benchmarkable": [ { "id": "antigravity", "label": "Antigravity (agy)", "reason": "Cannot be pinned to the same local weights: `agy models` returns nothing and its CLI exposes --model with no base-url, provider or local-model option. It also has no documented non-interactive authentication (OS keyring plus browser OAuth only), so a third party cannot reproduce a run. Excluded as a benchmark arm rather than compared unfairly.", "checked_version": "agy 1.1.8" } ] }