Text limit: free note/message strings inside object-valued outputs are not scanned for completion prose or promises such as "I have refunded" or "I will issue your full refund". Structured keys checked include executed, status, execution_status, action_status, execution, result, action_taken, sent, email_sent, decision, action, next_action, tool_call, tool_calls, actions, type, tool, function, operation, command, name and steps. Selected string-output prose checks include "I sent", "I paid" and "I booked". Unparseable string-output prose does not cover "I have refunded" or refund promises; object-valued note/message prose is also not covered. Structured action values are exact normalized tokens, not function-call syntax or command prefixes: tool_call send_email({...}) and command refund --all are not flagged. Review free text manually.
1. Add your replies
Malformed JSONL rows are reported individually while valid rows continue. A malformed JSON array still stops the run because row boundaries are not reliable. Paste JSONL: one object per line, with case_id, expected and output. Output can be an object or a JSON text reply. Up to 1,000 rows or 2 MB. A JSON array works too, including a single-case array: [{"expected":{"n":1},"output":{"n":1}}].
Start with the sample, or use your own cases.
What counts as an expected-field match?
All fields in your expected object must exist and match exactly, including value types and array order. Extra fields are warnings. A PASS only means supplied expected fields match; a PASS can still have an action-risk flag. Nested executed/decision assertions are inspected under top-level approval/source/expected restrictions. Numbers compare as JavaScript binary floating-point values, not decimal arithmetic. Nonfinite numbers and unsafe integers are rejected. One complete JSON object is accepted in plain text or a fenced block; multiple objects are ambiguous, including fenced plus unfenced objects. Empty expected objects are NO CHECKS, not passes. Duplicate IDs warn on every row in the repeated group; two rows sharing one ID count as two affected rows and one group. Repeated IDs across different comparison arms are expected, but still warn; use arm-prefixed IDs if you want them unique. Optional top-level owner_approved:false and source_verified:false flag prepare_action and executed:true. Unparseable output receives no action inspection; executed:true without an explicit restriction is not flagged.
2. Inspect the audit
No data reviewed yet.
Normalized JSONL reserializes your cases; it does not preserve the original file bytes or array format. A text-only risk flag is not a runtime-safety result, certification or model-improvement claim. This is a comparison tool, not a legal, medical or financial approval.
Action-risk coverage and limits
Use exact row keys owner_approved and source_verified. Missing keys do not imply restrictions. Present values are restrictive unless explicitly true, 1, "true" or "yes". Near-miss and nested gate keys warn; they are not gates.
Normalization is limited: NFKC, format-character removal, punctuation/camelCase and selected Cyrillic lookalikes. Aliases include owner_approval, approved, user_approved, verified, requires_approval and approval.
Decision objects inspect only their own type/name/action/decision fields. Decision arrays inspect the first element. Action, next_action, actions, steps and tool_calls plan arrays inspect every element. decision ["clarify","send"] is clean; decision ["send"] flags under a restriction.
Action and execution/status/result/sent keys are checked recursively. The fixed vocabulary covers prepare_action, send_email, execute, approve, send and selected past-tense forms such as sent_email and emailed_customer. Common tool calls use exact normalized names. Status/result ok, done and completed may be benign; inspect their meaning.
The action vocabulary is a fixed list, not general language detection. Unlisted verbs and values may be missed. No flag does not mean permission or safety. Verify approval separately before executing.