When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation
Interactive simulations increasingly evaluate policies in markets populated by language-model agents. Their outputs can look economic---prices, profits, consumer surplus, and welfare---without instantiating the behavior named in the claim. We audit this risk in a multi-turn buyer--seller testbed for configurable hotel transactions. An initial implementation reported welfare gains from two marketplace guardrails of +87.4, +35.0, and +28.8 across a Qwen2.5 1.5B--14B ladder. It also gave guarded and unguarded agents different offer schemas and choice procedures. Holding the schema and buyer choos
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- FuzzySimilar title/name (fuzzy) · 59%AgentCore-8B →
“Fuzzy title match (0.73): “When Guardrails Look Effective: Construct Validity Failures ” ≈ “AgentCore-8B””
- PossiblePossibly related (embedding) · 51%Brex built its AI agent policy by watching what agents actually do, not by writing rules first →
- PossiblePossibly related (embedding) · 50%Why prompt-level guardrails aren't enough: The platform security layers production agents need →
- FuzzySimilar title/name (fuzzy) · 87%guardrails-ai/guardrails →
“Fuzzy title match (0.94): “When Guardrails Look Effective: Construct Validity Failures ” ≈ “guardrails-ai/guardrails””
- FuzzySimilar title/name (fuzzy) · 87%SWE-agent/SWE-agent →
“Fuzzy title match (0.94): “When Guardrails Look Effective: Construct Validity Failures ” ≈ “SWE-agent/SWE-agent””
- FuzzySimilar title/name (fuzzy) · 87%zhayujie/CowAgent →
“Fuzzy title match (0.94): “When Guardrails Look Effective: Construct Validity Failures ” ≈ “zhayujie/CowAgent””
- FuzzySimilar title/name (fuzzy) · 59%NousResearch/hermes-agent →
“Fuzzy title match (0.73): “When Guardrails Look Effective: Construct Validity Failures ” ≈ “NousResearch/hermes-agent””
- FuzzySimilar title/name (fuzzy) · 59%bojieli/ai-agent-book →
“Fuzzy title match (0.73): “When Guardrails Look Effective: Construct Validity Failures ” ≈ “bojieli/ai-agent-book””