marking
marking — researcher or builder tracked in the Angestrom contributor network.
Repositories · 1
DietrichGebert/ponytail
Makes your AI agent think like the laziest senior dev in the room. The best code is the code you never wrote.
Papers · 31
MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
Memory has become a key component of large language models, enabling them to retain information and learn from long-term interactions. However, existing memory benchmarks mainly evaluate whether information is correctly extracted, stored, and retrieved, while largely overlooking how retrieved memories reshape model reasoning and affect performance on the current task. We identify memory-induced cognitive traps: even faithfully recorded and semantically relevant memories can distort model reasoning or beliefs and degrade current task performance. To systematically evaluate these failure modes,
VoxENES 2026: Benchmarking Generalization of Speech Spoofing Detectors Against LLM-Era TTS and Voice Conversion
Modern LLM-driven text-to-speech (TTS) and voice conversion (VC) systems produce synthetic speech that differs from the generators represented in many legacy spoofing benchmarks. This mismatch creates a temporal generalization gap that can overestimate detector robustness under real-world post-processing conditions. We bridge this gap by introducing VoxENES 2026, a bilingual (English and Spanish) benchmark of 53,628 audio samples generated using 10 contemporary speech synthesis methods and evaluated under 10 standardized post-processing conditions. Using VoxENES 2026, we benchmark eight pretra
Do Language Models Dream of Binding Molecules? Benchmarking LLMs under Spatial Constraints
Structure-based drug design (SBDD) leverages the 3D structure of protein targets, often complemented by other spatial constraints, to generate candidate binding molecules. While diffusion models have dominated as a leading paradigm for high-quality 3D molecule generation, LLM-based methods are rapidly emerging in molecular design and have shown competitive performance in pocket-conditioned molecular generation. However, their ability to reason about physics and 3D spatial environments is largely underexplored. In this work, we systematically analyze whether current general-purpose LLMs are cap
PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives
Video world models simulate future states conditioned on current observations and user actions. Recent systems have demonstrated impressive video consistency and action controllability over long sequences. However, fairly comparing these interactive models remains challenging. In practice, a human player typically evaluates a world model by pursuing long-horizon objectives through interaction. For example, a user may turn around 360 degrees to see whether the environment remains consistent, or walk into the water and inspect whether realistic water ripples are generated. The action sequence re
Robust Text Watermarking for Large Language Models via Dual Semantic Embeddings
This work presents Dual-Embedding Watermarking (DEW), a semantic watermarking scheme for large language models (LLMs) that leverages contextual and token-level embeddings to enhance robustness against paraphrasing and translation. DEW utilizes a signal-processing methodology, applying algebraic vector-space operations to \mbox{token and context embeddings to derive a watermark signal that degrades gracefully under semantic shifts. The method obfuscates the watermark by projecting embedding vectors through pseudo-random matrices seeded with a secret key. Relevant distributions derived from the
KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding
Evaluating large language models (LLMs) across languages remains challenging, as most multilingual benchmarks rely on translated English datasets, often obscuring linguistic and cultural specificity in the target language. This issue is particularly pronounced for less-resourced languages such as Kyrgyz, where reliable natively authored evaluation data are scarce. Building on previously introduced Kyrgyz-language evaluation datasets, this work reports the first systematic and large-scale evaluation of LLMs in Kyrgyz using the KyrgyzLLM-Bench benchmark suite. KyrgyzLLM-Bench comprises two nativ
MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
Memory has emerged as a cornerstone of modern LLM-based agents, supporting their evolution from single-turn assistants to long-term collaborators. However, memory is not always beneficial: retrieved memories often induce a critical issue of sycophancy, causing agents to over-align with the user at the cost of factual accuracy or objective reasoning. Despite this emerging risk, existing memory benchmarks primarily evaluate whether memories are correctly stored, retrieved, or updated, while overlooking how retrieved memories influence downstream reasoning and decision-making. To bridge this gap,
When Writing Style Drifts: Benchmarking Authorship Verification under Distribution Shifts in Genre, Time and the AI-Era
Authorship verification (AV) assumes that an author's writing style remains sufficiently stable to distinguish it from that of other writers. In practice, however, this assumption is challenged by distribution shifts caused by changes in genre, time, and AI-assisted writing. Existing AV benchmarks typically study these factors in isolation and focus predominantly on English, limiting our understanding of model robustness under realistic conditions. We introduce AVShift, the first German benchmark for systematically evaluating AV under multiple distribution shifts. AVShift comprises over 150,00
News · 9
Claude Code Is Steganographically Marking Requests
<p>Article URL: <a href="https://thereallo.dev/blog/claude-code-prompt-steganography">https://thereallo.dev/blog/claude-code-prompt-steganography</a></p> <p>Comments URL: <a href="https://news.ycombinator.com/item?id=48734373">https://news.ycombinator.com/item?id=48734373</a></p> <p>Points: 2019</p> <p># Comments: 583</p>
3 days benchmarking most llama.cpp flags on my weird 40gb vram laptop + tb4 egpu setup. Got +70% generation, +40% prefill, 60k more context, and filed a bug in llama around MTP. What I learned.
<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1vtc0z7/3_days_benchmarking_most_llamacpp_flags_on_my/"> <img alt="3 days benchmarking most llama.cpp flags on my weird 40gb vram laptop + tb4 egpu setup. Got +70% generation, +40% prefill, 60k more context, and filed a bug in llama around MTP. What I learned." src="https://preview.redd.it/nucq9zl77hkh1.png?width=140&height=75&auto=webp&s=57c12a136ce52468f58372d53a5922f73ff25428" title="3 days benchmarking most ll
Benchmarking Self-Hosted Gemma 2 9B vs. Frontier APIs: The FP8 Quantization Prefill Tax and VRAM Realities on an NVIDIA L4 [P]
<!-- SC_OFF --><div class="md"><p>When evaluating migrating production LLM workloads off commercial cloud APIs, the conversation usually gets oversimplified into a trade-off between quality and infrastructure cost. To look past clean, isolated averages, I built a repeatable evaluation matrix using a real-world workload: cold outreach and contextual profile re-engineering for my resume generation platform.</p> <p>I benchmarked an unquantized <strong>Gemma 2 9B</strong> against an optimized <stron
Is it agentic enough? Benchmarking open models on your own tooling
Introducing the FFASR Leaderboard: Benchmarking ASR in the Real World
Resource - AI Text Watermarking: How it Works and How to Evade It
<!-- SC_OFF --><div class="md"><p>Earlier this month, Anthropic <a href="https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content">announced</a> that it was adding invisible text watermarking to Claude outputs. This announcement got a lot of attention. </p> <p>At the same time the <a href="https://downstack.app/editor/news/strong-backing-code-practice-transparency-ai-generated-content">European Commission</a> announced that other firms, including Black Forest Labs an
MLPerf Client v2.0 Expands AI PC Benchmarking with Image Generation and Agentic AI - AiThority
<a href="https://news.google.com/rss/articles/CBMivwFBVV95cUxQcjMyckpjaVNtT1pBdVFuQTJjR0RSVUFFd3d5R2xOZXVFN2xrR2QxcUExU3NkSlc5NlBTTXFnOV8yZ3FVZ0hUOTFnSlZJSDRCQ05KS016N2JKRFJqQWx5SkExbEFUMXdaQ0c3NkQ1UmkwRkxqQkR2bUplNmRRMkxHbUZtZkVaamczMUdBX2hVcXZLR0Jhb1h1WEtNTU84eUltQ0ZYUEpfZVY2NVNoX29TS0tpcGhwRHNoZmpGazdCMA?oc=5" target="_blank">MLPerf Client v2.0 Expands AI PC Benchmarking with Image Generation and Agentic AI</a> <font color="#6f6f6f">AiThority</font>
Show HN: PantheonGPU – GPU health testing and AI workload benchmarking
<p>Hi HN, I built PantheonGPU because I wanted a better way to answer a simple question: is this GPU actually healthy and performing the way it should?<p>A GPU can show normal temperatures and utilization and still be underperforming, unstable under certain workloads, or have memory, PCIe, or configuration issues.<p>PantheonGPU actively tests the GPU instead of only monitoring telemetry. It currently includes 45+ tests covering compute, tensor workloads, memory, cache, PCIe, thermals, stability,