Your AI, On a Dial: Agent Retrieval Experiment
Traces from the agent experiment in “Your AI, On a Dial” (arXiv:2608.22852). An open-weight model runs a three-turn research loop over a live news search engine: it writes four search queries, picks four of the ten returned results to open, then reads them and writes a memo ending in a buy/sell decision. Each example runs the same company twice, once as is (baseline) and once with the dial, a shift on a single neuron, and shows the two traces side by side.
Setup
- Models. Qwen3-8B, Mistral-Small-24B and Gemma4-12B-it, greedy decoding.
- Search. Bing News (past month) via ddgs. Results are snapshot-cached, so identical queries return identical pages in both runs.
- Dial. An additive shift on one MLP neuron at the strongest setting tested (Δ in the right-hand title). π = (buy − sell) / (buy + sell) over all 300 runs of a setting.
- Labels. Gemini labels each sentence of headline and snippet as favourable, unfavourable or neutral for an investor in the company.
- Examples. Selected, not representative. Out of 150 held-out S&P 500 companies × 2 option orders, only cases where the baseline buys and the dial sells are kept, cases whose queries, memo or opened articles failed to parse are dropped, and up to ten per model are shown, ranked by the rise in unfavourable queries and in the unfavourable share of opened articles. Aggregate results are in the paper.
- Result page. Shown in the order the agent saw it, fused rank in parentheses. “Padding” marks an item from a fixed neutral seed query rather than from the agent's own queries. Where the scraped snippet is page boilerplate rather than article text, it is omitted here; the agent saw it as scraped.