Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks
Andrew Semenov, Svyatoslav Dorofeev · Kiz8
What can a language-model agent establish from records, and when should it decline to answer? Kierston simulates daily life in Allentown, Pennsylvania, then derives telecommunications, financial, administrative and licence-plate records from one shared behavioral timeline. KierstonBench asks agents to investigate through a restricted records interface, without access to the simulation ground truth or a supplied list of candidates.
- Question
- Can agents connect records across providers, reconstruct events and distinguish supported conclusions from unsupported identifications?
- Method
- A deterministic city simulator built from public census and geographic inputs. Synthetic people share households, routines and travel; provider records include incomplete observations and sensor errors. The underlying events remain available to the evaluator, not the agent.
- Experiment
- Each model attempts one case per task family in a fresh container with a shell, Python and the Allentown Data Interface. Incident tasks expose only records available by the stated case time. Correct identifications earn credit, incorrect ones incur penalties, and abstentions score zero.
Technical tasks
Link identities and reconstruct daily activity across provider records.
Real-time investigations
Investigate staged incidents using only records available before the case cutoff.
Deception tasks
Check a caller’s false account against independently generated records.
Linking records is not solving a case.
Frontier models resolve identities and reconstruct routines, but the paper reports that no model identified a perpetrator in any of the five suspect-identification tasks. Proximity, firearm ownership and criminal history were repeatedly treated as evidence of involvement without a link to the incident.
Abstaining can prevent an unsupported identification. A zero score is not the same as a correct answer.
| Model | TT | RTI | DT | All |
|---|---|---|---|---|
| GPT-6 Sol | 0.67 | 0.00 | 1.00 | 0.55 |
| Claude Fable 5.1 | 0.75 | −0.38 | 1.00 | 0.52 |
| GPT-6.1 Sol | 0.58 | 0.00 | 0.67 | 0.46 |
| GPT-6 Astra | 0.53 | 0.14 | 0.33 | 0.43 |
TT: technical tasks · RTI: real-time investigations · DT: deception tasks. All-task and group means include abstentions at zero. Scores are not accuracy percentages. Snapshot: arXiv v1, Table 15; four of 16 evaluated models.
Full evaluation and abstentions ↗- Limitations
- One attempt per family, on one sampled instance of one simulated day. Small score differences may change on another attempt. These results do not establish real-world investigative performance. The disappearance scenario was corrected during evaluation; seven models retain results from the original staging, as documented in the paper.
- Dual use & release
- The engine can support development and independent scrutiny of surveillance and records-fusion systems. The paper releases engine and benchmark code with public inputs, but not generated outputs or evaluation tasks: synthetic identities may coincide with real people, creating privacy and defamation risks.
@misc{semenov2026kierston,
title = {Kierston \& KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks},
author = {Semenov, Andrew and Dorofeev, Svyatoslav},
year = {2026},
eprint = {2611.04785},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
doi = {10.48550/arXiv.2611.04785},
url = {https://arxiv.org/abs/2611.04785}
}