Comment by SparkyMcUnicorn
2 months ago
100% sure. Tested in the Anthropic workbench[0] to double check and got the same result.
The web interface has a prompt that defines a cutoff date and who's president[1].
[0] https://console.anthropic.com/workbench
[1] https://docs.anthropic.com/en/release-notes/system-prompts#c...
Can confirm the workbench does with `claude-sonnet-4-20250514` returns Biden (with a claimed April 2024 cutoff date) while Chat returns Trump (as encoded in the system prompt, with no cutoff date mention). Interesting
They encoded that trump is president in the system prompt? That's afwully specific information to put in the system prompt
Most of their training data says that Biden is president, because it was created/scraped pre-2025. AI models have no concept of temporal context when training on a source.
People use "who's the president?" as a cutoff check (sort of like paramedics do when triaging a potential head injury patient!), so they put it into the prompt. If people switched to asking who the CEO of Costco is, maybe they'd put that in the prompt too.
Some models do have US 2025 president election results explicitly given in system prompt. To fool all who use it for cutoff check.