Comment by einrealist

16 hours ago

So when Anthropic releases a new model that "breaks compatibility" with some Markdown files, do we call it "refactoring" to find (guess) the required changes to have the desired outcome again? Don't we create brittle specifications to fit a version of a model?

2 comments

einrealist

Reply
Leynos  16 hours ago

Use evals

Coming soon, unit, behavioural and regression tests for your prompts and skills :P

  • stingraycharles  16 hours ago

    How do you use evals when you’re using Claude Code, given that Claude Code also changes their prompts all the time?

    You’ll have:

    * Claude model version

    * Claude Code prompts and tools

    * Your own prompts and skills and whatnot

    * Your repository’s source code (= the input)

    All of those change constantly, it’s not like it’s some kind of SWE benchmark.