← Back to context

Comment by josephg

17 days ago

Yeah I’m having a similar experience. I’ve been wanting a standard test suite for JMAP email servers, so we can make sure all created jmap servers implement the (somewhat complex) spec in a consistent manner. I spent a single day prompting Claude code on Friday, and walked away with about 9000 lines of code, containing 300 unit tests for jmap servers. And a web interface showing the results. It would have taken me at least a week or two to make something similar by hand.

There’s some quality issues - I think some of the tests are slightly wrong. We went back and forth on some ambiguities Claude found in the spec, and how we should actually interpret what the jmap spec is asking. But after just a day, it’s nearly there. And it’s already very useful to see where existing implementations diverge on their output, even if the tests are sometimes not correctly identifying which implementation is wrong. Some of the test failures are 100% correct - it found real bugs in production implementations.

Using an AI to do weeks of work in a single day is the biggest change in what software development looks like that I’ve seen in my 30+ year career. I don’t know why I would hire a junior developer to write code any more. (But I would hire someone who was smart enough to wrangle the AI). I just don’t know how long “ai prompter” will remain a valuable skill. The AIs are getting much better at operating independently. It won’t be long before us humans aren’t needed to babysit them.

8 comments

josephg

Reply
salawat  16 days ago

So what'd your prompt look like, out of curiosity? I hear about all these things that sound quite impressive, but no one ever seems to want share any info on the prompts to learn or gain insight from.

  • josephg  16 days ago

    It was nothing special. I can't seem to pull up the initial prompt, but it was something like this:

    > Build a thorough test suite for JMAP in this directory. The test suite will be run against multiple JMAP servers, to ensure each server implements the JMAP spec consistently and correctly. In this directory are two files - rfc8620.txt and rfc8621.txt. These files containing the JMAP core and JMAP email specs. Read these files. Then make a list of all aspects of the specifications. For each, create a set of tests which thoroughly tests all aspects of a JMAP server's behaviour specified by the RFCs, including error behaviour. The test suite should be configurable to point at a jmap server & email account. The account will contain an empty mailbox (error if its not empty). The test suite starts by adding a known set of test emails to the account, then run your tests and clear the inbox again. Write the test suite in typescript. The test runner should output the report into a JSON file. Start with a project plan.

    If you haven't tried claude code or openai's codex, just dive in there and give it a go. Writing a prompt isn't rocket science. Just succinctly say the same things you'd say to a very competent junior engineer when you want to brief them on some work.