Comment by anon-3988
2 days ago
I have used Codex (Sol 5.6 or whatever) to solve this problem. It turns the problem into Z3, then iteratively work through the problems until it figured out the solution.
Personally, I did not learn that much from that experience. So I am glad that there's other people working on it as well. I am mostly interested in the techniques used to solve this.
> Personally, I did not learn that much from that experience.
What did you expect?
I understand the sentiment, but in a world where these kinds of problems (And problems at work) that can trivially be solved by LLMs; what kind of value can I provide?
That itself is a learning experience. What is even the point of technical interview questions or take home questions? This means that I am now open to hiring completely non-technical person, as long as they have a good personality and management skills more than a competent developer.
"I understand the sentiment, but in a world where these kinds of problems (And problems at work) that can trivially be solved by LLMs; what kind of value can I provide?"
This is sad conclusion. Maybe you're more right than wrong, but my answer to the "value" question would be:
Solving even harder problems based on lessons from solving easier ones? Pretty similar to the trajectory in this blog post. Perhaps aided additionally by llms and other tools that can solve the subproblems so you can focus on the less obvious/automatable aspects of the problem?
But note the tension, only by being involved in the problem solving to some extent do you become better at it. So if your default is to say "An LLM can or will soon be able to solve it, why bother?", then your situation will become more desperate and your outlook more negative in a self-reinforcing way.
The fact that agents can solve these is telling and the Infosec Capture The Flag community is trying to figure out how to approach.
I recently solved a (in)famously hard challenge (disobey conference hacker ticket) more or less by accident. I say by accident because I have always ignored this challenges as they generally require a lot of patience and motivation to solve. Some years they haven't been solved at all. This year I had a GPT sub with some unused quota so I thought let's see how far it gets.
And it crunched through the whole thing in an evening and morning (occasional poking from me to keep going and steer it right).
Like anon, I learned nothing except that the agents have become really good at solving puzzles. Last time I had thrown a puzzle on them was Advent of code, with GPT3 I think and it struggled so much I gave up my experiment on day 7 or something.
>I have used Codex (Sol 5.6 or whatever) to solve this problem.
>Personally, I did not learn that much from that experience
Fire is hot, water is wet, etc
If learning is the point then obviously that was a missed opportunity.
If success was the intent, then it's a win.
While challenges are fun, sometimes the requirement is simply to achieve the end goal, with no other point than that.
If your job is to stop terrorists, and that includes hacking into a system and extracting their plans, the "fun" of it isn't the point, only the end goal it's important. Sometimes when we do things like this for fun we forget that someone else out there absolutely needs to achieve the result and doesn't care how it's achieved.
It's the part of penetration testing some people miss. (It's not generally a fun job, almost everyone I know that did it got out as soon as possible. They weren't solving problems, they just running audit scripts and generating reports.)