Comment by anygivnthursday
19 hours ago
But you still have to review that 2FA code and that involves thinking through the implementation, right?
19 hours ago
But you still have to review that 2FA code and that involves thinking through the implementation, right?
No.
Haven't typed a line of code or read any code for over 6 months now.
And I used to love coding and be a competitive programmer, but this is how "coding" goes nowdays.
I have a mental model of what it would do, and how it would work, and I ask questions to confirm things and tell it to watch for specific gotchas. Then simply test the feature myself a bit.
Security-wise, I think the latest cyber models are better than me anyway at finding vulnerabilitates and pentesting such features.
Plus 2FA is a very common pattern, so it likely has in the training dataset many really good implementations.
> Security-wise, I think the latest cyber models are better than me anyway at finding vulnerabilitates and pentesting such features.
I don't doubt that, but they are equally good in making mistakes, over-engineering, or adding things you never asked for. They have all sorts of patterns in their training data from excellent to inadequate and I find them challenging to guide them consistently in one direction. Also with questions and tests, they can add something extra you didnt need and you dont know about, so your scrutinizing questions and test cases could miss that.
At least for myself, I didnt find them reliable enough yet to do what you describe and just not look at the code at all.
As someone who has an ide open with claude/codex running, I can never imagine leaving security up to models.
Models have found vulnerabilities that i wasnt aware of sure, but their fixes to the bugs they found often included "overengineering". In this case by "overengineering" i mean optimizing for passing test cases related to said vulnerability they found. eventually i have to step in to make things coherent and make sure that future agent can look at this part of my code and copy it to not introduce that particular class of vulnerability. Otherwise if i dont do that similar vulnerability and codesmell keep appearing throughtout the codebase.
I have increasingly automated encrypting and rotating secrets and setting permissions on them including better network level practices. Thanks to AI which helped me quickly implement those. So security wise i am better because of AI? But I also attribute it to my know how rather than the AI because I have never seen AI suggest robust but simple security postures.
This app[0] is entirely "vibe-coded", started on bolt.new which was like the OG way to vibe code and slowly improved over 1-2 years using various models.
I have no idea how the code looks like, and barely even tested the app entirely, because it is still not released yet, but I do think a lot about new features, tweaks, improvements, etc. Years of programming and game development did help, but I don't think anymore that code is relevant, as long as it looks ok and feels good.
[0]: https://ultimidi.com
11 replies →
It’s equally good at catching mistakes as it is at making them. So you put it in a loop, write code, make mistakes, catch mistakes, write code, make mistakes… etc. Eventually it converges and stabilizes. That process can take weeks. So the question is: when is it worth it versus not?
You are abdicating your responsibility, which is fine for toy apps for yourself, but less fine when people expect that 2fa implementation to protect their accounts.
I think it's the opposite, with the right guideance, testing, frameworks, and using the top models today, implementing something with AI is most of the times better than what most developers would do.
I basically only manually test e2e myself, other tests are automated, code review is automated. I can tell the model to test for me too specific things or to add tests for specific potential issues, performance benchmarks, compared different implementations, etc.
The focus is a lot more around the code than on the code.
13 replies →
I genuinely believe that the multiple dedicated agents reviewing security will do a better and faster job than me and than 99.99% of people.
You're acting as if code was incredibly secure before LLMs because humans were reviewing it.
I trust a modern model implementing a standard feature like this much more than 99% of the people I've worked with.
People bash LLMs for overengineering but for this it's what you want. Taking extreme edge-cases into account that a human would never bother with and obsessing over security.
5 replies →
> Haven't typed a line of code or read any code for over 6 months now
> I ask questions to confirm things
Oh my.
As someone who reads the code, I can tell you, asking questions to confirm things is inadequate. The models lie to me, daily.
Every day I have two experiences:
1. I’m blown away by what it can do
2. I say, ”wait, you said this, but the code shows that, so you were just going to leave that endpoint without requiring any authentication??” and I get the “you’re absolutely right, that was my mistake, and you’re right to call it out” song and dance. Daily.
It also adds all kinds of bloat to code, tests, and “documentation”. I’d say I spend ~30% of my dev time picking lines of code or documentation and asking, “why does this exist?” and “what would break if we deleted this line?” and then arguing with it and removing things.
Yeah, I agree, they are far from perfect and make a lot of mistakes.
Truth is, modern software was already quite shit and full of bugs. All major apps had bugs, issues, going down, etc, so users did get used to things not working. I honestly beleive AI coding nowadays, for better or worse, does things better than the average developer.
Yes, it is overly defensive and verbose, but the end result is in general ok and fully functional. Yes, it adds 30 tests and "release gates", and they are not even that useful, most of the times they just act as an extra safety mechanism to make parts of the code immutable, so release fails if the model accidentally changed things.
Another issue with looking at code, is that it's very hard to manually change things anyway. I can't just change a variable from 10 to 20, because I don't know where it is used. I have to ask the model to set that value to 20. It is quite stupid and inefficient, but this is one cost of coding using AI. But, if you do this, things will likely work.
That being said, I've mostly used Astra xhigh since it was released and things just work.
2 replies →
Current models love shoving in defaults where correct code would otherwise fall over with an exception.
I dread to think what that means in security conscious code.
Hmm, my experience is different. I noticed Astra always asks me to add credentials in a secure way, to never pasted them in chat, to rotate any potentially exposed tokens, etc.
Can you give an example of unsafe defaults used?
Be brave and post a GitHub of your code that you haven't written a lick of and haven't read.
EDIT: Come on? Won't post your code for everyone to see? Why not just put it all in a repo, client and server both?
EDIT 2: Amazing. If you punch in notes on the keyboard for like 10 seconds then click the keyboard-icon button on the bottom right the website crashes
EDIT 3: If you click the main CTA then click "Let's start" the website hangs and you need to manually refresh the page for the content to load
I could probably look over, and get some snippets, but currently only the client-side is live, so the code is sort of there, albeit I assume it's minified for production.
The game seems quite bug free though, including minigames. The UI could be better, but it's not done yet.
I do for example have an automated system that simulates progression, takes screenshots of the game to find potential hidden buttons or overlapping elements, to test for performance, etc.
If it looks like a duck, and quacks like a duck, I honestly don't see why I would review 100k's of lines of code.
EDIT: I might have replied in a wrong thread, but it was about this entirely "vibe-coded" app: https://game.ultimidi.com
10 replies →
Interesting, what browser are you using?
Also, what do you mean punch in notes? Like mashing keys and pressing 10 buttons at once?
Never had the app/website crash, tested only on Brave (desktop and mobile) so far.
2 replies →
[flagged]
[flagged]