Comment by spacedcowboy

16 hours ago

I’m slowly coming around to the idea that if it works, I’m good with that. I never used to personally check all my colleagues work, either, I just trusted that they’d done a good job.

That’s not to say I blindly trust an LLM - my project is an autovectorising compiler. I have it write a bazillion tests, and I have real, non-trivial programs that it compiled. I have it run a ‘gate’ before releases that takes literally hours on 5 different architectures. To say this code is tested is … masterful understatement.

I think what helps is a long planning session, going into details, working out corner cases, designing test-cases, you know work and effort. Once that’s done it’s easier to let it work through the night and do a major production feature.

I still review the code, but hand on heart I couldn’t say I’d reviewed everything, sometimes I miss something in a complicated check-in. I try to keep checkins small and contained, but sometimes that’s not easy to manage alongside the “don’t break stuff” rule as well.

So it comes down to: I’ve been working on this for 6 months or so, and so far this has worked out for me. You know, kind of like getting used to a colleague’s idiosyncrasies over a period of time.

I will say that in my experience the code quality is fairly high. Maybe that’s down to the effort put in ahead of time in the planning stages, I don’t know, but I don’t seem to suffer from the ‘slop’ that people complain of. YMMV.

28 comments

spacedcowboy

Reply
yulaow  15 hours ago

I am not in META but for what I read on blind, assuming it is true, they are not spending any time in long planning sessions or in getting into details of the implementation plan at all, and they could not anymore considering how many PRs in parallel they are requested to manage

  • spacedcowboy  15 hours ago

    Yeah, I'm retired. The only person I have to answer to regarding progress is me. I'm a stubborn son of a bitch when it comes to choosing quality over speed...

    That doesn't mean I don't like progress mind :) This morning I got the preliminary results of a 'C' front-end to the compiler (it uses clang to compile to LLVM IR, then translates to our IR, then feeds that IR into the autovectoriser as if it had parsed the code itself). So for the exact same source code I now have:

         ┌─────────────┬────────────┬──────────────────────┬──────────┐
     │  Benchmark  │ clang -O2  │ xcc --par-auto (GPU) │ Speed-up │
     ├─────────────┼────────────┼──────────────────────┼──────────┤
     │ saxpy       │ 1,546 µs   │ 726 µs               │ 2.1×     │
     ├─────────────┼────────────┼──────────────────────┼──────────┤
     │ Mandelbrot  │ 394,238 µs │ 2,110 µs             │ 187×     │
     ├─────────────┼────────────┼──────────────────────┼──────────┤
     │ nbody       │ 37,090 µs  │ 1,035 µs             │ 36×      │
     ├─────────────┼────────────┼──────────────────────┼──────────┤
     │ perlin      │ 78,016 µs  │ 1,850 µs             │ 42×      │
     ├─────────────┼────────────┼──────────────────────┼──────────┤
     │ mandel_deep │ 73.2 s     │ 55.8 ms              │ 1,312×   │
     └─────────────┴────────────┴──────────────────────┴──────────┘

    Those are some pretty fine speedups, from naive C, with no alterations from the standard "C" code - it automatically finds places that can be made data-parallel, tracks the data-hazards to make sure its safe to "GPU" and writes all the kernels, boilerplate and data-stream-management if necessary. Obviously the checksums have to match, so I know it did the work correctly too :)

    Hacker News isn't interested - any post I make linking to the webpages is immediately blocked/dead so I'll wait until the C++ front-end is ready (some time next week) and announce it somewhere else. A compiler that takes standard C/C++ code and speeds it up by automatically finding data-parallel areas and scheduling them on the GPU with no programming overhead is pretty rare.

    Oh, and it manages the local network as well, so you can run xcd on machines on your lan, or via the TB4/5 bridge between Macs and it'll schedule on other GPUs, just like it manages multiple GPUs on the local machine. It characterises both latency and compute-time to figure out which devices are worth scheduling on.

  • mepiethree  1 hour ago

    Do you think they are actually shipping anything useful any faster? Muse is pretty impressive. I’m not seeing radical progress otherwise

jimbokun  7 hours ago

Yes, compilers are a good candidate for verification through thorough testing. If a comprehensive test suite passes, reviewing every line of code isn’t so important.

  • ben-schaaf  3 hours ago

    That's why Anthropics C compiler worked so well; because they had gcc's huge repository of tests to lean on. Oh wait...

chrisjj  9 hours ago

> I’m slowly coming around to the idea that if it works, I’m good with that.

A.k.a. "The concrete didn't crumble when I hit it with a hammer."

Process quality has been a thing for ages because it is essential.

  • rixed  26 minutes ago

    Are we speaking of the same industry? The "ship fast and break things" one? The one that has traded everything from performances to security in exchange for "better developer experience" (aka "easy" aka "cheap") ??

  • spacedcowboy  6 hours ago

    I mean, honestly ? Yeah.

    Except it’s about 1000 “hammers” (aka tests), x 9 different architectures, x 2 different (a reference and a production) variants, and x 3 different optimisation levels. Every time I find something that slipped through, it gets at least one test written to cover it in the future. My main time-sink is the hours it takes to run the ‘gate’ before a release across the test-farm. And that’s after I’ve been playing with it myself for a while, pushing corner-cases through whatever I’ve been working on.

sixothree  7 hours ago

Very similar here. For internal projects the test is - if it works, unit and integration tests were created, and it follows our architecture correctly.

dmitrygr  6 hours ago

> never used to personally check all my colleagues work, either,

Jesus F Christ! No wonder most software lately (~decade) has been shit. Have y'all not heard of code review? Or did you think it just for show...you sit on a change for a week and then press "Accept" without reading it?

  • spacedcowboy  5 hours ago

    My dear fellow, the statement was that I never used to personally check all my colleagues work. Of course I took part in code review (but then, that’s pretty much the same as me reviewing the LLM code, no ?) The statement is that I personally did not review every line of code in the repo. Some of it was authored, reviewed and committed by others.

    FWIW I was a software engineer for the last 35 years or so. Amongst other claims to fame, pretty much all Apple devices on the planet (those actively running Apple OS’s, anyway) are running my code, within AuthKit in fact, and one code review was insufficient by policy when making changes. Despite all that, I’d be lying if I told you I thought there were 0 bugs in the actually-rather-complicated repo… Maybe these days, running it by an LLM would be a good idea…