Comment by the__alchemist
5 days ago
That's because you're doing web stuff. (I/O limited). So much of our computing experience has been degraded due to this mindset applied more broadly. Despite a steady improvement in hardware, my computing experiences have been stagnating and degraded in terms of latency, responsiveness etc.
I'm not going to even go into the comp chem simulations I've been running, or that about 1/3 the stuff I do is embedded.
I do still use python for web dev, partly because as you say, it's not CPU-bound, and partly because Python's Django framework is amazing. But I have switched to rust for everything else.
As a java backend dev mainly working on web services, I wanted to like python, but I have found it really hard to work on a large python project because the auto complete just does not work as well as something like java.
Maybe it is just due to not being as familiar with how to properly setup a python project, but every time I have had to do something in a django or fast api project it is a mess of missing types.
How do you handle that with modern python? Or is it just a limitation of the language itself?
That's 100% an IDE thing. I use Zed (or Emacs or anything else supporting an LSP) and autocomplete is fast and accurate.
Pycharm has been fine. Just disable the AI stuff and you get accurate completion. It even has completion for Django ORM stuff, which is heavily dynamic.
Pycharm has "AI stuff" now?
I won’t completely argue against that, and I’ve also adopted Rust for smaller or faster work. Still, I contend that a freaking enormous portion of computing workloads are IO bound to the point that even Python’s speed is Good Enough in an Amdahl’s Law kind of way.
I hear this a lot, but can you really say that you're consistently saturating a 1Gbps line for netcode or 6+ Gbps nvme for disk data? In my experience this doesn't really happen with code that isn't intentionally designed to minimize unnecessary work.
A lot of slow parsing tends to get grouped in with io, and this is where python can be most limiting.
I don't personally use Python directly for super IO intensive work. In my common use cases, that's nearly always waiting for a database to return or for a remote network API to respond. In my own work, I'm saturating neither disk nor network. My code often finds itself waiting for some other process to do that stuff on its behalf.
It's been said that Python's greatest superpower is that it's the second-best language at the most stuff.
No one's really developed an ecosystem for a language that's more performant that can match it, and that's all it needs to assert dominance.
I've never understood this. Python cannot be optimized like C, C++ or Rust. It cannot do advanced functional things like OCaml, Haskell or Scala. It cannot run in browsers like TypeScript. It cannot do games programming like C# and it can't do crazy macro stuff like Clojure. I don't think it's even second best at those things.
1 reply →
> That's because you're doing web stuff.
I guess you didn't notice where he talked about running numpy?
And 300ms for a DB call is slow, in any case. We really shouldn't accept that as normal cost of doing business. 300ms is only acceptable if we are doing scrypt type of things.
> in any case.
In some cases. Are looking up a single indexed row in a small K-V table? Yep, slow. Are you generating reports on the last 6 years of sales, grouped by division within larger companies? That might be pretty fast.
I'm not sure why you'd even generalize that so overly broadly.
To put in perspective, 300ms is about looping over 30GiB data from RAM, loading 800MiB data from SSD, or doing 1TFLOPS on a single core computer.
300ms to generate a report would be able to go through ~100M rows at least (on a single core).
And the implicit assumption that comment I made earlier, of course is not about the 100M rows scan. If there is a confusion, I am sorry.
4 replies →