Comment by fooker
1 day ago
Counterintuitively: program in python only if you can get away without knowing these numbers.
When this starts to matter, python stops being the right tool for the job.
1 day ago
Counterintuitively: program in python only if you can get away without knowing these numbers.
When this starts to matter, python stops being the right tool for the job.
Or keep your Python scaffolding, but push the performance-critical bits down into a C or Rust extension, like numpy, pandas, PyTorch and the rest all do.
But I agree with the spirit of what you wrote - these numbers are interesting but aren’t worth memorizing. Instead, instrument your code in production to see where it’s slow in the real world with real user data (premature optimization is the root of all evil etc), profile your code (with pyspy, it’s the best tool for this if you’re looking for cpu-hogging code), and if you find yourself worrying about how long it takes to add something to a list in Python you really shouldn’t be doing that operation in Python at all.
"if you're not measuring, you're not optimizing"
I agree. I've been living off Python for 20 years and have never needed to know any of these numbers, nor do I need them now, for my work, contrary to the title. I also regularly use profiling for performance optimization and opt for Cython, SWIG, JIT libraries, or other tools as needed. None of these numbers would ever factor into my decision-making.
.....
You don't see any value in knowing that numbers?
That's what I just said. There is zero value to me knowing these numbers. I assume that all python built in methods are pretty much the same speed. I concentrate on IO being slow, minimizing these operations. I think about CPU intensive loops that process large data, and I try to use libraries like numpy, DuckDB, or other tools to do the processing. If I have a more complicated system, I profile its methods, and optimize tight loops based on PROFILING. I don't care what the numbers in the article are, because I PROFILE, and I optimize the procedures that are the slowest, for example, using cython. Which part of what I am saying does not make sense?
1 reply →
As others have pointed out, Python is better used in places where those numbers aren't relevant.
If they start becoming relevant, it's usually a sign that you're using the language in a domain where a duck-typed bytecode scripting-glue language is not well-suited.
Exactly. If you're working on an application where these numbers matter, Python is far too high-level a language to actually be able to optimize them.
Why? I've build some massive analytic data flows in Python with turbodbc + pandas which are basically C++ fast. It uses more memory which supports your point, but on the flip-side we're talking $5-10 extra cost a year. It could frankly be $20k a year and still be cheaper than staffing more people like me to maintain these things, rather than having a couple of us and then letting the BI people use the tools we provide for them. Similarily when we do embeded work, micro-python is just so much easier to deal with for our engineering staff.
The interoperability between C and Python makes it great, and you need to know these numbers on Python to know when to actually build something in C. With Zig getting really great interoperability, things are looking better than ever.
Not that you're wrong as such. I wouldn't use Python to run an airplane, but I really don't see why you wouldn't care about the resources just because you're working with an interpreted or GC language.
> you need to know these numbers on Python to know when to actually build something in C
People usually approach this the other way, use something like pandas or numpy from the beginning if it solves your problem. Do not write matrix multiplications or joins in python at all.
If there is no library that solves your problem, it's a great indication that you should avoid python. Unless you are willing to spend 5 man-years writing a C or C++ library with good python interop.
People generally aren’t rolling their own matmuls or joins or whatever in production code. There are tons of tools like Numba, Jax, Triton, etc that you can use to write very fast code for new, novel, and unsolved problems. The idea that “if you need fast code, don’t write Python” has been totally obsolete for over a decade.
2 replies →
From the complete opposite side, I've built some tiny bits of near irrelevant code where python has been unacceptable, e.g. in shell startup / in bash's PROMPT_COMMAND, etc. It ends up having a very painfully obvious startup time, even if the code is nearing the equivalent of Hello World
What exactly do you need 1ms instead of 14ms startup time in a shell startup? The difference is barely perceptible.
Most of the time starting up is time spent seartching the filesystem for thousands of packages.
4 replies →
These basically seem like numbers of last resort. After you’ve profiled and ruled out all of the usual culprits (big disk reads, network latency, polynomial or exponential time algorithms, wasteful overbuilt data structures, etc) and need to optimize at the level of individual operations.
Not at all.
Some of those number are very important:
- Set membership check is 19.0 ns, list is 3.85 μs. Knowing what data structure to use for the job is paramount.
- Write 1KB file is 35.1 μs but 1MB file is only 207 μs. Knowing the implications of I/O trade off is essential.
- sum() 1,000 integers is only 1,900 ns: Knowing to leverage the stdlib makes all the difference compared to manual loop.
Etc.
A few years ago I did a Python rewrite of a big clients code base. They had a massive calculation process that took 6 servers 2 hours.
We got it down to 1 server, 10 minutes, and it was not even the goal of the mission, just the side effect of using Python correctly.
In the end, quadratic behavior is quadratic behavior.
List membership check being significantly slower than set membership check is freshman computer science 101.