Comment by weberc2
8 years ago
I'm a professional Python developer and I run into performance problems a lot. Python makes things really hard for even specialists to "hold right". Contrast that with Go, which (for all the hate it gets) writes very alike well-formed Python in single-threaded applications, and writes how you would like to write Python in parallel applications. And all the while being two orders of magnitude faster. If we don't start taking performance seriously in the Python community, Go (or someone else) will eat our lunch sooner or later.
Go offers faster performance with code that is up to 50% longer - with the commensurate added maintenance burden.
And, go is still very slow compared to C, C++ or Rust.
Since performance is usually a power law distribution (99% of the performance gains are made in 1% of the code), it's frequently more effective - in terms of speed and maintenance burden - to code up hot paths in a language like C, C++ or Rust and keep python.
I accept that Go is more verbose than Python, but your maintainability claim doesn’t match my experience at all. I find that Go is more maintainable for a few reasons: magic is discouraged in Go, everyone writes Go in pretty much the same way and with the same style, Python’s type system is still very immature (no recursive types, doesn’t play nicely with magical libs). Further, in my experience with working with large Python and Go codebases, Python becomes less maintainable very quickly as code size increases, especially in the face of more junior developers. Go seems to be more resistant to these forces, probably because of the rails it imposes. Lastly, any maintainability advantages Python might have had are quickly eaten up by the optimizations, which are necessary in a much greater portion of the code base because naive Python is so much slower than naive Go.
Go is ~100X faster than Python and about half as fast as C/C++/Rust, and I find it to be at least as maintainable as Python for most (but not all!) applications.
As for your power law claim, I agree with the premise but not the conclusion—-“rewrite the hotpath in C!” is not a panacea. This only works when you’re getting enough perf gain out of the C code to justify the marshaling overhead (and of course the new maintenance burden).
I don’t like bashing on Python, but it doesn’t compete well with Go on these grounds. It needs to improve, and we can’t fix it by making dubious claims about Go. We should push to improve things like Pypy and MyPy, as well as other tooling and improvements.
Have you watched David Beazley's talks about using generators to implement coroutines? That might give you a similar pattern to goroutines. If non-blocking IO isn't the challenge, do you make use of the concurrent.futures module?
While I also encounter efficiency issues, most of them are frustrations with the overhead of serialization in some distributed compute framework or the throughput of someone else's REST API. As much as so many people complain about the GIL, it's never been a blocker for me (pun intended). Perhaps it's because my style in Python is heavily influenced by Clojure.
Now that I think about it, Python's string processing is often my bottleneck.
Coroutines aren’t parallelization, so they’re quite a lot worse than goroutines in terms of performance. If you want parallelism in Python, you’re pretty much constrained to clumsy multiprocessing. Besides parallelism, Python makes it difficult to write efficient single threaded code, since all data is sprinkled around the heap, everything is garbage collected, and you can’t do anything without digging into a hashmap and calling a dozen C functions under the hood. And you can’t do much about these things except write in C, and that can even make things slower if you aren’t careful.
Probably the best thing you can do in Python is async io, and even this is clumsier and slower than in Go. :(
I'm getting confused. Are you trying to do parallel compute or parallel networking?
If parallel networking, the benchmarks I've seen set Python asynchronous IO at about the same speed as Golang. The folks at Magicstack reported that Python's bottleneck was parsing HTTP (https://magic.io/blog/uvloop-blazing-fast-python-networking/). Note their uvloop benchmark was about as fast or faster than the equivalent Golang code.
If parallel compute, then multiprocessing is the way to go and Python's futures module ain't clumsy. It's just ``pool.submit(func)`` or ``pool.map(func, sequence)``. If you're asking for parallel compute via multithreading, you're going against the wisdom of shared-nothing architecture. Besides, pretty soon you'll want to go distributed and won't be able to use threads anyway.
In contrast to your experience, I find Python makes it easy to write efficient code. Getting rid of the irrelevant details lets me focus on clear and efficient algorithms. When I need heavy compute, I sprinkle in a little NumPy or Numba. My bottleneck is (de)serialization, but Dask using Apache Arrow should solve that problem.
2 replies →