Comment by std_throwaway
8 years ago
This is truly a problem. Even more so if you host your application on a network directory. Loading all the small files takes ages. I really wish there would be a good way to compile the whole application with all the modules into one package once you're ready to release. I really wish the creators of Python would have given such use-cases more consideration.
Edit: I'm aware that there are solutions that put everything a program touches into a kind of executable archive. A single file several hundred Megabytes in size. I've tested it. It doesn't really pre-compile the modules. The startup time was exactly the same.
Nuikta (http://nuitka.net/) already does that and much more:
- it compiles your program and make it stand alone so you can distribute just the exe
- it makes it start faster
- it makes it run faster
- it's fully independant of the system python. Actually your system doesn't even need a python at all
I don't get why it's not used, it's very robust, compatible with 3.6 and on some of my script I get about x4 speed up just on start up alone.
This is different from the package that I've tested (PyInstaller or py2exe).
Is Nuikta compatible with numpy, pickle, etc? I remember that numpy was very problematic with compilers like pypy for a long time.
In my experience it's easier and more reliable than PyInstaller or Py2exe to use, and cross plateform (but no cross compilation). It doesn't pack python files with an executable. It translates the Python code to C then compiles it.
Nuikta supports numpy officially, you can even see it in change logs: http://nuitka.net/posts/nuitka-release-0521.html
I haven't tried pickle.
>This is different from the package that I've tested (PyInstaller or py2exe).
In an ancestor comment you say:
> A single file several hundred Megabytes in size.
Are both points referring to PyInstaller? Asking because I've tried out PyInstaller with small CLI as well as GUI (wxPython) programs, and the resulting EXEs did not reach near that size, IIRC.
2 replies →
First time I hear about this, and I've looked for alternatives to cxfreeze and its cousins in the past.
Any time I see something like this, I feel like I'm hearing about some homeopathic cancer cure. If Nuitka actually does what it says it does, it's solving a big recurrent problem for the Python community, so why is nobody talking about it?
Having followed Nuitka since it started, I can offer my perspective:
- Before Nuitka, someone already did a "Python->C" compiler along similar lines: translate the Python source code into the C calls that the interpreter will make, eliminating any interpreter overhead and providing C compiler optimization opporunities. That thing sort-of-worked (with v1.5.2 IIRC), but was cumbersome to use and delivered a meager 5% performance improvement for the cases it did support; it was abandoned.
- Nuitka's plan had the same thing as a starting phase; people told the Nuitka guy that he's wasting his time based on prior experience. When he actually delivered a mostly-robust working version (much more usable than the previous attempts ever were), it indeed delivered only a small performance gain compared to CPython.
- As a result, it seemed like the community believed both that the whole thing is futile, and that the developer is fighting windmills.
- a lot of time passes, Nuitka keeps improving with better analysis, translation, compilation, etc - but the community has already cemented its opinion.
- Nuitka remains a useful magic system known to few.
I would say that the early Nuitka versions (and the prior attempt) gave it a SEP field that has never been lifted, and short of e.g. DropBox or Facebook adopting it, nothing will lift it either.
3 replies →
That's the question I'm asking.
Not only it's a beautiful tool, but the author has been quietly and steadily working on it for 8 years. Compatibility is the number one goal.
The guy has a lot of rigor and humility, so maybe communication suffered ?
4 replies →
I collect ideas, especially weird and powerful ideas.
I've learned not to try to talk about it because of that question: "If foo is so great, why isn't everybody using it?"
It's one of the single greatest frustrations of my life. I don't know. I've never made any progress on it. The best you can say is, "Well, that seems to be human nature." The world is full of "magic beans" and most people seem interested in banging their heads against the wall.
(Did you know, you can make a 140hp engine that fits in the volume of two stacked pizza boxes and has only one moving part?)
Anyhow, Nuitka is great, it does do all that.
And the creator is a freakin' saint for putting up with the way he's been treated by the Python community, is my opinion.
4 replies →
It's a function of how many people really need that solution. It's high enough that it exists. It's low enough that it's not a thing you do by default or even talk about much. But if you need it and it's compatible with your code - it's there.
3 replies →
Does nuitka build a static executable or do you still need to supply shared libraries with the executable?
It builds a static executable if you pass it the --standalone option.
2 replies →
Nuitka has not been able to compile any Python code I've written myself. It's not used because it's incredibly limited.
When was the last time you've tried it?
Is it a perl2exe descendant, packing the interpreter into an executable wrapper?
No it compiles python to C, then compile the C.
I know this isn’t everyone’s favorite but Cython has a way to convert your python code Into an executable with Python embedded and I bekievr it also Packs your imports
Cython is a complicated beast but I feel like it just needs a more friendly wrapper for this to be more widespread.
https://stackoverflow.com/questions/22507592/making-an-execu...
https://github.com/cython/cython/wiki/EmbeddingCython
Why Cython isn’t in the stdlib (I think it could easily replace ctypes) is beyond me sometimes
I worked on one Python application that had a startup time problem because it was on a network filesystem with slow metadata/stat times. It took several seconds to start Python.
We were able to solve most of the problem by zipping up the Python standard library and the our application.
That is, if you look at sys.path you'll see something like:
If you zip up the python3.6 directory into python36.zip then it will use that zip file as the source of the standard library, and use the zip directory structure instead of a bunch of stat calls to find the data.
This should also include getting access to the pre-compiled byte code.
You can also have Python byte-compile all of the .py files in a directory as part of your build/zip process.
Don't forget
right afterwords.
Also note calls to imp.load_source need to change to imp.load_compiled, and any .py files references directly in code need to be changed to .pyc (this is with 2.7, not sure about 3.x)
Thanks, I will try that!
cool!
I think design choices made in Python simply don't allow for comprehensive ahead of time compilation. For what it's worth, they have recently landed snapshots in Dart that do what you want:
https://github.com/dart-lang/sdk/wiki/Snapshots
It's what Flutter uses on iOS since you can't run JITed code; AOT compile it and load it as just another shared library.
Check out subpar: https://github.com/google/subpar
also it’s not static linked, so you need to make sure all of the shared libraries exist on the host, requiring to install a whole bunch of trash.