Comment by maitrungduc

4 days ago

The persistent caching part is what I would watch most closely, because Numba's on-disk cache has an invalidation rule that this environment can break completely silently.

Functions defined in a notebook cell are fine. The IPython locator stamps the cache with a sha256 of the cell source, so it is content addressed and nothing about the filesystem enters the check.

Functions inside installed packages are the ones to worry about. Those use the file backed locator, whose stamp is (st.st_mtime, st.st_size) of the .py file, and _load_index just returns an empty dict when the stamp does not match. The only trace is a _cache_log line nobody sees unless NUMBA_DEBUG_CACHE is set. So if a later session has its packages unpacked fresh by mamba, every recorded mtime is wrong on arrival, every cached compilation is silently thrown away, and the user simply experiences a slow notebook with cache=True dutifully set.

This is the same problem that led CPython to hash based .pyc in PEP 552, and the reason pip writes them: an installer cannot promise anything about the mtimes of the files it has just written, and a package manager unpacking into a browser filesystem is in exactly that position. The content addressed path already exists in Numba for the IPython case, so the pieces are there.

Worth measuring whether mtimes survive a mamba install into the persistent filesystem across sessions, because that answer decides whether cache=True does anything at all for the PyTensor and PyMC users, who are the ones with compilations expensive enough to care.