← Back to context

Comment by pshirshov

3 days ago

By the way, I'm wondering why unsloth (a goddamn python library) tries to run apt-get with sudo (and fails on my nixos). Like how tf we are supposed to use that?

71 comments

pshirshov

Reply
danielhanchen  3 days ago

Oh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell.

There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel`

Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if all fails, it just fails. We need `build-essential cmake curl libcurl4-openssl-dev`

See https://github.com/unslothai/unsloth-zoo/blob/main/unsloth_z...

  • pxc  3 days ago

    It seems Unsloth is useful and popular, and you seem responsive and helpful. I'd be down to try to improve this and maybe package Unsloth for Nix as well, if you're up for reviewing and answering questions; seems fun.

    Imo it's best to just depend on the required fork of llama.cpp at build time (or not) according to some configuration. Installing things at runtime is nuts (especially if it means modifying the existing install path). But if you don't want to do that, I think this would also be an improvement:

      - see if llama.cpp is on the PATH and already has the requisite features
  - if not, check /etc/os-release to determine distro
  - if unavailable, guess distro class based on the presence of high-level package managers (apt, dnf, yum, zypper, pacman) on the PATH
  - bail, explain the problem to the user, give copy/paste-friendly instructions at the end of we managed to figure out where we're running

    Is either sort of change potentially agreeable enough that you'd be happy to review it?

    • danielhanchen  2 days ago

      Thanks for the suggestions! Apologies again I'm pretty bad at packaging, so hence the current setup.

      1. So I added a `check_llama_cpp` which checks if llama.cpp does exist and it'll use the prebuilt one https://github.com/unslothai/unsloth-zoo/blob/main/unsloth_z...

      2. Yes I like the idea of determining distro

      3. Agreed on bailing - I was also thinking if doing a Python input() with a 30 second waiting period for apt-get if that's ok? We tell the user we will apt-get some packages (only if apt exists) (no sudo), and after 30 seconds, it'll just error out

      4. I will remove sudo immediately (ie now), and temporarily just do (3)

      But more than happy to fix this asap - again sorry on me being dumb

      12 replies →

  • Balinares  2 days ago

    I'll venture that whoever is going to fine-tune their own models probably already has llama.cpp installed somewhere, or can install if required.

    Please, please, never silently attempt to mutate the state of my machine, that is not a good practice at all and will break things more often than it will help because you don't know how the machine is set up in the first place.

    • danielhanchen  2 days ago

      Oh yes so before we install llama.cpp we do an path environment check and if its not defined then it'll install.

      But yes agreed there won't be any more random package installs sorry!

      2 replies →

  • elteto  3 days ago

    Dude, this is NEVER ok. What in the world??? A third party LIBRARY running sudo commands? That’s just insane.

    You just fail and print a nice error message telling the user exactly what they need to do, including the exact apt command or whatever that they need to run.

    • danielhanchen  3 days ago

      Yes I had that at the start, but people kept complaining they don't know how to actually run terminal commands, hence the shortcut :(

      I was thinking if I can do it during the pip install or via setup.py which will do the apt-get instead.

      As a fallback, I'll probably for now remove shell executions and just warn the user

      17 replies →

    • lyu07282  2 days ago

      on a meta level its kind of worrying for the ecosystem that there is nothing in PyPI that blocks & bans developers who try to run sudo on setup. I get they don't have the resources to do manual checks, but literally no checks against malicious packages?

      1 reply →

  • lambda  3 days ago

    [flagged]

    • danielhanchen  3 days ago

      I added it since many people who used Unsloth don't know how to compile llama.cpp, so the only way from Python's side is to either (1) Install it via apt-get within the Python shell (2) Error out then tell the user to install it first, then continue again

      I chose (1) since it was mainly for ease of use for the user - but I agree it's not a good idea sorry!

      :( I also added a section to manually compile llama.cpp here: https://docs.unsloth.ai/basics/troubleshooting-and-faqs#how-...

      But I agree I should remove apt-gets - will do this asap! Thanks for the suggestions :)

      6 replies →

    • pxx  3 days ago

      > What this whole thing hallucinated by an LLM?

      probably not, because LLMs are a little more competent than this

exe34  2 days ago

hey fellow crazy person! slight tangent: one thing that helps keep me grounded with "LLMs are doing much more than regurgitation" is watching them try to get things to work on nixos - and hitting every rake on the way to hell!

nixos is such a great way to expose code doing things it shouldn't be doing.

  • pshirshov  2 days ago

    In my experience LLMs can do Nix very well, even the models I run locally. I just instruct them to pull dependencies through flake.nix and use direnv to run stuff.

    • exe34  2 days ago

      oh yes they do nix very well, but I asked cursor to set up a firecracker vm with networking for exposing a port on the host, and use conda inside to install a certain version of python with some libraries. I asked for a firecracker-vm.nix, a build.sh, a run.sh and a close.sh. it kept trying to run code inside its own fhs-env, which would run, and then when I tried it outside of the fhs, it would fail. I'd paste in the errors and it would without fail say oh let's try the proper nix version of python - which I explicitly didn't want, because I wanted to run conda versions on other machines. I tried to guide it through conda-shell but didn't get very far. in the end I ended up using docker instead, which it did set up without fail.

      but when it was failing on my original idea, it kept trying dumb things that weren't really even nix after a while.

  • danielhanchen  2 days ago

    I'm glad someone commented and tried it out - appreciate it immensely - I learnt a lot today :) I'm definitely gonna give nixos a spin as well!