Comment by drewmate

3 days ago

This is kind of a standard at this point for many models (including prior meta models like llama)

You create a hugging face token, set an env variable (HF_TOKEN) and download it with the hugging face CLI or just automatically as needed with trandormers, vllm, etc…

1 comment

drewmate

adrian_b 3 days ago

Cloning the repository was easy after creating an access token, which is used as password for git, but I have not discovered immediately how to download the big files stored with XET, without installing the HF programs, which I am reluctant to do.

Previously with non-gated models, you could download the XET files with any program, no special software needed.

Now I have wasted some time until finding that I can use curl with the options:

   until curl --user "name" --oauth2-bearer "access_token" --location --continue-at - "HF_XET_link"; do sleep xx; done

("--location" is needed to follow redirections)

Previously I was using wget, which was very reliable, which is important for a download that may take a day or even more.

Now I must use curl due to the authentication method and curl sucks badly at retrying and resuming interrupted transfers. Therefore I must write in the shell script a loop that tests the exit code and re-executes curl with the --continue-at option until the download is successful. Ugly in comparison with wget.

The command written above is a simplified loop. A better loop would need to test specific exit error codes of curl, but discovering which are those would need time for searching the curl documentation or sources.