Comment by homebrewer

Generate builder images and reuse them. It shaves minutes off each CI job with projects I'm working on and is non-optional because we're far from all major datacenters.

Or setup a caching proxy, whatever is easier for your org. I've had good experience with nexus previously, it's pretty heavy but very configurable, can introduce delays for new versions and check public vulnerability databases for you.

It's purely an efficiency problem though, nothing to do with security, which is covered by lock files.