Comment by sebzim4500
3 years ago
I'm trying to get the thread model here. Is the concern that Youtube will inject JS into the payload which tries to break out of the youtuble-dl js sandbox using some zero day in whatever js engine they would use instead?
One of the reasons people use yt-dlp/youtube-dl (and nitter.net/etc) is to transform the modern proprietary JavaScript web into something more suitable for enthusiasts of the old document web and of FOSS. If the web switched to plain <video> then yt-dlp/youtube-dl would become completely unnecessary. Your browser should not have to run JS to watch an embedded video.
On my Ivy Bridge laptop running Linux, enabling hardware video decode in mpv took installing one package and adding one line to mpv.conf. Enabling hardware decoding in Firefox took multiple attempts of Googling frantically, toggling flags in about:config, passing logging environment variables to Firefox, recording a Pernosco trace of multi-process communication, and even asking for help in the gfx-firefox Matrix chat where they pointed out I had disabled media.rdd-process.enabled causing Firefox to print a misleading error message in about:support saying HARDWARE_VIDEO_DECODING was available, but failing at runtime saying WebRender was disabled even though it was enabled. And to my knowledge, hardware decoding in Chromium is simply not possible on Linux right now (maybe possible on Chromebooks, I haven't checked).
Even after I fixed hardware acceleration, playing a 1080p YouTube video in Firefox using hardware H.264 decoding took more CPU energy (40% of a core) than playing the same video in mpv using software H.264 decoding (20% of a core). Web browsers are just horrifically complex, intractable to understand, and inefficient.
Google attempting zero days on client computers would be something. It's not totally without precedent (Sony CD rootkits - https://en.wikipedia.org/wiki/Sony_BMG_copy_protection_rootk...) but would still be major news.
While they likely wouldn't do a zero-day, their JS files, particularly for automated captchas, do push the boundaries of whatever JS engine they're executed inside. See https://github.com/neuroradiology/InsideReCaptcha#the-analys... and note that this analysis is 8 years old. While there's minimal risk if you're either using a full-fledged modern JS engine or a limited-subset interpreter like the OP, an older or non-optimized spec-compliant JS engine might hit pathological performance cases and result in you DOSing yourself.
It's interesting to speculate about why they don't use this much more powerful technology to stop ytdl but instead use this much weaker yt specific thing.
Most likely the reason is that they keep the botguard system for the stuff that matters to them a lot more like account signups and click fraud, and don't want to incentivize the ytdl guys to break it on behalf of spammers/clickfraudsters.
I mean the DOS would be that your youtube-dl invocation hangs, and then you kill it.
Let's say they end up using Node. Node has a quite complete standard library that lets you access files and everything.
Now if they do it right and only embed some bare JS interpreter, it's still way harder to audit than these < 900 lines, for which it is quite easy to convince oneself that the interpreted script cannot do much.
Nowadays they could probably use Deno. Without permissions it doesn't allow network or file access etc.
Embedding a whole js engine and then interopping with it from python would be non trivial. Good luck fixing any bugs or corner cases you hit that way. The V8 and spidermonkey embedding apis are both c++ (iirc) and non trivial to use correctly.
Having full control like this +simple code is probably lower risk and more maintainable, even if there's the challenge of expanding feature set if scripts change.
The alternative would be a console js shell, but those are very different from browsers so that poses it's own challenges.
Fwiw there are python bindings for QuickJS and Duktape:
https://github.com/PetterS/quickjs
https://github.com/stefano/pyduktape
https://github.com/amol-/dukpy
I can't speak to the quality of those bindings, but they do seem maintained.
> Embedding a whole js engine and then interopping with it from python would be non trivial.
Cue libv8-node+mini_racer from which PyMiniRacer was born. It is non-trivial but not as hard as one might think.
The most painful part is the libv8 build system and Google-centric tooling (depot tools!), which makes it an absolute PITA for libv8 consumers that are not Google/Chrome.
This is why the libv8 gem was atrocious to keep up to date and to build for several platforms, and why libv8-node was born, because the node build system and source distribution are actually sane (props to their relentless work on which we piggyback on)
Disclaimer: worked at Sqreen, now maintainer of libv8-node and collaborator of mini_racer
https://github.com/sqreen/PyMiniRacer
https://github.com/rubyjs/mini_racer
https://github.com/rubyjs/libv8-node
Very cool, I'll have to remember that this exists! Looks useful.
apparently yt-dlp is somehow calling out to a js engine if available
Yeah, it's possible to install v8 or spidermonkey shells and use them to run code - we use them to run parts of the .NET wasm test suite - but they have a bunch of arbitrary limitations, so if you're trying to emulate a browser I'm not sure I'd bet on them. It's certainly going to be easier than a C++ embedding, so it makes sense that they took that route.
Another option is to use node, but it also has weird limitations/behaviors when running code.
youtube-dl targets a lot of websites other than Google properties, many of which are a lot sketchier (think, uh, NSFW streaming sites).