Comment by alightsoul
12 hours ago
I want to believe most of what they're missing is material held in national and public university libraries and is material specific to the country the books or papers come from. Most of their collections are not digitized and that's what I keep seeing, and then some books that are profession-specific.
They're basically rebuilding Google books, and keeping it private. That's privatizing all of humanity's knowledge. Then from what I see some other missing books are just rephrasings of other books, which can already be done with synthetic data using another LLM to generate it. I would assume a lot of those books are school text books as well, which are almost never digitized and are rephrasings of university text books. Maybe they are not able to generate synthetic data of sufficient quality and are resorting to human books that rephrase information instead
i would also assume, most of the cheap books are donated books from all over the world. that are imported into the us for resale as book lots. it would be similar to how donated clothes are packaged into lots and then resold to third world countries, which then sell them in fashion or clothes outlets. then, they really would be importing books from all over the world into the us for scanning and shredding for ai.
there is an amazing subplot in the book "service model" that describes a set of robot librarians doing exactly this kind of thing, plus sorting every bit.