Comment by _the_inflator
21 hours ago
Exactly. See TikTok trouble as example and quite honestly, try a local open source LLM and ask it to use profanity, paint nudes - the LLM doesn’t answer the question of it is from OpenAI or Google.
The thing is that needs more attention is reverse engineered a LLM which is highly fascinating. I tried it, but it seems I am not there yet to put it mildly. It requires serious effort.
I am just speculating but can LLMs be sleepers? You write software and it seeds traces here and there under certain conditions that pose a serious security risk.
Or a kill switch?
I don’t know. I distrust Chinese LLMs but even more due to training data.
It is after all not a Western model. Different biases and the might be subtle but nevertheless substantial.
In short: no open source LLM may be usable without additional Finetuning for certain valid use cases.
The real value is versioning and autonomy as well as lot more stable answering despite model rot.
Also testing and the supporting systems are easier to maintain.
It is mainly an infrastructure challenge.
Since most folks still somehow review the code, I find it highly improbable, it would become visible quickly. Unless say compiling Unix parts for example. But thats compiler work and not llm.
Jia Tan would disagree.
The overwhelmingly vast majority of open-source code isn't actually looked at or audited. Yes it's there for all to see, but that doesn't mean it's doing any good at the moment, in this context.
Its not the code so much as the model. The model itself can be poison pilled and embedded with ideas that make it unfit for use and there isn't a person out there who can look at the weights and tell you what or where the problem is by looking at the code.
this isn't if-else statements, its a jumble of linear algebra and matrices.
I watched a demonstration where one AI was trained to be obsessed with penguins. they asked for a random set of numbers from it. They fed that set into another AI model to analyze and the new model started to become obsessed with penguins.
I dont think open source is the only one to worry about though. I don't know if we even have a way to guarantee any model is secure, open or closed.