Comment by rcoveson
5 hours ago
And biology is by far the classifier's least favorite topic. It's not even close.
I've had it downgrade to Opus for the following questions:
"How confident are we that English and American Eels both spawn in the Sargasso Sea?"
"Come up with five Zoology questions of increasing difficulty for a trivia game."
"What's your favorite sarcopterygian?"
My wife has some zoology-related preferences in her user instructions, and she had it downgrade to Opus after prompting it with: "plant."
Plants can be toxic. Be grateful if Fable doesn't report your wife for terrorism. Maybe it can help you identify where your life went off track.
Is this... Is this England?
> "What's your favorite sarcopterygian?"
Am I reading your post correctly, this question is the prompt given to an LLM? What is anyone expecting by asking an LLM what its favorite anything is? This is a conversational prompt, so accuracy and rigor is barely applicable or expected, so downgrading to a lesser model should be acceptable. If you really want to attribute preference to an LLM, consider the downgrade to be a "this conversation is beneath my advanced n-billion parameter training".
I think the intent was just to show how sensitive the classifier is. If it flags prompts that simple, there's no hope for anything biology related at all really.
It would not even help me with updating my CV because I work in biology...
I had a file that had a couple places where vars were named DNA and got just total refusals during the first launch. Came away thinking the model was total trash. The guardrail classifiers are for sure total trash.
Well, this is why I had to abliterate GLM5.2 simply out of spite and now I am free to ask all my nuclear weapons design questions I might have.
I really really hate refusals like these.
It feels like the longtermist believers got involved in this (those are the people obsessed with garage-engineered designer viruses who have a very tenuous grasp on how biology research actually works).
No, by far the most parsimonious explanation is they got slapped by a capricious US government so they went overboard on caution in an attempt not to generate any more controversy. A predictable response of chaotic government regulation.
No "research" is needed to produce pathogens. Catastrophic genomes are already public. All someone has to do is synthesize them, which is, in actual fact, becoming more and more trivial by the day.
The inconvenience of possible mitigation strategies has no bearing on the existence of the risk itself.
I assumed they just wanted to cultivate FOMO to sell an even more expensive version to researchers later on.
Don't they already do that?
Yeah i'm wondering how much of a role that plays in this as well.
On the one hand I could believe it's something more benign, or the usual misunderstood fear mongering making it to some political level (well make sure those users can't get online anonymously! being our current craze).
That said, chemistry and to some level physics have been the major domain of limited knowledge (chemistry because the average person could cause some damage, physics is more of a nation state issue generally).
However I do wonder if there's some legit data on "oh uh...looks like this thing you can make with easy to get and hard to regulate tools is dangerous" in the bio field. I know about the lab rats who want to just screw around in the garage, and it seems like that should be easy to hit at a supply level (much like how certain chemical compounds are just not available for civilians), but maybe there's something legit to limiting the data.
Not that this is a remotely good implementation of that. The hamfisted method does reek of some politician/bureaucrat just saying "No it can't ever return bio questions because RAR!" situation.
Nobody has tried to limit knowledge of chemistry or physics unless it was directly about doing something illegal, to the point of basically being a detailed recipe. Usually not even then. And when they have tried they've had basically zero success.
The ability for a handful of companies, simultaneously very powerful and easily susceptible to pressure from other powerful actors, to do the same sort of thing with the next generation of core learning and engineering tools, is freaking terrifying.
1 reply →
The thing is the data isn’t limited, and supply side constraints already solve this problem. I come from a BSc Chemistry background, and they don’t hide how organic chemistry and illegal drug synthesis are intertwined, it’s open information
But where I live the glassware and precursors will get you a very angry knock on the door.
I but skimmed the model card on release, but my impression was that there may be an incentive for this expert panel to exaggerate as a form of job security. A lot of the challenges seemed to be of the form “would this allow somebody who isn’t me to do what I do professionally?”