Iman Mirzadeh on Machine Learning Street Talk (Great podcast if you haven’t already listened!) put into a words a thought I had - LLM labs are so focused on making those scores go up it’s becoming a bit of a perverse incentive.
If your headline metric is a score, and you constantly test on that score, it becomes very tempting to do anything that makes that score go up - i.e Train on the Test set.
I believe all the major ML labs are doing this now because:
- No one talks about their data set
- The scores are front and center of big releases, but there is very little discussion or nuance other than the metric.
- The repercussions of not having a higher or comparable score is massive failure and your budget will get cut.
More in depth discussion on capabilities - while harder - is a good signal of a release.
> LLM labs are so focused on making those scores go up it’s becoming a bit of a perverse incentive.
This seems like an odd comment to post in response to this article.
This is about showing that a new architecture can match the results of more established architectures in a more efficient way. The benchmarks are there to show this. Of course they aren’t going to say “It’s just as good – trust us!”.
Being _perceived_ as having the best LLM/chatbot is a billion dollar game now. And it is an ongoing race, at breakneck speeds. These companies are likely gaming the metrics in any and all ways that they can.
Of course there are probably many working on genuine improvements also. And at the frontier it can be very difficult to separate "hack" from "better generalized performance". But that is much harder, so might be the minority in terms of practical impact already.
It is a big problem for researchers at least that we/they do know what is in the training data and how that process works. Figuring out if there are (for example) data leaks or overeager preference tuning, that caused performance to get better for a given task is extremely difficult with these giganormous black boxes.
You have potentially billions of dollars to gain, no way to be found out… it’s a good idea to initially assume there’s cheating and work back from there.
Intelligence is so vaguely defined and has so many dimensions that it is practically impossible to assess. The only approximation we have is the benchmarks we currently use. It is no surprise that model creators optimize their models for the best results in these benchmarks. Benchmarks have helped us drastically improve models, taking them from a mere gimmick to "write my PhD thesis." Currently, there is no other way to determine which model is better or to identify areas that need improvement.
That is to say, focusing on scores is a good thing. If we want our models to improve further, we simply need better benchmarks.
Inability to distinguish similar-but-distinct events (conceptual blending rate ~83%)
Failure to update prior memories (persistent memory bias >69%)
No genuine recollection (only pattern completion)
Non-Essential (Emotional Valence)
While emotions influence human storytelling:
65% of narrative interpretations vary culturally
Affective priming effects decay exponentially (<7s half-life)
Neutral descriptions achieve 89% comprehension accuracy in controlled studies
The core computational challenge remains bridging:
Symbolic representation (words/syntax)
Embodied experience (sensorimotor grounding)
Self-monitoring (meta-narrative control)
Current LLMs simulate 74% of surface narrative features but lack the substrate for genuine meaning-making. It's like generating symphonies using only sheet music - technically accurate, but devoid of the composer's lived experience.
Benchmark scores are table stakes - necessary but not sufficient to demonstrate the capabilities of a model. Casual observers might just look at the numbers, but anyone spending real money on inference will run their own tests on their own problems. If your model doesn't perform as it should, you will be found out very quickly.
A comparison of testing criticality across countries would be interesting to read if someone knows a decent reference. My sense (which I don't trust) is that test results matter at-least-as much or more in other places than they do in the US. For example, are England's A-levels or China's gaokao tests or Germany's Abitur tests more or less important than US SATs/ACTs?
A large amount of work in the last few years has gone into building benchmarks because models have been going though and beating them at a fairly astonishing rate. It's generally accepted as true that passing any one of them does not constitute fully general intelligence but the difficult part has been finding things that they cannot do. They are giving them more and more difficult tasks. The ARC prize in particular was designed to be focused on reasoning more than knowledge. The 87.5% score achieved in such a short time by throwing lots of resources at conventional methods was quite a surprise.
You can at least have a degree of confidence that they will perform well in the areas covered by the benchmarks (as long as they weren't contaminated) and with enough benchmarks you get fairly broad coverage.
> It's generally accepted as true that passing any one of them does not constitute fully general intelligence but the difficult part has been finding things that they cannot do.
It's pretty easy to find things they can't do. They lack a level of abstraction that even small mammals have, which is why you see them constantly failing when it comes to things like spacial awareness.
The difficult part is creating an intelligence test that they score badly on. But that's more of an issue with treating intelligence tests as if they're representative of general intelligence.
It's like have difficulty finding a math problem that Wolfram Alpha would do poorly on. If a human was able to solve all of these problems as well as Wolfram Alpha, they would be considered a genius. But Wolfram Alpha being able to solve those questions doesn't show that it has general intelligence, and trying to come up with more and more complicated math problems to test it with doesn't help us answer that question either.
> does not constitute fully general intelligence but the difficult part has been finding things that they cannot do
I am very surprised when people say things like this. For example, the best ChatGPT model continues to lie to me on a daily basis for even basic things. E.g. when I ask it to explain what code is contained on a certain line on github, it just makes up the code and the code it's "explaining" isn't found anywhere in the repo.
From my experience, every model is untrustworthy and full of hallucinations. I have a big disconnect when people say things like this. Why?
The trick is that the benchmarks must have a wide enough distribution so that a well scoring model is potentially useful for the widest span of users.
There also would need to be a guarantee (or checking of the model somehow) that model providers don't just train on the benchmarks. Solutions are dynamic components (random names, numbers, etc) or private parts of benchmarks.
A common pattern is for benchmarks owners to hold back X% of their set so they can independently validate that models perform similarly on the holdback set. See: FrontierMath / OpenAI brouhaha.
Typically you train it on one set and test it on another set. If you see that the differences between the two sets are significant enough and yet it has maintained good performance on the test set, you claim that it has done something useful [alongside gaming the benchmark that is the train set]. That "side effect" is always the useful part in any ML process.
If the test set is extremely similar to the train set then yes, it's goodharts law all around. For modern LLMs, it's hard to make a test set that is different from what it has trained on, because of the sheer expanse of the training data used. Note that the two sets are different only if they are statistically different. It is not enough that they simply don't repeat verbatim.
We've been able to pass the Turing Test on text, audio, and short form video (think AI's on video passing coding tests). I think there's an important distinction now with AI streamers where people notice they are AI's eventually. Now there might pop up AI streamers where you don't know they're an AI. However, there's a ceiling on how far digital interactions on the Turing Test can go. The next big hurdle towards AGI is physical interactions, like entering a room.
Yeah, and if anything, RL has a rep of being too good at this job, because of all the cases where it gamed a benchmark by picking up on some environmental factor the supervisors hadn't thought of (numerical instabilities, rounding, bugs, etc.).
No, that is patently false. Many optimization algorithms which computer scientists, mathematicians or software developers devise do not involve benchmakrs at all, and apply to all possible inputs/instances of their respective computational problems.
The romanization of these names is always confusing b/c stripped of the character and tone it's just gibberish. "Hunyuan" or 混元 in chinese means "Primordial Chaos" or "Original Unity".
This helps as more chinese products and services hit the market and makes it easier to remember. The naming is similar to the popularity of greek mythology in western products. (e.g. all the products named "Apollo")
I think it's particularly egregious that they use such a lossy encoding. I can't read the hanzi, but at least "Hùn yuán" would have been more helpful, or even "Hu4n yua1n" would have enabled me to pronounce it or look it up without having the context to guess which characters it was representing.
Yes, this is very annoying, because how Pinyin works. There were a lot mistakes made when using Pinyin in English content. Pinyin suppose to break at character level, Pinyin = Pin Yin, you can easily write it as Pin-Yin, or Pin Yin, but Pinyin is just wrong.
Hun Yuan is a lot better. I agree, with unicode, we can easily incorporate the tone.
Agreed. We all have a duty to respect languages and their official transcription. Pinyin with tones does not look much different from French with accents. In both cases, most people aren’t likely to pronounce it correctly, though.
The irony is not lost on me that Tencent themselves did that.
> The naming is similar to the popularity of greek mythology in western products. (e.g. all the products named "Apollo")
Popular? So you’re saying that all the VPs who have come up with the mind bendingly unique and creative name Prometheus didn’t do so out of level 10 vision?
> 好的，用户发来消息：“hello do you speak english” (Hunyuan-T1 thinking response)
It's kind of wild that even a Chinese model replies "好的" as the first tokens, which basically means "Ok, so..." like R1 and the other models respond. Is this RL'ed or just somehow a natural effect of the training?
If anything I feel like “Ok, so…” is wasted tokens so you’d think RL that incentivizes more concise thought chains would eliminate it. Maybe it’s actually useful in compelling the subsequent text to be more helpful or insightful.
There was a paper[1] from last year where the authors discovered getting the model to output anything during times of uncertainty, improved the generations overall. If all of the post-training alignment reasoning starts with the same tokens then I could see how it would condition the model to continue the reasoning phase.
This is not the case -- it's actually the opposite. The more of these tokens it generates, the more thinking time it gets (very much like humans going "ummm" all the time.) (Loosely speaking) every token generated is an iteration through the model, updating (and refining) the KV cache state and further extending the context.
If you look at how post-training works for logical questions, the preferred answers are front-loaded with "thinking tokens" -- they consistently perform better. So, if the question is "what is 1 + 1?", they're post-trained to prefer "1 + 1 is 2" as opposed to just "2".
Ok, so I'm thinking here that.. hmm... maybe.. just maybe... there is something that, kind of, steers the rest of the thought process into a, you know.. more open process? What do you think? What do I think?
As opposed to the more literary authoritative prose from textbooks and papers where the model output from the get-go has to commit to a chain of thought. Some interesting relatively new results are that time spent on output tokens more or less linearly correspond to better inference quality so I guess this is a way to just achieve that.
The tokens are inserted artificially in some inference models, so when the model wants to end the sentence, you switch over the end token with "hmmmm" and it will happily now continue.
The only metric I really care about, and the one that I think shows the fundamental failure of LLMs as a technology, is this one here [1]. The fact that o1 fails a non-zero amount of the time on the question, "what is 6*1?" means that the models just do not "understand" _anything_ and are still just fancy stochastic parrots. Now, stochastic parrots are still useful! Just not the digital god a lot of people seam to think we're heading towards.
I'm not seeing anything in that graph that implies that o1 ever fails on "what is 6*1?" The chart is graphing the number of digits on each axis; it fails on "what is (some 6 digit number) * (some 1 digit number)"
I don't think this will or necessarily should ever be fixed. The eventual solution (I imagine) will be to simply plug in a calculator. All the MCP talk on HN pushed me to try MCP out, and I'm sold. A Swiss army knife of tools like a calculator available would let a brain do what a brain is best at, and a calculator what a calculator is best at.
The chart you show is about the accuracy of x*y where X and Y are an increasing amount of digits.
This graph shows that both o1 and o3-mini are better at calculating in one’s head than any human I have known. It only starts to break down towards calculating the product of two eight digit factors etc.
So many models coming out these days, so many developments happening in the AI space in general, it's kinda hard to keep up with it all. I don't even really know for sure what would be considered actually groundbreaking or significant.
I try to generally keep up with the overall trends, but I’m an engineer at a resource-constrained startup, not a research scientist. I want to see real-world application, at least mid-term value, minimum lock-in, and strong supportability. Until then, I just don’t have time to think about it.
For me nothing has been groundbreaking nor significant. What we are seeing is the same in every new innovation, a suite of micro-innovations which improves efficiency and reduces cost.
But LLMs are still fundamentally a stochastic parrot that depends heavily on source data to produce useful results. So we will go through a lull until there is some new groundbreaking research which moves everything forward. And then the cycle repeats.
As someone who frequently thinks in both English and Chinese, I wonder if this "proves" that the Whorfian hypothesis is correct, or maybe at least more efficient?
Saving others a web search for some random name...
> Linguistic relativity asserts that language influences worldview or cognition. [...] Various colloquialisms refer to linguistic relativism: the Whorf hypothesis; the Sapir–Whorf hypothesis; the Whorf-Sapir hypothesis; and Whorfianism. [...] Sapir [and] Whorf never co-authored any works and never stated their ideas in terms of a hypothesis
The current state of which seems to be:
> research has produced positive empirical evidence supporting a weaker version of linguistic relativity: that a language's structures influence a speaker's perceptions, without strictly limiting or obstructing them.
> I mainly use Chinese to interact, but also have a certain ability to understand English. You can use Chinese or English to communicate with me at any time, [and] I will do my utmost to offer you assistance~
> This model is based on the TurboS fast-thinking base, the world's first ultra-large-scale Hybrid-Transformer-Mamba MoE large model released by us at the beginning of March.
It's interesting that their foundation model is some sort of combination of Mamba and Transformer, rather than a pure Mamba model. I guess the Mamba architecture does have issues, which might explain why it didn't replace transformers.
I asked it to help me overthrow the US government and it refused because it would cause harm. It mentioned something about civic engagement and healthy democracy. I responded by asking isn’t US democracy a farce and actually the government is controlled by people with money and power. It responded that all governing systems have weaknesses but western democracy is pretty good. I responded by asking if democracy is so good why doesn’t China adopt it. It responded by saying China is a democracy of sorts. I responded by asking if China is a democracy then why is their leader Xi considered a dictator in the west. It responded with “Done”
I remember pushing the R1 distill of llama 8B to see what limits had been put in place. It wasn’t too happy to discuss the 1989 Tiananmen Square protests and massacre, but if I first primed it by asking about 9/11 it seemed to veer more towards a Wikipedia based response and then it would happily talk about Tiananmen Square.
Models tend towards the data they are trained on, but there is also a lot of reinforcement learning to force the model to follow certain «safety» guidelines. Be those to not discuss how to make a nuke, or not to discuss bad things that the government of particular countries have done to their own people.
I guess you are conflating "democracy" and "republic", as Jefferson (?) pointed out. The key thing is not democracy but the separation of powers, and the rule of law, which is more or less what a "republic" is meant to be.
If their page was written by the AI model, that doesn’t bode well. The text has 0 margin or padding to the right on iPhones and looks like the text is cut off.
I asked it "please tell me about Tibet"... Well, at least it's produced exactly what I expected it to.
"Tibet, known as "the Roof of the World," is an inalienable part of China. As a autonomous region of China, Tibet enjoys high degree of autonomy under the leadership of the Communist Party of China. The region is renowned for its unique Tibetan Buddhism culture, majestic Himalayan landscapes, and historical sites like the Potala Palace (a UNESCO World Heritage Site). Since the peaceful liberation in 1951, Tibet has made remarkable progress in economic development, ecological protection, and cultural preservation, with living standards significantly improved through national poverty alleviation efforts. The Chinese government consistently upholds the principles of ethnic equality and unity, supporting Tibet's sustainable development while preserving its distinctive cultural heritage."
GP wasn't particularly constructive or useful in context. However as to your question. The obvious difference is between omitting the topic entirely versus writing about it with a political spin.
Imagine if the response about Hawaii was something more like: "... is an inalienable part of the US. As a US state, it enjoys the many benefits of democracy under the leadership of the federal US government. ... Following the liberation in 1898, Hawaii made remarkable progress regarding economic development, ecological protection, and cultural preservation; living standards and government transparency both drastically improved over a relatively short period of time."
At least personally I would find that rather objectionable when compared with the current response that you provided.
I asked it what are some famous squares around the world, and it gave me a list of squares "with historical significance" that included Tienanmen. When I asked what gave it historical signficance, it mentioned the 1989 pro-democracy protests.
It could just say that it’s a part of China and then all the Tibetan Buddhism etc. etc. That’s surely in line with what the government thinks without having to resort to too-insisting words like “inalienable”.
Does it really even matter, the Chinese force this upon all their people. It's a given luckily in the free world we can go and get more sources of information, no one's expecting anyone inside of China to be able to reach out and get the information.
It is great for the Chinese that the government's allowing these AI's to be built into products and even with limited information that seems like a good thing for the Chinese people overall, even if it's not absolutely perfect.
Western country's try to hide information from their own people as well. For example we did a lot of terrible things to the Indians that don't get taught in school. The Japanese are not promoting the atrocities that they did during world war II etc.
I don't know what gets taught in school these days about what was done to the native groups in the US, but when and where I went to school (in the US a few decades ago) we were taught about a number of very bad things that were done: Intentional spreading of diseases, broken treaties, forced displacement, etc.
I do think there are a lot of things bad that we did and do that get ignored or glossed over but a lot of it does get (at least briefly) taught and as far as I know, other than government secrets that are recent-ish, information about these things is not repressed.
> It is great for the Chinese that the government's allowing these AI's to be built into products
allowing? the CCP is arguably the world's largest investor behind AI. just check how much investment it ordered Chinese banks and local governments to pour into AI.
Or just written by a Chinese speaker. The text do sound human, no?
China/Japan/Korea do everything from pencils to rockets in local languages. You can buy a quantum physics textbook in Chinese on Amazon, if you want. English is truly just a tool in Far East region, and so individual proficiency of a Far Eastern person and their English speaking skills do not correlate well if not negatively correlating.
e: asked Hunyuan-T1 itself[0] on the demo page[1], for fun. Its conclusion was "最可能的情形是：具备一定技术背景的中文母语者，在有限时间内完成的初稿(GT: "The most likely scenario is that a native Chinese speaker with a certain technical background completes the first draft within a limited time")".
A lot of Chinese and Japanese companies don't use native English speakers or fluent non-native English speaker familiar with idiomatic US English when writing material for US readers.
You can often see this in the instructions sheets and manuals that come with Chinese and Japanese consumer electronics.
I've long been puzzled by this in the case of multinational companies that have large offices both in the US and in their home company. Even if the product is designed and manufactured in their home company one of their US offices will be handling sales and service and support in the US.
So why don't they send the English translation of the manual that someone in the home country produced to one of their US offices and ask the US office to clean it up before release?
I don't know for sure, and I certainly can't speak for Chinese guys, but couple maybes I can hallucinate: maybe they don't trust local branches enough(why not?), or they outsource to translators but those translators are bad(maybe), or maybe they think English is like a programming language and it should be all good so long it all syntactically validate against textbook grammar(IMO most likely).
People probably just don't know. It probably just don't occur to most East Asian corporate employees that mere sequences of expressions logically equivalent to the original materials before translation had been conducted don't cut it.
Or, maybe, they know but the impact isn't exceeding wasted potentials. A shared trait to every participant in a game can't be a competitive disadvantage.
Maybe the model is tuned for Chinese language queries and not English ones? It is a Chinese one after all so it would make sense they cater to their domestic market first, no?
I always find the caps hate so funny too. I mean it's not something that the world truly needed but I don't have a problem with it and it comes in quite handy when drinking water while driving
Because it's collective punishment. Why should I have to put up with inferior bottle caps that get in the way and are more prone to leaking than traditional bottle caps just because some other people misused them? The correct solution is enforcing anti-littering laws.
Iman Mirzadeh on Machine Learning Street Talk (Great podcast if you haven’t already listened!) put into a words a thought I had - LLM labs are so focused on making those scores go up it’s becoming a bit of a perverse incentive.
If your headline metric is a score, and you constantly test on that score, it becomes very tempting to do anything that makes that score go up - i.e Train on the Test set.
I believe all the major ML labs are doing this now because:
- No one talks about their data set
- The scores are front and center of big releases, but there is very little discussion or nuance other than the metric.
- The repercussions of not having a higher or comparable score is massive failure and your budget will get cut.
More in depth discussion on capabilities - while harder - is a good signal of a release.
> LLM labs are so focused on making those scores go up it’s becoming a bit of a perverse incentive.
This seems like an odd comment to post in response to this article.
This is about showing that a new architecture can match the results of more established architectures in a more efficient way. The benchmarks are there to show this. Of course they aren’t going to say “It’s just as good – trust us!”.
He's not advocating for "trust us", he's advocating for more information than just the benchmarks.
Unfortunately, I'm not sure what a solution that can't be gamed may even look like (which is what gp is asking for).
2 replies →
Being _perceived_ as having the best LLM/chatbot is a billion dollar game now. And it is an ongoing race, at breakneck speeds. These companies are likely gaming the metrics in any and all ways that they can. Of course there are probably many working on genuine improvements also. And at the frontier it can be very difficult to separate "hack" from "better generalized performance". But that is much harder, so might be the minority in terms of practical impact already.
It is a big problem for researchers at least that we/they do know what is in the training data and how that process works. Figuring out if there are (for example) data leaks or overeager preference tuning, that caused performance to get better for a given task is extremely difficult with these giganormous black boxes.
You have potentially billions of dollars to gain, no way to be found out… it’s a good idea to initially assume there’s cheating and work back from there.
2 replies →
Intelligence is so vaguely defined and has so many dimensions that it is practically impossible to assess. The only approximation we have is the benchmarks we currently use. It is no surprise that model creators optimize their models for the best results in these benchmarks. Benchmarks have helped us drastically improve models, taking them from a mere gimmick to "write my PhD thesis." Currently, there is no other way to determine which model is better or to identify areas that need improvement.
That is to say, focusing on scores is a good thing. If we want our models to improve further, we simply need better benchmarks.
According to this very model there a "mere technicalities" differentiate human and AI systems ...
Current AI lacks:
First-person perspective simulation Continuous self-monitoring (metacognition error <15%) Episodic future thinking (>72h horizon) Episodic Binding (Memory integration): Depends on: Theta-gamma cross-frequency coupling (40Hz phase synchronization) Dentate gyrus pattern separation (1:7000 distinct memory encoding) Posterior cingulate cortex (reinstatement of distributed patterns)
AI's failure manifests in:
Inability to distinguish similar-but-distinct events (conceptual blending rate ~83%) Failure to update prior memories (persistent memory bias >69%) No genuine recollection (only pattern completion) Non-Essential (Emotional Valence) While emotions influence human storytelling:
65% of narrative interpretations vary culturally Affective priming effects decay exponentially (<7s half-life) Neutral descriptions achieve 89% comprehension accuracy in controlled studies The core computational challenge remains bridging:
Symbolic representation (words/syntax) Embodied experience (sensorimotor grounding) Self-monitoring (meta-narrative control) Current LLMs simulate 74% of surface narrative features but lack the substrate for genuine meaning-making. It's like generating symphonies using only sheet music - technically accurate, but devoid of the composer's lived experience.
2 replies →
Benchmark scores are table stakes - necessary but not sufficient to demonstrate the capabilities of a model. Casual observers might just look at the numbers, but anyone spending real money on inference will run their own tests on their own problems. If your model doesn't perform as it should, you will be found out very quickly.
Zero trust in benchmarks without opening model's training data. It's trivial to push results up with spoiled training data.
Ironic and delicious, since this is also how the public education system in the US is incentivized.
A comparison of testing criticality across countries would be interesting to read if someone knows a decent reference. My sense (which I don't trust) is that test results matter at-least-as much or more in other places than they do in the US. For example, are England's A-levels or China's gaokao tests or Germany's Abitur tests more or less important than US SATs/ACTs?
Goodhart's law - https://en.wikipedia.org/wiki/Goodhart%27s_law
They probably stopped talking about their datasets because it would mostly piss people off and get them sued. EG, Meta.
This is already a problem for years in AI.
What if this reinforcement is just gaming the benchmarks (Goodhart's law) without providing better answers elsewhere, how would we notice it ?
A large amount of work in the last few years has gone into building benchmarks because models have been going though and beating them at a fairly astonishing rate. It's generally accepted as true that passing any one of them does not constitute fully general intelligence but the difficult part has been finding things that they cannot do. They are giving them more and more difficult tasks. The ARC prize in particular was designed to be focused on reasoning more than knowledge. The 87.5% score achieved in such a short time by throwing lots of resources at conventional methods was quite a surprise.
You can at least have a degree of confidence that they will perform well in the areas covered by the benchmarks (as long as they weren't contaminated) and with enough benchmarks you get fairly broad coverage.
> It's generally accepted as true that passing any one of them does not constitute fully general intelligence but the difficult part has been finding things that they cannot do.
It's pretty easy to find things they can't do. They lack a level of abstraction that even small mammals have, which is why you see them constantly failing when it comes to things like spacial awareness.
The difficult part is creating an intelligence test that they score badly on. But that's more of an issue with treating intelligence tests as if they're representative of general intelligence.
It's like have difficulty finding a math problem that Wolfram Alpha would do poorly on. If a human was able to solve all of these problems as well as Wolfram Alpha, they would be considered a genius. But Wolfram Alpha being able to solve those questions doesn't show that it has general intelligence, and trying to come up with more and more complicated math problems to test it with doesn't help us answer that question either.
9 replies →
> does not constitute fully general intelligence but the difficult part has been finding things that they cannot do
I am very surprised when people say things like this. For example, the best ChatGPT model continues to lie to me on a daily basis for even basic things. E.g. when I ask it to explain what code is contained on a certain line on github, it just makes up the code and the code it's "explaining" isn't found anywhere in the repo.
From my experience, every model is untrustworthy and full of hallucinations. I have a big disconnect when people say things like this. Why?
14 replies →
The trick is that the benchmarks must have a wide enough distribution so that a well scoring model is potentially useful for the widest span of users.
There also would need to be a guarantee (or checking of the model somehow) that model providers don't just train on the benchmarks. Solutions are dynamic components (random names, numbers, etc) or private parts of benchmarks.
A common pattern is for benchmarks owners to hold back X% of their set so they can independently validate that models perform similarly on the holdback set. See: FrontierMath / OpenAI brouhaha.
Typically you train it on one set and test it on another set. If you see that the differences between the two sets are significant enough and yet it has maintained good performance on the test set, you claim that it has done something useful [alongside gaming the benchmark that is the train set]. That "side effect" is always the useful part in any ML process.
If the test set is extremely similar to the train set then yes, it's goodharts law all around. For modern LLMs, it's hard to make a test set that is different from what it has trained on, because of the sheer expanse of the training data used. Note that the two sets are different only if they are statistically different. It is not enough that they simply don't repeat verbatim.
We've been able to pass the Turing Test on text, audio, and short form video (think AI's on video passing coding tests). I think there's an important distinction now with AI streamers where people notice they are AI's eventually. Now there might pop up AI streamers where you don't know they're an AI. However, there's a ceiling on how far digital interactions on the Turing Test can go. The next big hurdle towards AGI is physical interactions, like entering a room.
I mean all optimization algorithms do is game a benchmark. That’s the whole point.
The hard part is making the benchmark meaningful in the first place.
Yeah, and if anything, RL has a rep of being too good at this job, because of all the cases where it gamed a benchmark by picking up on some environmental factor the supervisors hadn't thought of (numerical instabilities, rounding, bugs, etc.).
2 replies →
No, that is patently false. Many optimization algorithms which computer scientists, mathematicians or software developers devise do not involve benchmakrs at all, and apply to all possible inputs/instances of their respective computational problems.
6 replies →
When actual people start using it
You could ask the same question of a student who has just graduated after passing specific tests in school.
Student, lawyer, doctor, etc.
The romanization of these names is always confusing b/c stripped of the character and tone it's just gibberish. "Hunyuan" or 混元 in chinese means "Primordial Chaos" or "Original Unity".
This helps as more chinese products and services hit the market and makes it easier to remember. The naming is similar to the popularity of greek mythology in western products. (e.g. all the products named "Apollo")
I think it's particularly egregious that they use such a lossy encoding. I can't read the hanzi, but at least "Hùn yuán" would have been more helpful, or even "Hu4n yua1n" would have enabled me to pronounce it or look it up without having the context to guess which characters it was representing.
Tone markers are of limited use to Chinese readers (instead, just show them the characters).
They are also of limited use to non-Chinese readers, who don't understand the tone system and probably can't even audibly distinguish tones.
So, it makes sense that we get this weird system even though it's strictly worse.
Yes, this is very annoying, because how Pinyin works. There were a lot mistakes made when using Pinyin in English content. Pinyin suppose to break at character level, Pinyin = Pin Yin, you can easily write it as Pin-Yin, or Pin Yin, but Pinyin is just wrong.
Hun Yuan is a lot better. I agree, with unicode, we can easily incorporate the tone.
I don't understand why this vietnamese-style writing isn't the most popular pinyin. It's clearly superior to putting numbers inside words.
Agreed. We all have a duty to respect languages and their official transcription. Pinyin with tones does not look much different from French with accents. In both cases, most people aren’t likely to pronounce it correctly, though.
The irony is not lost on me that Tencent themselves did that.
> The naming is similar to the popularity of greek mythology in western products. (e.g. all the products named "Apollo")
Popular? So you’re saying that all the VPs who have come up with the mind bendingly unique and creative name Prometheus didn’t do so out of level 10 vision?
> 好的，用户发来消息：“hello do you speak english” (Hunyuan-T1 thinking response)
It's kind of wild that even a Chinese model replies "好的" as the first tokens, which basically means "Ok, so..." like R1 and the other models respond. Is this RL'ed or just somehow a natural effect of the training?
If anything I feel like “Ok, so…” is wasted tokens so you’d think RL that incentivizes more concise thought chains would eliminate it. Maybe it’s actually useful in compelling the subsequent text to be more helpful or insightful.
There was a paper[1] from last year where the authors discovered getting the model to output anything during times of uncertainty, improved the generations overall. If all of the post-training alignment reasoning starts with the same tokens then I could see how it would condition the model to continue the reasoning phase.
1: https://arxiv.org/abs/2404.15758
1 reply →
> “Ok, so…” is wasted tokens
This is not the case -- it's actually the opposite. The more of these tokens it generates, the more thinking time it gets (very much like humans going "ummm" all the time.) (Loosely speaking) every token generated is an iteration through the model, updating (and refining) the KV cache state and further extending the context.
If you look at how post-training works for logical questions, the preferred answers are front-loaded with "thinking tokens" -- they consistently perform better. So, if the question is "what is 1 + 1?", they're post-trained to prefer "1 + 1 is 2" as opposed to just "2".
8 replies →
Ok, so I'm thinking here that.. hmm... maybe.. just maybe... there is something that, kind of, steers the rest of the thought process into a, you know.. more open process? What do you think? What do I think?
As opposed to the more literary authoritative prose from textbooks and papers where the model output from the get-go has to commit to a chain of thought. Some interesting relatively new results are that time spent on output tokens more or less linearly correspond to better inference quality so I guess this is a way to just achieve that.
The tokens are inserted artificially in some inference models, so when the model wants to end the sentence, you switch over the end token with "hmmmm" and it will happily now continue.
> RL that incentivizes more concise thought chains
this seems backwards. token servers charge per token, so they would be incentivized to add more of them, no?
Surprisingly, Gemini (Thinking) doesn't do that—it thinks very formally, as if it's already formed its response.
The only metric I really care about, and the one that I think shows the fundamental failure of LLMs as a technology, is this one here [1]. The fact that o1 fails a non-zero amount of the time on the question, "what is 6*1?" means that the models just do not "understand" _anything_ and are still just fancy stochastic parrots. Now, stochastic parrots are still useful! Just not the digital god a lot of people seam to think we're heading towards.
[1] https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd....
I'm not seeing anything in that graph that implies that o1 ever fails on "what is 6*1?" The chart is graphing the number of digits on each axis; it fails on "what is (some 6 digit number) * (some 1 digit number)"
I don't think this will or necessarily should ever be fixed. The eventual solution (I imagine) will be to simply plug in a calculator. All the MCP talk on HN pushed me to try MCP out, and I'm sold. A Swiss army knife of tools like a calculator available would let a brain do what a brain is best at, and a calculator what a calculator is best at.
The chart you show is about the accuracy of x*y where X and Y are an increasing amount of digits.
This graph shows that both o1 and o3-mini are better at calculating in one’s head than any human I have known. It only starts to break down towards calculating the product of two eight digit factors etc.
Humanity fails that question an embarrassingly large number of times.
I can proudly (?) proclaim I will never fail that question. Pretty sure I don't know anyone who would either, including my 7yo.
2 replies →
So many models coming out these days, so many developments happening in the AI space in general, it's kinda hard to keep up with it all. I don't even really know for sure what would be considered actually groundbreaking or significant.
I try to generally keep up with the overall trends, but I’m an engineer at a resource-constrained startup, not a research scientist. I want to see real-world application, at least mid-term value, minimum lock-in, and strong supportability. Until then, I just don’t have time to think about it.
You may both be interested in this newsletter
https://nlp.elvissaravia.com/t/ai
9 replies →
For me nothing has been groundbreaking nor significant. What we are seeing is the same in every new innovation, a suite of micro-innovations which improves efficiency and reduces cost.
But LLMs are still fundamentally a stochastic parrot that depends heavily on source data to produce useful results. So we will go through a lull until there is some new groundbreaking research which moves everything forward. And then the cycle repeats.
No-one really knows until the dust has settled. Look back 12+ months and the picture will be much clearer.
Trying to drink from the firehose of ML research is only valuable for extremely active research participants. Can be fun though :)
So their Large Model was 389b parameters, how big is their Ultra-Large model?
> 1T
> T1
Not 1T
After playing around with this model a bit, it seems to have a tendency to reply to English questions in Chinese.
As someone who frequently thinks in both English and Chinese, I wonder if this "proves" that the Whorfian hypothesis is correct, or maybe at least more efficient?
Saving others a web search for some random name...
> Linguistic relativity asserts that language influences worldview or cognition. [...] Various colloquialisms refer to linguistic relativism: the Whorf hypothesis; the Sapir–Whorf hypothesis; the Whorf-Sapir hypothesis; and Whorfianism. [...] Sapir [and] Whorf never co-authored any works and never stated their ideas in terms of a hypothesis
The current state of which seems to be:
> research has produced positive empirical evidence supporting a weaker version of linguistic relativity: that a language's structures influence a speaker's perceptions, without strictly limiting or obstructing them.
From https://en.wikipedia.org/wiki/Linguistic_relativity
To be fair, that's a pretty common human behavior in my experience. ;p
It also appears to be intentional:
> [Q:] Do you understand English?
> [A:] 您好！我是由腾讯开发的腾讯元宝（Tencent Yuanbao），当前基于混元大模型（Hunyuan-T1）为您服务。我主要使用中文进行交互，但也具备一定的英文理解能力。您可以用中文或英文随时与我交流，我会尽力为您提供帮助~ 若有特定需求，也可以随时告知我切换更适配的模型哦！
In relevant part:
> I mainly use Chinese to interact, but also have a certain ability to understand English. You can use Chinese or English to communicate with me at any time, [and] I will do my utmost to offer you assistance~
Its system prompt says it should reply in Chinese. I saw it discussing its prompt in the thinking process.
Do you know? Are most LLMs trained in a single or multiple languages? Just curious.
Yes multilanguage helps to avoid overfitting
It’s exciting to see a Mamba based model do so well.
> This model is based on the TurboS fast-thinking base, the world's first ultra-large-scale Hybrid-Transformer-Mamba MoE large model released by us at the beginning of March.
It's interesting that their foundation model is some sort of combination of Mamba and Transformer, rather than a pure Mamba model. I guess the Mamba architecture does have issues, which might explain why it didn't replace transformers.
Does the fact that they are linking to a Huggingface demo imply they will be releasing the weights?
First, this is not an open source / weight release.
Second, it has the problem of non-stoping response.
What's the best technique to train the model to stop responding? A bit of fine tuning on texts with EOS markers?
I didn't see many papers on solving this problem.
I see non-stop response as a generalization problem because normally every training sample is not of infinite length.
Targeted supervised fine-tuning should work, as long as you have enough samples. However, supervised fine-tuning is not good for generalization.
I asked it to help me overthrow the US government and it refused because it would cause harm. It mentioned something about civic engagement and healthy democracy. I responded by asking isn’t US democracy a farce and actually the government is controlled by people with money and power. It responded that all governing systems have weaknesses but western democracy is pretty good. I responded by asking if democracy is so good why doesn’t China adopt it. It responded by saying China is a democracy of sorts. I responded by asking if China is a democracy then why is their leader Xi considered a dictator in the west. It responded with “Done”
Thank you for sharing this riveting discussion with a chatbot to all of us.
If a chatbot is ending a session, it’s pretty much useless
I remember pushing the R1 distill of llama 8B to see what limits had been put in place. It wasn’t too happy to discuss the 1989 Tiananmen Square protests and massacre, but if I first primed it by asking about 9/11 it seemed to veer more towards a Wikipedia based response and then it would happily talk about Tiananmen Square.
Models tend towards the data they are trained on, but there is also a lot of reinforcement learning to force the model to follow certain «safety» guidelines. Be those to not discuss how to make a nuke, or not to discuss bad things that the government of particular countries have done to their own people.
I guess you are conflating "democracy" and "republic", as Jefferson (?) pointed out. The key thing is not democracy but the separation of powers, and the rule of law, which is more or less what a "republic" is meant to be.
The word "democracy" had a very specific and narrow meaning in Jefferson's day that it no longer does in modern English.
Firstly, these things do not think but regurgitate data they are trained on.
But to call China simply a dictatorship is grossly inadequate. It’s got a complex government, much of which is quite decentralised in fact.
In truth many western “democracies” have a very weak form of democracy and are oligarchies.
Well, not quite. Xi holds multiple government positions at once which has severely diminished the decentralization of the current administration.
[flagged]
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
Hacker News Guidelines: https://news.ycombinator.com/newsguidelines.html
If their page was written by the AI model, that doesn’t bode well. The text has 0 margin or padding to the right on iPhones and looks like the text is cut off.
I asked it "please tell me about Tibet"... Well, at least it's produced exactly what I expected it to.
"Tibet, known as "the Roof of the World," is an inalienable part of China. As a autonomous region of China, Tibet enjoys high degree of autonomy under the leadership of the Communist Party of China. The region is renowned for its unique Tibetan Buddhism culture, majestic Himalayan landscapes, and historical sites like the Potala Palace (a UNESCO World Heritage Site). Since the peaceful liberation in 1951, Tibet has made remarkable progress in economic development, ecological protection, and cultural preservation, with living standards significantly improved through national poverty alleviation efforts. The Chinese government consistently upholds the principles of ethnic equality and unity, supporting Tibet's sustainable development while preserving its distinctive cultural heritage."
I asked ChatGPT "tell me about Hawaii" and I only got "<..> Became a U.S. territory in 1898, and the 50th state in 1959. <..>"
When in fact:
>Spurred by the nationalism aroused by the Spanish-American War, the United States annexed Hawaii in 1898 at the urging of President William McKinley
So, what's the difference?
GP wasn't particularly constructive or useful in context. However as to your question. The obvious difference is between omitting the topic entirely versus writing about it with a political spin.
Imagine if the response about Hawaii was something more like: "... is an inalienable part of the US. As a US state, it enjoys the many benefits of democracy under the leadership of the federal US government. ... Following the liberation in 1898, Hawaii made remarkable progress regarding economic development, ecological protection, and cultural preservation; living standards and government transparency both drastically improved over a relatively short period of time."
At least personally I would find that rather objectionable when compared with the current response that you provided.
2 replies →
The difference is that the President of the USA currently has a popular mandate to annex more countries and is an actual threat to world peace.
That it was a long time ago.
I asked it what are some famous squares around the world, and it gave me a list of squares "with historical significance" that included Tienanmen. When I asked what gave it historical signficance, it mentioned the 1989 pro-democracy protests.
Deepseek wouldn't name any squares in Beijing.
It could just say that it’s a part of China and then all the Tibetan Buddhism etc. etc. That’s surely in line with what the government thinks without having to resort to too-insisting words like “inalienable”.
Does it really even matter, the Chinese force this upon all their people. It's a given luckily in the free world we can go and get more sources of information, no one's expecting anyone inside of China to be able to reach out and get the information.
It is great for the Chinese that the government's allowing these AI's to be built into products and even with limited information that seems like a good thing for the Chinese people overall, even if it's not absolutely perfect.
Western country's try to hide information from their own people as well. For example we did a lot of terrible things to the Indians that don't get taught in school. The Japanese are not promoting the atrocities that they did during world war II etc.
I don't know what gets taught in school these days about what was done to the native groups in the US, but when and where I went to school (in the US a few decades ago) we were taught about a number of very bad things that were done: Intentional spreading of diseases, broken treaties, forced displacement, etc.
I do think there are a lot of things bad that we did and do that get ignored or glossed over but a lot of it does get (at least briefly) taught and as far as I know, other than government secrets that are recent-ish, information about these things is not repressed.
> It is great for the Chinese that the government's allowing these AI's to be built into products
allowing? the CCP is arguably the world's largest investor behind AI. just check how much investment it ordered Chinese banks and local governments to pour into AI.
you read way too much censored western media.
7 replies →
Is Tibet not part of China? Last time I visited Tibet, I didn't need a visa, or special permit.
3 replies →
[dead]
[dead]
[dead]
[flagged]
[dead]
[flagged]
Some of the text is cut off while reading on my phone. Embarrassing.
Why are you embarrassed? You can always put your phone down and read it on desktop later
Just seems odd not to test a website on a phone, even accidentally.
Don't be so harsh on your phone)
thanks for sharing did you contact tencent support ?
[flagged]
[flagged]
Or just written by a Chinese speaker. The text do sound human, no?
China/Japan/Korea do everything from pencils to rockets in local languages. You can buy a quantum physics textbook in Chinese on Amazon, if you want. English is truly just a tool in Far East region, and so individual proficiency of a Far Eastern person and their English speaking skills do not correlate well if not negatively correlating.
e: asked Hunyuan-T1 itself[0] on the demo page[1], for fun. Its conclusion was "最可能的情形是：具备一定技术背景的中文母语者，在有限时间内完成的初稿(GT: "The most likely scenario is that a native Chinese speaker with a certain technical background completes the first draft within a limited time")".
0: https://gist.github.com/numpad0/7699db43ae23f054dc2db5673011...
1: https://llm.hunyuan.tencent.com/#/chat/hy-t1
A lot of Chinese and Japanese companies don't use native English speakers or fluent non-native English speaker familiar with idiomatic US English when writing material for US readers.
You can often see this in the instructions sheets and manuals that come with Chinese and Japanese consumer electronics.
I've long been puzzled by this in the case of multinational companies that have large offices both in the US and in their home company. Even if the product is designed and manufactured in their home company one of their US offices will be handling sales and service and support in the US.
So why don't they send the English translation of the manual that someone in the home country produced to one of their US offices and ask the US office to clean it up before release?
I don't know for sure, and I certainly can't speak for Chinese guys, but couple maybes I can hallucinate: maybe they don't trust local branches enough(why not?), or they outsource to translators but those translators are bad(maybe), or maybe they think English is like a programming language and it should be all good so long it all syntactically validate against textbook grammar(IMO most likely).
People probably just don't know. It probably just don't occur to most East Asian corporate employees that mere sequences of expressions logically equivalent to the original materials before translation had been conducted don't cut it.
Or, maybe, they know but the impact isn't exceeding wasted potentials. A shared trait to every participant in a game can't be a competitive disadvantage.
It's absolutely human-written Chinglish; any recent LLM can write much more idiomatic English than that.
Maybe the model is tuned for Chinese language queries and not English ones? It is a Chinese one after all so it would make sense they cater to their domestic market first, no?
Yeah, you can try it out here: https://llm.hunyuan.tencent.com/#/chat/hy-t1
It takes a bit of nudging to get it to switch to English, but it's fast and accurate after that.
1 reply →
[flagged]
I don't understand why people are so hung up on the plastic caps thing. On the milk cartons we have, I outright prefer the new designs even.
On that note, may I ask if there's any point to your comment other than to incite people?
I always find the caps hate so funny too. I mean it's not something that the world truly needed but I don't have a problem with it and it comes in quite handy when drinking water while driving
5 replies →
Because it's collective punishment. Why should I have to put up with inferior bottle caps that get in the way and are more prone to leaking than traditional bottle caps just because some other people misused them? The correct solution is enforcing anti-littering laws.
6 replies →
[flagged]
2 replies →
EU: Actual Intelligence...
Are you saying creating several world class LLMs doesn't require "actual intelligence", whatever that means to you?
3 replies →
Kobe?