Comment by nightpool

> The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code

The Aider Polyglot website also states that the benchmark " ...asks the LLM to edit source files to complete 225 coding exercises".

However, when looking at the actual tests [0], it is not about editing code bases, it's rather just solving simple programming exercies? What am I missing?

[0] https://github.com/Aider-AI/polyglot-benchmark

>I try not to let perfect be the enemy of good. All benchmarks have limitations.

Overfitting is one of the fundamental issues to contend with when trying to figure out if any type of model at all is useful. If your leaderboard corresponds to vibes and that is your target, you could just have a vibes leaderboard

That's my perception as well. Most of the time, most of the devs I know, including myself, are not really creating novelty with the code itself, but with the product. (Sometimes even the product is not novel, just a similar enhanced version of existing products)

If the resulting code is not trying to be excessively clever or creative this is actually a good thing in my book.

The novelty and creativity should come from the product itself, especially from the users/customers perspective. Some people are too attached to LLM leaderboards being about novelty. I want reliable results whenever I give the instructions, either be the code, or the specs built into a spec file after throwing some ideas into prompts.

Having the verbatim answer to the test is not a "limitation" it is an invalidation.

[dead]

No need for thinking, that question can be found discussed and explained many times online and has almost certainly been part of the training data.

The fact that the answer is interesting makes me suspect that it's not a good test for thinking. I remember reading the explanation for the answer somewhere on the internet years ago, and it's stayed with me ever since. It's interesting enough that it's probably been written about multiple times in multiple places. So I think it would probably stay with a transformer trained on large volumes of data from the internet too.

I think a better test of thinking is to provide detail about something so mundane and esoteric that no one would have ever thought to communicate it to other people for entertainment, and then ask it a question about that pile of boring details.

> I like to make up my own tests

You just ruined your own test by publishing it on the internets

I’m not sure how much this tells me about a model’s coding ability though.

It might correlate to design level thinking but it also might not.

I have another easy one which thinking models get wrong:

"Anhentafel numbers start with you as 1. To find the Ahhentafel number of someone's father, double it. To find the Ahnentafel number of someone's mother, double it and add one.

Men pass on X chromosome DNA to their daughters, but none to their sons. Women pass on X chromosome DNA to both their sons and daughters.

List the Ahnentafel numbers of the closest 20 ancestors a man may have inherited X DNA from."

For smaller models, it's probably fair to change the question to something like: "Could you have inherited X chromosome DNA from your ancestor with Ahnentafel number 33? Does the answer to that question depend on whether you are a man or a woman?" They still fail.

I asked this to QwQ and it started writing equations (newton's law) and arrived at T_2 < T_1, then said this is counterintuitive, started writing more equations and arrived to the same, starts writing an explanation on why this is indeed the case instead of what it is intuitive, and concludes to the right answer.

It is the only model I gave this and actually approached it by writing math. Usually I am not that impressed with reasoning models, but this was quite fun to watch.

Obviously you would prepare cold brew the night before.

Yes absolutely this! We're working on these problems at FlyShirley for our pilot training tool. My go-to is: I'm facing 160 degrees and want to face north. What's the quickest way to turn and by how much?

For small models and when attention is "taken up", these sorts of questions really send a model for a loop. Agreed - especially noticeable with small reasoning models.

how do you know that this test doesn't already exist somewhere on the internet and LLMs were trained on it?

Can’t edit my comment - this was meant to say NON-thinking models can’t do it. The thinking ones can and that’s what is interesting (face palm)

[dead]

> Phrased this way without any help, all but the thinking models get it wrong

I C&P'd it into Claude 3.7 with thinking, and it gave the correct answer (which I'm pretty sure is #2). Including the CoT, where it actually does math (which I haven't checked), and final response.

# THINKING

Let's analyze the two options.

Option 1: Add cold milk immediately, then let it sit for 2 mins.

Option 2: Let it sit for 2 mins, then add cold milk.

To determine which option leads to the lowest final temperature, I need to consider the physics of cooling.

First, let's understand how cooling works in these scenarios:

1. When a hot liquid sits, it cools according to Newton's Law of Cooling, which states that the rate of heat loss is proportional to the temperature difference between the object and its surroundings.

- The rate of cooling is faster when the temperature difference is greater

- As the liquid cools, the rate of cooling decreases

2. When we add cold milk to hot coffee, we get an immediate temperature drop due to mixing liquids of different temperatures. The final temperature after mixing can be calculated using the principle of thermal equilibrium.

Now, let's analyze each option:

Option 1: Add cold milk immediately, then let it sit for 2 mins

- The coffee temperature drops immediately when milk is added

- Then this coffee+milk mixture cools for 2 minutes

- The cooling rate will be based on the temperature difference between the coffee+milk mixture and room temperature

Option 2: Let it sit for 2 mins, then add cold milk

- The black coffee cools for 2 minutes

- Then milk is added, causing an immediate drop in temperature

- The cooling rate during the 2 minutes will be based on the temperature difference between the black coffee and room temperature

The key insight is that hot coffee cools faster than warm coffee because the rate of cooling is proportional to the temperature difference. Since black coffee starts at a higher temperature than coffee with milk, it will lose more heat energy during the 2-minute cooling period.

Let's make this more concrete with some theoretical numbers:

- Let's say black coffee starts at 90°C

- Room temperature is 20°C

- Cold milk is at 5°C and makes up 10% of the final volume

Option 1:

- Adding milk immediately might bring the temperature to about 81.5°C (90 × 0.9 + 5 × 0.1)

- This mixture then cools for 2 minutes from 81.5°C to some lower temperature, let's call it T1

Option 2:

- The black coffee cools for 2 minutes from 90°C to some temperature, let's call it T2

- Then we add milk, which further reduces the temperature to 0.9 × T2 + 0.1 × 5

The question is: which is lower, T1 or 0.9 × T2 + 0.1 × 5?

According to Newton's Law of Cooling, the black coffee in Option 2 will lose more heat during the 2 minutes because it starts at a higher temperature and thus has a greater temperature difference with the surroundings. This means the drop from 90°C to T2 will be greater than the drop from 81.5°C to T1.

Let's think about this mathematically:

According to Newton's Law of Cooling: T(t) = Ts + (T0 - Ts) × e^(-kt) Where:

- T(t) is the temperature at time t

- Ts is the surrounding temperature

- T0 is the initial temperature

- k is the cooling constant

For Option 1:

T1 = 20 + (81.5 - 20) × e^(-k × 2) = 20 + 61.5 × e^(-2k)

For Option 2, before adding milk:

T2 = 20 + (90 - 20) × e^(-k × 2) = 20 + 70 × e^(-2k)

After adding milk in Option 2, the temperature would be:

0.9 × T2 + 0.1 × 5 = 0.9 × (20 + 70 × e^(-2k)) + 0.5

= 18 + 63 × e^(-2k) + 0.5

= 18.5 + 63 × e^(-2k)

Now we compare:

T1 = 20 + 61.5 × e^(-2k)

Final temp in Option 2 = 18.5 + 63 × e^(-2k)

For any positive value of k (which it must be for cooling), e^(-2k) is between 0 and 1.

If e^(-2k) = 1 (no cooling at all):

T1 = 20 + 61.5 = 81.5

Final temp in Option 2 = 18.5 + 63 = 81.5 (They would be the same)

If e^(-2k) = 0 (instant cooling to room temperature): T1 = 20 Final temp in Option 2 = 18.5 (Option 2 would be better)

For values in between, we need to compare 20 + 61.5 × e^(-2k) vs 18.5 + 63 × e^(-2k)

Subtracting: (T1 - Final temp in Option 2) = 1.5 - 1.5 × e^(-2k)

This will be positive (meaning T1 is higher) when e^(-2k) < 1, which is true for all positive k.

So, mathematically, Option 2 (letting it sit, then adding milk) results in a lower final temperature.

This makes physical sense because:

1. The hotter black coffee loses more heat during the 2 minutes than the cooler coffee+milk mixture would

2. The cold milk provides the same cooling effect whether added at the beginning or end

Therefore, Option 2 (letting the coffee sit for 2 minutes, then adding cold milk) will result in the lowest final temperature.

# ANSWER

# Coffee Cooling Analysis

## Option 2 wins: Let it sit for 2 mins, then add cold milk

This will give you the lowest final temperature, and here's why:

### The physics behind it:

1. *Newton's Law of Cooling*: The rate at which something cools is proportional to the temperature difference between the object and its surroundings.

2. *Key insight*: Hotter objects cool faster than cooler ones because the temperature gap with the environment is larger.

### Comparing the options:

*Option 1 (Add milk first, then wait):* - Milk immediately lowers coffee temperature - The coffee+milk mixture cools for 2 minutes, but at a slower rate because it's already cooler

*Option 2 (Wait 2 mins, then add milk):* - Black coffee starts hotter and cools faster during the 2 minutes - More total heat is lost during the waiting period - Then milk provides the same temperature drop at the end

The black coffee's higher initial temperature difference with the room causes it to lose more total heat energy during the waiting period, making Option 2 the more effective cooling strategy.

As far as I can tell the only way of doing a comparison of two models, that cannot be easily gamed, is being having them in open weights form and then running them against a benchmark that was created after both of the two models were created.