Comment by godelski
2 days ago
> In a sense you can do the same thing to yourself.
Of course. I said you can do it unknowingly too.
> The degree to which it's a problem is proportional to the size of the bureaucracy.
Now take a few steps more and answer "why". What are the reasons this happens and what are the reasons people think it is reasonable? Do you think it happens purely because people are dumb? Or smart but unintended. I think you should look back at my comment because it handles both cases.
To be clear, I'm not saying you're wrong. We're just talking about the concept at different depths.
I don't think the premise that everything is a proxy is right. We can distinguish between proxies and components.
A proxy is something like, you're trying to tell if hiring discrimination is happening or to minimize it so you look at the proportion of each race in some occupation compared to their proportion of the general population. That's only a proxy because there could be reasons other than hiring discrimination for a disparity.
A component is something like, a spaceship needs to go fast. That's not the only thing it needs to do, but space is really big so going fast is kind of a sine qua non of making a spaceship useful and that's the direct requirement rather than a proxy for it.
Goodhart's law can apply to both. The problem with proxies is they're misaligned. The problem with components is they're incomplete. But this is where we come back to the principal-agent problem.
If you could enumerate all of the components and target them all then you'd have a way out of Goodhart's law. Of course, you can't because there are too many of them. But, many of the components -- especially the ones people take for granted and fail to list -- are satisfied by default or with minimal effort. And then enumerating the others, the ones that are both important and hard to satisfy, gets you what you're after in practice.
As long as the person setting the target and the person meeting it are the same person. When they're not, the person setting the target can't take anything for granted because otherwise the person meeting the target can take advantage of that.
> What are the reasons this happens and what are the reasons people think it is reasonable? Do you think it happens purely because people are dumb? Or smart but unintended.
In many cases it's because there are people (regulators, corporate bureaucrats) who aren't in a position to do something without causing significant collateral damage because they only have access to weak proxies, and then they cause the collateral damage because we required them to do it regardless, when we shouldn't have been trying to get them to do something they're in no position to do well.
I said every measurement. That is a key word.
I know we're operating at a level that most people never encounter, but you cannot in fact measure a meter. You can use a reference tool like a ruler to try to measure distance which is calibrated. But that's a proxy. You aren't measuring a meter, you're measuring with a tool that is estimating a meter. You can get really precise and use a laser. But now you're actually doing a time of flight measurement, where a laser is bouncing off of something and you're measuring the time it takes to come back. Technically you're always getting 2x the measurement but either way you're actually not measuring distance you're measuring a light impulse (which is going to have units like candles or watts) and timing it, which we then convert those units to meters. You can continue this further to even recognize the limits of each of those estimates and this is an important factor if you're trying to determine the sensitivity (and thus error) of your device.
So I think you really aren't understanding this point. There is no possible way you can directly measure even the most fundamental scientific units (your best chance is going to probably be a mole but quantum mechanics is going to fuck you up).
If you pay close attention to what I'm talking about then you might find that these aren't as different as you think they are.
Which is my point. It isn't just that you can't because they are abstract, you can't because the physical limits of the universe prevent you to in even the non-abstract cases.
I am 100% behind you in that we should better define what we're trying to measure. But this is no different than talking about measuring something with higher precision. Our example above moved from a physical reference device to a laser and a stopwatch. That's a pretty dramatic shift, right? Uses completely different mechanisms. So abstract what you're thinking just a little so we can generalize the concept. I think if you do then we'll be on the same page.
I think you misunderstood my point here. Those were rhetorical questions and the last sentence tells you why I used them. They were not questions I needed answering. Frankly, I believe something similar is happening throughout our conversation since you are frequently trying to answer questions that don't need answering and telling me things which I have even directly acknowledged. It's creating a weird situation where I don't know how to answer because I don't know how you'll interpret what I'm saying. You seem to think that I'm disagreeing with you on everything and that just isn't true. For the most part I do agree. But to get you on the same level as me I need you to be addressing why these things are happening. Keep asking why until you don't know. That exists at some depth, right? It's true for everyone since we're not omniscient gods. My conclusion certainly isn't all comprehensive, but it does find this interesting and critical part where we run into something you would probably be less surprised about if you looked at my name.