Comment by josalhor
5 years ago
Just the other day I was in my logistics class and the professor started deviating into something that I believe was pure nonsense.
He started the lecture by analyzing how many pieces a machine could manufacture per day. Fair enough. He extended the model to measure different ratios of capacity. Makes sense.
Then he tried to extend the model to all machines, including humans. His example was: "How do you measure the capacity of a legal team?". I thought it was a trick question, so I answered (paraphrasing) "You can't answer that question the same way you answer for the machine. You can't give a single metric." He told me I was wrong and that the _right_ measure would be (total number of working hours/day).
I was tempted to try to convince him otherwise. The analogy was deeply flawed. He certainly measured the machines in (number of pieces / day) but measured the legal team in (hours/day). So, in analyzing a machine, you take into account its efficiency, but you don't do the same thing for humans.
I believe that is exactly the same thing that is going on in the post. Managers/Logistics/Economists are very susceptible to this kind of generalization pitfalls.
Edit: Given that this answer has generated some discussion I feel the need to expand on it. The legal team was not expected to sell their services "by the hour". In fact, any discussion about how their services were sold was shut down by the professor. From his point of view, the lawyers were machines and he was asking the question "how much can this machine produce?"
Yes, other students also suggested taking the number of billable hours/revenue into account, but that's not the answer the professor was looking for.
I don't criticize whether his answer is not technically right, but I feel it holds no real-world meaning. It was a purely academic question that leads nowhere instead of having a debate about how you measure the productivity of a group of human beings. And on top of that, his final answer was definitive and (from his point of view) was irrefutable.
There's a simple mathematical argument to thwart your prof. Imagine an equation of two or more variables (such as "capacity"). For example the "capacity" of a storage unit is an equation relating height, width, and depth to volume (3 independent variables).
Any such equation can only be represented by a single number ("capacity" or "productivity") if all variables are dependent (and therefore, there is only one independent variable in the equation).
So the assertion your professor is making is that the "capacity" of a team is always exactly dependent on hours worked per day, and any other proposed dimension of capacity (such as years experience, field of study, languages spoken, cases won, relationships with judges) are dependent on "hours worked per day". If he agrees any one of those variables affects capacity, but does not depend on "hours worked per day", then a single number can never reduce the dimensionality of the output (you need at minimum 2 numbers to represent two independent variables, you can never "collapse" the data).
> Imagine an equation of two or more variables (such as "capacity"). For example the "capacity" of a storage unit is an equation relating height, width, and depth to volume (3 independent variables).
> Any such equation can only be represented by a single number ("capacity" or "productivity") if all variables are dependent (and therefore, there is only one independent variable in the equation).
This doesn't seem right. You can have storage units with varied combinations of height, width, and depth, sure. But whether that matters depends on what you want to use them to store. An example of an approach that doesn't work would be storing unboxed fragile antique dollhouses. They have weird shapes, so you can't fill the floor area, and you can't stack them, so adding height to the storage unit doesn't add any capacity.
Except that of course you wouldn't just toss them into a garage and call it a day. (They'd break!) You'd keep them in boxes. Those pack and stack perfectly. Suddenly volume is what matters again, and increasing the width, length, or height of the unit by 10% will increase the amount you can store by about 10%.
This is even more obvious if you're storing water or oxygen. Fluids take the shape you give them. Your unit might have length, width, and height (though it really shouldn't... you want to store fluids in cylinders), but the only thing that matters for how much water you can put in there is volume.
The Knapsack problem is arguably a counter argument to measuring storage space by volume.
However, air freight is a much more direct one. You have 2 largely independent measurements for weight and volume with either being the limiting metric for each load.
1 reply →
If you can barely store a box in a storage slot, increasing the space doesn't allow you to store more boxes.
You can't really assume a 100% packing rate along with increasing x dimension by y% meaning you can store y% more stuff
1 reply →
The math here is not necessary. It’s just common sense that if only the hours matter, then nothing else matters, by definition.
Even in terms of widgets per hour, you need to be VERY careful to include measures of quality.
1 - Underlying defects will absolutely sink your downstream production rate.
2 - If you only measure widgets per hour, the machine will make more, but smaller widgets (See Soviet Nail factory story - https://skeptics.stackexchange.com/questions/22375/did-a-sov... )
3 - Quality has a quantity all it's own. Many times, better widgets will improve efficiency many times over their own cost of production.
4 - Your professor was real dumb.
Responding to your edit: It's ironic that your professor seems to have taken pains to rule out all of the framings that would render their assertion correct, in (I presume) an effort to try and come up with some sort of universal rule that works in any industry and any context.
I'm pretty sure it's due to exactly that sort of hubris that business school folks have invited so much disdain. The domain in which you're operating simply cannot be dismissed as an inconsequential detail.
Tangentially, there is a subset of law firms that do operate as if total hours worked is the only thing that matters. Over the past decade or so, they've been rapidly losing ground to law firms that, by not thinking that way, manage to do a better job of producing the kinds of output that clients actually want.
I can see why, my limited experience with white shoe law firms was they billed us 100 hours for the 10 minutes it took a legal secretary to do a search and replace on another contract they did for someone else.
To be fair to my professor, the question does make sense in the context of the subject. That is, the subject focuses on answering questions like: How much I am producing? How much could I produce? How do I measure that? Etc. The subject intentionally ignores business models.
So, the question "How do you measure the capacity of a legal team?" (note it says capacity), makes sense. It's the answer I disagree with.
But doesn't capacity imply some kind of fungible unit, whereas outputs of intellectual labor tend to be non-fungible?
2 replies →
You were both right, and it all depends on your definition of productivity. In economic terms, I believe the standard way of looking at this is to consider how much economic activity resulted.
For lawyers, most of them sell hours. The more hours they bill, the more productive they are.
Most businesses who hire programmers do not make their money by billing programmer hours. So that metric wouldn't work. Lines of code seems reasonable until you think it through. Honestly, I don't know that anyone has come up with a good solution for measuring programmer productivity.
But lawyers? They're in the business of selling time in 15 minute increments. Their productivity is simple to measure in this respect.
Well this gets back to the question of the output. For the law firm itself, this might be a valid approach. For the client, though, this is a terrible way to measure productivity. At a certain point hours may even be inversely proportional to productivity (assuming the client's output is desirable legal outcomes). This works very much the same as software engineering, except that usually (ignoring the case of consulting firms) all of the work is done in-house.
But the question in the grandparent post was about capacity - and in this regard, if some customer needs "X much" of legal services performed, then it's reasonable to state that if your legal team that can devote twice as much hours to that customer, it has twice as much capacity.
Completely agreed.
What about the general counsel of an in-house legal team?
Surely they care about their underlying activities and not just number of legal hours worked, right?
I have to believe there is a legal team somewhere in the world that is measured on productivity beyond just "number of billable hours" generated.
I imagine an in-house team is being paid a salary or a retainer, in which case their work hours are moot. (Not a lawyer, so I could be wrong.)
I think the problem is the word "productivity". When we say that word, we're implicitly suggested there's a simple integer or decimal that can capture whether a person's wages are money well spent or not. For most professions, programming included, I am highly skeptical of the existence or even potential for such a number.
6 replies →
> You were both right, and it all depends on your definition of productivity
Sure, but at some point, you can pick a definition that is so far removed from what was intended, that this exercise is utterly meaningless.
> Sure, but at some point, you can pick a definition that is so far removed from what was intended, that this exercise is utterly meaningless.
You could say that anytime there's any lack of clarity about what is meant by any given term,
With "productivity", you could reasonably mean any number of things.
It's not like someone said "pizza" and I said "that depends on what you mean by pizza". You could say that (is a calzone a pizza?), but it wouldn't be reasonable to do so.
In the case of productivity, I think it's reasonable to clarify what is meant.
P.S. Was your use of "utterly meaningless" an intentional pun?
> I don't know that anyone has come up with a good solution for measuring programmer productivity.
Well, the only people who could meaningfully search for such solution - programmers - have all the incentive in the world not to find it. Not very surprising they didn't find it yet (and won't ever.)
This may explain why lawyers are motivated to stir things up, rather than settle them. They're motivated by the wrong metric.
As an Engineer, I sell my time by the hour too. No different than the lawyer. Yet I try to finish things efficiently. Huh.
> As an Engineer, I sell my time by the hour too.
As in you literally bill for hours, and the more hours you work, the more you get paid?
Most programmers that I know (which is obviously not a great metric) either get paid a salary (which is divorced from actual hours worked) or they get paid by the hour but have no say over how many hours they will. In both cases, time is independent from productivity. Therefor, there's no harm (and really only benefits) to coding efficiently.
But if you a) control how much time you work (like lawyers do, to an extent), and b) get paid for your time, then yes the incentives are setup to encourage you to be inefficient. Completely agreed.
4 replies →
Paradoxically, if you optimize for billable hours on your first job, you might never get to that highly-compensated 100th job. A selection effect of sorts.
I'm sure lawyers think they try and settle things efficiently as well.
The software engineering profession is riff with wasted work: rewrites, new bullshit services and tech, insanely complex clustering and cloud deployments, etc.
> This may explain why lawyers are motivated to stir things up, rather than settle them. They're motivated by the wrong metric
Most lawyers I know have many clients and are swamped with work. They have little incentive to "stir things up". It's similar with accountants and plumbers in my city. They aren't trying to make more work for themselves because they already have their hands full.
But in regional markets where supply isn't so constrained relative to demand then, sure, there's an incentive to make-work once you've wrangled a client, just as with any other profession.
The underlying question is: what is the firm's utility from the lawyers? If the firm is not doing anything themselves but outsourcing the team, person-hours is the correct answer.
If the team is doing work for the firm, but you don't want to complicate the model, you can stick a labor-enhancing constant (to allow for heterogeneity between workers) and use "work" as a unit. Sure this model is wrong, but all models are wrong. We're just trying to create some useful ones.
See my response above, but this suffers the same problem. Economics people talk about "utility" as a single number, when in reality its a multi-dimensional (perhaps infinitely dimensional) number. Because it depends on a multitude of independent variables (age, health, experience, intelligence, expertise, efficiency, relationships, persausion / charisma, etc.), it can never be simplified. This also makes it impossible to compare two utility values, because there's no way to strictly order variables in many dimensions (without arbitrary reduction in complexity that loses information like calling "cost" or "hours worked" the primary axis and sorting on that).
You're right that the map is not the territory, but it doesn't change the fact that you need a map to navigate. No science is comprehensive enough to fully model the territory (even physics)
This makes me think of something that's intrigued me for a while: what's the productivity of a yacht racer? The better they are, the less time they spend actually racing in competitions.
Doesn't this mean that by your professor's metric they are becoming less productive?
I would say that a yacht racer, or other athlete's, "output" is their wins/ranking in competitions. There are devils in the details of how you assign a simple number to that, and Goodhart's law is always lying in wait, but that seems to be the right kind of thing to measure.
More cynically, you could measure a racer by the amount of revenue generated by sponsorships, ad placement on the yacht hull, endorsement fees/kickbacks, etc.. If you have two equally competitive racers, but one is more mediagenic, perhaps that one has higher "productivity"? If a racer often loses, but does so in engaging, nailbiting ways that create a following, perhaps that one is "productive"? A wrestling "heel" may lose their bouts but be a successful character, say.
> There are devils in the details of how you assign a simple number to that, and Goodhart's law is always lying in wait
Yup, they could start sabotaging their competition or bribing judges to disqualify other competitors, etc.
You would be interested in Sabermetrics, which is the use of statistics to quantify the contribution of baseball players to the outcomes of the games they play in. Yacht racing is also a competitive team activity and you could define the productivity of a racer in terms of how much that person's efforts contributed to the position or time their yacht finished a race in. It's a relative measure and would have to be defined relative to other racers or to a fictive 'standard' racer.
For something high risk like that it might be you only “produced” something if you placed.
I was on the in-house legal team at a manufacturing company. You described my life. The push was always to reduce what we did down to something measurable. It's understandable, because they're manufacturing people and it's how they think, but that's very difficult to do for legal work.
I always argued that the value we provided may not be realized for years, when some clause we put in a contract prevented us from getting sued. We may not even know it had happened! Ultimately, for non-litigation legal work, your value as a lawyer is often in preventing bad things from happening. How do you measure that?
The machines are from mediocristan while the lawyers are from extremistan[1][2]
So yeah, your professor compared apples and oranges.
1. https://people.wou.edu/~shawd/mediocristan--extremistan.html
2. https://kmci.org/alllifeisproblemsolving/archives/black-swan...
Your professor clearly knows nothing about legal work. Don't tell him that though, you'll just piss him off.
Thank him for his "wisdom" but make sure you do it in a convincing way. Finish the class, get your A, move on with your life. You don't need him to acknowledge he's wrong, you just need him to give you good marks.
No, I think he is right. Assuming the aim is to optimise for profitability, the machines output items are chargeable but it is a lawyers hours that are chargeable - so optimise (and measure) hours!
(There are obviously some “jobs” a lawyer does that tend be charged at a fixed rate such as conveyancing, which you would want to optimise for throughout)
> No, I think he is right. Assuming the aim is to optimise for profitability, the machines output items are chargeable but it is a lawyers hours that are chargeable
I have updated the post to expand on your observation.
As someone in academia myself I think this is plain bad teaching. If you make students guess what specific strange idea you had in your brain, instead of having them using theirs, you are not only wasting your time, you are wasting theirs as well.
If you want everybody to guess what you are thinking, become a quizmaster.
It's not the generalization pitfall, but a completely wrong metric. "Hours spent" is an expense, not a revenue. If you were measuring machines by KWh spent, you'd never have an insight to optimize that number down instead of up.
Lawyers are often billed by hour, but that's not what they are paid for. You will quickly lose customers if you optimize for that metric. Instead, if you figure out how to spend less hours for the same results, you may charge even more per hour, because you are saving your customers' time, plus handle more customers over a given time frame.
But as with lawyers, programmers, managers and all other non-machine-like workers produce "customer happiness" that's not easily measured, other than at the level of overall competitiveness of the firm.
The capacity of a legal team approaches infinity as the ambulance they are chasing approaches the speed of light.
Really curious what school is this?
> Edit: Given that this answer has generated some discussion I feel the need to expand on it. The legal team was not expected to sell their services "by the hour". In fact, any discussion about how their services were sold was shut down by the professor. From his point of view, the lawyers were machines and he was asking the question "how much can this machine produce?"
Pricing the value of in-house counsel is an interesting problem in itself (because they typically don't bill by the hour). One could use an insurance policy pricing model (the worst that could happen is a very costly loss in a lawsuit) to determine what's the counsel protecting the company from.
> Really curious what school is this?
My local university. If you want the specific school, my handle is associated with my real-world identity. It won't be hard to figure out.
> Pricing the value of in-house counsel is an interesting problem in itself (because they typically don't bill by the hour).
Absolutely!
> One could use an insurance policy pricing model (the worst that could happen is a very costly loss in a lawsuit) to determine what's the counsel protecting the company from.
Also true. The lack of cost analysis in my classes worries me very much.
> Also true. The lack of cost analysis cost in my classes worries me very much.
Time to transfer somewhere else?
1 reply →
It seems to me that one measure of productivity in a law firm is billable hours, however in the case of a litigator - it’s also successful case outcome. Again, it’s billable hours towards an outcome. So I can understand this heuristic based approach, on the other hand it seems like you’re tending towards the quantitative side?
Generalizations or rules of thumb can actually outperform complex quantitative approaches to decision making. Look at the 1/N heuristic for portfolio management for example.
Anyhow just my opinion - something for your curious mind to consider !
Wow. Even with such a basic analogy, there are still more useful metrics you could pick... contracts closed, cases won, client files completed, cigars smoked.
Yeh you cant really apply production engineering mathematical analysis to this sort of work.
You could to say production line workers using piece rate system but a a legal team consists of many different people with different skills and who perform many different tasks.
Turns out, machines also have some sort of duty cycle. Most mechanical contraptions can run faster than spec, at the expense of wear, heat, jams, and ruined parts. So you can't even measure machines in widgets/hour without more info.
Out of interest - how would one judge a professor then?!
Classes taught? Students taught? Students getting degrees (undergrad/Masters/PhDs)? Research grants won? Nobel prizes won?
Well, lawyers are famously expensive. And time is money. So presumably a sufficiently expensive legal team will actually produce hours, and the measurement is perfectly reasonable.
A machine generates income by the number of widgets (value $x) produced per day.
A legal team generates income by billable hours (value $y) worked per day.
This sounds like the beginning of one of those stories about the Soviet economy not working.
This way of thinking has originated with scientific management, also known as Taylorism.
how many pieces a machine can manufacture = output
working hours = input
Would the professor be okay to pay me for the time I spend reading web? I mean, from his perspective, it is the time I spend that is important, not what I do.
There is certain need for people to be able to simplify problems in terms that they can understand an manage.
This is necessary, because humans have very limited capacity to understand world around them and otherwise it would not be possible to make informed decisions, as gathering all relevant information would necessarily take practically infinite amount of time.
From that point of view I understand people like your professor is mostly result of bias also called Dunning Kruger effect. This is basically lack of education in a given area. You need at least some knowledge in an area to be able to appreciate complexity and unknown unknowns.
If you don't want to be that guy, the best medicine is first to learn to be self aware, second to be aware of various biases (including Dunning Kruger effect) that you are subject to and third to get some knowledge/experience in an area you are trying to make decisions in.
The real answer is revenue generated/hour