Comment by mrandish
1 day ago
To me the interesting question about a job like this is "How can you tell if you're doing it well?" It involves such high-stakes, high-uncertainty and highly variability that it has to be nearly impossible to know. I mean you're predicting distant outcomes from creative pursuits which must first survive a gauntlet of wicked complexity and randomness.
Only a few percent of your judgements are ever tested (by surviving being optioned, produced and released) and, of the ones that are, at best you only get a small sampling of false positives over a sea of potential false negatives. I imagine he's incredibly interested in the fate of any titles he didn't recommend which end up being produced (perhaps by another studio). Having filled a similar role in a different industry with similar high-stakes 'unknowables', I thought a lot about this. It was pretty obvious what practically mattered was how much my output "felt right" to downstream decision-makers vs actually being right.
While my stakeholders were quite happy with my work, actually targeting such ephemeral and uncorrelated feedback felt unproductive and dumb. Eventually, I settled on making the evaluation process fully transparent and consistent. I ensured all objective criteria were documented and each subjective judgement had clear confidence intervals. This was more challenging than it sounds. In the end, it was still hard to know if I was really improving year to year. For that, I still had to rely on my own, mostly subjective, self-assessment but at least I had some objective tracking data to calibrate on. That at least helped me feel like I was executing with diligence and integrity. It also increased my confidence no one else in the industry was doing it any better.
I suppose it's mostly about clear communication. If you are reviewing books for a movie, the job does not seem to be "will this become a successful adaptation?" so much as "what is the strongest movie latent in this book, and how do I communicate that to the people who can act on it?". Those people would then try to imagine how that script would be portrayed on the screen, what the acting would be like, what the scenes would look like and where the material would break under translation. Given you all have some understanding about what makes a great script and what makes a great movie, you make a pipeline that has multiple experts controlling different aspects of the transformation and generating the strongest final product based on the original book, which, from book adaptations I saw, most of the time is just a thin seed rather than a forced blueprint.
If it later turns out the material was not adaptable in the way you thought, I'd imagine that is not just a binary miss, since the reader, producer, writer and executives can discuss and try to see where their judgement failed and what went wrong. I get that the hard feedback is sparse, but it doesn't have to be researche-grade measurements as much as it has to be good judgement, constant reality checks, even if just from proxies, and good taste. I'd be curious if this sounds close to what you were doing.
PS: there's this Dalton + Michael YC advice for startups which seems relevant: when outcomes are highly uncertain, you can't judge the result-only whether you acted logically, ethically and treated people well along the way.. (https://www.youtube.com/watch?v=XgcdvIj5I-k)
> I'd be curious if this sounds to what you were doing.
Yes, there were definitely aspects of teasing out how much a poor outcome was related to the initial call and how much was in the execution. The execution breaks down into the early definition and scoping, which can be part of the initial decision, and then all the downstream decisions.
Assessing all this is inevitably clouded by confounding factors like unforeseeable external factors and even human variability. That's pretty easy if a film you green-lit hits screens the week the COVID shutdown happens but in my domain post mortems more often came down to non-binary judgements like the degree to which an unlikely outcome was unforeseeable. In my post mortems I focused a lot on how well I modeled and surfaced the more relevant risks. Unlike an IPO prospectus, just listing all the possibilities isn't enough. The value is in identifying and surfacing the factors below "obvious" but above "very unlikely", then plugging them into scenarios and playing them forward.
To my company's credit, it wasn't uncommon that during a detailed, often painful, post mortem of a horrendous failure, my work would be singled out for being outstanding. Why? Because I'd surfaced and appropriately weighed the risks, including the ones that sank the project. When the calls weren't obvious (and many aren't), my job was to ensure the stakeholders had situational awareness, including relevant risk factors. As I got better at modeling the intractably 'wicked' nature of my role, I included fewer "Go / No Go" calls and more provisional judgements like "If we choose Go, we're betting we can execute a combination of these factors better than their median probability and that the following less-likely externalities won't occur."
Conversely, I found it frustrating that in the celebratory review of one especially huge win, which I'd endorsed with as big a "Go" as I'd ever give, my esteemed stakeholders failed to notice that my "Go" was right but for the wrong reasons. My private self-assessment was quite brutal because the excellent outcome wasn't due to an unforeseeably rare "Golden Goose" (the opposite of a Black Swan). It was a less likely factor that ended up being critical for reasons I'd seen and assessed but weighed incorrectly. The silver-lining was this was one of those rare times that exactly how I'd blown the eval of that factor was discernible. It ended up being one of the single most instructive events of my career because in parsing how I'd failed, I uncovered a process error which significantly leveled up my skills. It's like finding a math mistake wasn't a calculation error but an error in the formula or, in coding, a mythical compiler bug.
It's ironic the legendary 'big win' I was probably most known for was actually one of my biggest preventable errors. Once I'd had time to really study it, I did an internal talk on the whole episode, labeling it as my best-ever 'teachable moment.' I've heard the video of that talk is still regularly shown.
Please tell more!
Not sure what you want to know. To me, the interesting aspect is the unique challenges of making high-stakes decisions in ultra high-uncertainty situations where you never receive any feedback signal on most of your calls. And the little you do get is greatly delayed or buried in ambient noise. Yet, due to the size of the infrequent prize, the game can still be worth playing... if you can find and hold a slight edge.
There aren't a lot of professional careers which require skill and years of experience yet are flooded with so many false positives, false negatives, and "we'll never even knows". Domains where playing at a world-class level only takes being right 5% of the time - are just hard to reason about. It can feel like a sadistic casino where 97% of blackjack hands have no clear winner, yet sometimes hitting on 20 is the optimal call. But other times standing on 12 is the best strategy. But it's not entirely random. There are real signals. It's just hard to identify which are real, which are red noise and which are just mapped backward.
With so many false positives and false negatives it's easy to end up chasing black swans (random outlier events). Or to just settle for trying to please your boss, whose own track record is probably closer to astrology than strategy. My best meta-takeaway is to focus on thoroughly mapping the decision space, carefully track and map all the signals, even build a taxonomy of signal types if you can. Then relentlessly optimize the decision making process over the actual outcomes. Why? Because in such 'wicked' domains, sometimes the wrong decision process can still score winning results. And other times, an optimal decision process can yield a string of losses. Your job depends on figuring which is which before it's obvious to other expert players.
As for the book reader in the TFA, I suspect a lot of his value isn't in his a binary "go / no go" call. It's accurately mapping the strengths and weaknesses of a particular title and suggesting where to place it in the studio's current decision matrix. And, on a good day, maybe spotting non-obvious ways the property could be developed.
Sounds to some extent like advertising and marketing in a market like India which is still predominantly offline and driven by visibility.
1 reply →