Comment by mrandish

The video tech one of my startups made was used to do "On-Set Video Playback" exactly like this for a bunch of movies and TV shows. We didn't make the product for that purpose and only learned this was a thing when a playback company contacted us asking for a change in the firmware to enable external synchronization to a 48fps source clock.

Since it wasn't a difficult change and the use case was neat, we made a custom version for them. That's how I got to know some of the people who do this work and even got to visit some movie and TV sets. So, based on that, here are some insights relating to TFA.

The first thing to know is on-set video playback for film and TV production is a specialized service because it can require arcane knowledge to properly interface various video displays with 24 fps film cameras. This used to involve a lot of custom modified displays, hand-built or modified interface boxes and various arcane cables/adapters but it's gotten somewhat easier with the advent of variable frame rate displays and GPUs.

Since time is money on-set, productions just hire this out on a project basis to one of a few specialized firms in Hollywood. As it's a niche thing, these firms are usually just a handful of knowledgeable A/V guys who've acquired a variety of customized interfacing gear over many projects along with different types of displays and have practical experience in making it work quickly and reliably with various cinema cameras.

There are two main parts to a project: 1) Making the screen look right on camera, and 2) Getting the right images on the screen at the right time.

1) Making it look right breaks down into three parts: A) Getting the source on the screen, B) Synchdronizing the source with the cinema camera so there's no screen flicker or rolling, and C) Adjusting the screen's brightness, contrast, gamma, saturation, etc to 'read' well on-camera along with minimizing any light glare and reflections. Depending on the ambient scene brightness and the camera's shutter speed, iris, etc these adjustments can sometimes be more extreme than the display's native adjustments allow. The playback team has tools for this including stand-alone signal processing boxes that range from simple knob adjustments all the way to 3D LUTs that can remap any pixel value to any other. They might also use old-school tricks like covering the screen with neutral density film similar to window tinting.

2) Getting the right images on the screen at the right time breaks into two parts. A) Creating the source imagery, and B) Triggering the playback on cue. For most projects the production will just have the playback team create the source imagery. The only exceptions tend to be shows where on-screen shots are frequent and integral to the story. In those cases, the on-screen content will usually be the responsibility of a designer working under the production's art director and the playback team's job will be getting it on the screen. The on-screen imagery for Tron: Legacy is pretty minimal and contained to a few scenes so it was probably designed and created by the playback team as a per-hour line item on their project bid.

In those cases, the playback team will receive the relevant script pages and a few storyboards. Based on those and perhaps a phone call with a line producer or AD, they'll make some comp stills and send them over for pre-approval. Once approved, they'll do the actual source content and send clips for approval. Depending on the production, this may just be signed off by the line-producer or an AD but, in other cases, the director will want to at least see it. If the playback team is providing the display they'll send over photos so the production designer and set dressers know what will be coming.

On the shoot day the team gets there early and coordinates with the set dressers to get the display in place, then electricians for power and finally the DP and camera crew to test sync, brightness, etc. The type of content going on-screen and how interactive it needs to be will determine if they've already pre-recorded the source and just play it back on-set, use an interactive video source to sequence or animate the content or actually use a "live" source. They tend to use whatever software can do the job, is easy, reliable and flexible. This can range from as basic as Powerpoint to more sophisticated presentation tools to scripting tools and, when necessary, even custom command line apps they've cobbled together over the years. For video clips, they'll record what they can and then modify or composite elements together with standard tools like Photoshop and AfterEffects.

In the case of Tron: Legacy, it's hard to tell if they pre-recorded the sequences from a Linux computer and triggered playback in steps on-set or used a live Linux computer since, done properly, they can look essentially identical. There's a strong preference to pre-record everything and sequence or animate it for playback but sometimes that's not possible due to keyboard/mouse activity. The reason is that any live computer might crash or respond at slightly different speeds from take to take complicating editing, especially when there's repeated re-takes or on-set heat from lights, etc. Power on set can also sometimes be from generators and very dirty, even on studio sound stages and back lots. Any delay due to playback not being ready, or worse killing a take, can have severe reputational consequences.

In cases where a live computer is unavoidable, the strong preference is for someone from the playback team to do all the operation off-camera while the actor uses a dummy keyboard and mouse. This usually works fine because there's rarely a need to hold on an all-in-one medium shot showing the screen and keyboard/mouse during actual interaction. On-screen interaction is almost always shown in an insert close-up of the just the screen and bezel. Either way, the sound of keypresses and clicks are dubbed in later by foley artists just like footsteps and doors closing, etc. The absolute nightmare scenario for any playback artist is when a live computer is required that an actor actually operates. It never goes well. Not because the actors are dummies but because they have to focus first on hitting their marks, saying their lines and actually, you know, acting. Under those conditions, typing the exact same techno-gibberish in exactly the same way, with the same timing, while repeating lines over 30 takes would likely give most of us trouble too.