Back in 2016, Jay Redd explained in detail the work of Sony Pictures Imageworks on Alice Through the Looking Glass. He then worked at Riot Games to directed a short film. Today he returns to talk about his work on the second season of the Apple TV+ series, For All Mankind.
How did you get involved on this series?
I had just finished directing a short film at Riot Games in Los Angeles, and was taking a little break from making feature films. I got a call from my long-time friend and producer Deborah Giarratana, letting me know that the show was looking for a VFX Supervisor to take over production and post production. I loved Ronald D. Moore’s Battlestar Galactica series, and knowing that the show was about the space program in its early days, and that it was an alternate history, was shooting in Los Angeles where I live, back at Sony Pictures Studios where I spent much my career making movie with Sony Imageworks – well it all just was like a dream come true. I’ve been a life-long space nerd, and an amateur astronomer, so this just felt like the perfect fit!
In the beginning of 2019, I had an in-person conversation with Ron and the other executive producers on the show, and we all clicked really well, shared the same aesthetic vision for the show, and the very next day, I was on set, shooting big VFX sequences! It had been shooting for a couple months already, so I took things over, made some changes, and off we went, the rest is history, as they say.
How was the collaboration with the showrunners and directors?
Honestly, working with Ron Moore, Ben Nedivi, Matt Wolpert, and Maril Davis and the rest of the team has been wonderful. The Executive Producers, writers, and Directors of this show respect the VFX process and what we bring to the table. There is a deep trust and understanding that every team will bring their best foot forward. The writers often times come to us early in the process and we discuss what’s expensive, not so expensive, what’s hard, etc. Ron and rest of the EPs trust me and my team to conceptualize and complete complex visual effects that stand up to feature-quality work, on a TV schedule and budget. It’s a really collaborative and supportive relationship.
What was their approach and expectations about the visual effects?
In my first interview with Ron, I asked “Is this show photo-real and grounded in reality?” Ron answered, “Absolutely 100 percent.” That was exactly what I needed to hear, and my hope came true in that instant. The approach to the show was an ‘alternative history of the space race’ – meaning there would be parts that were grounded in real history, and other parts that were fictional, often blurring the lines in unexpected ways. We were going to be using historical footage, news footage, stock footage, fully CG creations, real and fictional space crafts, landing on the moon, etc. Stock and historical footage would be manipulated, location photography would need to be changed to reflect the period, the late 1960s and 70s. In Season 2, we would jump 10 years into the early 80s, so again, the look of footage would change, and we needed to keep up with that.
The approach on our show has always been ‘photo-real’ and also ‘of the time’. There would be different aspect ratios from historical footage, but the show was shot in 16:9. We would sometimes need to degrade our modern footage to fit in with historical pieces, and even put our cast into historical settings, often mingling with political figures and cultural icons of the past. The visual effects needed to be top-shelf, feature quality, photo-real, 4k, and often cut back-to-back with archival footage.
How did you organize the work with your VFX Producer?
Barbara Genicoff, who has great experience in producing VFX for other TV shows and brings a strong know-how in being able to manage hundreds and hundreds of shots, and dozens of sequences, across multiple VFX companies and teams. We both know how to manage thousands of shots on big shows, so we work really well together. I couldn’t do my job without her, and without the rest of our teams coordinators, Josh Fritchie and Niña Pastores. I mention their names because support teams don’t get enough credit!
Barbara and I look at the type of work, per episode, and we break things down by style of work, repeated assets and work across multiple episodes, one-offs, 2D work, touch-ups, etc. Since this is a relatively fast TV post-production schedule, we have to work on a lot things in parallel. Most of my career is in feature films, and there is a bit more of a linear process there. On TV, things move much much faster, so we spread the work out more than I might normally on a film.
For instance, most of our Moon work was done with Method Studios Montreal and Method Vancouver, with Zoic Studios doing an isolated lunar sequence for Season 2. I don’t want to spend a lot of time and effort getting multiple companies to match the look of things, so we try to keep the types of work assigned to companies that do the best at what they do.
We look at each studio from a skillset, capability, past work styles, location, cost, team and talent – not in that order necessarily. We try to approach the show holistically – balancing quality and schedule, time and money, the usual. Always the first and most important thing – QUALITY.
What are the main changes you have made since the first season?
Since there is more of everything in Season 2, I made some changes on how we shoot Lunar landscapes, by building a bigger plot of regolith (lunar soil) on two different locations. One location, a smaller plot of lunar soil, was on the stages at Sony Studios. The other, much larger plot, was built down in an old Boeing jetliner hanger right next to the Long Beach Airport. This was a HUGE warehouse where 747s and other planes were serviced. This giant space allowed our DPs to put up a 100k ‘sunlight’ 80ft in the air, and also creating a giant 360-degree black-cloth enclosed space. We don’t shoot with green or blue screen for the lunar work, as the spill is too much, and the Moon has no atmosphere, so the sky is always black.
We had 5 times more astronauts and many more cosmonauts in Season 2, so we had a lot more elaborate visor work to deal with as well. Workflow-wise, we made some efficiencies in the way we manage shots and assets across multiple partner studios.
The biggest unexpected thing that hit us(and the world), was COVID-19. We had finished filming episodes 1-8, and were just about to start filming the biggest VFX episodes 9 and 10. Then the lockdown came. Everything stopped for a bit, while everyone figured out how to move their artists to working from home. Different countries were dealing with the pandemic in different ways, so it was a ripple-effect of slow-downs, and sometimes complete stopping. But this only lasted a short time, ultimately. We still had to keep working on episodes 1-8, so we could try to keep on a delivery path as much as possible. All of our VFX partner companies did a pretty amazing job getting up and running again, so we only lost a few weeks in total.
In late summer/early fall, after much debate, we were one of the first shows to try filming under the new, very strict COVID safety protocols. Mask wearing, constant COVID tests for cast and crew, physical and social distancing for catering, no eating on set, shorter shooting hours, dividing the crew up into two groups – those that had access to cast, and those that didn’t. Sony/Apple and our show did an amazing job under these new, strange conditions. The crew dedicated themselves to staying safe and focused, and we were able to complete the two episodes’ filming in early fall. We were then full-on into post throughout the remainder of the fall of 2020 and into early 2021, before Season 2 started airing on February 19th.
Can you explain in detail about the creation of the Moon Base?
The main designs came from Art Department, headed up by our Production Designer Dan Bishop. He and his team worked with the writers in designing an expanded base that built upon the Season 1 base that we showed being auto-piloted to land on the surface of the Moon. Season 2 picks up about 10 years later, and the space program didn’t slow down in any way – rather it picked up, so the Base needed to show how it could accommodate 15+ astronauts. The discovery of ice in Season 1, led to a major expansion of the mining operation in Shackleton crater as well.
We needed to show that there was a well laid-out plan that could accommodate Operations for mining, experiments, LSAM landing pads, transport, rovers, lighting at night (when the sun is gone for two weeks), communications, crew quarters, etc. Dan and his team designed a pretty amazing, and functional expansion, both inside and out – multiple modules. We ended up building a huge physical set on the Sony lot that had OpsCom (Operation Command), extensive hallways, sleeping quarters, a galley, labs, medical bay, Captains quarters, service hubs, etc. The physical sets are really incredibly detailed and appointed.
For the exteriors, those would all be done in CG. We did build physical doors on both of the airlock chambers on set, so we could do some cool hookups from outside to inside, and vice-versa. We worked with Method (headed up by Mat Krentz in Vancouver, and Philippe Desiront in Montreal) extensively on the digital build of Jamestown, starting from concept work from the Production Design team. As the show progressed, and new shots were designed that weren’t originally planned, we extrapolated from the initial designs and followed style and structure in adding details where we didn’t have artwork. For the big opening shot, pushing into the Moon and to the Jamestown base at night, I worked with The Third Floor, and Previs Supervisor Barry Howell. He and I would sit down and discuss ideas for numerous different ‘establishing shots’ for the entire Season 2. We had to change things up as episodes progressed – moving rovers around, adding more rover tracks, keeping continuity on some things, and changing others. Barry’s got a great film movement sensibility, so it was a pleasure developing a bit of a shorthand with him on this season.
Look-wise, Jamestown also needed to appear new and sleek, but also utilitarian and used. The Moon is a dirty place. Regolith (lunar soil) gets everywhere, and sticks to everything, so I worked with Method and their texture and shader teams to make sure there was enough ‘grit’ in places on the Base itself. There were many types of metals, alloys, kevlar wrapped modular units, wiring, radiation deflection material, radar dishes, antenna’s, etc. For the surrounding lunar environment, we did a few cineSync calls and I would annotate on images, where we needed to add Rover tracks, boot paths, piles of crates, gear, lights, cables, and space gack, etc… to make it look like it was really lived in and a busy, functioning mini-colony. The Method teams did a terrific job of very high levels of detail and surfacing, along with some brilliant optical treatments – blooms, specs, color bleeds, lens distortions, even matching to our physical lenses and filters.
Can you tell us more about the various Moon environments?
We had many more environments this time around, so we built on what we did for Season 1, as far as process goes. Again, initially working The Third Floor, we would take early designs from Production and start building different topologies of the various landscapes – some from just concept work, some from elevations and blueprints. We would then visualize different sequences, and start to figure out which areas connected, and which areas were one-offs and could stand alone. The Jamestown Base expanded a whole lot, so we needed to make sure we could connect the Base to the LSAM landing pads, the edge of Shackleton Crater, the expanded mining operations, the solar panel array, etc. This was a huge terrain. We also had the big claim site, some journeys through mountains, cliffs faces, LSAM/Marines flying through flat-lands and the in to deep Canyon terrains, only to land in a surprise location behind the claim site. These were treated as separate environments.
There was also an extensive sequence in a ‘Rock Forest’ leading up to an encounter with the Soviet Cosmonauts. It took a lot of back and forth with the Method teams, on getting everything to connect where it needed to. They have some great tools for terrain building, and also a super strong matte-painting team, so we could use projections often where needed.
Because of schedule, Zoic took on one sequence in that ‘Rock Forest’ that had it’s own terrain style, so it worked great to split up some of the Moon work this season.
Ghost also did an isolated shot in Episode 10 of the lunar surface with crashing debris, and that turned out great.
Getting the Moon to look right, and not just a monotonous black-and-white landscape, takes a LOT of work, patience, and artistry. All the teams were instrumental in getting all the details of regolith, rocks, craters, boulders, boot prints, etc. I would often get very specific with placement of boulders, small hills, distant mountains – all in an effort to make each shot look well composed, with interesting and variety. Big kudos to the environments teams and supervisors for making it all look fantastic.
How did you create the digital doubles and the various Moon vehicles?
These were created with pretty straightforward LiDAR and 3D scanning, then built at our respective VFX partners. Gentle Giant did the scanning for us. We had to pay attention to different markings on different astronaut and cosmonaut suits, name-tags, military rankings, etc. The Rovers needed to work from wide angles to close-ups, and often cut against live-actions shots of the same, so lots of attention to details there.
Can you tell us more about the reflective visor challenges?
The visors are one of the most prevalent ‘invisible effects’ in the show. Our audiences won’t realize just how much work goes into making them look good, and that’s okay! We don’t want anyone to notice, but they are indeed, a huge undertaking, and are very tweaky and nuanced. Season 2’s visors are essentially almost 360-degree mirrors – they see EVERYTHING – like a fish-eye. Because of this immense area of reflection, this forced us to shoot often times with no visors at all, due to shooting on stages with gear, lights, crew, etc. I came up with a basic set of ‘rules’ that helped everyone on set understand which visors we use for which kind of shots, or no visors at all, or a combination of things. In the show, we often use the visors as a story-telling device, frequently showing action directly in the reflections, so we had to be very careful about what we shot practically, and what we didn’t. We asked the crew to dress in black so as to not be seen, but this was kind of moot if we were shooting on the lunar surface – crew would just show up as black blobs against the surface. Those shots, when they worked with the visors down, had to be repaired.
With the clear visors, trying to get rid of big camera rigs and cables and ceilings and lights would be very problematic, especially seeing the actors through the clear visors, so we would often shoot with nothing, and then put the visors and all the reflections in, in CG. The gold visors were a whole different thing, in that you didn’t see through them, and they acted more like perfect mirrors, reflecting literally everything in 270 degrees. We often shot WITH the gold visors down, since we would either completely replace them, or we would paint out rigs/sets, and do lunar surface reflections, sun speculars, and even other astronauts as CG additions.
These shots went through elaborate 3D tracking phases of the scanned helmets. There are nuances like flexing and warping of visors in real-life, and that had to be dealt with my the amazing teams at Method, Refuge, Zoic, etc.
For wide and medium shots, we would make the call on set, to either shot with or without the clear visors. Sometimes we could get away filming with them down, saving a bunch of CG work.
Todd Sheridan Perry, my on-set supervisor for Seasons 2, got super proficient at making the calls on visor conditions throughout the shoot. He was also a VFX Supervisor on Season 1, at Method studios in Montreal, so he knew the process on the artist side. I brought Todd on to help me out with on-set work and supervision, when I can’t be there, due to me prepping future episodes. He brings with him a ton of experience both on set and on the box, and is instrumental in making the show.
How does the lighting on the Moon affects your work?
The Moon has no atmosphere, and therefore has a black sky – no reflected or refracted light, as there is no ‘air’. All you get is the Sun, or if you’re on the side of no sun, just a bit of starlight and maybe a little Earth-shine. There’s no Moon in the sky on the Moon, so you don’t get that ‘glow’.
For the Season 2 Episode 1 opener, we start on the moon in complete ‘night’. Jamestown Base is located on the South Pole, so much like here on Earth, the sun disappears for long periods during different seasons – often being down or up for weeks at a time. At Shackleton Crater (a real crater on the moon, by the way), the sun comes up for two weeks, and sets for two weeks. Ron wanted to show the majesty and magic of this recurring event, and how it is different than that on Earth. Here, we get to enjoy long, colorful and beautiful sunrises and sunsets – it’s one of the great joys of being human. On the Moon, there is no atmosphere or clouds, so there is no refraction of light creating dazzling colors and intensity changes. On the Moon, it’s almost like a light switch! Once the sun peeks over the horizon, you’re being blasted with really intense light, and when it’s shining down fully, it is literally dangerous to be exposed to it for too long.
For the night look, we had a challenge, in that we still needed to ‘see’ our astronauts and cast, but still be dark, and not your typical ‘move night’, blasted with cyan blue. I worked closely with Steven McNutt and Ross Berryman, our DPs, on defining a ‘look’ and base-level exposure so that later, I could work with my VFX teams in doing set extensions and CG environments ‘in the darkness’. We need some flexibility in dynamic range so we could push or pull exposure as necessary. Steven and Ross came up with a physical lighting rig that was essentially helium-filled balloons, with lightweight LED light panels embedded in white silk elongated horizontal columns – almost like a giant life-raft. They floated this above the stage and lunar plot, and it gave a beautiful diffuse and ambient feel, without much directional shadow. If we had any directional shadows, the gag would be gone. We needed even, flat-ish lighting, but still be able to make out shapes and contours of our cast and surrounding ground and objects.
For the sunrise itself, our rigging grips, led by Curt Griebel and John Warner, created an elaborate ‘eyebrow/eyelid’ rig consisting of hundreds of yards of black cloth on ropes and pulleys in 360 degrees, that allowed the sun (100k light), to be revealed on cue, and repeatable. It needed to feel like the sun was blowing you away, and the enormous 360 black curtain warehouse they created was really effective. Shooting in this giant black cave, helped us have accurate lighting on all the astronauts, and often allowed us to shoot with many visors intact on wide shots.
When you watch the sunrise scene, if you look in the visors, you’ll notice that they reflect the ridge of Shackleton crater, the astronauts standing next to each other, and the extensive lunar environments, and of course the sun itself – all added by Method in CG.
Shooting with a light source that bright made all the dust in the air show up on set, and that was a no-no, because again, the moon as no atmosphere! The teams at Method worked really hard in managing that. We deliberately shot with diffusion on the lens, so that it looked more beautiful, and we even embraced the atmosphere on the set sometimes, even though we know that it wasn’t technically right. Sometime you need to go for the poetry of an image, and not the technicality of it. The sun was going to be a major character in this Season, so it was important to really design the look of it and keep it consistent across the episodes, and VFX partners.
I take an ‘optical’ approach to the show, so we did a lot of work on keeping things feeling analogue. Lens flares and aberrations are very important, but I never let anything get ON the lens. We don’t want to take viewers out of the experience. My philosophy is that if you have dust or liquid or whatever on the lens, you’ve just said ‘look at me, i’m a camera!’. I’m not a big fan of that gag.
Can you elaborates about the terrifying and yet beautiful solar storm on the Moon?
This is was a big and exciting undertaking. In the script, Ron wrote: “…the surface of the moon, boils”. That was the cue for me to dive in to some serious technical research and artistic design.
Michael Morris (Better Call Saul, Locke and Key), the director of Episode 1, gave me some initial boards and ideas, inspired by film Westerns, Lawrence of Arabia, etc. He, and Ron and the EPs, really wanted to create that sense of desert, of heat and dread – a real struggle and impending doom – all while being bombarded with intense solar radiation. The Sun was both beautiful and deathly. It was a character in our whole Season.
We took those boards and kicked-off Barry and his team at The Third Floor to create the initial previs. Since our stage space was limited, and we needed to create a vast open lunar landscape, we utilized previs to help us envision what we would shoot, what would be digital, and what would be a hybrid approach.
Meanwhile, I dug into scientific research, electrostatic energy, solar radiation, wind and dune physics, water dynamics, waveform modulation, cymatics – all in and effort to come up with a unique, scary, yet beautiful visual system to portray the ‘invisible killer’ that is solar radiation. Since you can’t normally see radiation, we knew we needed to somehow make it visible, dramatic, yet not over-the-top fantasy.
There are actual accounts from Apollo 17 astronauts that repeatedly saw, and even made sketches of ‘twilight rays’ or ‘streamers’ that they would witness for about 10-15 seconds before lunar sunrise and sunset. Apollo 8, 10, and 15 also shared these accounts. We also know that electrostatic charges can elevate dust particles, and charged particles can levitate. I was so inspired by this idea, that I had to run with it…
In the script and story, the Solar Storm that is ejected from the sun is also many many many times more powerful than any that NASA had recorded, so we knew we had some room to make this a REALLY intense storm on the lunar surface. And again, because the Moon as no atmosphere, there would be nothing to slow the radiation down – it would be pure, raw, deadly energy.
I’ve always been fascinated with sound waves, magnetic fields, wavelets, patterns, dunes, etc. I set out to come up with a combination of these things, that we could apply to the very fine dust particles on the lunar surface, that we could then SHOW the waves of unrelenting radiation waves hitting the astronauts. I put together a ‘pitch video’ comprised of footage of dunes, time-lapse river beds, dust devils, sound waves played through metal plates of sand (cymatics), incense smoke in slo-mo, sound waveforms and spikes, and a few concept art pieces I created from lunar footage from Season 1. I pitched these ideas to Ron, Ben, Matt, Maril, and Michael, and got their blessing to pursue. I worked with Method Vancouver, and brought on Craig Wentworth, who I had worked with on two different films at Imageworks years ago (Men in Black 3, Alice 2). He knew my aesthetic sensibility and had a strong background in leading teams in creating never-before-seen effects. I met with the team over cineSync, kicked-off all the ideas, supplied them with a ton of reference and my pitch video, and off they went!
As tests started coming through, we quickly realized that each shot – wide. medium, or close-up – would have to have special treatment to details, motion, readability, etc. Not just one approach was going to scale to all the different shots. I would show some tests to the show runners, and they’d say “more spikey” or “it’s a little too pretty” or “we need more activity”, etc…. so we learned early on that each type of shot would need its own handling. We developed a bit of language for ‘spikes and waves’, ‘towers and columns’, etc.
We also needed to show that the regolith surface was changing, and being altered by the relentless radiation waves, so in many shots, the entire surface from the live-action was completely replaced, in order to make the surface feel ‘alive’. Special attention was paid to boot-marks and rover tracks melting away progressively. I literally annotated on every single shot, specifically placing where columns of dust would be, how tall they were, how fast or slow they would move, and keeping track of continuity from shot to shot. About halfway through we developed a pretty good short-hand of how the shots needed to feel. Lots of love and care from the effects and simulations teams went into this sequence, not to mention the giant compositing undertaking to integrate everything.
Additional elements like lens flares, blooms, diffusions, visors, extensions, etc were added to make the silent killer visible and tangible. I’m really proud of how this sequence turned out, and it’s been nice to hear the positive reaction from people.
Can you explain in detail about the creation of the various shuttles?
We had two shuttles this season, one which was essentially based on the STS shuttles – in this case Shuttle Endeavor. Hybride Montreal did a great job of building that shuttle, and showing it docked to the SkyLab space station. In our alternate timeline- the STS shuttles were a very commonplace way for astronauts to travel to low-Earth orbit and to the Moon – almost like a taxi service. Many people online say “the shuttled could never get to the moon and back, they weren’t designed for that, didn’t have enough fuel, etc.” Well, we actually thought about that, and if you pay close attention, in the first episode of Season 2, when you see the Shuttle KonTiki docked to SkyLab, you can see the cargo bay doors open, and in there is a giant fuel tank – lots and lots of extra fuel. We don’t say exactly what kind of engine or fuel is in there, but we did make sure we covered out tracks!
Hybride in Montreal, also built the initial new ‘Pathfinder’ shuttle, that was revealed behind big red doors in one episode- they did a great job with the dramatic reveal with the spotlights in the entirely digital warehouse! The design for the Pathfinder came from Dan and his team, inspired by some never-built designs from NASA, plus a ton of unique detail work. Due to schedule demands, we had Ghost also work with the Pathfinder asset, and they took that, and added working cargo bay doors, and an elaborate missile launch system inside. Avoiding spoilers, this would play out in a later episode, and be seen up close, full-frame. They brought some great design ideas, based on a few notes I gave them.
The Pathfinder launch sequence was designed to make the audience think it was a ‘standard’ vertical launch, with astronauts on their backs… but in the camera move, we rotate 90 degrees and pull-back from the window to reveal that in fact we are horizontal, and the Pathfinder is mounted on the back of an enormous C-5 military cargo plane, in the air at 36,000 feet. Ghost created the entire environment and C-5 plane, and Pathfinder. We researched very specific paint schemes for the C-5, and ended up making a unique scheme based off a few different versions of the plane found in reality.
Highly detailed textures, showing individual heat tiles, bolts, seams, windows, etc were created by the talented team at Ghost. This sequence needed to also feature the brand-new ‘NERVA’ nuclear engine, that allowed the Pathfinder to efficiently travel to the Moon and back. A version of the ‘NERVA’ engines were actually designed and tested in the late 60’s, but never used anywhere. The Pathfinder is the first ‘NEW’ ship for For All Mankind, and is a stepping stone for the show heading into the future, and a real diverging from history, deeper into our alt-history timeline.
Can you tell us more about the shaders and textures work?
I won’t be able to tell you specifics about how each company works in their own workflow, but I can tell you that we use many different tools, rendering engines, etc… all to get the job done in the best way. Our show is delivered in 4k HDR, so everything has got to hold-up in high-fidelity viewing situations. We use Houdini (Mantra), Arnold, RenderMan, Vray, etc. It’s my job to make sure everything tracks and looks consistent, and our VFX partners are very good at working with and analyzing footage from other houses, so we can make things co-exist.
The texture and shader teams at our partner studios have done an outstanding job on making everything look as detailed and realistic as the show demands.
Our on-set data wrangler Matteo Borgomanero, along with Todd, our on-set VFX Supervisor – make sure we get ample photographs and texture reference for each and every set, prop, and costume. We also work with Gentle Giant studios in doing LiDAR scanning and texture acquisition, so we can hand-off solid packages to our VFX partners.
Let’s go back to Earth. How did you create the various locations and the anachronism removal?
Most of the locations on Earth are real, and it was mostly ‘period clean-up’ – removing signs, security cameras, cars, people – anything that wasn’t accurate to the early 80’s, we would remove. We also added digital signs to buildings, for specific names of companies from the show. For the JSC (Johnson Space Center) in Houston, we actually shoot downtown LA, at LA Center studios, and our Production Design team goes in and dresses the lobby, and we update the exterior with digital paint-cleanup and removal. The main building there was built in the 50’s, so it looks of the period.
On our locations scouts, we would evaluate how usable a location was or wasn’t. I would asses with Dan Bishop and the other art directors, if we could change an awning, or remove and replace a sign, or add an extension – digitally or physically. We would also call upon our stock/historical footage team to find footage of establishing shots of different cities, like Houston, New York, DC, NASA, etc. For different characters homes, these were a combination of location shooting, and stock footage. For instance, Larry and Ellen’s house, we only had a daylight shot, and we also needed a twilight shot, so I worked with Crafty Apes on turning that one daylight shot into an early-morning twilight look. I did a quick and dirty ‘day-for-might’ look in Photoshop, then sent it over to the very talented team there, and they did some seamless matte-painting work, added some porch lights, removed the cast shadows on the lawn, reapplied the sky though big trees, etc. – making it look like it was shot at 5am. There are numerous shots like this in the Season – all ‘invisible’ effects, but very cool when you know what was done. Barnstorm, Refuge, Crafty Apes, Union VFX, Studio 8, all did terrific work.
How did you approach the planes sequences for the filming and lighting?
This was very exciting and involved sequence to bring to life. Our writer for this episode, Bradley Thompson, also has a background in flight, so he wrote some extensive notes for us to start. Andrew Stanton (Finding Nemo, WALL-E, Toy Story, etc) directed the episode, so we immediately started working together on what was possible – for on-set work, and digital. Andrew being such an experienced animation director, created some terrific initial story-boards, and then we took those, and I sat down with Barry Howell, previs supervisor at The Third Floor, and we mapped out how we would create the world in which the characters of Ed and Gordo flew.
We established that the time of day from the script, and for beauty would be about an half-hour to sunset. This would create a nice and deep sense of directional lighting, color, depth, and drama. I gathered some visual reference of clouds and sunsets from 20,000 ft up, and then Barry and his team started building the previs environments. Meanwhile, I worked with Mark Byer (Special/Practical effects) in determining what kind of practical rig we would need to make the cockpits feel like they were really moving. We also worked with Ross Berryman, our DP for this episode, in determining light direction, changes of light during certain, shots, etc.
Barry and his team created a super-tight previs version of this sequence, working with me, Andrew, Bradley, and Garret Reisman (our real-life NASA astronaut and technical advisor). The Third Floor contributed so much to Season 2, and with such a strong cinematic sense from The Third Floor crew, this sequence came together in a really effective way.
In our alternate timeline, we created a one-seater T-38 – when in reality the T-38s were two-seaters. Some would say this is anachronistic or inaccurate, but hey, alternate timeline! We ended up shooting with two different physical cockpits – one that would allow front-on views of the character, and one that would allow profile views and rear-views. This came down to some limitations of what we could do with each rig, the easy removal of the glass/acrylic canopy, etc. The ‘box cockpit’ was relatively simple, and had no usable exterior. Mark and his team created a ‘motion’ rig that wasn’t actually hydraulic or pneumatic, but rather operated by real live humans – giving us the ability to shake and rumble, bank and turn, at will. That physicality, combined with clever camera moves to compensate for banking, rotations, inclines, add some acting, some good directing, and amazing digital backgrounds, and you sell the illusion. Our stunt team, led by Todd Schneider also worked with the actors to help create realistic movement. When it came time for our character Ed Baldwin to emergency eject, stunts took over, and gave us some great footage to work with in final VFX. We added booster fire, smoke, explosive bolts, a digital parachute, clouds and mist, massive camera shake, lighting changes, digital ocean, digital chair backs, etc… huge amount of VFX work to make this come together.
Ghost VFX in Copenhagen, Denmark, and led by VFX Supervisor Martin Gårdeler – absolutely killed it on this sequence. This was the first time i’d worked with them, and I am super happy with the beautiful work they did on this and other sequences in the Season. We had some tricky plate work and green screens to deal with, as the pilot helmets were like the visors – round and reflective of everything. We needed to show reflections of the pilots in the canopy, show the sun specular, show the intensity of the speed of the T-38s, and do some extensive plate work in order to make the whole sequence come together in a uniform manner. The sun was setting during the sequence, so we had to map out the look from beginning to end. Ghost did an amazing job of giving us the volumetric backlit and side-lit clouds, a terrific sense of speed and depth, and the creation of the heads-up displays. I got very specific about continuity from shot-to-shot, in terms if speed, roll, sweep, and motion-blur, so I really appreciated their patience. Martin and the Ghost team brought some wonderful optical/lens effects to the shots, and worked closely with me on color-timing each shot to look its best. Ghost also created the T-38s on the ground as well. Each time you see them on the tarmac, those are digital. They added a huge amount of detail to the planes for the dogfight sequence, so we could be up close!
The series is full of fake archives footages. How did you handle this challenge?
LOTS of discussion about what was usable and what wasn’t. And, we had to ‘get messy’. Footage from the time was either film, or crappy early ‘VHS-style’ video. We had to degrade some of our modern filmed footage to fit in with the archival footage. We had a dizzying array of formats to deal with – 4:3 film, 4:3 video, PAL, NTSC, compressed VHS at 30fps, telecine’d film, poorly scanned film, re-recorded early video, etc.
In the final episode, we had to integrate our characters into archival footage that was a combo of film, video, etc. I worked with Crafty Apes on making our 4k stuff, degraded enough to look seamless. They really had to smash the crap out of the footage to get it to match the archival – blurs, color bleeds, blown-out highlights, color depth removal, diffusions, compression, etc. They did a great job, working with some really challenging material and source footage. I had to coax them a little in that it was okay to make our actors ‘look worse’. The team really came through, and the whole sequence looks as if it is archival.
We also did many ‘deep fakes’ for this Season. I had been following a company called ‘Canny AI’ in TelAviv, for a couple years, and so when Season 2 started, I reached out to them to see if they would be interested in working with us on a few shots. They did a couple tests, and we felt really confident with them. Again, because we use some pretty sketchy archival footage, there were some problems to overcome in making the deep fakes read properly. This was all because a lot of the news footage was shot in video, interlaced, then compressed and archived online. The deep fake AIs need clear data and detail in order to do the best job. When you’ve got a piece of video that is so low-res, mouth shapes sometimes take up maybe 10 pixels, leaving very little to change. People might think, “well it’s low-res, so that should be easier” – it’s actually the opposite. There was a lot of massaging of the footage to get it work on low-quality source material. Canny AI worked diligently to try different options, and they really delivered. We also would sometimes resort to some paint-work on a frame here and there to get the phonemes and mouth-shapes to read clearly. Refuge stepped in on a few shots to do some emphasis on particular mouth shapes.
We came up with our own methods for recording our own actors and voice actors. I can’t reveal a few of our secrets, but let’s just say that sometimes the actor making the mouth shapes is NOT the same actor making the voice! We even had Barbara, our VFX Producer create a few new original mouth-mimics, that didn’t come through strong enough in our initial recordings of our voice-actors.
Which sequence or shot was the most challenging?
The Solar Storm sequence was a challenging sequence from physical production, stunts, set work, all the way through to VFX. Lots of R&D and design work went into the look of the ‘boiling regolith’ there. Episodes 9 and 10 have huge sequences there too, but I don’t want to spoil anything here. We worked with Mark Byers and his Special Effects team on getting a ton of awesome in-camera physical effects work, that we then added to in VFX. I always strive to get in-camera effects as often as possible, and Mark and his team gave us great material.
The very end shot at the end of Episode 10 was a big giant beast. It looks months from concept to delivery. I started with some quick photo-boards I gave to Barry and his team at The Third Floor, and they created an elaborate previs version(there were MANY!) – all timed to music. I utilized a bit of my own experience from WAAAAY back on Contact, and my astronomy background, when I designed and animated the opening shot from Contact, so I had some pointers I could give to the teams at The Third Floor, and later Zoic, who did the final final final versions of the shot, led by VFX Supervisor Jeff Baksinski. They did a great job on a really long and involved shot that cliff-hangs and leaps us into Season 3! Utilizing a ton of matte-paintings, projections, 3D environments, volumetrics, digital re-creations of a plate, etc…. it was a beast, and it turned out great.
Did you want to reveal any other invisible effects?
I’d like to just give a shout-out to the incredible roto and paint teams that make all our stuff look seamless. The Zero-G stuff wouldn’t work without the ability to remove cables and fix cloth. Almost every shot of an astronaut on the Moon has elaborate roto to isolate from the background to allow digital extensions, etc.
Crew and set pieces are removed from sunglasses and eyeglasses, picture frames, etc. Shooting in 4k shows EVERYTHING, and so even sometimes we need to repair a wig-line, or a microphone wire, etc. It’s not often talked about, but almost every show these days as dozens of ‘fix-it’ shots in there, that would normally never considered a VFX.
These repair teams do some amazing work to hide all things that go into creating our illusions.
About 65% of the time you see an image on a TV – that has been composited in. So much of the storytelling happens on TVs in our show, and lots of the content can’t be created in time for shooting, so we do it later on. Each television has a specific look and feel, and each ‘network news’ feed also has a different look and feel. Refuge, Barnstorm, Teaspoon, Union, Zoic – all did great compositing into these TVs. Details like tube distortion, noise, warp, color bleed, broadcast noise, glass reflections, etc. all were created to make things look in-camera.
The sequence of our astronauts landing in Star City, Russia – that was actually shot at Van Nuys airport in California, and there were no airplanes there, just a stairway and a ton of green screen. Zoic did an incredible job of completely creating the jet, removing the original airport, and adding all the surrounding flat environment, the tarmac and snow, all the way down to the cold breath of each actor!
How was the collaboration with their VFX Supervisors?
We had super strong teams, and I would work with each Supervisor on specific details about the assigned sequences. We would set-up initial creative and technical kick-offs over cineSync, and we would spend as much time as was necessary to be as clear as possible what the goals and desires were for each sequence, going over each cut, artwork, etc. While we have a lot of organization on the show, there are always always, times when we need to ‘make stuff up’ that may have not been in storyboards, or fixed in previs, or even having a design from which to go. I sometimes do my own paint-overs and concept work to give the teams something to start from, but there are other times, because of schedules or workload, I might just ask the different teams to present a couple options on a note, and we would go from there. I love seeing what different Sups and Artists can bring to the table. If it’s a good idea, it’s a good idea, doesn’t matter where it comes from.
I like to create an open atmosphere of sharing ideas, asking questions, and trying things out – so open dialogue and communication is key on our show, and we like nice people who are passionate about the work, so we’ve been very lucky to have terrific teams.
Is there something specific that gives you some really short nights?
I tend to think A LOT about what we’re going to shoot and how we’re going to shoot it, during principal photography. I go over the plans many times in my head, and also come up with ‘plan Bs’, as things can always change on the day. A lot of money is being spent on set shoots, so I like to be prepared for anything and everything, so some nights I get into thought-loops about how we’re going to shoot some of our more difficult sequences. I love being on set, and working with our directors, SFX, stunts, lighting, art department, props, costume, camera, DPs, cast – all that energy and collaboration of everyone to get the shots ‘just right’ is still a thrill even after 25 years of doing this.
Also, it needs to be said again, out loud – without my amazing team on the production side, my job would be much much harder. Big kudos to my producer and our coordinators and managers, and at those same types of teams teams at all our partner studios.
What is your favorite shot or sequence?
Impossible to pick, and I have to be careful so as to reveal any spoilers, but the most challenging sequences both creatively and technically were the Solar Storm on the moon with Molly and Wubbo, the LSAM flying through the canyon to the Claim Site, the T-38 dogfight, and the 3 big sequences in episodes 9 and 10…that will be obvious when you watch them….and hundreds of other shots, that are really cool and interesting in their own ways. All the work we did to get our modern actors to interact with historical figures is something i’m proud of as well, as those took focused set-up and practice on set to get just right, and then handed over to our VFX partners to do their incredible compositing and integration work. Our 2D teams at every company really nailed it on those pieces.
One of the things I love about For All Mankind, is the massive variety of VFX work. We never just do one thing over and over again. There’s always something new and challenging ahead!
What is your best memory on this show?
Hard to pick one memory. For me, it’s the holistic experience of the entire season, and the entire team. I’m basically doing my dream job, working in Los Angeles, on set at Sony Studios, a great cast, with a super talented team of writers, directors, creative people, a real astronaut and technical consultants from NASA, on a show about space, set in an alternate timeline, full of history and discovery. I was deeply affected by 2001: A Space Odyssey and Close Encounters, so here I am like a little kid surrounded by his favorite toys! Being able to work and collaborate with artists all over the world is really wonderful using art and technology to tell stories. I never stop learning, and I’ve learned more about orbital mechanics, fuel types, rocket engines, launch sequences, gravitational forces, Delta V, space suits, physics, and geography of the moon, than I ever imagined I would learn in my lifetime.
The hours can be long, and of course there are always challenges along the way, but that’s what makes it all interesting. I’m having a great time with this great team, and hope to continue into future seasons. It’s an exciting series with some big, creative ideas.
How long have you worked on this show?
I’ve been with the show since early 2019, through Season 1 and Season 2, and we are now filming Season 3! Happy to be back… we didn’t have a break from Season 2, so we started right away on pre-production of Season 3, literally as we were finishing Season 2!
What’s the VFX shots count?
For Season 2, we came in around 1600 shots across 10 episodes.
What was the size of your team?
My direct team is me, plus 3 others, and with our VFX partners, i’m sure it numbered in the hundreds!
What is your next project?
For All Mankind, Season 3!
A big thanks for your time.
© Vincent Frei – The Art of VFX – 2021
