Back in 2020, Justin Ball explained the visual effects work on the first season of Perry Mason. He went on to work on The First Lady, Creed III and Barry.
How did you get involved on this show?
Skydance reached out to me when they were ramping up the show. They had already landed the VFX Producer Lisa Kelly, who I’ve worked with before when they approached me. Simon and I had worked together a couple of times over the years back in his TV days. Both on the first seasons of Boardwalk Empire and Magic City. So it was great to collaborate with him again.
How was the collaboration with director Simon Cellan Jones?
This was great. As I mentioned we had worked together before over the past 10 years on a couple shows. But that was a bit more limited capacity. So it was a nice reunion to see him again with this project. This was certainly quite a bit more ambitious. So it was a fun endeavor.
What was his approach and expectations about the visual effects?
The expectations were high, to say the least.
How did you organize the work your VFX Producer?
Lisa Kelly, the VFX Producer was actually on the show before I joined. Lisa and I had worked before on another film called the Current War back in 2017, so this was wonderful reunion. Lisa is someone I trust implicitly and is a very talented producer. Having that bit of shorthand between us was a massive benefit in general, but more specifically on this show was a massive help. With all the moving and shifting this show went through, we needed to rely on each other to keep our team positioned in the correct way to ensure we were able to collect everything we needed.
How did you choose and split the work amongst the vendors?
Cinesite London – Ben White & Martina Silharova – They were our Lead Vendor handling basically all of the 3rd Act. All of the Vegas CG Hotel, Science Lab Fight & Final Fight sequences, along with the stunt actor face replacements.
Weta FX – Charlie Tait & Marvyn Young – Handled Hero Baby Max AI/ML face replacements.
Trixter – Joern Grosshans & Rufin Wiesemann – Additional Baby Max AI/ML – Including 6 months older Max at the end of the film.
Outpost VFX – Jason Evans & David Brown – All the daytime Driving comps and some additional Environments like the Nevada Airport.
NVIZ – Moti Biran & David Cordon – Motorcycle Chase sequence
Viridian FX – Ben Louden & Yvette Yang – Las Vegas Motel Sequence along with additional swing vfx work.
Milk VFX – Fernando Tortosa & Jasmine Ford-Elgood – Environment work like Buffalo Establishing, and Cleveland Motel scenes. Along with the Aria Suite Hotel Scenes and Fight Sequence.
Host VFX – Mike Cosgrave – Postviz and additional VFX.
Where were filmed the various sequences of the movie?
Principal photography for The Family Plan was shot in Atlanta and in Las Vegas. We had a traveling road unit for drive-bys and driving plates, along with some plate units.
Can you elaborate about the tilt shift shots in the opening of the movie?
The director wanted to use this as a little bit of tone-setting for the film. As we are trying to establish Dan (Mark Walberg) character as being hidden in every-day middle America, that he has disappeared into the sameness of cookie-cutter homes and a boring life (compared to his past).
What was your approach for the CG baby?
Initially this was not something we were planning to do. We (VFX) were encouraging the production to be as pragmatic and practical as possible with the kids. We had cast twin girls to plan the Role of Max, Illiana and Vienna, who were great. Rapidly it became apparent that VFX would need to become involved to help solve certain filming limitations.
Based on the footage we had shot to that moment we knew that we’d have to pull out every trick in the book. In our initial shoot plan, we had always planned on scanning the twins multiple times over the duration of the shoot. So we had always planned for some potential digi-double work if it came up, and we also wanted to have that bit of a safety net in post to have moment-in-time captures of the children as they were growing so rapidly.
While in prep, we had discussed the plan of having stunt babies generated (dolls) along with a hero stuffy to be on hand. These were useful during the stunt sequences and 2nd unit filming. But VFX also pushed for having additional babies on hand to be used as body-doubles or photo doubles, as for us it was always best to have the weight and mechanics of a real child and perform face replacement vs replacing the entire child. So as we started filming we always had 4 babies scheduled for each filming day. The 2 hero twins and their doubles.
Pretty early on in the shoot it became apparent that we would need to increase our involvement in regards to solving the baby shots and sequences. So along with our scanning plans for the children, we also started attempting FACS sessions with the children, which were more successful than anticipated. We also launched into a series of pickup shoots against bluescreen with the babies in various positions and lighting situations. But I knew our success here would be minimal. I launched into a plan to solve the potential needs for the baby in post using an all AI/ML solution. So in addition to the bluescreen shoots, we also launched side-shoots with the children with a VFX rig we assembled.
I referred to it as the “Poor-man’s Lightstage”. We created a mobile setup with the help of the electrical department to create a 7-camera Blackmagic rig with animated lighting for us to film the children and feed an ML library. We would place the children in the middle of the horseshoe of cameras and film them closeup eating snacks (babies are always happy when they are snacking!), watching movies or interacting with their parents at 4k or 6k for about 15 minutes at a time. All the while we would animate the lighting from a 5-light setup. The cameras were placed at alternating heights and angles to help capture as much information about the faces as possible.
We did this about 5 times over the duration of the shoot to capture the children at various stages of their development. We were also concerned about needing to age-lock the children when in post, so we wanted to be covered. The ML solution became our predominant way of addressing any Baby Max related face issues in post. And there were plenty.
Can you elaborate about the creation and animation of the CG Baby?
When locking in my position on the way to tackle the baby, we had to source a vendor who had the means to achieve what we were after, as I knew we’d be pushing the envelope a bit in terms of the needs for an AI-type solution. Lisa and I reached out to several companies all across the globe who potentially had the means to pull this off. I was pretty adamant that the vendors that could pull this off needed to be in the AI/ML/DeepFake space for a while. I know everyone has been dipping their toes into the water, but I needed teams that had been doing it successfully for over a year / year and a half at this point in time. That pool of options was very very small.
We issued a test while still in principal photography of a hero shot of the child that needed to be fixed. We sent this to about 5 vendors who I’d worked with before in this capacity, teams that had done ambitious performance-based face replacements, or enhancements over the years. We sent them the mountain of data from one of our capture sessions and the shot to see what they could pull off.
We knew this test was going to take a long time just to try to get something workable. We worked with the various vendors on their ideas and plans and ways to tackle, knowing the other potential issues in other footage, I could see where the roadmaps would fail in terms of designing performance for the child and the limits of the proposed strategies.
In the end, only 2 vendors rose to tackle the issue in a seamless way. Trixter and Weta FX. Through their various projects to date they had learned ways to manipulate the data, to groom the data that they could almost guarantee the performance and resulting comp in ways that no-one else had cracked to that moment. This manipulation along with the mountains of Max data we were able to create seamless ML max faces with specific performance moments.
We even had a VFX capture while in post of the twins when they were approximately 6 months older than their last capture during principal so that we could use their updated likeness for the older child at the end of the film, who was entirely ML for the face & performance.
What was the main challenges with the CG Baby?
I’d say the biggest challenge became when we proved we could pull off the ML face replacement successfully, as this opened up the wider scope of shots to address and moments to fix.
Can you elaborate about the environment work and especially for the final sequence?
I’m a big believer in trying to film everything possible, so for all of the environmental work we filmed plates to serve as the base lighting for DMP or whatever we needed. If possible we would film with production cameras, but it became increasingly difficult to get a camera split off for VFX with the demands on Main and Second Unit, so a lot of the time I would break away from set and film plates either with my Blackmagic or still camera. This allowed us to capture the various locations we needed to reach on the show.
We also had a road unit that traveled from Buffalo to Vegas making various planned stops to film drone or array car plates to sell the traveling aspect of the film.
The rooftop sequence was shot in a warehouse in Atlanta. We had fought hard to have the sequence filmed in exterior daylight but as we moved closer and closer to the build dates for the set, the scale of the build (and supporting camera and lighting needs) became so great that we had to bring it inside.
Again I had filmed these plates ahead of time in Vegas. We had scouted the rooftop with the DP and director in prep and had a good sense of the lighting plan, so when I went with the plate team to Vegas, we had a good attack plan for the time of day and lighting needs for the plates.
As we prepped for the stage shoot I was able to review proxy stitches of the array footage with Michael Burgess our DP and we could strategically plan the Soft Sun where the real sun was in the plates. Cinesite handled this scene and really knocked it out of the park.
Did you use LED walls for the hotel interior sequences?
No, all of this was done with bluescreens that we shot in Atlanta. We only used LED walls for lighting when we were doing some motorcycle-chase pickup shots in Vegas at the end of principal. As we were near the end of the shoot, Simon and the editor wanted to go back and pick up some details for that scene, so we found an LED stage in Vegas that could accommodate our needs. But we did not photograph the walls, we just ran the array footage from the Chase shoot for lighting, and shot the cast against bluescreen.
For the Aria suite, which was Vegas at night, I did film the array plates ahead of the bluescreen shoot, so I had done a pass of stitching the plates and gave them to the lighting teams to reference. Michael our DP and Nick the gaffer were able to come up with an animated lighting scheme for outside the windows to time with the plates that were shot. This worked very well in the end and really helped tie the plates together.
How did you recreate the various Las Vegas environments?
All of this was done with plate array Units that I shot while in Vegas. During prep I had sorted out a few key positions that we could use for camera positions. In December of ‘22 I traveled out with the plate unit DP Doug Chamberlain and had camera and drone teams meet me there to help film. We were using a 3 head array rig that Jason Toth at Revered Cinema had developed that could be flown on a heavy lifter drone. We had also hired Chris Sheuster of Aerial Mob and his team to come up from LA as they had the clearances to be able to fly and get the permits for Vegas airspace quickly as our timeline was short. We used the same array rig on the drone as we did for the ground based work.
For the Aria Hotel Suite we filmed from the Cosmopolitan, utilizing one of the hotel room balconies with a Jib arm to hang the array over the edge of the building on a wrap around balcony. For the Restaurant scene, I was able to get on the rooftop of Caesar’s one night to get that vantage. For the Poseidon Hotel the plate unit used the Trump Tower as its filming platform. I utilized the drone array to capture the elevator traveling shots and view for the Poseidon Suite view, since this height was not reachable any other way. The rooftop fight sequence had the plates filmed from the roof of the Tower using the jib arm & array to capture 360 of the city at various times of day.
Can you tell us more about the light work during the night shots?
The motorcycle chase was filmed largely practical with pod vehicles and stunt drivers and tow rigs etc.
We did a lot of light removal in the main run of the chase sequence, but that was more something I wanted to do aesthetically. During all those night driving and stunt sequences we had a special 360 array plate camera vehicle that could drive the action of our hero van. So we ran plate routes on all those sequences. This was useful for when we did the pickups in Vegas. We projected that material on the LED walls for lighting and then used the plates in comp to dial it all in and make sure we matched the principal photography.
How did you work with the SFX and stunts teams especially for the final sequence?
I always took the stance with the stunt coordinator Sean Graham that this was at heart an action movie, not a VFX movie. That VFX was there to support what the action team needed in regards to safety and rigging etc. So we really just let them do their thing. They would come to us with their plan and confirm it was all achievable with our budget and plans etc, and we would sort out any adjustments to be made, but really just let them roll with what they do best.
Did you want to reveal to us any other invisible effects?
Well, hopefully there are lots of invisible shots. I think one of the main ones for me was in the final scenes when the family is walking out of the hotel with the police waiting outside. There is a slo-mo shot closeup of Dan carrying Max, where we totally changed the performance there, and really changed the moment in the story from what we had filmed. Charlie Tait and the Weta FX team killed it on that one. We had actually not even presented any of the ML baby Max shots in post to the director, but when this came in, I slipped it into the preview cut to see if audiences would pick it up. The feedback was tremendous on how great Max was and the end of the film with him etc. We knew we had the solve then.
Which sequence or shot was the most challenging?
I’d say the rooftop fight sequence. It took a lot to get that whole thing working well and with the intense stage lighting to make it really feel dialed in. Cinesite really killed it on this sequence.
Is there something specific that gives you some really short nights?
Driving comps. The work I hate the most. We had to do a lot of work to get all the practical driving and bluescreen driving to match. We even were altering the practical footage so that all the car work would have the same look. We then had a lot of work in DI with the DP and Colorist to make sure they were happy. I hate car comps.
What is your favorite shot or sequence?
I’d say the work we did with Baby Max was my favorite. It was an uphill fight for me to get everyone behind the ideas I was pitching for how to solve the face replacements. We had a lot of meetings including pitches to the studio to make sure everyone could believe we could pull it off. Revealing to people who had seen the dailies and were there when we filmed the child that the performance we delivered was not what they had filmed was pretty great. That we were that successful in delivering a seamless hero performance.
What is your best memory on this show?
I think my best/craziest memory on the show was when we lost our key Poseidon Vegas Hotel filming location due to circumstances outside of our control about 5 days before we were to film it. The company was in Vegas filming the final portion of the shoot and we lost our key set. Mark, the magic man he is, made a few calls and contacted the owners of the Stratosphere hotel to see if we could film there. Late that night the production designer, art director and I were called, told to be at the Strat before dawn to assess if we could make the location work as our replacement Poseidon hotel. We took pictures, checked it against the hotel designs we had built, roughed up some mockups that morning to confirm it would work. The only day we could film at the location was Valentine’s Day of all days, a big day for the hotel. We had to dress, film and strike the set all within 10hrs while couples were arriving for dinner at the valet stand. It was a crazy day filming all that work. The art department worked overnight to dress the set. As we shot, we cleaned up the set behind us. It was hot and sunny, rained, SNOWED, and was about freezing when we left that day. It was one of those great filming days where everything just clicked and somehow we pulled it off.
How long have you worked on this show?
I’d say it was about 15 months in the end.
What’s the VFX shots count?
I think we landed about 1370 in the end.
What is your next project?
2023 was a pretty busy year. So after we wrapped the post at the end of September, I’m pretty happy to just relax and be home for the holidays. Looking forward to what’s coming in 2024, but not in much of a rush.
A big thanks for your time.
WANT TO KNOW MORE?
Cinesite: Dedicated page about The Family Plan on Cinesite website.
Outpost VFX: Dedicated page about The Family Plan on Outpost VFX website.
Apple TV+: You can watch The Family Plan on Apple TV+.
© Vincent Frei – The Art of VFX – 2024
