In 2018, Dan DeLeeuw explained his work on AVENGERS: INFINITY WAR. He describes today his work and the challenges on AVENGERS: ENDGAME.
Infinity War and Endgame were filmed back to back. What are the main advantages on the VFX side?
I think creating our infrastructure early – staffing the teams with super talented people and creating dynamic systems was the greatest advantage. After finagling 5119 shots over the course of two films, we have one of the best teams in the business!
After Infinity War, does the directors wanted to change anything about their VFX approach?
The director’s didn’t having any specific requests. The infrastructure for the show was quite robust and flowed seamlessly into ENDGAME.
How did you organize the work with VFX Producer Jen Underdahl and the Marvel VFX Supervisors?
Jen worked to split the work by sequences as much as possible. This methodology worked better for INFINITY WAR because characters and assets slotted into specific battles throughout the film. For ENDGAME, characters and assets were shared across sequences. As a result, there was much more sharing between VFX houses. I took a more high level overview on these two films. I oversaw pre-vis and preproduction planning; as well as running shots in post production. Swen Gilberg covered first unit and Marten Larsen covered second unit. When photography was complete, Swen and Marten moved into post production to help push shots to final.
How did you split the work amongst the VFX vendors?
We had the good fortune of working with many of the same vendors from INFINITY WAR. As a result many of the VFX houses already had assets required for ENDGAME. We tried to cast the show based on work we could leverage from INFINITY WAR, as well as, each house’s strengths.
Did you update some models or techniques such as Thanos thanks to the experience of Infinity War?
Digital Domain and Weta Digital updated their Thanos rigs. Weta used a new technique called Deep Shapes to provide a procedural way to get a finer level of complexity as the face moves from one expression to another. Digital Domain modified their process for tracking the dots on Brolin’s face. The new system used machine learning to track the dots in a few hours rather than one to two weeks.
How did you design and create the Quantum Space suits?
The Quantum Suits were designed by Ryan Meinerding in Marvel’s look development department. From their designs, Framestore, ILM, and DNEG worked in parallel detailing out the suit and making them functional. The suits could build over the characters body using Stark’s bleeding edge technology. We based the effect on Framestore’s work from INFINITY WAR on the MK 50 Iron Man armor.
A new CG character makes his appearance, a new version of Hulk. Can you explain in detail about his design and creation?
We received designs from Ryan, and ILM ran with the sculpt of Smart Hulk’s face. We decided on a great design pretty quickly which allowed us to spend more time perfecting the rigging and performance. We used systems created by the Disney Research Team in Zurich named Medusa and Anyma. Medusa would capture incredibly detailed meshes of Mark Ruffalo’s face and Anyma would interpret the movements of the face from the FAC’s session. For ENDGAME, the research team rewrote Anyma to work from the dots captured in the helmet-cam footage. ILM rewrote their retargeting software to capture as much of Ruffalo’s performance as possible. Framestore worked on Smart Hulk as well and created a machine learning system that could be trained to recognize Mark’s performance.
How did you use your experience with Thanos for this new Hulk?
All of our work with Weta and Digital Domain focused on retaining the smallest details as possible from Josh Brolin’s subtle performance. With Mark Ruffalo, it was almost the opposite extreme. Where Josh’s face was subtle and intense – Ruffalo’s face was broad and elastic. The teams needed to focus on translating the intent of Mark’s performance onto Smart Hulk’s larger features. So for Endgame we had a great new character with his own set of challenges.
Can you tell us more about your work on-set with the crew to get all the necessary infos for the post?
One of the many benefits of working with Marvel is the continuity of the VFX tools across 22 feature films. For the most part, the VFX teams use of the same databases, HDR tools, and the same shooting crews. As a result, everyone on set is familiar with the data we need. That familiarity is the only way to get films this big done!
Some sequences gives us a different look on the first Avengers movie. What are the main challenges about that?
Sequences from the older films shared original and new photography. For the original films we unarchived the final VFX shots and converted them into ENDGAME’s color space. In the DI the new and old shots were color timed to match to create the illusion of a new perspective on a scene. The great advantage is that we have access to all the assets from the other Marvel films. For instance we had the old LIDAR data of Morag from the GUARDIANS OF THE GALAXY set. When we went back and recreated some of the shots, we could re-build the live action set and incorporate it into our new photography. For scenes like the Battle of New York, ILM unarchived their old shots and data to help complete their new shots.
How did you approach the fight between the two Captain America?
For Cap vs Cap, we shot over green screen with two stunt performers. Chris Evans would step in fighting and dialogue. When we see two Cap’s, Chris would perform against a stunt double. Lola then created the face replacement to complete the illusion of the two Captain Americas. Originally, we shot the sequence without Cap wearing his helmet, but it was confusing as to which Cap was which. DNEG completed the shots with an awesome all CG background and added the helmet to reduce the confusion.
The final battle is really epic! What was your approach with such a big sequence?
We always try to map out as much of our big sequences as possible in pre-vis. The Third Floor did an amazing job of creating road map for the entire final battle. For specific beats, the stunt team would create fight choreography for the hand to hand combat. This stunt-vis would then be cut into the pre-vis and used by production to shoot the sequence. Any all CG shots would get turned over as soon as possible because they didn’t have to wait for the live action photography.
We shot on stage with minimal sets and dressing. Ninety percent of what you see in the final battle is created in CG. Weta created a crater shape and layout from The Third Floor’s model. The rough crater was shared with ILM and both houses created their own incredibly detailed virtual sets.
We tried to split the sequence as logically as possible. We knew that the sequence would grow in post-production and we needed two of the biggest VFX houses in the world to complete it.
Can you tell us more about the previs work with the directors on the final battle?
As soon as we get the script, I will start working with pre-vis to flesh out the big fight sequences. The awesome team from The Third Floor’s team was headed up by Gerardo Ramirez – they have artists that go all the way back to the first THOR movie. Gerardo and I will sit down with the script – or even just concepts if the script isn’t written yet – and start to figure out the main beats. We typically over animate a sequence, but that allows us to trim down the edit and keep the best parts.
The most important scenes were Cap picking up Thor’s hammer, the portals opening, and Tony’s sacrifice at the end. Once these main sequences were laid out, we would show the Russo brothers and would get input and notes. I think the biggest note was always to reduce – we wanted to make the sequence bigger than it already was!
How did you help the cast and the crew to visualize this epic sequence?
We would show the pre-vis and concept art to the cast and crew to help the lighting as well as set the mood for the actors. For CG characters, we would always dress up stunt performers in mocap outfits so that everyone knew where are digital characters were on set.
The battle has a crazy amounts of CG assets (characters, FX, environment, …). What is your secret to don’t be lost with so many informations?
The MVP’s of the show were the VFX on-set crew, production coordinators, and the VFX editorial team. Matt Lloyd, David Bosco, and Emily Denker headed up each of those groups respectively. The amount of data that they flawlessly handled was amazing. It all comes down to great tracking systems and crews with the experience and talent to keep it straight.
Can you elaborates about the complex process to shoot all the elements for the final battle?
Much of the final battle would be completed digitally which offset the complexity of the shoot. The set itself was pretty simple. It consisted of a dirt floor and wreckage that could be moved to dress in elevations and foreground for camera. In some cases we would remove set dressing because it would hamper the stunt performers movements. SFX created fire effects for interactive light and to add complexity to the set.
What are the main challenges with full CG shots?
For this film, it was the density of digital characters and effects. When we shot on stage we knew we wouldn’t get the scale and scope we needed. As a result VFX would have to carry a heavy load. When INFINITY WAR ended, we still had to go back and finish photography for ENDGAME. This caused us to have to turn over all CG shots later than we normally would have. The VFX houses had to back load their staffing to create enormous teams to get the work done!
What the VFX shots count?
2496.
What was your best memory or moment on Endgame?
I have wanted Captain America to pick up Mjolnir since the first AVENGERS. The fact that I had to opportunity to design and bring the Cap vs. Thanos fight to the big screen was the best!
Now that those two huge Avengers are done, how did you feel?
I feel like we climbed the movie equivalent of Mt. Everest. It’s an amazing view from up here. Everyone is seeing the movie and we don’t have to keep secrets anymore!
A big thanks for your time.
WANT TO KNOW MORE?
Digital Domain: Dedicated page about AVENGERS: ENDGAME on Digital Domain website.
DNEG: Dedicated page about AVENGERS: ENDGAME on DNEG website.
Framestore: Dedicated page about AVENGERS: ENDGAME on Framestore website.
Industrial Light & Magic: Dedicated page about AVENGERS: ENDGAME on ILM website.
Weta Digital: Dedicated page about AVENGERS: ENDGAME on Weta Digital website.
© Vincent Frei – The Art of VFX – 2019
