AVENGERS – ENDGAME: Matt Aitken (VFX Supervisor), Sidney Kombo-Kintombo (Animation Supervisor) & Gerardo Aguilera (FX Supervisor) – Weta Digital

In 2016, Matt Aitken explained the work of Weta Digital on INDEPENDENCE DAY: RESURGENCE. He then worked on AVENGERS: INFINITY WAR. He speaks to us today about his work on AVENGERS: ENDGAME with Sidney Kombo-Kintombo (Animation Supervisor) and Gerardo Aguilera (FX Supervisor).

How was this new collaboration with directors Russo Brothers and VFX Supervisor Dan DeLeeuw?
We feel like we really consolidated our working relationship with the Russo Brothers and VFX Supervisor Dan DeLeeuw on this film. INFINITY WAR was the first time Weta Digital had worked with Joe and Anthony, Dan, Jen Underdahl and their visual effects team so it involved a process of us getting to know each other. By the time ENDGAME came around, we had established an element of trust with the filmmakers off the back of the success of INFINITY WAR and our work on it, so we were given more autonomy to develop the visual effects work in ENDGAME. There was also a shorthand that developed over time around discussing the work, so by the time we got into the last month or two on ENDGAME our calls were shorter and ultimately less frequent as well.

INFINITY WAR and ENDGAME were filmed back to back. How does that effect your work?
In many ways we treated this as one overall project with two films. Most of the team stayed the same from INFINITY WAR to ENDGAME. We used the few months downtime between the two films to make some improvements to our Thanos facial animation rig and enhance our simulation toolset ready for the work on ENDGAME. At one point I was concerned that I would have to miss a lot of the filming for our scenes on ENDGAME because, while the second film was being shot in the second half of 2017, we were busy working on the effects for INFINITY WAR. But as it turned out, most of the filming for the end battle took place in a 5-week block of additional photography in September and October of 2018, after INFINITY WAR had opened so the filmmakers had had a chance to circle back and focus on the end battle. So I got to be present on set for that shoot after all.

How did you organize the work with your VFX Producer?
I have a great working relationship with my Visual Effects Producer Marvyn Young, a close collaboration with a lot of overlap in our various roles. So I get very involved in bidding the work, resourcing and scheduling the show. In turn Marvyn will contribute to the creative direction of the work.

What are the sequences made by Weta Digital?
Weta Digital was the main contributor to the 3rd act battle. Our first shots are of Thanos’ H-Ship attacking and destroying the Avengers compound, and our last shots are the scene where Tony Stark dies. We did most of the sequences in between, including Thanos fighting Thor, Iron Man and Captain America; the portals sequence; the charge and clash; and Tony’s snap.

Can you tell us more about the filming on the end battle?
Plates for the end battle were filmed entirely on set at Pinewood Atlanta Studios in Georgia, U.S.A. The sets were often minimal, sometimes just a dirt floor and a 360º greenscreen. Production filmed with Alexa 65 large-format cameras from Arri and we worked at native IMAX format for every shot.

How did you used your experience of THE LORD OF THE RINGS armies for this sequence?
It was exciting for those of us working on AVENGERS: ENDGAME at Weta Digital who had also worked here on THE LORD OF THE RINGS to be back in the world of creating spectacular large-scale battle scenes. A lot of the challenges for ENDGAME were familiar to us from our work on THE LORD OF THE RINGS, including how to deliver on the scale and spectacle while also keep the story grounded in the personal experience of our heroes. As with THE LORD OF THE RINGS, we used Massive, the crowd simulation software developed at Weta Digital to create the battle scenes in RINGS, to animate the many thousands of warriors in the ENDGAME final battle. We felt strongly that we wanted the individual fighting styles of each of the different warrior types: Wakandans, Asgardians, Sorcerers and Ravagers on the hero’s side, and Chitauri, Outriders and Sakharans on Thanos’ side, to be reflected in the background fighting action in the battle. We wanted to present all that visual complexity and not simply have the battle be made up of pairs of warriors standing hacking away at each other with swords. To enable this, we spent several days on the mocap stage with a team of stunt performers recording battle vignettes: 3 Wakandans taking out 3 Chitauri, a horde of Outriders savaging a group of Sorcerers and so on. This material also allowed us to vary the tone of the battle and show that Thanos’ army is dominating at the point where Tony decides to use the stones.

A beautiful scene brings back all the characters using the portals. How did you create the various environments?
The environments we see through the portals that open up behind Captain America are entirely CG. This was necessary so we could render them with the same camera that was being used on the crater side of the portal, ensuring that their parallax matched that of the ruined compound battlefield as the camera drifted past. Our digital matte painting department created and rendered the environments in a software package called Clarisse, using reference footage provided by the production including key shots from previous movies. The view of each location: Wakanda, New Asgard, Kamar-Taj, Titan, Contraxia etc was carefully orchestrated for each shot to ensure that the angle on the environment gave an immediately recognisable view of the location through each portal, even though this required cheating the view to be different from shot to shot.

Can you tell us more about the lighting work on the end battle?
The environment is predominantly CG throughout the end battle sequence which meant that we could relight the environment, we are not constrained by the lighting they got on set. We can make it feel more like we are outside, with low lighting from a distant sun, and we can change the look of the environmental lighting from action beat to action beat. So, it’s sunny at the start of the sequence, and then after the compound is destroyed, it’s overcast; there’s a huge pall of smoke and dust hanging over the area. And when the portals open up, it becomes brighter again because we’re playing the environment lighting to match the mood of that sequence.

How did you manage your render farm to do not burn?
We have a very big render farm!

How did you enhance the dramatic makeup of dying Tony Stark?
Exposure to the combined power of the six infinity stones turns out to be fatal for Tony Stark. When we see him post-snap, the suit is very damaged and Tony is mortally wounded. We created the burns on his head and neck as a digital prosthetic. This involved developing a variant of our Tony Stark digital double that had the deeper wounds modelled into the geometry, a revised hair groom where his hair was singed away at his temples, and detailed texture work for the scarring and blood. This CG wound was rendered onto a match-move of the actor’s head and then tracked into place to ensure that the CG picked up all the fine details of the actor’s facial performance. A balance had to be achieved between making sure that the wound looked serious enough to be ultimately fatal to Tony, but also not being so gory that it would distract from Robert Downey Jr’s performance. The filmmakers were very clear that they wanted Tony to be able to retain his dignity through this sequence. Creating the wound as a digital prosthetic, as opposed to filming the actor with a practical prosthetic, allowed for the level of damage to be explored in the context of the final cut of the sequence. A practical prosthetic would have locked the filmmakers into a certain look which might’ve been too little damage, or too much. We welcomed the opportunity to work with the filmmakers to explore the nuances of Tony’s appearance at this pivotal moment in the movie.

Thanos is back in a younger version. Can you explain in detail about his design and creation?
The Thanos we see in ENDGAME is four years younger than the Thanos of AVENGERS: INFINITY WAR, arriving at the battlefield from the GUARDIANS OF THE GALAXY era. This Thanos is at the peak of his physical power and ready for battle. Compared to Thanos on Titan in INFINITY WAR who was more of a philosopher, meditating on past actions, this Thanos is dressed in his armor and planning to destroy the whole Universe then rebuild it from scratch. As well as creating his armor, we made animation changes to his physical fighting style to reflect his increased strength and agility. We also updated Thanos’ facial rig, building on and improving Weta Digital’s INFINITY WAR Thanos asset. We once again used a digital actor puppet as an intermediary stage, mapping Josh Brolin’s onset performance to this first before transferring it to our digital Thanos puppet. By doing this, we can confirm that we have captured all the nuances of Josh Brolin’s facial performance before migrating the facial animation from the actor puppet to the Thanos puppet.

For AVENGERS: ENDGAME, we further refined the Thanos facial rig to get around some shortcomings that we had experienced on INFINITY WAR but didn’t have the chance to fix. In particular, we added additional controls to the corners of his mouth to achieve greater fidelity and complexity in the performance. We also took advantage of some new tech that our facial animation team had developed which we call ‘Deep Shapes’. Deep Shapes add another layer of realism to the facial solve without adding any extra overhead to the task of sculpting the set of facial targets. Deep Shapes can be thought of as adding another octave of complexity to the transition of the face from one expression to the next, without influencing the shape of the face at either the start or end point of those transitions. This is important because a crucial aspect of our approach to facial animation at Weta Digital is that the facial animator should be able to retain complete control over the shape of the face.

Which sequence or shot was the most complicated to create and why?
The whole end battle was hugely challenging to achieve but in particular the fight between Scarlet Witch and Thanos was surprisingly difficult. Many of those shots are entirely or predominantly CG, including a hero Scarlet Witch digital double. But the real challenge was in creating Scarlet Witch’s magic power. Scarlet Witch takes her powers to the next level in ENDGAME. For Wanda, it is only 5 minutes since Thanos killed the love of her life, Vision, and so she is seething with anger. Since she first appeared in the MCU, Scarlet Witch has been developing her powers and in ENDGAME she becomes powerful enough to triumph over Thanos, stripping the armor right off him, until he retaliates by ordering his H-Ship to strafe the battlefield. Working off reference images provided by Marvel, including key frames of comic book art, we worked up the effect as a combination of volumetric and particle simulations developed by our fx team. The renders of these then went to our compositing department as raw material from which to create the final effect. This went through many iterations before we settled on the final look, which had to be immensely powerful but also recognizable as the same Scarlet Witch magic power that the audience had seen her use in previous MCU films. These iterations took time to turn around and this effect was in development for several months before we felt we had achieved the desired look.

Is there something specific that gives you some really short nights?
The show actually went really smoothly, something about us having a super-awesome team working on it. It was hard work, but we also managed to have a good time while we were doing the work.

What is your favorite shot or sequence?
I love everything we did on this show, the quality is excellent across the board. Having said that, the moment when the portals open up and all our heroes show up for the battle gets me every time. I’ve seen the film a few times since it opened and the reaction from the fans at that part of the film is always incredible.

What is your best memory on this show?
Working on these films, INFINITY WAR and ENDGAME, is up there with working on THE LORD OF THE RINGS for me. These are the kind of film projects that I got into visual effects to work on, and it’s crazy for me to think that I got to work on two of the biggest film franchises of the last 30 years.

How long have you worked on this show?
I worked on ENDGAME for 8 months, from August 2018 until early April 2019.

What’s the VFX shots count?
Weta Digital delivered 494 shots for AVENGERS: ENDGAME.

What was the size of your team?
The core team on the show at Weta Digital was 570 people.

What is your next project?
I’ve got something very exciting potentially lined up as my next project. I’m very sorry but I can’t say what it is just yet.

// Sidney Kombo-Kintombo – Animation Supervisor

How did you enhance your facial animations techniques since INFINITY WAR?
Thanos was already a well built and developed asset from our work on INFINITY WAR. But ENDGAME gave us the opportunity to push him even further. Thanks to the work done developing Alita for BATTLE ANGEL, we had a much better understanding of the behaviour of the orbicularis oris – the muscle that goes around the mouth. We integrated these developments from BATTLE ANGEL into our Thanos facial rig which gave a more convincing result of the lips sticking together and separating as Thanos speaks.

Another development is what we call Deep Shapes. The supervisor of our facial team originally developed Deep Shapes for Ang Lee’s GEMINI MAN, and we integrated the tech to be used on Thanos. Deep Shapes adds another layer of realism to the facial solve without adding any extra overhead to the task of sculpting the set of facial targets. Deep Shapes can be thought of as adding another octave of complexity to the transition of the face from one expression to the next, without influencing the shape of the face at either the start or end point of those transitions. This is important because a crucial aspect of our approach to facial animation at Weta Digital is that the facial animator should be able to retain complete control over the shape of the face.

How does his younger version affects the animation work?
In ENDGAME, Thanos is at the peak his condition. He is more agile and faster than he was in INFINITY WAR. But mainly, he is in a different state of mind. This young Thanos is not a philosopher. He is a warrior who only has his skills and his blade, not the infinity stones. So, in terms of animation, the goal was to have him emotionally and physically more involved in the battle. You can see that he is more focused because he knows he has to give it his all to defeat our heroes.

Can you tell us more about his animation during the fights?
Since INFINITY WAR, Dan Deleeuw constantly told us how much of an accomplished warrior Thanos is so illustrating this was our main goal. We had to make sure that his fighting motion was as convincing as possible. Marvel provided us with some really great mocap takes done by their stunties and some great footage referencing what they had in mind. We then took that motion and adjusted it to fit Thanos’ gigantic frame. We had to adjust the posing, his stance and his silhouette on pretty much every shot where he is fighting. We tried to convey a sense of rage and viciousness in his attacks. When you see him break Cap’s Shield, that is the most ferocious we have ever seen Thanos. He is also broad and sweeping in his movements, something we’ve not seen from him before. That was to sell the idea that his emotions are now involved. It’s not all business only anymore, as in INFINITY WAR. It’s personal.

The end battle is really impressive. How did you approach a massive sequence like that?
I remember the first time we received the previs. It looked like something we would need a year to put together, but we had 5 months! So we had to be really organised and efficient in our choices. The story had also had to make sense and be believable. So we broke the sequence down to ensure continuity, in terms of position, action and orientation. We made several maps that showed where each action was taking place and if it made sense within the storyline. This was a crucial exercise for the sequence where the portals opened. It looks really obvious now but there were a lot of discussions surrounding the overall placement of the portals in the layout.

Another thing to anticipate was the scale of the battle. We needed to give the feeling that a lot was going on in the background and while not taking away from the main characters. We needed more than noise, we needed a story. So with Marvel, we shot different vignettes of fight choreography showing factions of the hero’s army winning and losing battles. We also designed what we called the sky battle as a successions of long cycles that we reused and repositioned to better suit any given shot. A funny piece there, is when Gamora hits Quill, we placed two necrocrafs exploding just on screen right to match the timing of the hit. It’s nothing major, but it puts a smile on your face if you notice it. A fluid conversation with Marvel was key to the organisation of the sequence. Dan Deleeeuw and Jen Underdhal always made themselves available to answer any questions and adjust the sequence with us. That sense of collaboration is the main reasons behind the sequence’s success.

Did you used some tricks to help to understand the various actions during the battle?
I guess we can call them tricks, yes. But it is really just what is known to every animator as animation principles. Our recipe for the battle was solid posing, strong silhouettes, appeal and staging. The posing has to be solid because Jeff Ford’s editing style is very quick, making it difficult for the audience to read posing if it’s not clear enough. We shot lots of reference to understand how an action that lasts only 12 frames could be as clear as if it lasted for 30 frames. Once we had accomplished that on the puppets, we just had to make sure that the eyes of the audience were properly guided to the action. So we adjusted the framing of the actions to make the hook ups as fluid as possible, where needed.

Which sequence or shot was the most complicated to create and why?
Haha! Lots come to mind, but none can top the shot we called the ‘oner’. It’s the one where both armies clash in the crater. We had to animate pretty much all our heroes in a single shot. We had plates that we needed to integrate with our digital doubles as well as sell the scale of the battle. It involved over 20 artists just in motion. We had to show the main heroes having their moment, the sky battle, the Leviathan getting punch and dropping to the ground; Wakandas, Sorcerers, Asgardians and Ravagers fighting the Chitauris, the Sakaraans and the Outriders…. Coordinating the guys from Massive, motion edit, animation, layout and roto departments was quite a headache. It took us 3 months to put that one together. But I can tell you that seeing the shot assembled for the first time was quite emotional for us. It goes by really fast cause it’s only a 600 frame shot. But definitely the most challenging I have ever accounted…. So far.

Is there something specific that gives you some really short nights?
When you have the opportunity to work on something this big – and I mean literally big – you can lose yourself in an ocean of details. You need to keep your eyes on the bigger picture, on the keys pieces that will glue the details together in a natural fashion. Figuring out how to tackle any given challenge before any artists even touch the shots, planning to make sure that we wouldn’t have to redo or change the concept, making sure that we requested the right asset or rig or terrain adjustment, making sure that we would not burn the artists energy by making the wrong call… That gave me some short nights yes!

Did you create some previs for the end battle?
Yes we did! We previsualised quite a few shots mainly around Thanos’ final moment. But the most significant work was on the Women of Marvel bit, where Captain Marvel lands and rushes to the van helped by all the others female heroes of Marvel. I remember that it was really close to the deadline for the animation team, and we had already had a crack at this segment following the original previs. But the result didn’t convince the directors so Dan Deleeuw asked us if we could propose something knowing that there wasn’t much time left. We spent some time discussing what the best options were and what the story needed to be. I brainstormed with Matt Aitken and some of my animation team and we came in on a Saturday to previs some 25 shots. I edited it and we sent it to Dan that same evening. We had such positive feedback from the client stating how happy they were with our proposition and that they were goimg to keep it. We executed it and the result is what you’ve seen.

What is your favorite shot or sequence?
On this movie, we had a lot of input in term of composition, and story-telling, which makes one enjoy their job even more. But I have a soft spot for the shot where Thanos says to Cap “I‘m gonna enjoy (destroying earth) very, very much!” If you watch closely, there’s a lot of subtext going on there that you can read as he tenses and relaxes. We referenced a scene from the THE GODFATHER: PART II where you see Al Pacino as the Godfather for the first time as he explains how he would kill a corrupted cop. We pushed in as Thanos talks and it made the scene a lot more dramatic than with the original static camera. Really proud of that shot!

What is your best memory on this show?
Lots of moments that I cherish from the production window. If I had to pick a moment, it would be after we were done. When my wife watched it in a theatre. She had no idea how involved people were with the characters and the MCU. It was a surprise to her. All the cheering, shouting, crying, laughing made the moment unique. Hearing her talking to me about what I’ve known for a long time was a shared experience that I found most rewarding.

How long have you worked on this show?
If I recall correctly I’ve been on it for just over 6 months.

What is your next project?
Not really thinking about it just yet. Still enjoying vacation time!

// Gerardo Aguilera – FX Supervisor

Can you explain in detail about the design and creation of the Nano suit for Iron Man?
Iron Man’s nano suit design was one that we had started developing in collaboration with Marvel for INFINITY WAR based on some concept art and development we did. For ENDGAME we had to update our setups to work with Ironman’s new Mark 85 suit. In this update we also needed to make sure our setup was flexible enough to have the suit be heavily damaged when it comes to the end of the battle and Tony does the snap with the infinity stones.

Can you tell us more about the FX work on Iron Man for the post-snap?
This was a huge moment for us, and these shots were some of the very last that we worked on. The infinity stones have such power that we needed to find a balance between how damaged Tony’s suit ended up post snap while still conveying how the suit was actually trying to protect Tony. We created the effect in two main stages. First, we got the timing of the energy coloured lines that we see coming off the stones and moving across the suit. Once that timing and look was approved, we used the same technique for the nano suit, but this time to create the damage. We used the timing of the colour lines to create small “trenches” on the suit that were creating damage as they moved across the suit. At the end we had to develop this effect in such way that the compositors could have control over how fast or slow this damage to the suit was occurring. It was important to convey the idea that the suit was trying to resist this damage. Ultimately it was a lot faster and more efficient for our compositors to be able to adjust and fine tuning the timing.

Can you tell us more about the environment creation and its destructed version?
For some of the big environment destruction moments such as the compound explosion, Giant Man coming up from beneath the rubble of the compound building and the rest of the destruction in the battle we used our in-house bullet implementation in Houdini that we called whRigid. With this workflow it allows artists to quickly setup the destruction shots no matter the scale and gives them the ease to quickly adjust parameters for the simulation and easily iterate to get the best result possible.

Can you explain in detail about the vanishing effect on Thanos army and on himself?
The vanishing effects are by far some of my favourite sequences in both AVENGERS films. They are such key moments so we had a big responsibility to deliver something amazing for the fans to enjoy. Internally at Weta, we refer to it as the blip effect. For Thanos’ blip we had to hit some really important timing cues for him. For efficiency we came up with a way we could show our VFX Supervisor, Matt Aitken and Marvel’s VFX Supervisor Dan Deleeuw how the blip would transition over Thanos’ body. It was represented with a simple 2 colour mask that was easy and quick for us to update if we got any notes regarding timing. The great thing about these masks was that they were driven by the same growth algorithms we used for the actual final blip effect. So once the client was happy with the motion and timing of the mask, this was then matched one to one as Thanos starts turning into ash and later on disintegrating.

For Thanos we also created custom velocity fields to hit some important blip notes on how he would vanish and when he would fully dissolve. This custom velocity fields not only helped us hit the timing notes but also added visual complexity to the effect. The other challenging aspect in ENDGAME was to blip out the entire enemy army, including large drop ships, giant Leviathans and Chitauri tanks to name a few. We developed a secondary blip setup that we called it the blip light which was a lot quicker to run and was specifically designed for larger amounts of army, and very large structures whilst maintaining the same visual complexity as the close-up hero blip of Thanos and The Black Order.

Which sequence or shot was the most complicated to create and why?
We had a few interesting challenges in ENDGAME. One interesting and challenging effect that comes to mind were the giant Dr Strange portals where all our heroes and armies emerge from. These sparks portals needed to maintain the physical behaviour of sparks floating in the air and rotating at very fast speeds and then cooling down causing some of the sparks to drop and some to float in the air. Back in INFINITY WAR, we developed a setup for the Dr. Strange portals, but they were a lot smaller, about the size of one adult human. For ENDGAME we had to extend our setup so we could create very, very large portals for entire armies and spaceships to come out of. We created custom pressure simulations that would act as velocity or wind fields for the large-scale portals. This would gave us a natural look and feel to the portals whilst maintaining their iconic look. Later on, we utilized the power of Houdini engine to deliver the portals to our lighting team whilst giving them some controls at render time if they wanted to reduce the amount of sparks, decrease or increase the scale of these sparks and extend or reduce the length of the sparks.

Is there something specific that gives you some really short nights?
Well I couldn’t pin-point one exact situation, but I would say it was the drive to always improve on each iteration that we presented to Marvel to achieve the best result we could. It was a challenging but very rewarding project in the sense of content and schedule and more than once the idea of trying to perfect a specific effect or shot definitely kept me up at night!

What is your favorite shot or sequence?
I would say the Avengers compound destruction section was one of my favourite parts to work on. It was such a shocking moment for me to find out that the Avengers compound was going to go down like that. It made me even more excited to work on it.

What is your best memory on this show?
Hands down it has to be when Marvel’s VFX Supervisor Dan Deleeuw gave us a note on a shot, “the fx team is really knocking it out the park”. It was such a proud moment for me to to have the great FX team we have at Weta recognised for all the dedication we had been investing into the project.

How long have you worked on this show?
I’ve been involved in the last two AVENGERS films since early 2017. After finishing INFINITY WAR last year I had a few months off the project before starting on ENDGAME, so about 7 months specifically on AVENGERS: ENDGAME.

What is your next project?
I’ve started working on another project that comes out next year that I’m sure will be another great blockbuster! Unfortunately I can’t share the name just yet!

A big thanks for your time.

WANT TO KNOW MORE?
Weta Digital: Dedicated page about AVENGERS: ENDGAME on Weta Digital website.

© Vincent Frei – The Art of VFX – 2019