In 2017, Stuart Penn explained to us the work of Framestore on ALIEN: COVENANT. He then took care of the effects of the films KINGSMAN: THE GOLDEN CIRCLE and PADDINGTON 2.
How did you get involved on this show?
Stuart Penn (SP): I joined just as we were getting out first turnover so it was full steam ahead from day one.
How did you feel being back in the MCU?
SP: Excited. We knew from the outset it was going to be epic. The culmination of over 10 years of films – the expectations were very high.
How was this new collaboration with directors Russo Brothers and VFX Supervisor Dan DeLeeuw?
SP: Framestore had previously worked with the team on INFINITY WAR. Being such a big project the work was divided up between multiple supervisors on the client side. We mainly had reviews with Mårten Larsson, and our reviews were always fun and entertaining. It’s good to work with a team with a great sense of humour.
How did you organize the work with your VFX Producer?
SP:We broke our crew into 3 teams to concentrate on different sequences and aspects of the work. We named our teams Hulk, Rocket and Nebula – it lead to some friendly competition between the teams.
How did you split the work amongst the Framestore offices?
SP: The work was mostly done in London with some prep work done in partnership with Anibrain, in Pune.
What are the sequences made by Framestore?
- We did a couple of shots with Rocket and Nebula when Tony first lands back on Earth. These were Rocket and environment extension shots.
- Thanos’ Yurt – shots of Rocket turning over the gauntlet and reacting to the events.
- Holograms – Captain Marvel, Rocket, Rhodey and Nebula talking with Nat as holograms – includes Captain Marvel suit replacement.
- Hawkeye testing the Quantum gate and travelling to his house. Hangar extensions, Quantum gate and time travel effects and Hawkeye’s Quantum suit.
- The Time Travel testing scene with Hulk and Antman testing the quantum gate in the van. The work was Smart Hulk, hangar/environment extensions and replacements, quantum van effects.
- Infinity Stone Planning – The team planning the time heist – Hulk, Rocket and Nebula, including Hulk eating ice cream!
- Back to the Past. The Team in their quantum suits walking through the hangar to the quantum gate and activating the gate. Quantum suits, hangar/environment extensions and replacements. Including the manifestation of the suits helmets.
- A single shot of Rocket & Hulk flying the Benatar into New Asgard.
- Asgard – Rocket in Asgard with Thor and the Asgard environment including the establisher and environment extensions. Also a couple of suit manifestation shots.
- Time Travel Return. – Returning to the Avengers hangar after the Time Heist – Hulk and suit de-manifestations, gate effects and hangar/environment extensions.
- Evil Nebula in the Hangar – Evil Nebula activating the quantum gate and Thanos’ ship smashing through the hangar roof.
- Asgardians of the Galaxy – at the end of the film when Thor teams up with the Guardians. Rocket, Benatar (exterior and interior extensions), Nebula fixes, Thor eye.
- Wakanda – shots of Wakanda at night at the end of the film.
- Tony holograms at the funeral.
How did you work with the art department to design Smart Hulk?
SP: The design of Smart Hulk was well developed when we started as ILM had been developing him for some time, so we were largely matching to existing turntables. Later in the day we received concept images of Hulk in 6 different costumes (plus the Quantum suit). We studied the costumes and brought real clothing that match for modelling, look development and movement studies.
Can you explain in detail about the creation of Smart Hulk?
SP: We started with a base model, textures and some face shapes from ILM who had been developing Smart Hulk for some time. To work in our pipeline it required additional work to refine the textures and displacement map and also a groom to match the reference.
How did you handle his rigging and animation?
Max Solomon, Animation Supervisor: We modelled hundreds of very specific facial shapes to help us reproduce every nuance of Mark’s performance. But often we couldn’t be literal with the lipsync. Somehow it felt over articulated when Hulk’s lips matched Mark, perhaps because they are much bulkier. In these instances we had to reduce the articulation without losing expressiveness.
SP: We also created a full dynamic muscle system for the body, it was important that we could see his muscles tensing underneath his clothes. We put a lot of work in to a new facial rig, pushing the level of the rig beyond anything we have done before to be able to get the subtleties of performance required.
Can you tell us more about his animation and especially his face?
SP: We used mocap of Mark Ruffalo’s performance on set as a starting point for Hulk’s body animation. It required considerable work by the animation team to adapt the performance to account for Hulk’s weight and different proportions. For the facial performance, we used machine learning to quickly generate first pass animation from the head mounted cameras Mark wore on set. The shots were then hand animated to achieve the nuanced performance that contained the essence of Bruce Banner. This involved painstakingly scrutinising the footage of Mark’s performance to try to reproduce every detail.
Finally we added a layer of micromovement extracted from the machine learning solves. The animators had control over the amount to micromovement that could be added so that it would enhance the performance.
How did you create the various shaders and textures?
SP: We received a package of textures from ILM which we adapted to work with our proprietary shaders that we use with Arnold. Hulk’s skin was quite complex with multiple layers through to an underlying subsurface layer influenced by Banner’s skin tones under the outer green Hulk skin.
What was the main challenge about Smart Hulk?
SP: The biggest challenge was his facial animation, giving him Mark Ruffalo’s performance while keeping the essence of Hulk. Smart Hulk had to give a wide range of emotions from deep sadness to humor, both which required subtlety and careful timing.
Max Solomon: For me staying “on model” was really tough. The face rig had so many controls it meant that there were thousands of possible combination expressions and it required considerable discipline and rigour to keep Hulk looking like Hulk. It was very easy to make him not look like himself and this was always very subjective as we were not exactly matching Mark.
How did you enhance your Rocket model?
SP: We started from our asset from GUARDIANS OF THE GALAXY VOL 2. But made adjustments to his groom to match changes that had been made during INFINITY WAR. We also built an entirely new costume – his flight suit. We updated his eyes to give better caustics, wetness and highlights for the close up performance moments.
Can you tell us more about the creation of his fur?
SP: Framestore has a robust fur pipeline from grooming tools to fur shaders to fur dynamics solvers that deal with groom collision. For Rocket we started with our asset from GUARDIANS OF THE GALAXY VOL 2 and updated his groom to match some adjustments made for INFINITY WAR and also made adjustments to work with the new flight suit and quantum suit costumes.
Can you explain in detail about the design and creation of the Avengers suits?
SP: The look and design of the suit wasn’t decided on until long after the shoot so there was no on-set reference or practical suits built. We started with concepts provided by Marvel. The suits were developed in parallel by multiple vendors and models and turntables were shared and the design refined. Since we had the most close-up shots the design of the upper part of the suits and the shoulder epaulettes in particular was driven by our shots. Once we had locked down most of the design elements – which were done on Captain America, we had to create custom suits per character to fit their individual body shapes. There was quite a range of body shapes – Captain America, Rocket, Thor, Black Widow, Nebula, Tony Stark, Hawkeye, Hulk, Ant Man and a special case for War Machine, where we created a version of his armour that matched the materials and patterning of the rest of the team’s suits.
Can you tell us more about the shaders and texture of the suits?
SP: We constructed the suit from a range of materials. The white parts of the suits were designed to be a stiff cloth material with an armoured feel and the black material was a more flexible carbon fibre. The trick was to create materials that would hold up over wide range of sizes in frame, while still maintaining a cloth look. The cloth dynamics settings on the different materials had a large impact on the way the different parts of the suit looked and moved.
How did you handle the lighting challenge?
SP: The lighting on the suits was matched to the lighting on the characters. In particular we rendered digi double heads for the characters and matched the lighting to these. This also gave us the option to you digital necks on the characters where they were covered by their original costumes. It also gave us interactive shadows between the suits and the actors. An optimised version of the hangar geometry was used to help light the characters in the hangar – so lighting the hangar to match the plate gave a good basis for the lighting on the characters.
Can you tell us more about the matchmove work?
SP: The suits required very precise body tracks, especially around the necks. The tracking rig contained a number of soft mods that allowed for precise neck tracking. The aim was to keep the performance of the actors in the plate – though some adjustments had to be made to account for the difference between the costume they were wearing and the structure of the suits.
How did you create the Quantum Gate?
SP: We used the look of the Quantum van as a basis for the Quantum Gate. The van was based on the FX in Ant Man and the Wasp and was a combination of 2d and 3d FX. On set there was a partial build of the Quantum Gate, though in a lot of shots we replaced it completely with a CG version. For the gate opening effect we deformed the floor of the gate to form a long tunnel. We rendered a lot of different passes of lighting and energy passes for compositors to create the look. Comp elements were passed back to create interactive lighting passes for the suits going into the gate.
Can you tell us more about your environment work and especially the hangar?
SP: We did a range of environment work on the film, from an Asgard establisher and window views, to Wakanda night cityscapes, to the Avengers hangar and compound. The hangar was shot on location however it was largely covered in green screens. We built the complete hangar and dressed it with a large number of props and vehicles to make if feel lived in. For the window views we added the CG exterior Avengers compound and replaced the window frames and glass. For a lot of shots the only things remaining from the plate were the actors and the set dressing around them. The majority of the hangar was CG. We developed the asset to a level that made if easy to just drop in the camera for a shot and render it. In comp we also had a template that kept it consistent and made it easy to make global adjustments to look across an entire sequence.
Which sequence or shot was the most complicated to create and why?
SP: The time travel testing sequence was the most complicated as it involved some of the more difficult closeup Hulk performances, often with a full CG environment and the quantum gate van FX.
Is there something specific that gives you some really short nights?
SP: Some of the Hulk facial performances were tricky.
What is your favorite shot or sequence?
SP: I can’t choose. So many good shots, some of my favourites aren’t heavy VFX shots. Some of the Thor shots with Rocket in Asgard or with the Guardians in the Benatar. They are great performances and kept us entertained.
What is your best memory on this show?
SP: Marvel sent us some Infinity Gauntlet doughnuts to keep our energy up!
How long have you worked on this show?
SP: 9 Months
What’s the VFX shots count?
SP: 301
What was the size of your team?
SP: 310 people worked on the show about 210 working together at the peak.
What is your next project?
SP: A BOY CALLED CHRISTMAS.
A big thanks for your time.
WANT TO KNOW MORE?
Framestore: Dedicated page about AVENGERS: ENDGAME on Framestore website.
© Vincent Frei – The Art of VFX – 2019
