In 2014, Kelly Port explained to us the work of Digital Domain on MALEFICENT. He then worked on FREE STATE OF JONES and BEAUTY AND THE BEAST. He is back in the MCU with AVENGERS: INFINITY WAR.
SPOILERS! MAJOR KEY ELEMENTS OF THE STORY ARE REVEALED BELOW!
// From Kelly Port, VFX Supervisor
How did you get involved on this show?
Marvel asked Digital Domain to participate in a test to showcase how we would tackle creating Thanos with Josh Brolin. We ultimately were awarded the majority of the sequences with Thanos. The scenes in Knowhere were some of the first shots we worked on.
What was your feeling to be back on the MCU?
I am such a fan of the entire Marvel Universe, and what Kevin Feige has been able to do with these amazing narratives and storylines. It was awesome being a part of it all, especially with this film. My initial excitement was definitely in being asked to participate in the test. We were all eager to explore the new technology that we have been researching.
How was the collaboration with directors Russo Brothers and VFX Supervisor Dan DeLeeuw?
It was an absolute career highlight to be able to collaborate with directors who focus on character building and story lines motivated largely by these characters. Since Digital Domain was tasked with creating Thanos in his most subtle and dramatic moments, we immediately had a level of confidence that what we were dreaming could be pulled off, and balanced with the rest of the characters.
How did you split the work amongst the Digital Domain offices?
Digital Domain studios in Vancouver and Los Angeles both played a role, with about 70% of the work coming from Vancouver.
What are the sequences made by Digital Domain?
Thanos character throughout the following sequences (over 500 shots and over 40 minutes of screen time):
- Ark Ship Massacre
- Knowhere Betrayal
- Meditation chamber
- Torture chamber
- Vormir Stone Quest
- Zehoberi Flashback (young Gamora)
- Waystation
- End Battle in Wakanda with Thanos
- Thanos’ Yurt
(FYI, Thanos during the Titan sequence was done by Weta Digital)
How did you work with the art department?
Production Designer Charles Wood and his team generated a tremendous amount of extraordinarily beautiful artwork, including a lot of conceptual pieces, and even some looks they wanted us to avoid. All meant to generate ideas, and a back and forth dialog with Marvel and the directors. So much energy and passion came from that group. Charlie’s enthusiasm was infectious and definitely made its way through our team and into the final shots of the film.
Which character was the most challenging to create and why?
Without any doubt, Thanos was the most challenging to create for two main reasons. First and foremost, it was absolutely critical that the subtleties of Josh Brolin’s performance needed to come through more than anything because he’s such a key character in the film driving a lot of really emotional scenes. Secondly, he needed to look as photorealistic as possible because he was acting against live actors and he needed to believably fit within each scene.
Did you received specific indications and references for the animation?
We used complex motion capture setup while shooting on set and for Thanos, Josh Brolin worked with Terry Notary for guidance on body language and movement. When this motion capture got to Phil Cramer and his animation team, we would need to make certain adjustments to compensate for Thanos’ larger size and weight. For the facial animation, our primary goal throughout the process was to maintain the highest fidelity in maintaining Brolin’s performance as was technically possible.
How did you handle their various skins shaders and textures?
Our primary look development for Thanos was handled by Jeremy Buttell and Chris Nichols. They really are masters of their craft. Jeremy kicked off the early development of the test and the early stages of production, and Chris was able to shepherd the character through the remainder of the production. Additional work on Thanos and our many other characters were handled by our look development lead Fernando Brandao. They were all able to effectively balance all of the models with incredibly detailed textures and novel shaders. This would then be tested and ultimately handed off to our lighting department.
The characters have many different skin colors. How does that affect your lighting work?
As we began to experiment with lighting ideas, we soon discovered that Thanos looked his best with slightly more complex lighting with various colors rather than monochromatic light. Our CG Supervisors Hannes Poser and Martin Johansson worked with our lighting lead Chris Rickard to find really this balance. I worked closely with Kym Olsen, our look development lead, and we were able to quickly experiment with various lighting and compositing ideas that would ultimately feed back into our production lighting. We found that some warmer keys or accents would generally complement Thanos’ skin color. This worked quite well in most of the scenes that we were responsible for, given that they were generally darker and moodier. For the full daylight Wakanda sequence, we had less flexibility, but found that having Thanos walk through the dappled light of the trees was very effective for integrating him into the shot and giving him depth.
How did you share assets with the other VFX studios?
We shared shots with a number of other vendors. It’s always very collaborative working with other studios. Everyone in the industry knows each other, and many of them are former, and if not former, future colleagues.
Many FX are involved on your sequences. Can you tell us more about it?
A shout out to our effects leads is definitely necessary here. Victor Grant & Brett Ellis and their teams were integral to creating all the amazing effects in our portion of the film, from the electrical plasma effects of the gauntlet to Vormir’s exotic atmosphere. These teams were responsible for explosions, fire, smoke, weapons fire and debris for the Gamora flashback sequence as well. They had quite a small team for the amount of work they were responsible for and really pulled it off!
Which sequence or shot was the most complicated to create and why?
The most complicated sequence was the scene that takes place on Vormir where Thanos sacrifices Gamora. Many of the shots of Thanos in this sequence are very close and very emotional, so it was extremely important that Brolin’s performance come through for this scene. We also had to ensure that the environment as a whole all fit together and complemented the dramatic nature of story as it played out. The environment included a wide variety of photographic and digital elements that needed to be seamlessly blended together, including aerial plates shot in Brazil and Iceland, the practical set build, all combined with a stormy digital atmosphere and mountain environment.
What is your favorite shot or sequence?
This is such an impossible question! There are so many! Each one had its own unique qualities that made it special. I would have different answers to this depending on successful completion of technical challenges which is certainly rewarding, or perhaps from a film and story perspective. From that angle, I would say that one of my favorite scenes, was a very small but important scene that occurs immediately after Thanos snaps his fingers. We called this the metaphysical Waystation. It took us a while to discover the look of this, but ultimately ended up with this abstract, dreamlike environment that really served what I think is a quite brilliant dramatic pause in the story. It allowed Thanos (and the audience) to take a moment to reflect on what he had just done. I think having this moment makes the subsequent scenes (seeing the effects of the snap) all the more effective. There were incredibly subtle performances here by Brolin and so we had to make sure those micro expressions and subtleties came through to Thanos. It’s a short scene, but definitely one of my favorites.
What is your best memory on this show?
I’ll share two memories that nicely bookend my experience on this project. The first was early on after we had completed our character test for Thanos. It was the first shoot day for Brolin, and Dan DeLeeuw, the Marvel executives, the Russo brothers were all in a trailer showing Josh Brolin the test for the first time. They were all very excited about it, and I think there was a collective sigh of relief that we could pull this off. We had put so much pressure on ourselves to make this initial test the best it could be. Thanos ultimately had more than 40 minutes of screen time as the main character, so he really had to carry the film. It was a very proud moment.
More recently, the excitement of the audience’s reaction seeing the film for the first time was really, really cool for me. We do it for the love of the craft, but we also do it for the love of entertaining. The reception from the audience is so rewarding. It means a lot.
How long have you worked on this show?
Almost two years.
What’s the VFX shots count?
513 Finaled shots.
What was the size of your team?
Just over 340 individuals including artists, supervisors, facility and production support.
// From Phil Cramer, Animation Director
Can you tell us more about the animation?
Traditionally in visual effects, there has been a big visual jump between animation & the actual fully rendered submission. That process often leads to kickbacks as unexpected results might occur,which in turn could make it challenging for filmmakers to make the best creative decisions and give animation approvals in confidence. Animation approvals can become a leap of faith for the filmmakers, as they have to trust that the play-blasted performance will still work down the line.
To avoid this, we created a new process at Digital Domain called AnimViz. Within animation, we are rendering all submissions via GPU accelerated rendering that in turn gets comped with temp versions of all downstream departments. This drastically improves the overall look and introduces a seamless progression between submissions. It gives the filmmakers the needed confidence in the performance, because they can view something closer to final product at a much earlier stage. On AVENGERS: INFINITY WAR, this ended up being so successful, that we will embrace this for all projects moving forward.
// From Eric Scott, VFX Producer
How did you organize the work with your VFX Producer?
From the beginning, we wanted to make our animation deliveries more efficient, in a way that has never been done before. Kelly, Phil and our DFX Supervisor Scott Edelstein had a plan to work in “groups” of shots that were selected from each sequence that made the most sense to work on concurrently and in continuity. We would group several shots of animation and would present them to the client as a mini-cut. These mini-cuts would always be kept together, until animation was approved on all of the shots in the group. If we had an update to one of the shots in the group, the whole group would be re-submitted for review.
This process proved to be a little challenging to schedule on the production side of things but made a whole lot of sense from the creative side. My production team did a great job adjusting to this new process and learned to schedule shots in a whole new way. It ultimately ended up being very successful and allowed the directors and Marvel to see the shots as they would play in the movie, not just as stand-alone shots. It gave everyone a better sense of clarity on how the animation would work, across cuts, and in continuity.
// From Darren Hendler, Head of Digital Humans
Can you tell us more about the face animations?
Digital Domain created an all new two-step system to handle the facial work: Masquerade and Direct Drive. These two process work in unison to create the highest quality creature facial animation from an actor’s live on-set performance.
Masquerade is an all new proprietary application that allows us to capture facial performances with more flexibility and fewer limitations than anything we’ve seen or done in the past. Masquerade has dramatically improved the quality and the subtlety of what we are able to capture from an actor.
Essentially, we take frames from a helmet-mounted camera system, and create a high-resolution, and accurate, actor face scan. Masquerade uses machine learning to take previously collected high-res tracking data, and turns the 150 facial data points taken from a motion capture session into roughly 40,000 points of high-res 3D actor face motion data.
Here’s how it works.
Training data collected from high-resolution scans (e.g. USC’s ICT, Disney’s Medusa, Digital Imaging) show exactly what the actor’s face is capable of doing through a regiment of facial movements. This allows the computer to see many details including how the actor’s face moves from expression to expression, the limits of the actor’s facial range, and how the actor’s skin wrinkles.
Then, we do a motion capture session with the actor during their performance. They wear a motion capture suit with a helmet-mounted camera system, and perform live on set with their fellow cast. During this session, we are able to do body capture and facial capture of the actor at the same time. Prior to Masquerade, high-resolution, and accurate, data for the actor’s face from their live performance alone was impossible.
The second process in the system, Direct Drive, takes that data from Masquerade and transfers it to the creature–in the case of AVENGERS: INFINITY WAR, from Josh Brolin to Thanos–by creating a mapping between the actor and the creature.
The mapping includes defining a correspondence between the actor and the character, including how different elements of each unique anatomy align. Direct Drive then figures out the best way to transfer Brolin’s unique facial performance to Thanos’ unique face. During Direct Drive, we transfer a range of performances and facial exercises from the actor’s face (Brolin) to the creature’s face (Thanos). We have the opportunity to modify how it transferred. This is also a huge moment for Digital Domain to add a level of additional control to ensure that the performance is portrayed as accurately as possible on the character.
For animation, we then built a specialized rig that transfers as much of the performance as possible to animation controls. It still maintains all the high-res data already captured and ensures that we do not lose any of the subtleties of the actor’s performance. Then it gets passed on to the animation group–led by Animation Director Phil Cramer–to do the fine-tuning and make any director-driven changes.
A big thanks for your time.
// WANT TO KNOW MORE?
Digital Domain: Dedicated page about AVENGERS: INFINITY WAR on Digital Domain website.
© Vincent Frei – The Art of VFX – 2018
Great interview!
The work on Thanos is very well done.