Jeff White joined ILM almost 10 years ago and has participated on projects like VAN HELSING, STAR WARS EPISODE III, TRANSFORMERS: REVENGE OF THE FALLEN or TRANSFORMERS: DARK OF THE MOON. He received a VES Award for Outstanding Visual Effects in a Special Venue Project for TRANSFORMERS: THE RIDE 3D.
What is your background?
I grew up where the cows roam in upstate New York and went to Ithaca College for Cinema and Photography. From there I got my MFA from Savannah College of Art and Design before starting at Vinton Studios (now Laika) working in commercials. I started as a Character TD at ILM doing rigging and simulation work. It’s a great department to start in because it’s right in the center of the pipeline.
How was the collaboration with director Joss Whedon?
I’ve been a huge Joss fan for a long time and feel very lucky to have been able to work with him. He’s amazing at working with actors and there was a natural extension to working with our animators. He brought a fresh perspective to the work and was sure to let you know when he was happy the shot was heading in the right direction or if we’d missed the mark he would offer very specific feedback to get us back on track. All the humor that comes through his writing in the films is a part of working with him everyday. The entire crew looked forward to shot reviews with him.
What was his approach about the VFX?
The great thing about working with Joss is you know that he’s going to keep the visual effects working for the story and character development and not the other way around. He came up to ILM when we were in the trenches of character development on Hulk and had a very in-depth discussion about who this Hulk is, how he moves, literally what his motivation is. He showed us comic book reference of poses he liked. It was enormously helpful for the entire team and the animators at ILM did an incredible job interpreting that into the character on screen. Joss was in our shot reviews with the rest of the great team from Marvel, it was a true collaboration. He was able to identify the big picture items a shot needed and let us work through the details.
Can you tell us more about your collaboration with Production VFX Supervisor Janek Sirrs?
We were very fortunate to be working with Janek on this project. He was deeply involved in the previz work for the film which we used extensively in planning our New York City photography shoot. He has a great eye for the work and brought a wealth of knowledge from his experience on IRON MAN 2. He was great at keeping the various VFX houses working on the film on the same page with the extensive amount of asset sharing and intercut sequences that we had. I look forward to working with Janek again in the future.
Can you tell us more about the Avenger’s Jet creation?
The Quinjet was one of the first models that we started on. We had an excellent design from the Marvel art department to start with as a reference. We did several rounds of texture work on the completed model to ensure that we had a very natural feel to the weathering. We almost always positioned a light to get a nice specular roll across the wings to bring out the breakup in the maps. One of the great features is that it can go into hover mode and actually change its profile to look more like a bird of prey. On location, we used a helicopter as a stand-in for the jet to give the camera operators something to frame up on as it landed and to give our animators something to reference for flight dynamics.
How did you create the Helicarrier?
The Helicarrier was by far our largest asset on the show and was built by Rene Garcia and painted by Aaron Wilson. We started with a design from the Marvel Art department. Once we had the major forms correct, we started into all of the detail work. The Helicarrier is seen from almost every angle and each time we’d start a new shot we needed to add additional geometry and texture detail. We shared the asset with Scanline VFX and Weta. As each of the vendors would add damage or additional detail for a given shot, they would send it back to ILM to be folded in for continuity. We spent most of our time working on details that we’d pull from photos of aircraft carriers like the spec breakup on the hull or waterline staining of the paint. We added carrier catapult launch strips, arrestor cables, blast doors, moving vehicles and digital double crew on the set, all with the goal of selling the scale.
Can you tell us more about the asset sharing with the other vendors?
We had several assets that were required by other vendors for their sequences. For instance, we built the Helicarrier and sent it to Weta. They created all of the damage when Hawkeye blows up engine 3 and passed it back to us for Helicarrier shots we had which occur later in the film. It was a very smooth process and Janek made sure all of the vendors were staying on the same page with the look of the assets.
On this show, Tony Stark got his own Tower. Can you tell us more about it?
Like the Helicarrier, Stark Tower is seen from a variety of angles and needed to be created at a variety of resolutions. We used it for wide shots but the profile was significantly narrower than the Metlife building it was replacing in the plates which resulted in a lot of reconstruction work on the backgrounds. Additionally, we had numerous scenes that took place on the balcony outside Tony’s apartment that required a separate high resolution asset in order to seam together perfectly with the set piece that was built and shot in New Mexico.
Can you explain to us step by step the creation of the impressive shot showing the suit-off process?
Each film with Iron Man seems to add a new, cool way for him to de-suit and THE AVENGERS is no exception. Joss’s idea was that Robert wouldn’t have to be locked into the machine; instead it would work around him to take the suit off so that he walks naturally and never breaks his stride. We started the shots with a very tight Imocap track on Robert that we’d use to constrain the Iron Man suit to and use for shadowing. We worked out how much of the suit was removed in each of the shots and then Michael Easton and Bruce Holcomb did several rounds of adding secondary animation of arms moving, mechanisms turning, etc. We ended up replacing the entire walkway that he’s moving down so we could animate the floor opening up as the “carwash” (as we referred to it) moves with Tony Stark. The backgrounds are all constructed from nighttime photography captured from the top of the MetLife building with moving traffic that we recorded on Canon 5D Mark IIs while we were up there.
How did you improve the Iron Man model and animation for this show?
In addition to fleshing out the design and building the under-suit, for THE AVENGERS we had an opportunity to add a bunch of new gadgets to Iron Man’s Mark VII suit. The most significant change was the addition of the jetpack. Joss wanted Iron Man to be able to hover and fire without always having to engage his hand RT’s. This freed up the animators to come up with some great new poses while he’s in the air, especially when he confronts Loki after suiting up for the first time.
Can you tell us more about the shot in which Tony Stark put his new armor during a free fall?
Expanding on the idea of more complex ways for the suit to be put on Tony Stark, Joss built a sequence around the idea that the new Mark VII suit could fly on its own and save him as he plummets down the side of Stark Tower. Nigel Sumner supervised our team at ILM’s Singapore studio on the sequence and like the carwash, we had to work out how much of the suit would be deployed in each shot to save him in the nick of time while keeping the duration of the fall believable. Foreground plates were shot with Robert Downey Jr. on a wire rig, though in several shots we added CG legs with cloth simulation so that we could get more movement and wind flutter in the pants. The backgrounds are constructed from our Metlife building photography while Stark Tower was a CG asset. We also added a bit of flutter to Tony’s cheeks, atmosphere and objects close to Stark in the shots to help maintain a good sense of speed.
Did you create some procedural animation for the Iron Man armor to help your animators?
No, each shot ends up being so customized to the given camera angle and cut that it’s quickest to do it by hand. Michael Easton, Keiji Yamaguchi and Bruce Holcomb did all of the animation for the suit up.
How did you create the digi-doubles for the super-heroes?
We did Lightstage and Mova capture sessions for all of the digital doubles knowing they were going to have to hold up close to camera. We also had full body scans from Gentle Giant and our own photography shoots of the costumes to work from. All the time and hard work the crew put into building those the assets paid off when we needed to switch a number of shots from plates to digital doubles to achieve more dynamic camera moves.
About Hulk, how did you create this new Hulk and rig it?
We were fortunate to be working with Mark Ruffalo, who went the extra mile in partnering with us in creating this Hulk. One of the things that really works for this latest incarnation of the Hulk is the integration of Mark into the design. Every bit of Hulk stems directly from Mark, from the pores on his skin, to the grey hair of his temples, right down to using a dental mold of Mark’s teeth as a basis for Hulk’s teeth. Our strategy was to work out rendering and texture issues on the Banner digital double until it looked indistinguishable from Mark Ruffalo.
Because our Banner and Hulk models shared the same topology, we were able to transfer textures, material settings and the his facial library for animation. This gave a decent base to start from but with their significantly different proportions, there was a lot of retargeting work that needed to be done. We typically try to be economical with our poly counts but with Hulk we made a conscious decision that he was going to be extremely dense in his resolution. That way we never came up short on resolution for all of the close-ups and detailed shape work that was required to represent the anatomy under the skin. We then invested in a robust multi-resolution pipeline so that he was manageable for the artists to work with.
As the cut was coming together, Joss and Mark came up to ILM and Mark did a performance for every Hulk shot in the film in a full Imocap suit with integrated facial cameras. Joss was able to make selects on that Imocap performance data which we would apply to the Banner digital double to verify the accuracy of our capture and then re-target the animation onto Hulk. This gave the animators a great base to work from but there was a tremendous amount of work required to get the performance to read correctly and the weight of Hulk’s movement right. After animation, we would run three layers of muscle and skin simulations to get the dynamics and slide of real skin. There was a coarse tetrahedral mesh sim for large-scale ballistics, and then cloth and thin walled flesh sim on top of that for accurate slide and wrinkles. Additionally, there was extensive per-frame anatomy work done by the modelers as needed to make sure he was exactly right.
Can you tell us more about the amazing shots in which Hulk is chasing The Black Widow destroying the lab and all that in slow-motion?
That shot was supervised by my colleague and our associate VXF Supervisor on the show, Jason Smith, and was a great chance for us to do our version of the slow motion Olympic sprinter and really show off all the layers of dynamic simulation. On set, Dan Sudick, the special effects supervisor, had built piece of modern art metal sculpture roughly in the shape of Hulk that he pulled down the hallway to get all of the great destruction and interaction with the environment. Hulk was then animated to roughly match the pace of the mandrill and several passes of CG glass, debris and sparks were run to integrate him into all of the practical destruction. Black Widow was later composited in from another take.
Can you tell us more about the ILM Imocap system and what improvements you made to the technology?
For this film, Ronald Mallet and our engineers improved our solvers and supplemented the patented pattern bands with a geometric pattern that was screen printed onto the suits. Combined with the data we get from set this helped us get even quicker and more accurate solves on the motion.
About the New York final sequence. What was the real size of the sets?
There was a 300’ stretch of the Viaduct built as a set in New Mexico with 40’ green screens on each side and dressed with damage and damaged cars. We built the city around the set by shooting nearly 2,000 tiled spheres, akin to a high resolution Google street view. Using those photos the digital environment was spearheaded by Andy Proctor and David Meny. We had to paint out all of the cars, trees, people, streetlights, and anything that needed to parallax as the camera moves and replace them with CG assets. Using a custom shader we developed we ended up replacing every single window with a dynamic CG version that would take into consideration the appropriate reflection, add a window blind and randomly choose a room interior from a library we had built which would change perspective with the camera. We then built a library of 190 assets with thousands of variations to dress our synthetic New York streets including cabs, police cars, street lamps, awnings, and sandwich boards, even hot dog carts. For the viaduct we rebuilt the Pershing Square café as well as replacing the Metlife building with Stark Tower. To avoid having our rendered photographic environments from appearing too static we used the battle at the end of the film to introduce smoke, dust, debris, embers and ash to add texture and movement to the shots.
How did you create the huge New York environment and also what was your preferred method of destruction?
The flying shots were created using the same techniques as the Viaduct but required extensive planning for the photography shoot to make sure we had access to the right vantage points from building rooftops. The animators had representations of our photo spheres in their Maya scenes which they used as a guide to make sure our flight paths didn’t stray too far off nodal where the photography would break down.
Once all of that was built, we added the destruction from the alien invasion by adding damage patches onto the buildings. In cases where the camera was moving we built full 3D damage sections we could splice into the buildings. The Leviathan is a massive winged creature that doesn’t quite fit down a New York City street, so there were many opportunities for building destruction simulation which we did with a combination of rigid simulation of building debris with effects simulation for dust, glass and fire.
How did you design and create the Aliens and the Leviathan?
The original design for the aliens came from the Marvel art department and ILM’s VFX Art Director, Aaron McBride, did the final design for the Leviathan. They shared similar themes of gold armor and purple lights to connect them to their home world. In shots, however, the gold was too vibrant so we played up the patina on the metal and added battle damage and weathering.
What were the challenges with the Leviathan?
Selling scale was the big challenge. Marc Chu, ILM’s Animation Supervisor found the right balance of speed and a subtle swimming motion to keep them dynamic. The Leviathans serve as a transport for the foot soldiers and it was a real challenge to get that to read amidst all of the chaos. We ended up adding explosions, cables and goo as the foot soldiers burst out to help them read on screen.
Can you tell us more about the way you choreograph the numerous fights on the final sequence?
Joss worked with the stunt team to choreograph the fights on set. This was really helpful where the Avengers needed to interact directly with the Chitauri. As we moved into shot production we made changes to the animation to have the fight work better for the cut.
Most of the shots in New York involve an impressive number of elements. How did you manage these on the pipeline side in order to finish the sequence within the deadline?
ILM has an extremely robust production pipeline and asset management system. Set dressing New York City was split between our Digimatte and TD groups because the volume of work was so large. Ryan Martin wrote several tools that allowed us to dress in the traffic jam, randomizing the layout per block so there was enough variety to keep it from being repetitive. The key was to build as much movement back into the scenes as we could to overcome the static photography. We added driving traffic, parking cars, digital doubles, cyclists, sun reflections, boats on the water, anything we could find to add some life back into the scene.
Can you tell us more about the impressive continuous shot that shows each super-heroes around the city and ends on Hulk and Thor?
The idea was to have a continuous shot showing all of the Avengers working together, it’s a really turning point in the film where they set aside their differences and work together as a team. We shot a plate for Black Widow, Hawkeye and Thor, the rest of the shot is created in CG. We used our photography as a base and then spliced together several New York City streets to create a run that was long enough to sustain the shot. The animation and camera layout took months because there were so many interconnected pieces. After that, we added a huge number of elements and simulations to bring it all together.
Can you tell us more about the use of ILM’s Plume for this show?
Plume was used extensively in our effects work. Being GPU based, it has a very fast iteration time, which is key to developing the look of explosions and debris. On THE AVENGERS, we added deep compositing output into Plume which gave us the flexibility to make tweaks to the animation or rigid sim without having to rerun all of the effects passes.
What was the biggest challenge on this project and how did you achieve it?
The most challenging part of the project was getting the performance and look of Hulk just right. There were many technological advances that helped that happened but in the end it was a very dedicated and passionate team of artists that worked tirelessly to make him a believable character and bring him to life on screen.
Was there a shot or a sequence that prevented you from sleep?
The tie-in shot was pretty daunting and went right down to the wire. As far as loosing sleep, I’ve got one year old twins so they take care of the sleep deprivation department!
What do you keep from this experience?
It was an amazing experience, I feel truly lucky to work on a film with so many talented artists, great visual effects challenges and also be this caliber of a movie. The chance to collaborate with Joss, Mark and the entire team at Marvel is one I would jump at again. The fact that it’s been so well received critically is really a testament to Joss’ vision for the film.
How long have you worked on this film?
About a year.
How many shots have you done?
Just over 700 shots.
What was the size of your team?
Roughly 300 people.
A big thanks for your time.
// WANT TO KNOW MORE?
– ILM: Official website of ILM.
© Vincent Frei – The Art of VFX – 2012
