AVENGERS – AGE OF ULTRON: Ben Snow – VFX Supervisor – ILM

Ben Snow joined Industrial Light & Magic in 1994. He has worked on films such as TWISTER, JURASSIC PARK: THE LOST WORLD or THE MUMMY. He then became VFX Supervisor and take care of the visual effects of movies like STAR WARS EPISODE II – ATTACK OF THE CLONES, IRON MAN, PIRATES OF THE CARIBBEAN: ON STRANGER TIDES or NOAH.

What is your background?
I studied computing and film at Canberra University in Australia and started in the industry working for a boutique computer graphics house in the UK as a runner, then worked in London and Sydney on commercials and program openers before joining ILM in 1994.

How did you and ILM got involved on this show?
I worked on the first two IRON MAN films and so I knew the team at Marvel, and ILM has worked on several other Marvel projects. Chris Townsend, the studio side supervisor told us he felt that only a couple of VFX houses in the world were capable of the sort of work that was needed for the Hulk and Ultron and he was talking with both of us.

How was this new collaboration with Production VFX Supervisor Christopher Townsend?
Chris is ex ILM and we’ve worked together before and I think he felt the ideas I brought forward in discussing what we could to to improve the Hulk were good. As with every show of course we needed to get to the right price as well, but there was a level of trust and creative agreement between Chris and I that stood us well throughout the show.

That’s your third Marvel movies. How did you approach this new project?
I was comfortable dealing with the studio and bringing ideas to the table. Our animation supervisor Marc Chu, producer Jeanie King and co-vfx supervisor at ILM Philippe Rebours have all worked with Marvel before and so again there was a degree of comfort and trust. We know they like us to be bring forward ideas and help them solve problems and that’s very empowering. We also know that Marvel films have a tendency to evolve during post-production so to be prepared for changes, and to get extra material while we’re shooting to cover completely new shots and even pieces of action that might come up – so we do our best to be prepared for any changes after shooting wraps.

How was the collaboration with director Joss Whedon?
This is my first time working with Joss and I enjoyed it immensely. Joss was in the calls where we discussed shots with him and the studio, and open to ideas on set and during post production. He had a very clear idea of what he wanted but also reacted well to new ideas and suggestions. He’s very smart and well read and constantly drops references to film, theater and literature – it gets hard to keep up.

What was his approach about the visual effects?
He regards it as an important part of the picture but doesn’t let the film get overwhelmed by the effects. He was more concerned with animation and performance than lighting and simulations, but interested in it all. When it comes to performance he was very specific, working closely with Marc Chu on performance issues, just as he would with the actors on set. He had a lot of previz and conceptual art as a starting point and was clear on what he loved versus what he wanted us to explore further.

What is the work done by ILM?
We did the opening tie in shot where all the Avengers show their powers during the approach to the Sokivan Fortress where Strucker is working. Also early scenes with Hulk and Black Widow and Tony’s dream in the Leviathan chamber in the beginning of the film. We created the Avengers Tower exteriors in New York and built the Avengers Quinjet. We created Ultron Prime and did the vfx work for the encounter between Ultron Prime and the twins in the Church and in the Fortress, then the encounter between the Avengers and Ultron Prime in the arms dealer freighter. At the end of that sequence Wanda bewitches Bruce Banner which leads into the Hulkbuster sequence where Tony Stark dons the Hulkbuster armor to try and contain the even angrier than usual Hulk. We did all of that sequence. We also did the scenes between Ultron Prime and Dr. Cho in her lab, and then created the Ultron shots for a later scene where he creates Vision. Then we did much of the end battle – all the shots with Hulk or Ultron Prime, plus many of the other battle shots, the wider shots of the Sokovia city ripping out of the ground and lifting off and flying, and many of the closer shots during lift off as well. Then we did all the shots at the end as the Avengers fight Ultron and the city starts falling.

Can you describe one of your typical day during shooting and post?
Before shooting we get the storyboards, previz and breakdowns done by the studio and use those for bidding and also for planning what we’ll need on the shoot. That means that ahead of time we can try and give other departments like camera, SFX and costumes an idea on what we’ll need when shooting and review what elements we’ll need with the assistant director and his team. I also spent some time helping out directing the aerial unit which meant forward planning based on the previz and scrip to come up with shot lists. The goal is to arrive on set with a clear outline of what we’ll need for the actual shots, and anything we need to get photographs and data for to allow for recreating sets and new shots during post-production.

A typical day on set:
Arrive on set with the list of VFX needs for each setup outlined by the ADs (assistant directors) as being what we’d be shooting that day. If possible review it with the ADs and other heads of department then participate of the director’s walk though with us all of what we’d be shooting for that day. Then check in with our VFX on-set team and answer any VFX related questions from the other heads of department or crew. As we start shooting a shot, if there is a possibility of VFX I or Chris would discuss what we need with the ADs, D.P. and even the directors and actor if need be. Some of our VFX team members will take proxy objects (like a Hulk backpack or cutouts of different characters) that we’ll send into the set to help the camera crew line up and compose their shots. Others will be working with costumes to make sure any on-set motion capture suits are set up and photographed, and others will be taking reference photographs of the set.
Once the camera start rolling Chris or I will be around video village with the director to answer any questions and to keep an eye on what’s being shot to avoid unnecessary VFX shots, while the other of us will be onset watching from near the A.D. for any questions from camera, actors or them about what we’re shooting from a VFX standpoint. We’ll have other members of our team standing by with reference spheres and prop pieces and cameras to take HDRI photographs of the lighting and set. When the director has the shot he wants, we’ll allow the actors to step off the set and shoot clean plates and reference passes. Sometimes our skilled on set crew can get HDRI sets between takes but otherwise we’ll shoot those as well.

This goes on through the day. There might be meetings about upcoming scenes as well, plus calls back to ILM where they’re already building the computer assets and starting early shot work. After wrap we’ll often go to editorial and watch dailies (or sometimes before call time, depending on the production). After that is usually the time to review any shots from ILM and do any additional planning for coming days shooting. They are long days and you have to be fast on your feet but its immensely fun, and the crew are all dedicated and professional and we’re all in it together.

A typical day in post:
During post production I come in ahead of dailies and look at the shots that have run overnight to get my thoughts in order. Then we have group dailies (for AVENGERS it was in ILM’s largest theater) where the artists come in and we go though shot by shot the work they submit. We’ll look at the shots and pull up relevant references and cuts of the sequence to see the shots in context if need be. For dailies our Vancouver office is linked in looking at the same shots we’re reviewing in our SF theater and on a “lifesize” video transmission so I’m reviewing their shots and SF shots together. The facilities are so closely linked we’re able to have shots animated here and lit in Vancouver or vice/versa. Dailies usually takes an hour or two – we might review anywhere from 20 to 60 shots. Then we have a cup of tea. Animation dailies usually take place at the same time as TD/FX/Comp dailies so animation director Marc Chu and I will catch up and compare notes later in the morning. Then we have meetings for bidding new work, discussing creative problems, show scheduling etc. We might work out what new groups of shots we want to package up and send the client and possibly could have a call with them with a cineSync to review work previously sent or new work they’re turning over. The supervisors do desk reviews where artists submit takes of shots for further review and we do rounds where we visit artists desks including the modellers and art directors. We’ll have nightlies in the afternoon to provide another group forum. On AVENGERS we had work going on in Singapore which my fellow Visual Effects supervisor Philippe Rebours would review with Singapore’s in-house VFX supervisor Dan Seddon and his team, with Jakub Pistecky supervising animation. Philippe and Jakub would do a remote review of their shots with the Singapore team in the early evening a couple of times a week. Likewise I’d regularly review the work being done by our new ILM team in London, supervised in London by Mike Mulholland.

How did you design and created Ultron?
Ultron Prime’s initial designed by Marvel’s Vis-Dev team under Ryan Meinderding, and then further refined at ILM by our art director Alex Jeager and our lead modeler Bruce Holcomb and painter Tim Odell. Because Ultron Prime’s design needed to be modified to accommodate all of the acting and sophisticated motions James Spader was giving us on set we had an evolutionary process going on between Bruce, rigger Abs Jahromi and animator Mickael Coedel to refine the design to work for the motion. Joss and the Marvel team wanted Ultron to be the “most sophisticated robot ever”. We studied intricate clock mechanisms, robotics, high end engines for general feel and materials.

Can you explain in details about its rigging and animation?
Ultron Prime ended up the the most sophisticated character rig we’ve ever done at ILM, with about 2000 nodes in our proprietary block party tool – by comparison about ten times as many as one of our TRANSFORMERS characters. The main challenge was getting the subtleties and character of James Spader’s on-set Ultron performance onto a character whose face and body made up of rigid pieces of metal. The face alone had 600 nodes of rigging. While the animator could animate the face using the controls they might use with an organic looking creature, Abs rigged it so the shapes they animated would drive the individual plates making up Ultron’s face to slide over and other one another – allowing sophisticated expressions from a rigid base. The same intricate rigging was extended to his body so for example you’d see secondary actions like his chest plates sliding over one another as his pectoral muscles moved. We gave the animators control over these secondary actions and also the ability to animate parts moving inside the body.

How did you handle the performance capture process?
We did initial performance capture rehearsals with James Spader and Mark Ruffalo at The Imaginarium Studios in London with real-time capture so they could study and get comfortable with their characters and to help us with motion studies. ILM created a monster-mirror where the actor could sit in front of a computer and via real-time markerless facial tracking drive an animated version of Hulk and Ultron’s face via our muse facial animation system – so they could get a sense of how the character would look. This was really a rehearsal tool although data from the system can be used to drive the full animated rig as well. On-set we had the actors where ILM Imocap delta tracking suits with our VFX team filming them with HD witness cameras in addition to the shooting cameras. Robby Derry from video-hawks provided facial capture using twin HD cameras on a helmet rig for Ruffalo and Spader. We also did separate optical motion capture sessions with the actors and stunt people in a specially constructed motion capture studio at Shepperton and with our animation team back here at ILM.

How did you enhanced The Hulk from the previous Avengers movie?
It was important to the ILM team and Chris Townsend that we build upon the work done on the previous films to try and make Hulk better than ever. We did this by revamping textures, skin and hair. We revised his facial library to be even closer to Mark Ruffalo’s and we rebuilt him from the inside out, creating a skeleton and taking the sculpt of his outer skin surface and transferring that detail down to the muscle level, so we could improve our skin sliding and muscle flexing and make it more realistic.

Can you explain about its creation?
First off we tackled the even more crazy than usual Hulk from the fight with Hulkbuster. Joss had described this version of Hulk as being almost looking strung out like a Junkie. We made his eyes bloodshot, added Rosacia and made his veins pops. The animators and modelers added a bunch of tension shapes. A lot of the detail we added for angry Hulk were then folded back into our regular model. Creature model supervisor Lana Lan worked with texture paint supervisor Jean Bolte and Look development supervisor John Walker to revamp the model, textures and skin and hair of Hulk completely.

On set, we had a stuntman named Rob De Groot who is a bodybuilder and we painted him green for reference. We filmed him in almost every Hulk shot and also did a separate session where he did a series of exercises devised by our creature model supervisor Lana Lan and creature TD supervisor Eric Wong to show off all the different muscles in his body in various poses. We studied that footage and the way muscles hardened and became more defined through the skin. We also talked to local medical professors about how muscles tense and studied scientific literature on the amounts muscles change length based on tension. In the past we’d used muscle systems more for jiggle and simulation – relying on outer skin shapes for the detail. While the results were pleasing under the old system the system was very labor intensive and required a lot of shape fixes, which could sometimes affect the animation itself. Our rigger for Hulk was Sean Comer and he revamped the muscle and skin solve setup completely. He and Abs Jahromi proposed a more correctly inside-out system and used MRI/CT scan data to model the muscle flexing and determine accurate tissue thickness. In the resulting system muscles are physically solved with tetrahedral finite elements. The muscle tissue is dynamically attached to Hulk’s skeleton and flex data is cascaded into the muscles to drive directional and hardening properties and well as to being out striations and vein detail. On top of the muscles we have a fascia or connective tissue layer which in turn is connected to a subcutaneous solve model driving the skin surface. The result is a render-able surface that exhibits the physiological properties of real skin.

There is an impressive fight between The Hulk and Iron Man with his Hulkbuster. How did you approach this great sequence?
The sequence started with a detailed previsualization from The Third Floor. It was filmed in Johannesburg. The VFX team went in a week early and scanned and photographed the areas and vehicles we were planning to film. Because we had a good previz which the director liked we were able to plan it in detail. We had some practical SFX mainly for crowd interaction but most of the destruction was added later in CG. SFX did rig several of the market stalls to break when Hulk smashes through them and the art department had some aftermath rubble that we dressed in to some of the plates and photographed for reference. I used the previz to plan our aerial shoot and spent several days flying around Johannesburg in a helicopter shooting both material to cover the previs and other shots that might be interesting – this came in handy as the sequence evolved a little in post – we had plates we could use. Our Digital Matte Artist Paul Huston went up and photographed tilesets at different heights for some of the falling shots and also shot tilesets from the tops of buildings that we used as backgrounds for some of the shots. The building under construction that the characters smash down through was added in CG to an open public space behind the Johannesburg town hall. The SFX team were allowed to shoot off large smoke and ash events for the aftermath – which was quite a cleanup job for the location crew afterwards. The sequence was started in San Francisco, and then split between San Francisco and our team in London led by Mike Mulholland and Nina Fallon.

Can you tell us more about the lighting and shaders for the Hulkbuster?
Hulkbuster was built by Howie Weed with materials by Robert Marinic. We based the materials on the Iron Man costumes, but also matched two practical pieces of the Hulkbuster suit that production had for reference – a helmet and forearm.

What references and indications did you received from Joss Whedon for the fight choregraphy?
Joss had worked a lot of the Hulkbuster fight choreography out with the previz crew, and then our animation supervisor Marc Chu and his team pushed it further. For other sequences, some of the choreography came from previs, some was worked out by the stunt crew ahead of time, and some of it we made up during shooting. There was postvis done on the film during the editing process which also helped but quite often the animation team came up with new beats of the fight as we started the visual effects work as well.

How did you created the huge environment and the flying city for the final sequence?
The flying city was one of our largest environment assets ever. As well as creating a real looking city we had to have it tear out of the ground and rise into the sky and feel like it was flying. For the scenes of liftoff I sat down with Mike Balog, our creature TD supervisor, Florent Andorra our FX TD supervisor and Johan Thorngren our generalist supervisor in charge of digital environments. We discussed how we could make the city believable and particularly the destruction as it lifts off. We studied footage of building demolition, earthquakes and landslides as well as photos of earthquakes. In the past we’ve commonly dealt more with single or small groups of buildings getting damaged but this was a whole city. Rigid sim lead Ian Frost came up with a pipeline for taking our generalist city asset created in 3DS Max, fracturing and breaking it in Houdini, then being able to send it to our Zeno proprietary software for FX and back to Max for Vray rendering. In the past we’d have had to make reproduce generalist environment assets planned for destruction as break-apart models in our traditional pipeline which can get expensive but Ian Frost’s toolsets meant we could go straight back to the generalist pipeline. Florent came up with a bunch of shelf plugins to make it easier to set up all the secondary dust and fine debris and Mike’s rigid destruction team did some terrific work on breaking apart the buildings and adding interior detail as they broke apart.

The flying city was supposed to be an Eastern European ex Soviet Bloc type city, mixing older architecture with 70’s Soviet Bloc buildings. The art department found some locations in the Aosta Valley in Italy that had some of that mix and we filmed the street level scenes there and in the UK. Our VFX team took many photographs and scans and for all the aerials I tried to shoot some sort of plate albeit it sometimes just for lighting and materials reference. Johan Thorngren and Daniel Trbovic, the lead generalist artist on the flying city asset, used the scans and photographs to create buildings and rows of houses. We use the aerial footage as a reference for creating the city, and the compositors used it for looking at the atmospherics.

Ultron have an army of Iron Legionnaires. Can you tell us more about this army?
The sub-ultrons was a shared asset originally modelled by our colleagues and Double Negative. We imported the asset and matched their turntable with our own look development. We created variations and different levels of damage. Tim Belsher, our CG supervisor in Vancouver, did the materials development and Vancouver did many of the shots. The wider shots used massive crowds set up by Eric Wong.

How did you manage so many elements during the final battle?
It was a credit to the teams involved. Jon Alexander, our compositing supervisor, worked with his team to make sure all the comp scripts were clearly laid out and organized so multiple artists could work in the same composite script. We’d use the layout team to rough in positions but it definitely came down to groups of artists working together and meeting to make sure they’re on the same page. One of the big shots during the end battle was a shot with camera orbiting the Church interior and slowing down to see each of the Avengers in action. Postviz had done a great job of laying out the basic action, then Alex Lee our Vancouver lead animator worked out a bunch of the finer details, first sketching in different effects via paint-overs and then laying in placeholders for lighting, beams etc. in the animation take as a guide for the other artists.

Can you tell us more about the asset sharing with the other vendors?
This has become a fact of life in larger VFX projects. It can get to be a problem because assets get tweaked like everything else, so you can find yourself waiting for a different vendor to finish off an asset so you can finish your shot. Chris and the studio VFX team worked to minimize that. We’d share assets by packing up the geometry and materials and running a turntable for the other vendors to match. We shared our turntable environments to make it easier, and our artists would talk to artists at the other companies if there were questions. Materials and rigs are usually proprietary so there can be quite a few questions which we all answer the best we can.

How did you split the work amongst the various ILM offices?
Our office here in San Francisco is tightly bound with Vancouver, since they actually remotely tap into our servers here in San Francisco which means we can easily share work with them, but ILM has done a lot of work to make it seamless to transfer work to our other offices in London and Singapore. I’d automatically be able to look at the latest takes of their shots and we can get on the phone and use RV remote or HP remote to review work together. That said, we tried to break the work up in sequences.

Because we were starting up the London office we started by having them take sections of the Hulkbuster sequence which we’d established in San Francisco, but then they went on to do complete sequences like the liferaft/helicarrier rescue and dogfight with Iron Man and War Machine vs. the subs. They also did the lookdev themselves on assets like the liferaft. We have a great teams in all our facilities and we check in regularly. In Singapore they did a lot of the Ultron Prime shots – the freighter sequence where he meets with the arms dealer and the shots with the Twins in the fortress. They also did Tony Stark in the Leviathan chamber and his dream induced by the Scarlett Witch. Our Vancouver team did sequences around the edge of the floating city including a challenging sequence where Thor and Captain America rescue some people when their cars fall off a collapsing bridge. They did a variety of shots in the end battle.

Was there a shot or a sequence that prevented you from sleep?
Ultron Prime kept a few of us from sleep. It was a challenge creating a robot that could be articulated enough to capture the performance of James Spader and still feel like it was made of metal, and it took a long time. Luckily Abs and Bruce did a great job and that with James performance and Marc’s team’s animation led to a compelling character.

What do you keep from this experience?
I had great experiences filming in Johannesburg, Italy and the UK. The cast and crew onset were all great professionals – adaptable and working as a team. But it was still great to come home and work with our crews at ILM, whose enthusiasm and talent made for an enjoyable and satisfying experience. Working on the Hulk was a highlight in that it rekindled my interest in digital humans. We were able to push things forward on Hulk and had a lot of ideas on things we could push even further int he future so its exciting.

How long have you worked on this show?
I first met with Chris and Ron Ames, one of Marvel’s VFX producers, in December 2013 as I was finishing up NOAH, and we started shooting late January 2014 so its been around a year and a half.

How many shots have you done?
We did around 810 shots at ILM. I think we were the biggest vendor but there were around 3000 shots on the movie, I believe.

What was the size of your team?
We were 425 ILM artists and production staff worked on AVENGERS: AGE OF ULTRON over the course of the last year and a half.

What is your next project?
I’m relaxing, doing some publicity and staff reviews for the show and bidding future projects!
Some interesting possibilities but nothing I can talk about.

What are the four movies that gave you the passion for cinema?
My passion for the cinema was initially stoked by a book on Monster movies and so I started out as a horror fan although mostly read about the films more than saw them. The films I found particularly inspiring for getting into Visual Effects were seeing a double feature of GOLDEN VOYAGE OF SINBAD and SINBAD AND THE EYE OF THE TIGER at my local cinema in Queanbeyan Australia, then a classic horror double feature of the 1931 JEKYLL AND HYDE and 1932 MUMMY, once I could drive into the larger nearby city of Canberra to a repertory theater there. The STAR WARS, CLOSE ENCOUNTERS and particularly THE EMPIRE STRIKES BACK enforced the idea. My basic love of films probably comes from the years I spent going to the drive-in (we made our parents take us to CHITTY CHITTY BANG BANG and THE JUNGLE BOOK many times over the years) but in terms of big inspirations, the films LAWRENCE OF ARABIA, THE GOOD THE BAD AND THE UGLY and NIGHT OF THE HUNTER were classic influences, and APOCALYPSE NOW and MAD MAX 2 (THE ROAD WARRIOR) blew me a way in the movie theater.

A big thanks for your time.

// WANT TO KNOW MORE?

– ILM: Official website of Industrial Light & Magic.

© Vincent Frei – The Art of VFX – 2015