Boris Schmidt joined Pixomondo in 1999 and worked on many projects such as 2012, THE LAST AIRBENDER or SUCKER PUNCH. For RED TAILS, he will be VFX Supervisor as well as for THE AMAZING SPIDER-MAN. He talks about the challenges faced on this show and the collaboration with Sony Pictures Imageworks.
What is your background?
I studied design in Germany then began working at Pixomondo in 1999 where I started doing some CG for commercials and other projects. Roughly three years ago, Pixomondo came to Los Angeles to work on 2012, though the office was in Venice at the time. Since then, I have stayed in LA and have worked on a couple of feature films.
How did Pixomondo get involved on this show?
We had worked with SPI before on other projects and they called our executive producer, Joni Jacobson. We got involved with the film through our Toronto office, which did about a dozen shots for the winter trailer with a quick turnaround. They were really happy with our work so they gave us more shots on the actual film. Then the bridge sequence came up and was initially much smaller but if kept growing as the script evolved.
How was the collaboration with director Marc Webb?
We seldom worked with Marc but got our notes from Jerome Chen, the VFX supervisor on the film. They looked at the shots together and the Jerome would relay any notes back to us. Marc and Jerome were completely on the same page as far as their vision and methodology so we really felt like there was never any conflict once it got past Jerome.
How was the collaboration with Production VFX Supervisor Jerome Chen?
I really enjoyed working with Jerome. He has a good eye and was always to the point. I think we had a good understanding of what he likes and I think we had excellent teamwork.
What have you done on this movie?
The majority of Pixomondo’s contribution to the film were visual effects shots in a sequence that takes place on the Williamsburg Bridge in New York, and also several sequences that involved transforming human characters into lizards and then back into human form as well as some arm replacement.
Can you tell us more about the filming on the bridge sequence?
They built a small portion of the bridge to have on set, and then we had to digitally extend the bridge. Ultimately, we decided to use only the foreground elements of the plates, like the actors and some of the cars, and not use the filmed parts of the bridge. We basically replaced the whole bridge because it was easier for us to do the lighting, matchmove and stereo work. We had about a dozen practical cars on set, essentially one lane of the bridge so we had to add additional lanes and the traffic going the other direction. The production built girders and pylons up until about 15 feet up in the air and everything else was CG extension. Everything on each coast was also CG and matte paintings. There were very few plates where we got everything in live action plates.
How did you recreate this huge environment? Did you use specific bridge references?
When we got the first previs, we started analyzing the shots and came out with the certain angles the camera captures, like how many times we see the lower east side of Manhattan, Queens, etc, so we knew what needed to be covered. We made a map where we defined our special angles then we started matte painting the background. The backgrounds were all projected on simple geometry for the buildings. We started very early on rebuilding the complete Williamsburg Bridge, detailed down to every rivet. I spent a lot of time mapping out the camera angles from each of the shots in the entire sequence so we could see the frequency of each. From there, we could quickly maximize our bandwidth on the work that needed to be done because we could pinpoint where we were going to need the high detail. We knew exactly which shots were going to be close ups or medium shots or background CG stuff and we could focus our efforts accordingly. We knew where we needed to concentrate so we didn’t have an artist spend time painting in 6,000 rivets that would never been seen in a faraway shot.
How did you collaborate with Imageworks teams for the asset sharing?
We collaborated with SPI on all of the shots where there was a creature in it, either Spider-Man or The Lizard. Early on we had a couple of meetings at SPI to lay out the workflow. We did some early tests to figure out how to share assets between different software packages. After we determined how everything was going to be accomplished technically, there was a lot of coordination to make sure we got the approved animation and camera movement from SPI. They animated The Lizard and some cars then we imported them into our software and did the rendering and compositing. Whether the shot was creature-driven or environment-driven helped us to set the pipeline for each shot. If the primary action was the cars being thrown from the bridge, we were the lead house on getting the camera set up and getting the animation layout approved then we would pass it on to SPI to add the creature. If the main action of the shot was clearly about The Lizard or Spider-Man, SPI got their animation approved first and then we did the secondary animation. How things were handled was very much on a shot-by-shot basis but the whole process was pretty collaborative and the file sharing was seamless.
Can you explain to us in details the animation challenges for both Spider-Man and The Lizard?
Animation on the CG Spider-Man and The Lizard was done by SPI but we needed to make sure all secondary animation was integrated seamlessly. If The Lizard threw a car, we needed to animate the suspension of the car or debris and dust flying around.
How did you manage the impressive shots of Spider-Man holding the car over the river?
We analyzed the different camera angles early on so we had a good idea the passes on the bridge and which direction we are looking. We started doing matte paintings for those specific shots. When Spider-Man is holding the car, we decided not to use the plate with the practical car and instead replaced it with a CG car so it was easier for us to animate the weight of the vehicle. Also, there was a lot of smoke and fire interacting with the car so it actually looked more realistic when we used the CG car. Spider-Man is the only practical element of that shot. Since we didn’t use the plate cart, we were pretty free to create our own camera movements.
How did you create the fire in the car with the kid?
The only thing we got in camera was the boy on blue screen holding onto a rope. We had to matchmove and dimensionalize the boy in stereo. Then we had to create smoke and fire simulations so it would flow around his body. Because of the danger, the car wasn’t actually on fire with the boy inside of it. He was completely comfortable on set when it supposed to be on fire so we had to add in a ton of CG details such as the smoke, fire, sweat on his brow, and made his eyes seem more red from the smoke and from crying. We kept adding layer upon layer of detail in CG until it looked right.
Have you developed specific tools such as for the web or the water?
We did a lot of R&D on the water. We were looking at real plates from the east river and recreating the motion and the movement, like wind and waves. We started developing a water shader pretty early on as well. At first we thought we would be able to use some plates they shot from the bridge, but Marc and Jerome wanted a real sense of danger so they wanted the water to be further away. We ended up replacing the water in all the shots we had. Most of the helicopter plates were not shot in stereo and since the bridge has so many fine structures, like wires and cables and such, we had to create the stereo impression without having the right film. We were able to render the bridge completely in 3D to create the stereo.
Can you explain in details the first transformation for The Lizard?
We got a digital scan of the actor, Rhys Ifans, for the face recreation in 3D. For the skin, we got early shots from SPI where they did a similar task for reference. Based on this first look, we created our CG model then there was some was back and forth on details. We would give them an example and they gave us notes until we determined the final look. We were trying to control individual scale so we could more easily morph the characters.
Have you worked with the makeup department?
We got reference photos of the actors but we never directly worked with the makeup department.
Can you tell us more about the transformation reverse process on the SWAT team?
We made sure we got a lot of digital scans of the SWAT actors for the face recreation in 3D. Also, during the shoot, they placed markers on their faces. The little dots were easy to paint out later on but helped us in tracking their face movement. We had to do the complete CG face recreation for each actor. In the concept phase, we went back and forth on details like how many scales the face was going to have and such. We really modeled the whole lizard face until the director, Marc Webb, was satisfied with the look and then we began defining transition points. When the blue ash hits the face, this is where the healing starts so the underlying human face is revealed in that area. There was a lot of timing and concept work involved.
Can you explain to us in details about Dr. Connors arm replacements?
Most of the arm replacement shots were stereoscopic so we had two plates, one for the left eye and one for the right eye. The actor, Rhys Ifans, wore a green sleeve for arm coverage so we needed to paint it out and replace everything that was behind the green arm. These particular shots weren’t complicated but they were tedious since we needed to make sure everything worked in 3D. We first tracked and matchmoved the arm movement, then we developed a CG stump for the missing arm. There were also a lot of different reflections we needed to paint out. In the lab, there is a lot of glass, metal and mirror-like objects so we needed to make sure the original green sleeve didn’t show in any of the reflections. For Dr. Connors’ lab coat, we needed to do a cloth simulation. When an actor has an intact arm, he moves differently than if it was just a stump. We had to animate the arm to make it feel more like a stump and less like a full limb.
Which branches of Pixomondo have worked on this show?
Los Angeles primarily worked on the bridge sequence. Toronto handled much of the arm replacement. Frankfurt created some of the challenging shots where Dr. Connors becomes The Lizard in the taxicab as well as the transformation of the SWAT team from the lizard men back to their human selves. Berlin worked on complicated CG FX shots with the burning cars. Burbank did cloth simulations for Dr. Connors’ arm and CG shots featuring a realistic spider. Additionally, Beijing, Shanghai and London helped out with matchmove, modeling and matte paintings.
What was the biggest challenge on this project and how did you achieve it?
One of the biggest challenges on this project was the timeframe. I think it was all about efficiency and planning everything out because there wasn’t time for a lot of revisions. We worked on setting up good pipelines for the different types of shots like the arm replacement or the boy in the car. The bridge sequence was originally shot in stereo but there were a few reshoots that were flat so that added a considerable complexity to the shots. We had to make sure everything matched up.
Was there a shot or a sequence that prevented you from sleep?
All of them! Actually, none of the shots were overly complicated to the point where we could not solve the issues. Sometimes the shots evolved from a script point of view so just tried to be flexible and just focused on delivering great work on time.
How long have you worked on this film?
We started work on THE AMAZING SPIDER-MAN in November 2011 and delivered May 2012.
How many shots have you done?
In the final film, there are 272 shots we worked on.
What was the size of your team?
We had about 202 Pixomondo artists working on THE AMAZING SPIDER-MAN.
What is your next project?
As a company, we have a couple of big feature films in various stages of production.
What are the four movies that gave you the passion for cinema?
I think the first time I saw something in 3D, it was a music video from the 80s featuring a wireframe human head. That really impressed me when I was a kid. Then the first TRON came out and I think that film made the biggest impression on me. I really knew I wanted to do visual effects for a job at that time, and of course STAR WARS was a great influence as well.
A big thanks for your time.
// WANT TO KNOW MORE?
– Pixomondo: Official website of Pixomondo.
© Vincent Frei – The Art of VFX – 2012
