John Nelson supervised the visual effects of more than 20 movies such as IN THE LINE OF FIRE, THE MATRIX RELOADED, I, ROBOT and IRON MAN. He won an Academy Award for Best Achievement in Visual Effects for GLADIATOR.
What is your background?
I got a degree in General Studies from the University of Michigan, with a concentration in Film Studies. I studied every aspect of filmmaking in both the Art and Speech departments. I made films that won awards at festivals and graduated with high distinction.
After graduation, I moved to Los Angeles. I went to work for Robert Abel and Associates where I worked on hundreds of television commercials that used special effects. I started as a cameraman, then worked as a technical director and when I left eight years later, I was directing commercials. I was nominated for Clio awards many times, winning twice.
After that I moved to West Berlin, to help set up Mental Images GMBH. After two years in Germany I moved back to the States to work for Industrial Light and Magic. I got there at the end of THE ABYSS and stayed through TERMINATOR 2. My best shot in “T2” was the shot-gunned head of the T-1000 that resealed itself.
After ILM I moved back to Los Angeles to supervise for Rhythm and Hues, then Sony Pictures Imageworks. I left Imageworks, to supervise GLADIATOR for which I won the Academy Award for Best Achievement in Visual Effects (2001). I was nominated again for both I, ROBOT (2004) and IRON MAN (2008) but did not win. I have worked on 25 feature films. As the head of the visual effects department I am involved in the breakdown, planning, design, shooting and posting of every visual effects shot in each film I work on.
How did you get involved on this show?
When I was finishing POINT BREAK producers Andrew Kosove and Broderick Johnson of Alcon Entertainment asked to speak with me. My first response was “Am I in trouble?”. They answered no and that they were beginning a new film and would I be interested in it. I asked what was the project and they said it was the sequel to BLADE RUNNER I think I responded “Are you F%!!-ing kidding me? I am definitely interested!” I met with Denis Villeneuve and we connected on an artistic level.
What was your feeling to work on the sequel of a cult movie?
It was daunting to have everyone you spoke with poke you in the chest and say “ BLADE RUNNER is one of my favorite films, the sequel better be good!”. I think any fan of science fiction or cinema has a great deal of reverence for the original BLADE RUNNER (myself Included). We had a great team and everyone bore the burden of knowing we had to do something special. Fortunately we had a great director to lead us.
How was the collaboration with director Denis Villeneuve and Director of Photography Roger Deakins?
Denis and Roger both were both great and work extremely well as a team. We would meet in preproduction to discuss previs, design, how to shoot certain VFX scenes and the aesthetics of the visual effects. Roger was almost always with Denis and myself when we would review previs for blocking for the total CG shots.
During pre-pro and principal photography Denis and I would discuss every aspect of the visual effects. Roger would attend as many of those sessions as he could but he was quite busy preparing and testing the camera looks and lighting. For the numerous big VFX sequences we would all meet in the large room outside Denis’ office with all the departments to work out how we were going to shoot them. We always wanted to shoot as much practically as we could but in a film with so many effects shots of environments that were massive and futuristic, there were many total CGI shots.
In principal photography everyone is so busy things get more compartmentalized. Roger was working with his crew on the set refining the lighting and conferring with Denis. As the leader of the VFX department I made sure we got the elements we needed to do our work. Denis was interested in everything but his primary focus on the shoot days was getting the performances from the actors.
In post production we would meet with Denis every week to review VFX shots in the cut. I would try to give Denis three options that I felt were all good based on what he had told me he wanted. Once he would choose one of those options then we would drill more deeply into that shot or sequence to make it even more special.
Later in post when Roger came back on to time the film we would have three way reviews with the VFX vendor, Denis, Roger and myself on how the shot looked and what we still needed to do with it. The VFX department made sure creative refinements were executed as dealing with eight VFX vendors and 1200 VFX shots requires an immense amount of detail work. Many of the decisions happened at midnight after everyone with any sanity had gone home.
What was their approaches and expectations about the visual effects?
They expected each shot to look impressive but realistic while satisfying the emotional requirements of the story. To achieve this, I looked at each shot hundreds of times and always in the cut. I gave detailed notes to the VFX vendors on what I wanted and what was lacking. I wanted every shot we did to be as perfect as it could be. For this I gave up time with my family and friends but I think that if you have an opportunity to make something special you should bring to bear all of your resources and energy to do so.
How did you organize the work between you and VFX Producer Karen Murphy and the vendors?
First you have to understand that doing a big film like this requires a partnership between the visual effects supervisor and the visual effects producer. It is like a marriage. I had an incredible VFX producer in Karen Murphy and we were a great team. Karen and I both have a great deal of experience so we would look at what needed to be done and she would deal with part of it and I would deal with part of it. We would both scheme to get what the film needed within the time and budget we had. I can’t tell you how many times we would go to a meeting someone else called about a sequence and then when everyone got there they would all look at VFX and say ‘so tell us how we are going to do this”. The one thing being experienced is good for is that you know how to make decisions quickly and we certainly had to on this show. We covered what we knew to be difficult with an intense amount of focus. Karen and I broke down the film, with Karen doing the detailed budget of all things VFX. We figured out who would be the VFX vendors (and why), hired our internal VFX team and worked to plan each VFX shot and sequence. Often this would require we get to work three hours ahead of time to deal with something and stay three hours after everyone had gone to keep us on track.
As for the vendors, we chose VFX houses and people I wanted. I was blessed in that this was a show everyone wanted to work on. Paul Lambert at Double Negative, Rich Hoover at Framestore, Pierre Buffin at Buf, Deak Ferrand and Sebastien Moreau at Rodeo, Viktor Muller at UPP, Ryan Tudhope at Atomic Fiction and Richard Clegg at MPC and Weta Workshop for the miniatures. I had an incredible team of people. I had to get them to pull together as one force to make one vision.
Can you tell us how you choose the various VFX vendors?
I broke down 1000 VFX shots myself into what techniques and elements were required for bidding. I knew all the vendors and I wanted to keep the shot count under 250 at any single vendor. I knew UPP and want them on any show I do because they are so good. I knew Dneg, Framestore and MPC from previous shows I have done and knew they were deep and dedicated. I wanted Buf for the Holofunhouse and Sinatra because I have always admired their work for David Fincher and wanted to work with them. I had seen Atomic Fiction’s work on FLIGHT, THE WALK and DEADPOOL and liked it very much. I wanted to see who would rise to the occasion and deliver incredible stuff. In addition to that I know Rich Hoover from my days at Robert Abel and knew I could trust him. Paul Lambert and I became great friends. What he did at Dneg for us was just out of this world.
How did you split the work amongst these vendors?
I gave the opening of the film, the Trash Mesa (a mini film in itself) and Vegas to Framestore. As I said I knew Rich Hoover from Abels’s and William Sargent (the head of Framestore) from other films I have done and knew they would not give up till it was great.
I gave Dneg Los Angeles, JOI and the Sepulveda Seawall. I discussed the show with Paul Franklin and Alex Hope and I knew Dneg would do a great job. They had three of my most important items with the cities, JOI and the seawall. Paul Franklin introduced me to Paul Lambert who supervised for Dneg. We became good friends and creatively bonded. Dneg and Framestore were my biggest vendors with over 250 shots each.
I gave Buf special sequences that required a lot of creative research and execution. They did Ana’s memory lab where she creates memories (a five-camera array where we could control space and time), the Sinatra hologram juke box and the Holofunhouse with Elvis, Marilyn and Liberace. They also got the end sequence outside Ana’s with K and Deckard in the snow.
Atomic Fiction got Bibi’s bar which is one of the coolest sequences in the film.
MPC got Rachael reborn which of course was one of our hardest sequences. They had created Arnold for T-4 and I knew they could take it further. That’s where we needed to go.
Can you tell us more about your collaboration with their VFX supervisors?
Paul Lambert from Dneg and Rich Hoover from Framestore moved to Budapest during principal photography to work with me supervising multiple units. Pierre Buffin and his team of Olivier Cauwet and Jeremy Robert would fly in when Buf sequences were being shot. Ryan Tudhope (Atomic), Viktor Muller (UPP) and Richard Clegg (MPC) would do the same. For Rodeo I would work with Robert Bock during principal and Deak Ferrand and Sebastien Moreau during post. My relationship with the vendor supervisor is simple. I tell them what I want creatively and trust them to deliver it. I chose them because I thought they were good. I would review up to 100 shots a day (always in the cut) and give detailed notes on what was good and what needed improvement. I kept an eye on the large picture of how everything fit and what was in dire need for improvement. I worked with my coordinators and my visual effects editor Scott Anderson to review and improve all the shots until I felt we had achieved what Denis had asked me to do.
The vendors are all around the world. How did you proceed to follow their work?
We reviewed shots every day in the cut and made notes. I had five VFX coordinators for the eight vendors and they would be hovering waiting for the one before them to finish so we could review their shots. VFX editor Scott Anderson and I would review the shots in the cut and I would give notes to the coordinator covering that VFX vendor, then we would walk around the building while the next coordinator got set and do another 30-40 shots. We did cineSyncs in the morning or afternoon based on where the vendors were and what time it was there. For Weta miniature shoots which happened while we were shooting principal photography, we would come in at 5am, Skype until 7am, when we would start shooting with Denis and Roger, then Skype again from 7 to 9 pm with Weta in New Zealand because they are on the opposite side of the world from Budapest.
Can you tell us more about the previz and shooting process?
For previs we used the MPC team with Duane Flock supervising it. He had worked for Bill Carraro (our line producer) before and was quite solid. We would meet with myself, Roger and Denis and line up the shots and review them. For anything that was acting we would not previs, only the establishing or action shots without people.
This new movie is full of various environments and locations. How did you approach the creation of the places?
Every wide shot in the film is a visual effects shot. Each environment was approached differently but we attempted to start with photographic elements whenever possible. I started my career as a cameraman and I firmly believe if you can shoot it you must try to do so before you add VFX. Sometimes there was nothing that existed to shoot as the landscape was so unique. At those times we build the environment in totally in CGI.
In the beginning of the film for the trip to Sapper’s farm we shot aerial plates in Spain and Iceland and added CG elements to them. For Sapper’s farm we shot a backlot set of the house and greenhouses with set extensions of Iceland in the background.
For the trip to LAPD we shot aerial plates in Mexico City and then added a ton of CG fog and CG buildings to them. As we needed soft light our aerial DP Dylan Goss shot only in cloudy overcast backlit weather. For the LAPD building shots we used a large scale miniature and augmented it greatly with CG.
For the Trash Mesa that environment was all CG and built to look like the ship graveyard in Chittagong, Bangladesh. The Trash Mesa crash site was a backlot set about the size of half a football field that was extended greatly in CG.
Most of the views of Los Angeles were total CG shots with the exception being K walking toward his apartment where we shot a building in Budapest and added signs, buildings, ads, cars, Spinners and drone ads in CG. The flight to the seawall fight was also a total CG environment but the seawall fight was shot in a tank we built in Budapest (extended in CG).
What was the real size of the sets?
Dennis Gassner our production designer would build the practical sets to be as large as the stages we had in Budapest. They were a good size but not immense. In almost every instance what was close to camera was real and the background needed to be extended with CGI.
Can you tell us more about the use of miniatures?
Weta Workshop build the LAPD building and parts of the Trash Mesa backgrounds. They were shot in various lighting (backlight, different fog level, practical lights only, mattes, etc.). The photography was done on a Canon still camera shot go motion with motion control to get motion blur and repeatability for each pass. Alex Funke shot the miniatures in a computer controlled smoke room to keep the atmosphere constant during the long motion control passes.
Can you explain in details about the creation in CG of these locations and especially about Los Angeles and Las Vegas?
Los Angeles was made from Mexico City plates with tons of CGI on top of it. Several other Los Angeles scenes (Coke/Atari shot, Trip to Wallace Earth headquarters and shots to and over the seawall) were total CG shots.
How did you handle the lighting challenge for Las Vegas?
Las Vegas was a total CG environment done by Framestore. Syd Mead designed some of the buildings. We did extensive fog tests to establish how much into the distance we would see. The pilot fish view where we first see Vegas took weeks to render at 3.4 K so we had to test and confirm lots before we pushed that button. Framestore had to build and immense amount of detail (as Dneg had to for Los Angeles) including buildings, signs, different types and ages of buildings and ads.
Can you tell us more about the various holograms in Los Angeles?
The holograms in Los Angeles are either ads that have this sort of multi-plane projection or human shaped figures that walk out and engage their potential customers. Paul Lambert and his team at Dneg did a great job in creating these. Dneg art department did some early concepts where you see different planes of projection that are in different colors. This gave the image depth and the illusion of being both a projection and a hologram. For the human shaped holograms we shot actors (usually Ana De Armas as JOI) with the camera close while she looked at a cut out at K, then shot plates with the same lens but farther away when we shot Ryan Gosling as K. For some holograms JOI is full CG. We used our scans of JOI to create that. Paul and his team put back facing shells on all the holograms and layers of light projection details from the signs the holograms are projected from.
The iconic Spinner is back too. Can you tell us more about its creation?
The art department designed K’s spinner and built several vehicles practically. The ones you see outside K’s apartment or at Bibi’s bar are those real vehicles. For K’s spinner we photogrammed it to get rough geometry and build it in CG along with other variations of spinners for other vehicles so they would not always be the same spinner.
Did you receive specific indications and references for the Spinners animations?
We roughly matched what we saw in the original BLADE RUNNER and tried to improve upon it. The shot where the spinner lands in front of the Freysa’s church showed a particularly good transferral of weight and the force it took to keep the spinner in the air. For several flying shots we used a volume capture for the camera movement that Habib Zararpour had. That way Denis could be involved in doing the camera move.
The new love interest is Joi, an hologram. Can you tell us more about her design and creation?
Early in preproduction we looked at conventional holograms and other movie versions of holograms and did not like them. We did not want it to be too much like a hologram of today or too glitzy and sparkly like it was a CGI creation. We wanted everything in the film to seem analog (not digital) and photographically real. Once Dneg broke the “back shell” look we knew we could create a realistic volume for JOI that was a noticeable volume but was not too much to get in the way of the actors’ performances. Using the real photography of JOI for the front face of her “shell” gave us Ana’s performance and the realism of her being photographed (not CG) while the CG back shell gave us her volume.
How did you handle the touching sequence when Joi is synching with another girl?
This was very difficult. Early in preproduction I merged Ana de Armas’ face with Mackenzie Davis’ face and got some very interesting versions. We knew it would be cool but we also knew it would be very difficult to get them to overlap and give a performance as two women merged into one. We shot tests in preproduction with doubles to test techniques of how to get the actors to overlap and what would that look like. Paul Lambert and his team at Dneg would take the test footage and map it onto CG versions of the women to reposition and merge the geometries.
For the actual shoot first we shot both women separately with Marriette (Mackenzie) being shot first. Then Denis would pick the take he liked and I would take five minutes on set with Jessica our script supervisor to break down her actions (at two seconds you raise your hand, etc.) they we would shoot Ana as JOI. I would line up Ana over Mackenzie’ face by using an IPAD so Ana could get in a starting position. Then I would pull the IPAD out so she would not look at it but at Ryan. Then we would talk Ana through the performance with a stopwatch (now you are raising your hand, etc.) through the whole shot. The actors were very close but not perfect in their actions as far as matching each other. Between re-projecting them onto CG and repositioning them and letting them drift a bit we made shots that start out of sync but end with the eyes in sync which I think is magical.
Rachel is coming back in one sequence. How did you approach and create it?
This was also one of our hardest sequences. We knew this coming into the show. We looked at all the other digital human work from BENJAMIN BUTTON to TERMINATOR GENISYS and STAR WARS ROGUE ONE and combined techniques that looked the most promising. Early on we decided that a 3D CGI head replacement would be the best method for creating a Rachael that was realistic enough to serve our needs. Denis cast a body double actress (Loren Peta) who resembled a youthful Sean Young and would perform the scene with Harrison Ford on the day of shooting. We dotted up Loren’s face. She was in full hair, makeup and costume. Sean Young was on set as an advisor for the shooting days of that scene. We shot Loren as Rachael doing her performance with Harrison and then we brought both Sean Young and Loren into a DI4D facial motion capture stage where we had both actresses do the lines from the scene with Denis directing them.
We then went into post and replaced everything from the neck up with a CG head of the young Rachael (including CG hair complete with even small flyaway hairs at the edge of her hairline). To get the CG model correct Richard Clegg and his great team at MPC started with a life cast of Sean from when she was about twenty-eight years old (nine years after the original BLADE RUNNER) and made geometry of that and the underlying skull and muscles beneath the skin of the face. They amended this model by looking at scenes from the original film to get a better match on the facial lines, makeup and Sean’s mannerisms and gestures. Then they took several scenes from the original film and replace one shot in each sequence with the digital double. This was the litmus test in that when we showed these scenes to Denis and the producers they did not notice the digital double and bought her as real.
Now we knew we had a digital head that looked real and we needed to make her act real and give a performance. The animators at MPC used the DI4D facial motion capture as a reference but did not fully use it as Denis wanted something specific in the performance that it did not give. I felt first Rachael should be confident as she walks up so she is like the scene in the first film where she meets Deckard for the first time (“is it expensive?, very, I’m Rachael”). Once Rachael walks up to Deckard in our new film, her expression goes from confident to longing. Denis said this was like two people who have not seen each other for years and then meet at a train station with emotion flowing uncontrollably out of them. Then after Deckard rejects her she shows her emotion of rejection. Richard and his team at MPC took scenes from the original film that showed these emotions and along with DI4D poses of Sean to create a performance that looks real and hits the emotional moments that Denis wanted.
Which sequence or shot was the most complicated to created and why?
The threesome, Rachael reborn, Ana’s memories and Vegas Holofunhouse were the most complicated scenes to do. The threesome because of how hard it was to shoot and post and balance both women’s performance into the third merged woman. Ana’s memories because we shot them with a five-camera array to shoot the kids (so we could control space and time) and then we shot Ana as a separate pass months later. Rachael reborn as making a digital human look real and act is an incredibly difficult thing to do and Holofunhouse as it had so many different acts that were all shot separately from the FG action of K and Deckard fighting.
What is your favorite shot or sequence?
The threesome is my favorite VFX scene, especially when the eyes of the women line up. Paul Lambert and the Dneg team really nailed it.
What was the main challenge on this show and how did you achieve it?
I am proud of all the work but I think the Threesome with JOI and Mariette merging, along with the birthday party memory with Anna (Buf), Rachael reborn (MPC). Trash Mesa (Framestore) and the cities (Dneg and Framestore) are my favorites. Vegas Elvis and Sinatra (Buf), Bibi’s bar (Atomic Fiction) and the hall of replicants (UPP) are also very good. (Every vendor delivered great stuff so every sequence works well.) The main challenge was to create a world that was an extension of the first film but not a copy of it. We strove to achieve consistency between eight VFX vendors work and support Denis’ unique vision of the story.
What is your best memory on this show?
When Denis would say “I deeply love it” on any of our visual effects shots.
How long have you worked on this show?
From November 2015 to August 2017.
What’s the VFX shots count?
1190 VFX shots in the finished film. 1886 turned over but some dropped. Eight VFX vendors (Dneg 291, Framestore 281, In House team 192, UPP 188, Buf 105, Rodeo 64, Atomic Fiction 36, MPC 33 in the finished film of 1190 VFX shots).
What was the size of your team?
About fifteen people between who was on set and who was in the office making sure it all got done.
What is your next project?
Getting reacquainted with my wife and son after doing two projects in a row for the last three years. I promised I would not do a show for six months as we have plenty to do around the house. I need some time to rest and recharge. Sometime in the spring we will see what projects are out there and hopefully there will be one for me.
What are the four movies that gave you the passion for cinema?
I was the head usher at the Vogue theater on the east side of Detroit as a kid. It probably influenced me in ways I haven’t even figured out yet.
For me the films that shook my world were….
1) 2001 (Stanley Kubrick, the man)
2) LAWRENCE OF ARABIA (David Lean – big shot, small people)
3) BALL OF FIRE (Howard Hawks, written by Billy Wilder)
4) BLADE RUNNER (Ridley Scott)
… but I have several others that rank high (NORTH BY NORTHWEST by Hitchcock, THE LADY EVE by Preston Sturges, A HISTORY OF VIOLENCE by David Cronenberg, THE HUSTLER by Robert Rossen, PATTON by Franklin Shaffner, written by Francis Ford Coppola, THE GODFATHER, AMERICAN GRAFITTI and THE MATRIX and many more.
A big thanks for your time.
© Vincent Frei – The Art of VFX – 2017
amazing interview!