Robert Pinnow has been working in visual effects for almost 20 years. He founded RISE in 2007 with Florian Gellinger, Sven Pannicke and Markus Degen. He has worked on many films such as NINJA ASSASSIN, CLOUD ATLAS, CAPTAIN AMERICA: THE WINTER SOLDIER and THE MAN FROM U.N.C.L.E.
What is your background?
I used to be a 3D artist almost 20 years ago and went into VFX-producing commercials before doing the same for feature films. Today I am a managing partner of RISE Visual Effects Studios and mostly doing VFX supervision and production while also taking care of our technical infrastructure.
How did you and RISE get involved on this show?
We were in talks about the Berlin set extensions almost a year before a first version of the script was discussed. We had done similar work on Guy Ritchie’s THE MAN FROM U.N.C.L.E. before and were able to show some nice, mostly CG shots of old Berlin. Then, still before the preparation phase for the shot, the idea to do all the backgrounds for the car driving sequences was born and more detailed plans were made.
How was your collaboration with the various directors?
All three directors are very different and each has his own focus while directing. Tom was very experienced in working with VFX while Henk and Achim were quite new to it. The fact that Tom trusted us from our previous work on A HOLOGRAM FOR THE KING and CLOUD ATLAS was really helpful in reducing the others’ anxiousness. I think it is very special about those three that they have a high level of trust with each other, despite being so different. Whenever I saw them start a discussion, it was always focused on improving details serving the series as a whole.
What was their approach and their expectations about the visual effects?
I started with what I thought was a maximum list of things VFX could help with. While talking it through with the directors and producers and while having an eye on the cost estimate, a second list was created. It was focused on trying to achieve as much as possible in camera. The truth is somewhere in between and of course there were a lot of shots that we saw on neither list and that arose from changes in the story, weather problems and locations being unavailable. When you’re trying to make a virtual city come alive you need so many digital extras and cars – something unimaginable to even attempt with human extras and real vintage cars on an efficient shooting schedule. The most important premise was that the dirty look of the 1920’s was not to be disturbed and that all the VFX shots blend into the look seamlessly.
How did you organize the work with your VFX Producer Katrin Arndt and at the RISE offices?
During the prep and shoot, I was both supervising and producing simultaneously. When Katrin joined us in February 2017, I was able to focus more on the creative aspects of the show. We had not been working on this high of a shot count before – especially considering the diversity of shots. It really helped us that Katrin, who was both new at RISE and to the project, was pushing hard to keep things moving forward quickly.
The other RISE offices were involved from the very beginning. Our team in Cologne had their hands on almost every CG building we created – already while we were still shooting. They were also handling the whole LiDAR scanning process and finished 173 comp shots. The team in Munich was responsible for developing the whole showdown sequence with its CG train and the fast-moving 360 degrees CG environment. Both offices had their own department supervisors and once the basic setup was done we mostly saw shots in a pretty advanced state when they were presented in Berlin.
Can you tell us more about the previz and postviz work?
It all started with the storyboards. From there we were deciding together whether doing a previz for a sequence would be beneficial in how we approach improvements, either technically or creatively. The results were then given to all departments where they could be helpful.
The good thing about working on a series with a story-driven period Berlin is that the edit won’t be too dependent on a postviz or the final the VFX work. There were of course full CG shots and elements where it did help with the editing decisions. However, in most cases we had the freedom to start doing a postviz only after the editing for an episode was in a good shape. It made the process much more cost efficient but still enabled editorial to fine tune the cut once we dropped the backgrounds and CG queues in.
What was your approach to recreate the period Berlin?
We approached these shots in two groups. The first group were the ones that could be done with a set of generic Berlin houses that we had prepared for this purpose. This mainly applies to the scenes in the streets that that had been shot on the studio backlot. This group also includes a set of extensions we built for the Studio facades that are only partially built three stories high – but supposed to be five like almost everywhere in Berlin. The second group included all shots that were special locations like “Alexanderplatz” or “Potsdamer Platz”. These shots were mostly wide ones showing the more special, mostly bigger and more unique houses. The camera was always in motion, so we built a large set of digital models to stay flexible and avoid doing things for one shot only.
Can you explain in detail about the creation of Berlin?
The houses were modeled from scans done by our LiDAR department. We started with meshing the scan pointclouds, filled and fixed holes from scan shadows and applied textures and shaders. For facades that we could not scan in one piece, like the ones from the studio backlot, we used construction drawings and elements from the lower stories to create the upper part. The roofs were done in ROOFIE, our scripted roof tile generator. It lets you choose from a set of different tile styles and textures to give them a look everywhere between old Rome and modern family home. The more unique buildings were hand modeled while sometimes using some elements e.g. a balcony from other buildings. We had lots of historic pictures provided by the art department and tried to stay as close to the original historic feel of the city as possible.
Which location was the most complicated to recreate and why?
One of the most challenging locations was surely Alexanderplatz. The background throughout the whole sequence was shot on today’s Alexanderplatz as two bigger, rather modern buildings still have the same shape as they had back then, and it made framing the shots much easier. These offered some cover in one direction for the shots on eye level following in the edit. The challenge was to make the shots look as historically correct as possible without replacing the ground in the center that was important for continuity with the other shots where it could not be reworked for cost reasons. Technically challenging was also the Lipezk airfield in episode 11. It is a full CG landscape we see from many different angles and from very wide to quite close while flying over it.
How did you populated the streets with people and cars?
The cars were mostly built from original ones we scanned on set and then animated through our scenes. The models for the digital extras were created in “Make Human” and then dressed using “Marvelous Designer” according to the fashion we saw on set. This way we were able to add a cloth simulation selectively where this made sense, based on the distance to the virtual camera.
How did you create and animate the crowd?
The digital crowd was done using Houdini. Andreas Giessen, one of our Houdini TDs and FX supervisors is an expert for the built-in crowd tool (among all other FX things). Together with our pipeline and rigging departments he added lots of fancy features to control the different variations of look and movement. To be able to turn around these simulations quickly that we needed for so many shots, he distributed the individual cloth sims of the agents to the render farm. That way it took us a couple of minutes to generate cloth caches of even the largest of crowd scenes. He is planning to include details about this in his Houdini talk at the FMX conference this year. Depending on the requirements and settings of the tool, the animations from the library were automatically applied to a more or less randomized crowd. We were able to define the distribution between different cloth colors, hats, men and women, body height and weight plus several other parameters. In many of those shots the crowd was MUCH closer than we had ever anticipated. As a result, we had to improve the look significantly compared to our earlier crowd work throughout the project. The biggest gain of detail came from individually simulating cloth for many of the digital extras – 500 in one of the shots. But in the end the system saved us from shooting crowd elements for months and months to be able to match every possible perspective, timing and lighting situation. Most of the crowd work that you see in the series is done with full CG agents.
There are sequences that take place in the air. How did you create the plane?
There was an original JU52 in a hangar that we filmed all the interior shots in. It was even able and permitted to fly although it would have taken us hours to unrig. It wasn’t worth it just for the small takeoff sequence and the light on that day was fading anyway. We took the CG model that was being built for the wide exterior shots in the storm and action sequences and reutilized it to create the takeoff, too. The model was made from a LiDAR scan and was detailed enough to show small portions of the plane at over 100% screen size.
The plane is stuck in a big storm. Can you tell us more about the FX creation?
The biggest challenge of showing a plane in a storm at night is bringing together the demand to see as much as possible while keeping a realistic look. After all, you would not see anything but darkness aside from flashing lightning. All the exterior full CG shots were done with Houdini volumes for clouds with lights embedded into them for the lightning. Rain hitting the plane and heavy turbulence were key to getting an impression of the mayhem our heroes are in. We also added a layer of very fast, wispy clouds in the foreground to add some speed.
How did you enhance the stunts?
During preproduction there were plans to use digi doubles at various occasions including for action shots quite close to camera, but for cost reasons we ended up doing mostly wire removals. Everything was of course shot in the least dangerous environments for the actors so lots of plate or CG backgrounds were used to help portray in a more exciting context the action sequences.
Can you tell us more about your work on the underwater shots?
With the knowledge we gained from the huge number of underwater shots in RENEGADES, the short underwater sequence in BABYLON BERLIN was pretty straight forward. There was a mix of Actors and Stunts shot in a deep pool here in Berlin. Plenty of roto followed. Then it was about balancing the directors’ demand for underwater visibility with keeping a realistic look. We had a set of pre-simulated fine particles and different levels of bubbles rising from the car that just required a camera to be placed in the scene.
How did you handle the water simulations?
Our whole pipeline is built around Houdini as our main 3D software. Having done similar stuff previously, the biggest challenge was to balance the effort for this short sequence with the look. Recreating reality without the VFX being the vehicle for the story telling is always an unappreciated task as on such complex tasks it consumes a lot of the budget and in best case will look like being shot for real at the end.
There is big sequence on a train. How was it filmed and created?
The showdown on the train was planned to be shot on a real tank wagon while VFX would insert behind a BG plate. While planning this in detail, production understood that the security implications for the actors to work on an uneven surface more than 4 meters above ground were huge and all the approaches to solve this would have cost much more than doing the wagon in CG. Getting the right BG plates and retouching all modern elements proved to be complicated, too. We ended up shooting the whole sequence in a bluescreen studio using a blue mockup of the top part of the wagon with everything under and behind the actors being created in CG.
Can you tell us more about the creation of the CG trees?
The trees were originally created in SpeedTree, including the wind simulation. Once you load them into the scene to scatter them on a surface in Houdini, you’ll still be able to choose the color variation of the leaves plus a range of sizes to be used etc. We learned that there seems to be a natural density of approx. 70000 trees/km², but we also did successful render tests with more than one million trees in a scene. With a fast-moving train one of the challenges is still that you need to build a long stretch of environment including all the bits and pieces that fly by.
How did you create the big explosion?
The explosion of the train wagon was a Houdini Pyro sim. Watching it in the edit for the first time showed that this needed to be done with more than one simulation to match the lighting on one hand but also keep up the impression that Gereon Rath, the main character, will not be entirely grilled while still standing on the following wagon. We ended up having different sizes and decays per shot to make it obvious that he’s able to survive.
Is there any other invisible effects you want to reveal to us?
I am not sure if the result really qualifies for being an invisible effect. The opening shot of the first car driving sequence in episode one is supposed to be invisible and was quite complicated. During all discussions with the directors before the shoot we had been agreeing not to show the car from the outside in the VFX shots we were doing the digital backgrounds for. Then on the shooting day, Achim decided to do exactly this – start on the radiator grill and crane up to look into the car while we see the whole street behind it. That day, we once more agreed to shoot this in case it might work but not use it in the edit until we did a test. You know, editors tend to ignore this kind of agreement and so did ours. Everyone liked the opening moment so much that we finally had to make it work. The exterior of the car has been replaced entirely with it’s digital model to reflect the CG environment appropriately.
What is your favorite shot or sequence?
As a sequence, my favorite would be the riot and shooting scene from episode 4. It had been edited without even seeing a postviz. So its purpose was not to maximize the effect of the massive digital crowd or extensions for the wide shots. There are many VFX shots in this sequence but they are just shots among others, not the heroes.
My favorite shot is the moment when the plane leaves the clouds and we capture our first sight of the military camp in Russia. This is a full CG shot and we go from near total darkness – the sequence before was at night – to rather bright morning light within one shot while we’re flying through first denser clouds then wispier ones which open up the view.
What is your best memory on this show?
Working with the directors and also the DPs was a fantastic experience. Their respective, and even more so, combined knowledge paired together with their openness for suggestions, and capacity for discussing matters that were heavily spilling into their department were what made me feel privileged to be part of this project.
How long have you worked on this show?
We were in early talks about the city extensions in November 2015. A more detailed preparation of approaches started in February 2016. Shooting took place from April to December 2016. First turnovers were due in February 2017 and last shots were finalized in August 2017. So from prep to final our work took 17 months.
What’s the VFX shots count?
We were working on 925 shots, 812 of which made it into the series. You can see that turnovers happened rather late in the editing process to avoid wasting money on shots that are not sure to be needed.
What was the size of your on-set team?
Most of the time two of the three directors were shooting with their respective teams on different locations while the third one was preparing his next chunk of shooting days. Our team generally consisted of two set supervisors with additional support coming from a data wrangler or the LIDAR team when necessary.
What is your next project?
We just finished work on Marvel Studios’ BLACK PANTHER and also on the German feature film JIM KNOPF or JIM BUTTON. Looks like we are going to do some more work for Marvel soon and also start on a German TV series called BEAT.
What are the four movies that gave you the passion for cinema?
GATTACA and MEMENTO really caught me long ago. One of the reasons for working in VFX is probably JURASSIC PARK.
A big thanks for your time.
// BABYLON BERLIN – VFX BREAKDOWN – RISE
// WANT TO KNOW MORE?
RISE: Official website of RISE.
© Vincent Frei – The Art of VFX – 2018
