In 2019, Andreas Giesen had explained the work done by RISE on Hellboy. He then worked on several shows such as Doctor Sleep, WandaVision, Loki and Reminiscence.
How did you and RISE get involved on this show?
RISE worked successfully with Dan Glass on a few other projects in the past and we were also supporting the principal photography in Babelsberg with virtual production work. When Robert Pinnow, one of the owners of RISE and Executive Producer on this show, told me that we would also be doing the VFX work for several sequences of The Matrix Resurrections I was very excited and looking forward to get involved!
What was your feeling to be part of such an iconic universe?
Me and everybody else in the team were super thrilled to get the chance to work on a Matrix movie, as this is not an opportunity that presents itself every few years. I would say that’s more a once-in-a-lifetime-thing. The first Matrix movie influenced me a lot back in the day and was definitely one of the reasons why I wanted to work in VFX. When I heard that we even had to recreate the iconic bullet time effect…sounds cheesy, but actually a dream come true.
How was this new collaboration with Director Lana Wachowski and Production VFX Supervisor Dan Glass?
As mentioned we worked with Dan Glass already on a few other projects in the past but for me it was the first time and I enjoyed it a lot. The collaboration was really great and the immense experience Dan has was making the weekly cineSyncs always very productive and precise. Also he was very open to ideas from our side and that made the whole process very collaborative. We worked exclusively with Dan and then he then presented our work to Lana.
What were their expectations and approach about the visual effects?
When we came onboard, the pre-production and principal photography was already done and they had a pretty clear vision about the visual effects for most of our sequences. We also received concept art which we could use as a visual reference. And for sure, when working on a franchise like this, the expectations were pretty high, especially in matching the style of the effects seen in prior Matrix movies and also for the development of new effects that needed to fit seamlessly in the Matrix universe. The planning and on-set material provided by the production was just great and helped a lot to make sure to ground every VFX work in reality.
How did you organize the work with your VFX Producer?
Our VFX Producer Annelie Dangel took care of all the scheduling, resource planning and bidding, which gave me the freedom to focus completely on the creative process and look development. Production Manager Anne Barger kept the daily business going and made sure that we always had a complete overview about this huge amount of shots.
What are the sequences made by RISE?
We worked on 5 bigger sequences. First, there was the one in which we see Neo on top of a skyscraper during some flashback scenes. Then he is on another skyscraper at night where he gets rescued in the last moment by Bugs. The next sequence was the shoot-out on the Deus Machina office floor, starting with a big explosion. The longest sequence was taking place in an abandoned factory, in which Agent Smith corners Neo and the others and they start to fight while old “friends” like the Merovingian show up. Last but not least was the Sati/Construct sequence, a safe place program in which they meet and plan the next steps against the machines.
Can you elaborates about the city environment creation?
For the flashback sequence in which Neo is attending a rooftop party and tries to jump off the edge of the building we had to create a CG extension for the set part of the roof and then to think of a way to create the surrounding city.
Nando Stille, our CG Supervisor, and his team built a 3D model based on a Lidar scan of the remaining rooftop area and a few other buildings as we had to deal with several shots in this sequence. So we had to make sure that this environment would work from all of those POVs and also in different lighting conditions.
We also received a lot of drone footage and panoramic photographs from production, shot at the real skyscraper in San Francisco. This was very helpful to recreate the city around the skyscraper by projecting it onto simple geometry in combination with some matte painting for the far background. Jonathan Weber, one of the compositing supervisors on the show, made sure that we had a consistent look for San Francisco in all shots.
The environment is seen in various light conditions. How does that affects your work?
That was for sure a critical info to have from the beginning, so that we could plan the asset builds accordingly. We built the set extension of the actual skyscraper as a fully textured and shaded asset that would look photoreal in all these different light conditions. For the city background we used various HDR bubbles which were taken at different times of the day, to make sure that light angle matched the plate of the shot and merged seamlessly with the CG extension.
How did you populate the city with cars and crowd?
As we used a combination of drone footage and matte paintings for the city environment there were already several real cars and some crowd visible in the plates as reference. In the end we created several assets to scatter across the streets, especially in the night sequence, because the streets in the plates were quite empty. We used simple geometry for the buildings to create the illusion of light interaction caused by the CG cars passing by.
There is a lot of destructions that happens during the office shoot out. Can you elaborate about them?
This sequence offered several challenges. First it was important to have continuity throughout all the shots and as nearly every shot was retimed we had to find a way to quickly preview the destruction so that Dan and Lana could basically art direct every impact. To maximize flexibility, our FX supervisor Kristian Kück and his team built a system in Houdini that used pre-cached elements with different impact forces and debris flying in various directions. Using this system we could quickly place the destruction for a shot, render a clay shaded preview which Dan and Lana could then review. Once they were happy with it, we just switched the preview impacts to full resolution assets with additional smoke simulations and finer detail.
Another challenge was the lookdev for the so-called “water spear” element. As the sprinkler water is pouring down, Lana was looking for a cool way that the bullets would interact with those water droplets. So our FX team and lookdev compositors started with different approaches to explore this idea. Some were more stylized and others going for a realistic look. When looking at real references there was not much of an effect visible, but I think in the end we found a subtle and cool look that is still visible but realistic and not too stylized.
Also compositing wise this sequence was quite a task as for several shots different retimed plates had to be combined, the FX elements integrated and all that with the water raining down. Patrick Kreuser, our compositing supervisor in Berlin, and his team cared about every little debris piece a lot so that in the end this sequence became the enjoyable mayhem you have seen on the big screen.
How did you manage the slow-motion aspects of the explosions and destructions?
We already knew from the beginning that we had to play a lot with slow-motion and retimes, Jonas Sorgenfrei, FX TD at RISE, made sure that all our FX elements were set up for this purpose. Basically every cache had enough substeps to be slowed down to about 20% of its original speed. So in the end we were very flexible to choose the perfect speed for every element and every shot. Because depending on distance to the camera and the amount of camera motion, the necessary retime could vary a lot to get the desired “slow-motion feeling”.
Can you explain in detail about your work on the fight sequence?
In general the factory fight sequence had more shots than all of our other sequences combined. As most of the shots were pretty straight forward with retimes, punch enhancement, some destruction and cleanups, this sequence also offered a few very complex and interesting lookdev tasks: We see Neo the first time using his new powers which create some kind of force field.
Then there are two bullet time shots, several ones where Agent Smith uses the “recursion” to avoid bullets and punch super fast, as you know it from the subway fight in the first Matrix movie. Last but not least the fight continues once Smiths plunged Neo through the floor to the first floor of the factory.
Apart from the retimes and cleanups we also had to exchange the whole ceiling of the factory using another highly detailed CG asset. The look of the more subtle force field was very effectively tackled in compositing using 2.5D approaches to give it a very vivid look. Later when Neo has more power we added several FX elements on top like shockwaves and more complex distortion layers.
How did you recreate the bullet time and other homage?
Even though everybody in this industry knows the bullet time effect the first step was to analyze it in detail, looking at the different bullet time scenes in the first Matrix movie. But Lana was also looking for more than just a simple copy, it should be more of an updated version of it, so to say bullet time 2.0.
The FX department started to provide layers which could be used in comp to get the lookdev going, as a lot was experimenting with distortions and different IORs for the refractive look that was necessary. The base were several individual disc elements, basically like hundreds of lenses that form the complete trail. Already the shape and changes in the shape over time made a huge difference lookwise. During several review sessions with Dan and also Lana’s feedback, the effect evolved, always avoiding a too glassy look and feel. Once both were happy with the basic trail look we added a fluid like element trailing directly behind the bullet and forming one disc after another. This added another level of complexity to the whole effect.
The idea for Agent Smith’s recursion effect was to have a very similar look as in the original movies. So here we did a frame-by-frame analysis as well to be able to re-create this effect when Smith is avoiding bullets and punching Neo against the wall. In the end it was a clever combination of several retimed plates and individual “arms” which we used to create this staccato punch effect. For sure this also involved extensive roto and cleanup work.
One of my favorite shots was also the one when Smith is throwing Neo against the wall, just before the recursion starts. Lana wanted to see here a homage of the moment when Neo pushes himself off the wall and starts flying and creates a spiderweb-crack like pattern. Just that this time Neo creates it when he is getting thrown against the wall. For these shots we replaced the whole wall with a CG element that allowed us to create exactly the pattern that Lana was looking for. Some more FX simulations with detailed plaster breaking apart, billowing dust, a lot of fine tuning in comp and the new spider-crack shot was finished.
Can you tell us more about the set extension work for the construct?
Some of the shots in this sequence were shot at a beautiful outdoor location near San Francisco. These shots were the perfect reference even though these had to be enhanced with many CG flowers and god rays as well to create the atmosphere Lana was looking for.
But the majority of shots was additional photography done in the studio in Babelsberg, so basically just actors in front of greenscreen. We also received a lot of footage shot at the original location that we could use as background plates for those shots.
So the first step was to define for every direction what we would see in the background and then looking for background plates with the correct lighting. Unfortunately the lighting in the studio changed from shot to shot so basically we had to do a complete re-grade of the FG plates before even adding the environment to make this sequence work.
Once we had our selects for every direction, we exchanged specific parts based on Dan’s and Lana’s feedback and added CG elements like moving flowers and trees. So in the end some shots were quite straightforward using the background plates while others needed basically a complete recreation of the environment done in CG and compositing.
Which shot or sequence was the most challenging?
This was definitely the sequence with Sati and the others in the construct as the whole sequence tended to have an artificial look. I think to some extent that was also Lana’s idea, as it was a construct and not a real place, but for sure just to a certain degree. So we had to find the perfect balance between realism and stylization. Philipp Mekus, compositing supervisor for this sequence, did a great job bringing all those shots together.
Is there something specific that gives you some really short nights?
Again the Sati sequence as the footage shot in the studio was very different from shot to shot regarding the lighting as already mentioned. So making sure to keep a light and color continuity in those shots was definitely a challenge. Luckily Markus Degen, our CCO, was giving us a helping hand as he is definitely the master of colors and grading at RISE.
What is your favorite shot or sequence?
My favorite sequence is the office shoot out as it reminded me a lot of the lobby scene in the first Matrix movie and creating a choreography of destruction is always a lot of fun in my eyes.
What is your best memory on this show?
I think for me it was the overall atmosphere to work on such an iconic franchise and to be able to recreate and evolve effects like bullet-time or Smith’s recursion. And from what I’ve heard, I was not the only one in the team feeling like this…
How long have you worked on this show?
We started in early 2021 and most of the shots were delivered in July. Some shared shots even a bit later as we were waiting for elements. So overall it was about 7-8 months.
A small team was even on-set in Babelsberg during the shoot to support with virtual production.
What’s the VFX shots count?
We delivered about 400 shots.
What was the size of your team?
The total size of the team was around 90 people at its peak.
What is your next project?
After a short break I started to work on Last Voyage of the Demeter, which is also a long term project and I am very excited about it as it comes with several challenges like large scale ocean simulations. And I like oceans and simulations.
A big thanks for your time.
WANT TO KNOW MORE?
RISE: Dedicated page about The Matrix Resurrections on RISE website.
© Vincent Frei – The Art of VFX – 2022
