Shaun Roth has been working in visual effects for over 15 years. Before joining ILM in 2020, he worked at Zoic Studios and DNEG. He has worked on films such as Captain America: The First Avenger, Interstellar, Star Trek Beyond and The Fate of the Furious.
Adam Ferrall-Nunge has been working at ILM since 2007 and has worked on projects such as Warcraft, Black Panther, The Irishman and The Book of Boba Fett.
Rasely Ma has worked in many studios such as Digital Domain, MPC and ILM. He has worked on many projects like Godzilla, Transformers: The Last Knight, Aladdin and Jungle Cruise.
Marko Chulev started working at ILM in 2011 and has worked on projects such as The Avengers, Pacific Rim, Kong: Skull Island and Terminator: Dark Fate.
Before The Witcher, Robin Witzsche has worked on shows like The Mummy, Jurassic World: Fallen Kingdom and Space Jam: A New Legacy.
How did you get involved on this show?
Shaun Roth – CG Supervisor // For me, it was my very first show at ILM after having joined in the fall of 2020. I was a huge fan of Season One of The Witcher on Netflix, so I was very excited to be a part of Season Two.
How was the collaboration with the various directors, and the VFX Supervisor, Dadi Einarrson?
Shaun Roth – CG Supervisor // Most of our interactions were with VFX Supervisor, Dadi Einarrson. I found him to be great, and very reasonable. From a spec point of view, all of the set data we received (LiDAR, photography, element shots, Macbeth ColorChecker, etc.) from on set wranglers, and third party was top-notch which made our jobs much easier. Dadi very much knew what he was after, where he wanted to go, but was also completely open to suggestions and our creative input. I think we earned his trust early during the work on Nivellen which made the collaboration much more enjoyable.
What were their expectations and approach about the visual effects?
Shaun Roth – CG Supervisor // As mentioned above, I think we earned Dadi’s trust after the first few months of work, and this meant he allowed us to explore avenues and approaches to some of the work, but it also gave us concise notes and clear direction.
How did you organize the work with your VFX Producer?
Shaun Roth – CG Supervisor // We took the approach of identifying “hero” shots and working those up quicker when organizing the work. Alun Cummins and Bruno Baron – respectively VFX producer and VFX supe – knew we could amortize the work, and that required a pragmatic approach: keep working up the look in key angles, while the rest of the sequences trailed behind slightly but working towards the same goal or look.
How was the work split between the various ILM offices?
Shaun Roth – CG Supervisor // The majority of the show and shot work was done at ILM’s Vancouver studio, while research and development for our Flux Lite face capture technology (which was used on Nivellen) continues to be done at our San Francisco studio.
What sequences were created by ILM?
Shaun Roth – CG Supervisor // ILM was responsible for Nivellen in Episode One, and any shots with the Wild Hunt, and the Dark Elf Sphere environment. We had a string of shots in Episode Four when Ciri has her vision with the monolith, and we had a few shots in Episode Five where we introduced the Wild Hunt during a vision in the clouds, and then Episode Eight–the season finale–where Geralt, Ciri, and Yen are transported to the Dark Elf Sphere. The majority of our work though was focused on Episode One with Nivellen where we delivered around 106 shots.
What was your approach about the new character, Nivellen?
Shaun Roth – CG Supervisor // In very broad strokes, two things drove the initial approach. First, we knew Nivellen was a boar-like creature. So of course, we used real world reference, photographs, anything we could find to give us that “mood board” to go off of. Secondly–and most importantly–is that we knew Nivellen was going to be played by Kristofer Hivju. Kris is a very talented and expressive actor with a distinct look, and he has great facial expressions. Knowing the importance of maintaining his performance, we knew the creature had to be designed in a way that could handle that. So the first thing to do was to create a CG model of Kris’s face, and work backwards off of that. As for the approach, we knew we wanted to tackle the challenge through on-set facial capture principle photography and leverage ILM’s Flux technology that was originally developed for The Irishman and has since been used on Jungle Cruise and gotten better and better as development continues.
Adam Ferrall-Nunge – Senior R&D Engineer // Speaking to the facial performance part of the approach, we knew that we wanted to leverage facial capture to get the essence of Kris’ performance, and to convey some of the subtlety that comes from capture, while providing a basis to allow animation to focus on areas to really perfect the performance on Nivellen. The capture was done with a single head mounted camera on a rig designed by Technoprops. We wanted to leverage our proprietary markerless facial technology called Flux, to capture Kris’ performance and retarget onto Nivellen. Flux has the benefit of giving excellent results with very sparse facial markers, which was the case with Kris’ make up.
Building on technology developed by Stephane Grabli and Jeffery Yost from our R&D team, we integrated ML-based virtual landmarks designed by our colleague at Disney Research Studios’ Jacek Naruniec to provide the foundation for Flux Lite, a more approximate, lightweight, and automated version of our Flux solver. The goal was to allow facial capture supervisor Rasely Ma and his team to quickly capture Kris’ performance and transfer it to Nivellen, using the same animation rig controls that would be used to augment the final performance.
Can you elaborate about the creation of Nivellen?
Shaun Roth – CG Supervisor // The ILM Art Department played a key role upfront in trying to get a broad strokes look for Nivellen. Kouji Tajima, one of our amazing concept artists, did some incredible initial portraits for what Nivellen could be, and helped give a direction for the CG model. As the model progressed, the concept continued in tandem as well, helping to both shape the CG model, and to have any questions we had about the look answered by Dadi. It was a great collaboration between our Art and digital model shop departments. When it came to the CG model, we were not able to get Kris into a scanning booth due to the global pandemic. Since we could not wait for it either, our Modeling Supervisor, Marko Chulev, did it the old-fashioned way and modeled and groomed it based on images of Kris. Both the hair and facial hair are such a big part of Kris that you simply can’t evaluate the model without a groom that we could re-use as a starting point on the creature. Luckily he is a very photogenic actor! To make iterations quicker for client feedback, we implemented ILM’s “Rapid Prototyping” process. This allowed Marko to model, texture, and groom Nivellen in a bit of a bubble without having to be concerned with downstream work. Our look development lead on Nivellen, Hugo Debat-Burkarth, spent a lot of time researching boars and specifically, their skin. How does the skin react to light contribution? Is the skin leathery? Soft? fuzzy? Again, the boar was where we started, and then Nivellen would evolve from there. Does Nivellen have a natural wet look, or is he very dry? In the end, we got a very realistic skin texture that complimented all of the other elements in a very natural way. As we began to see Nivellen composited into the plates with animation, we would make adjustments as needed.
Lighting-wise, we had a very nice palette to work from. All of our work was rendered through Katana using PRMan. Markus Sterner, our lighting Supervisor, stitched all of our HDRI’s to give us a nice base lighting rig (IBL) which was also leveraged by our animation team for presentation purposes. Once we got into it, Markus would get into the nitty gritty of adding lights from fires, candles, etc. There were creative things we needed to address, like crumbs in the beard, that we could turn off in lighting; maybe reduce the width of certain hair layers, like bristles on the forehead. As I said, the dynamic lighting really played to our advantage and let us really emphasize Nivellen and all of his key attributes. Once we had a base setup for a sequence, we were able to render out whole sequences often with the same settings, from there we got amazing results from a version 1 lighting pass.
Compositing, supervised by Ryan Clarke, alongside comp lead Simon Herden, would rebalance as needed, and blend back to the plate. Our CG model and the onset prosthetic was basically a 1:1 match that we could dictate between a full head replacement, or a blend to the practical hair of the prosthetic on a shot-to-shot basis, Spec from the skin and how it reacted to certain lighting environments would come up, and need a rebalance from shot-to-shot. For example, in some shots he might look sweaty, others perhaps a little too dry, the artists had total control to tweak anything they needed to so the effect would be seamless. As for the eyes, even though they were CG, we would still pull the pings out of Kris’s actual eyes and add them into Nivellen’s eyes, removing studio lights in the process.
Can you tell us more about his animation, and especially his face?
Rasely Ma – Layout/Face Capture Supervisor // With Nivellen our goal was to capture every detail of Kris’s performance. He is very expressive, but as you can imagine, translating that performance onto a much different, larger face with tusks can be difficult. To do this, we worked together with R&D in ILM San Francisco to develop ILM’s Flux Lite facial capture process which is based on our Flux facial capture system. The facial capture team also worked very closely with the animation team. We paid extra attention to get the best curves possible for both departments, since Kris had to be retargeted onto our much larger Nivellen model which then would be passed onto the Animation team’s hands to get cleaned up and finaled. After each facial capture is solved, we always want to give it a cleanup-match pass to make adjustments, additions to make sure it is as close as we can get it.
Marko Chulev – Modeling Supervisor // Early on, due to restrictions as a result of the pandemic, we didn’t have access to full FACS scans for face shapes of Kristofer’s face at the phase we were building and developing his creature face, so we used interview footage for key expression to translate the likeness on the fantasy creature, Nivellen. We ended up choosing about six expressions (Anger, Disgust, Happiness, Surprise, Fear and Sadness), and did a side-by-side to show the resemblance and make sure we keep the essence of the character so you can recognize the actor doing the same expression). At a later stage we received a limited FACS shape to support the actor model, and then we built and maintained a face shapes library that had parity between Kristofer and Nivellen. We worked closely with our animation team and our face cap team in the development stage of the character so we could design his shapes to be able to hit the performance for Nivellen. After achieving parity, and calibrating both face shapes on the actor and creature, we handed over to our animation team to start shot production.
Shaun Roth – CG Supervisor // Animation would take the face capture data, which were curves on the rig, and do what was called a “beast” pass under the supervision of Michael Beaulieu and animation lead Mark Della Rosa. This beast pass was not only a clean up stage to account for things like the tusks around the lips, or larger features like the brow and nose, but to emphasize the performance Kris did to make sure it read on Nivellen. We needed to capture Kris’s performance so when the audience watched Nivellen they could tell it was Kris. There was a lot of work going into the face capture performances to make the dialogue read, the performance come across, and the Animation team had to make slight changes where necessary to accurately translate Kris’ performance to Nivellen. It helped that Animation had been working with the HMC footage for everything, and they knew all of Kris’s ticks and movements in his performance, so they were able to replicate it just by looking at the raw plates.
We developed a tool called “Sneakview” which allowed an animator to submit their animation to Sneakview, which inturn would render through a Katana template in order to get a lit version with textures, lookdev, and the groom, and then get it comped over the BG plate. The quality settings were tweaked to ensure quicker renders, so you might get a bit of noise, but for an animation submission it was perfect. This, along with representative environment lighting, gave us what was essentially a lighting slapcomp.
How did you work with Kristofer Hivju who played Nivellen?
Shaun Roth – CG Supervisor // We did not have much interaction with Kris personally, at least by the time I came on the show. At the beginning while we were in the process of building Nivellen, the prosthetics team was working on their end too. Both sides kinda worked in tandem. We would be able to show them what we had done, and advise on how large the opening was for Kris’s face should be to facilitate a better capture from the HMC – things like that.
How did you handle the eyes and fur?
Shaun Roth – CG Supervisor // The fur and hair was really a combination of Kris’s classic red hair look, and bristly boar hair. The wiry and bristly hair was great and didn’t require simulation due to its stiff nature; we just had to account for things like shoulder collisions. We had boar bristles on the forehead and brow which were very distinct to a boar. Basically they were hair follicles that were two to three times as thick. As we got into shot work, we found this to be a bit distracting, so we ended up making them thinner, and reduced the amount.
The eyes were a bit of a journey. Initially, we had very dark bloodshot eyes, very much in the realm of a boar or beast. As a concept, this looked great, but as we started to get animations and facial captures in, we found that it was hard to read at times, and in a way, made Nivellen much less relatable. One of the last changes we did on Nivellen was to move him back into a much more human white sclera eye. This really cemented him in and was the final touch. Once we saw those eyes in the shots, we knew we had nailed the look.
Robin Witzsche- Groom Artist // For Nivellen’s groom, it was important to keep as much as possible when it comes to the character of Kris’s brows, beard, and hair. We used photos of Kris, Marko Chulev’s rapid prototype as a 3D concept, real world boar photo reference, and the beautiful 2D concept art by Kouji Tajima (and lots of back and forth amongst the group) to create the final look. We added a bit of debris in his beard as a final touch to help sell his rough character. His hair also had to blend seamlessly with the hair on the back of the prosthetic. We had to be conscious of not putting too much fur on the parts of Nivellen’s’ face that needed to be expressive, so that Kris’ performance could easily come through. As Shaun noted, we ended up reducing the amount and thickness of the boar bristles on his face to do that.
Can you elaborate about the Nivellen transformation to his human form?
Shaun Roth – CG Supervisor // For a while, this effect went back and forth: was it going to be a full transformation, with all the bells and whistles? Or were we going to hide it behind a pile of fire and poof… he’s human again? We opted to meet in the middle. We wanted it to be fast, with hints of drastic grotesque changes to his features to make it obvious to the audience what was happening, but not give away too much. Again, we leaned on the amazing talent in ILM Art Department and had them mock up frames for us of what a few beats peeking through the flames might look like. This concept (done by Kouji Tajima) got the thumbs up from both Dadi and the showrunner, so we used our digi-double of Kris, with two drastic blend shapes modeled after the conceptBehind a moment we knew was obscured by flames, the Comp team would maintain what they could from the plate, especially mid-way through when he was looking more human, as we still had a matchmove we based it all off of. I think going this route was very successful – we could read what was happening, and it was quick and grotesque enough to fit the vein of the series.
How did you create the Dark Elf Sphere environment?
Shaun Roth – CG Supervisor // This environment was meant to be very desolate and unique in nature. Dadi did a shoot in Iceland to capture a lot of the landscape for us to use, augment, and inspire during our build. Since getting the ILM Art Department involved worked so well in the past, we again turned to them to give us some preliminary investigations and looks. We looked at more muted palettes: blue, orange; one sun, two suns; unique rock formations scattered throughout, trying to find what this world could be. What you see is what we opted for in the end, with a very orange environment, three suns, and sharp-like rock features. The surface of Mars was a bit of an inspiration in terms of colours and desolation, but we didn’t want people to look at it and say, “Oh, that’s Mars”. So we played around with the orange colours, to keep it in the same palette but far enough away from the traditional cinematic Mars-look. All the rock features played a huge role in keeping that distinction as well, as did the three suns. In terms of how it was shot, Geralt, Ciri, and Yen were on a soundstage, surrounded by greenscreen, standing in a pile of rocks.
Secondly, we had our Wild Hunt in full gear, riding down a beautiful beach in the United Kingdom on a nice sunny day. We then went in and did a full roto and a rework of the plates to get them in-line.
And then we had our environment to tie it all together. We spent a lot of time making sure our sequence CDL was working, and getting our neutrals in order to balance everything out. Our Comp Supervisor for this work was Miguel Macaya Ten – and he has a fantastic eye for colour and CDL’s. He created a great base for us to work with, and then pitched it back to the clients.
How does the specific light of the environment affect your lighting work?
Shaun Roth – CG Supervisor // The three suns made it interesting to light, especially when our plate elements had very strong key lights with the Wild Hunt on the beach. It’s also not appealing to see shadows being cast from each sun, so we would opt for one being the “stronger” one. We definitely played a lot with lighting direction, as our generalist team built it up while staying in tune with the plates we had.
In the end I think it’s successful. When you watch the sequence, you don’t question the lighting even though you see the three suns in the sky. It all compliments very well throughout, which in the beginning was the largest challenge to tackle in this sequence.
Can you explain in detail about the Wild Hunt creation?
Shaun Roth – CG Supervisor // The Wild Hunt came in two forms: one being the physical form of the riders in the Dark Elf Sphere world, and the other being the vision in the clouds. All of the elements were shot on a beach in the United Kingdom, with more-or-less all of the camera work done for them in Episode Eight captured on that beach.
For the work done in the clouds in Episode Five, each rider was shot separately riding towards the camera. The idea was to use the chosen takes from each rider, and repo them into position. Some were massive (the size of a building), while others were tiny. We played a lot with these positions, but even when it looked good from a composition point-of-view, we still needed to look at it from a 3D point-of-view, as we needed to emit fluids and particles, and have them fly through clouds. We needed to be spatially aware of each rider so we didn’t obscure them too much from all the other riders.
Believe it or not, the inspiration from Dadi was from an old Guinness Beer Surfer ad from 1998. It had horses superimposed into the waves as surfers were doing their thing around them. Very much a “1998” type of effect, but it really helped us identify the direction of what Dadi wanted to achieve with these shots. Lots of FX were used to integrate them into the clouds, and to give the impression that they themselves were made of clouds.They were somewhere in the middle of being physical beings and manifestations. Comp had the task of helping piece it together, blend it to the BG plate, and add in elements to help us integrate everything. We had to build this in a way that could be flexible, and to account for creative notes, as this was a big moment in the series – the first reveal of the Wild Hunt.
How did you enhance the Wild Hunt riders?
Shaun Roth – CG Supervisor // In general, the Wild Hunt actors were in full costumes and prosthetics, and looked fantastic. No CG augmentation was needed to enhance them at all. For Episode Five in the clouds, we pushed them back to as close to silhouette as you can get before losing detail. They had to feel very phantom like. For the Episode Eight work in the Dark Elf Sphere, we would do a lot of regrading, and relighting of the plate elements.
Originally we had planned to salvage what we could from the ground around the riders to try and minimize roto, but in the end we opted for a full roto, as the environment they were shot in just didn’t match what we were aiming towards. For the most part, the natural lighting direction and shadows were kept from the plates, and our environments lit to that. FX was used to enhance the horses’ impacts on the ground to really make them terrifying, like a charging stampede coming our way. They needed to feel powerful, heavy, and frightening.
Which shot or sequence was the most challenging?
Shaun Roth – CG Supervisor // Nivellen was a challenge but once we had the recipe down, we could push that out into all the other shots and we started to get a feel for it. The Episode Eight Dark World environment was interesting because we had to take three elements (riders on a beach, actors on a set, and a CG world) and get them to sit well together.
I think the Episode Five work was the most challenging from my perspective, because it was very subjective, and almost magical in nature. Getting everything to work together with the riders looking like clouds, some being much larger than the others, using the sky environment to our advantage, but trying to make sure it didn’t look too CG, all added up to the complexity of the shots. Luckily since it was confined to two shots, there was a bit more flexibility in how we approached it. We didn’t have to worry about this being a look that had to propagate into dozens of other shots, so we could use all the tricks at our disposal to achieve the desired look.
What is your favorite shot or sequence?
Shaun Roth – CG Supervisor // Oh, I loved all the shots with Nivellen. The quiet moments of Geralt and Nivellen discussing things over a drink is a great moment. I think the shot where Nivellan drops the bathtub in the background for Ciri is a fun one. We get a lot of expressions from Kris’ performance, and really get to show off the character. Plus seeing a bathtub dropped from a giant green mechanical arm was a fun moment!
What is your best memory from your time on The Witcher?
Shaun Roth – CG Supervisor // Hmmm, as cliche as it may sound, I loved working with the team on this one. Being a smaller group, all contained mostly in Vancouver, we got to know each other quite well, and really got along great, even though we were all working virtually from home. Lots of praises, laughs, and amazing work came off this show. I even got to get my hands dirty on this one and did the FX when Vereena bursts into flames at the end of Episode One. It was a nice throwback to my FX days! Also, I think Nivellen is one of the best looking creatures I have had the pleasure to work on. It was great seeing him come to life looking as good as he does.
Robin Witzsche – Groom Artist // I’m a big fan of the show and Nivellen is one of my favorite characters that I have gotten to work on so far! Everyone was really great to work with, especially transitioning to working from home in the middle of a show with barely a hitch! It was a real highlight to see everyone’s work come together and see him really come to life!
Rasely Ma – Layout/Face Capture Supervisor // I think the team collaboration and communication were the highlights of the show for me. With the amount of work and the speed of completion needed per shot for Nivellen we had to constantly talk, message and develop workflows to make sure there were as few bumps as possible between the departments. This was the case as early as the initial testing, devving, and setup-phase of the show. The teamwork was unforgettable!
How long did you and the team work on The Witcher?
Shaun Roth – CG Supervisor // The global pandemic pushed back the initial dates so the show went on for over a year and a half. I was on for about nine to ten months myself.
What’s the VFX shots count?
Shaun Roth – CG Supervisor // I think in total.. We had around 140 or so shots. The majority being Episode One and Nivellen, with that work making up for about 106 of that.
What was the size of your team?
Shaun Roth – CG Supervisor // I think we had roughly 40 total on the team, but not all at the same time.
A big thanks for your time.
WANT TO KNOW MORE?
Dadi Einarsson & Gavin Round: Here is my interview of Dadi Einarsson (VFX Supervisor) & Gavin Round (VFX Producer).
Netflix: You can now watch The Witcher on Netflix.
© Vincent Frei – The Art of VFX – 2022
