Marten Larsson have joined Digital Domain in 2004 as technical director. He worked on many projects such as THE MUMMY: TOMB OF THE DRAGON EMPEROR, 2012, TRANSFORMERS: DARK OF THE MOON or ENDER’S GAME. He talks to us today about his experience as VFX Supervisor on PIXELS.
What is your background?
I studied Computer Science in college (aimed towards media programming, computer graphics, image processing) in Sweden where I grew up. My first job in VFX was in the software department and from there I transitioned into FX and then on to become a supervisor.
How did you and Digital Domain get involved on this show?
Digital Domain’s production company used to represent Patrick Jean, the director of PIXELS short film; so we knew about this project very early on. Once Chris Columbus was attached, his people reached out to Senior Visual Effects Supervisor, Matthew Butler to work on the project.
How was the collaboration with director Chris Columbus?
Chris is great! He was a pleasure to work with both on set and in post. He is very focused and enjoyable to interact with especially on such a fun project.
How was this collaboration with Production VFX Supervisor Matthew Butler?
I have worked with Matthew for several years now at Digital Domain; it was great to have him on the client/production side our history enabled us work really well together.
What was the work done by Digital Domain?
We worked on the following sequences: PAC-MAN, Centipede and Donkey Kong along with many shots of people dissolving into voxels and reforming from voxels. We also did some work on the Paperboy and Max Headroom sequences.
Can you describe one of your typical days during the shoot and the post?
A day during the shoot was actually at night most of the time. Most of our shots were night scenes. When we were shooting the PAC-MAN sequence for example, we’d get to set around 6pm. PAC-MAN was shot in downtown Toronto, which was made to look like New York. We’d close the streets off from traffic at 7pm and the street was then dressed to look like New York. Around 9.30pm it would finally be dark, and we’d start shooting; production would then shoot until we started to see the sky go blue again in the morning, usually about 5.30am.
A typical day in post would be a bit different depending on the day of the week. Monday(s) for example, was the day for going through all the sequences with the sequence supervisor(s) Joel Behrens (PAC-MAN), Alex Millet (Centipede) and Bryan Smeall (Donkey Kong). Meetings started at 11:30am in our screening room at the studio, we’d go through dailies with the artists and give them feedback. In the afternoons we continued going through the sequences and then at 4.30 pm we’d do nightlies where we give a second round of feedback to artists so they had a chance to address notes and put things in the render cue for overnight rendering. I also had an early check in with our animation director Jan Philip Cramer almost every day.
Can you explain in details about the creation of these characters?
Our characters were modeled as regular skinned characters and rigged in a fairly standard way. Some extra constraints were in place to ensure that some motion was restricted in order to prevent the character from revoxelize too much when there wasn’t as much movement. On top of this we did some filtering of the point positions of the animated characters as a pre-step to the voxel conversion, to further calm down the revoxelization. The voxelized version of the character that was rendered was generated per frame using the animation output. The animators worked with the prevoxelized character but had the ability to turn on the voxelizeation process to check poses and facial expressions when needed. A lot of attributes were also transferred from the skinned character across to the cubes (or voxels as we called them, they were not actual voxels as in a volume format, they were just modeled cubes). This was helpful for using the skinned character to carry attributes that tracked with the character and stuck on the per-frame generated voxelized character.
What was the most complicated character to create and why?
I’d say Donkey Kong, because he is such an iconic character and the size of the voxels he was made of was so close to the size of the features of his face. His eyes are exactly four voxels big, his nose is three wide and there is exactly one row of voxels in height between the nose and the eyes. This meant we’d have to be very careful when we voxelized his head to make sure that we preserved all these constraints while still making sure the face revoxelized as he spoke so it didn’t look too static. It was also a bit challenging to make sure the facial expressions read and we had to exaggerate some of the animation to be able to read the expression we wanted after the face got voxelized.
Another tricky part was how we translated a 2D character into a fully 3D character that moves in a physically plausible way, while still trying to retain the spirit of the simple video game character. Donkey Kong has a very iconic look that was fairly easy to match when you look at him from the front and he is in a pose that matches the video game. But when you start looking at that same model from a three quarter angle it looks less like the game since you have never seen him from that angle before. Add motion to that and you’ll end up in a pose that he never does in the game, which makes him look even less like the familiar game character. In the game there is no in between motion, he only turns 90 degrees in one frame. So, when you move him in a way that is natural you’ll always have poses that are nothing like the game. There was no real silver bullet to solve this. We basically tried to get him looking as close to the game as possible in the iconic poses and angles and try really hard to not show him from any angles that are too different from the game.
Can you tell us more about the shading and textures?
It was a bit of a challenge to make the characters not look too simple. They needed to be made out of these very simple boxes, or voxels, that are very simple in their shape. The idea is that even though they are made out of the simple voxels to resemble the pixels from the original low-res games, they were still aliens with very complex technology. And it gave us an excuse to add some light emitting energy within them and make things look cooler and more interesting. So in order to make the character still look complex and more real we added a lot of detail and patterns to both the individual boxes and the overall character.
We also ran into the issue of flat surfaces early on. If you look at PAC-MAN he is a sphere. If you build that sphere out of boxes you essentially made a flat mirror. So the first test render had all these really odd looking reflections of lights and objects perfectly reflected on PAC-MAN and it looked really strange. Your brain is telling you that you are looking at a sphere but you are seeing flat mirror like reflections on the surface. We got around this on all characters by blending some of the pre-voxelised model normal into the normal on the voxels.
The PAC-MAN sequence has a lot of destructions. How did you handle and created them?
The destruction is fairly standard with the one difference, in addition to breaking things everything was also voxelized, i.e. turned into voxels. It created this odd looking mix of debris, sparks, smoke and energized cubes. Our general rule was that anything he bites turns into voxels or cubes and anything he bumps into is destroyed as if he was a normal earth made object. Of course we broke that rule a lot and did whatever we thought looked more interesting but in general we tried to stick to it.
The characters emit a lot of light. How was created the light interactions on-set?
Looking at PAC-MAN, he is essentially a giant yellow light bulb in the middle of a street at night, so we knew that he would throw a lot of interactive light. To help with this, we had a Mini Cooper on set that was rigged with big yellow light panels and generators on the roof. This car was then used as a stand-in for PAC-MAN. It also had a paint pole on the roof with an LED light up top to show the height of PAC-MAN to help with framing of the shots. For some shots the interactive light car was only used for lighting reference and in other cases it was present in the filmed plate. We ran into cases where the timing of the interactive light car didn’t work with what Chris wanted PAC-MAN to do in the animation and we did end up painting out both the car and the light it threw in some cases. Overall it was very helpful to have as a reference though, and in some shots it was all the interactive light we used in the shot.
We also did lidar scans of most environments to help our modeling and tracking departments. We knew we needed to model a lot of geometry that needed to line up very closely to buildings and parked cars in the plates in order to add reflections and interactive light from PAC-MAN in addition to anything that was already in the plate.
In the end we modeled a bit more than we what we originally had planned but it helps so much to really show off the interactive light to sell that PAC-MAN is really in the scene, since it is pretty obvious that PAC-MAN is fake, no matter how good we make him look.
What was your feeling to be created those legendary games characters?
I really enjoyed it. So many people know these characters so it is both challenging and exciting to be able to do a new take on them and how they could look if they were real. Right now they are advertising for the movie all over Los Angeles and it is pretty cool to see giant voxelized versions of the iconic characters up on billboards and having been part of generating those.
What kind of references and indications did you received for the pixelating and the beaming up?
We did a lot of our own testing, but we also referenced the original short by Patrick Jean. In some cases we followed very closely to how he had solved the voxelization and in other cases we deviated a bit.
Was there a shot or a sequence that prevented you from sleep?
I can’t think of any specific sequence. They all had their challenges. Donkey Kong had a lot of animation challenges and quite a bit of roto. Centipede had so many moving parts in each shot since there were a whole bunch of centipedes, debris from them, mushrooms and light cannons and tracers in most of the shots. PAC-MAN was a bit of a challenge to get to a look that made him look like he emitted light but still retained a round shape. The PAC-MAN sequence was also where we tried to set the look of what the voxels looked like in general in the movie, so that took some time too.
How long have you worked on this show?
I started early tests in October of 2013. Most of the team came on after the shoot finished in September of 2014. We delivered the show in July of 2015.
How many shots have you done?
We did a total of 361 shots.
What was the size of your team?
About 160 people, from both of Digital Domain’s studios were involved with the show in total.
What is your next project?
Vacation! We are going to take our 8 month old daughter to Sweden for the first to meet her extended family.
A big thanks for your time.
// WANT TO KNOW MORE?
– Digital Domain: Official website of Digital Domain.
© Vincent Frei – The Art of VFX – 2015
