Ethan Wilde: We're celebrating tonight. The the start of the Regional Virtual Production Academy here at Santa Rosa.
Ethan Wilde: We've got some students here in the room. We're eating some pizza. some junk food. We just spend a little while looking at
Ethan Wilde: epics meta humans with our faculty member.
Ethan Wilde: Shawn Nelson, who's here with me tonight, too? Who's working on his rig over here? Did some Demo Shawn's gonna be teaching our introduction to virtual production class third week of January.
Ethan Wilde: For that's great.
Ethan Wilde: We've had a lot of interest. So we're we're really excited. You're able to join us.
Ethan Wilde: And let's see, I think I made you Co. Host. So you should have.
Ethan Wilde: The controls at your fingertips.
Kon Wilms: Okay, let me
Kon Wilms: let me share something I put together. Give me 1 s.
Kon Wilms: Hopefully. Everyone can see this.
Ethan Wilde: Yes, yes, turn the volume up a little bit, too, on the big screen here. Okay, great. I hope the I hope the text is like legible.
Kon Wilms: so let's see, what can I start? Ii so II put this together sort of a
Kon Wilms: so I'm with Amazon. And just for, like an intro I work in a group called creative tools. And it's a product team within aws. Aws, does cloud computing basically stands for Amazon web services.
Kon Wilms: And some of the things that we do for creative tools are to create tools for artists. So we have deadline that's used for a lot of productions to render offline visual effects.
Kon Wilms: And then we have a couple of other products as well. And one of the areas that we sort of experiment in to see if it would be useful for other products or new products. Or you know, if we can help.
Kon Wilms: let's say, run things in cloud, because there you can just turn the infrastructure off, and you're not paying for it. There are a lot of advantages to do it doing it, but there are also a lot of challenges.
Kon Wilms: But one of the areas we have is virtual production. And
Kon Wilms: we've been doing this as a group for about like 3 or 4 years.
Kon Wilms: So I'll just, you know I'll get into. I kind of gave a you know, an introduction. I
Kon Wilms: I'm a technical lead for this. let's say, a virtual production product segment, if you will.
Kon Wilms: And I've been in this visual effects industry since about 1992 or so, working at various companies, company, 3 method deluxe, etc.
Kon Wilms: The interesting thing about
Kon Wilms: this is, you know, how do we put things that traditionally run on set? And
Kon Wilms: you know, you need a lot of hardware. You need low latency. It's very challenging to put a lot of this stuff into. Say, the cloud, just because of latency.
Kon Wilms: But we've been experimenting with this, and I'll sort of go through where we started from. So the initial
Kon Wilms: infrastructure we built and the the architecture and design, and what we were trying to do, etc.
Kon Wilms: And then I'll I'll go through what we came. You know we did this experiment a couple of years ago. It worked successfully. And then we took that. And we basically went from. We started with the green screen, and I'll go through what we did for that process. And this was on like unreal version 4.2 5, I think.
Kon Wilms: And then we went all the way to. Now we're running on a led volume with unreal 5.3 a whole bunch of different hardware. You know. A lot of things change over a couple of years.
Kon Wilms: So hopefully, this is useful for all of you, just in from, let's say, perspective of
Kon Wilms: where things were, where they are.
Kon Wilms: and also you can hopefully see through this that a lot of the techniques are not necessarily rendered, you know, antiquated or not used anymore or no, no benefit to them. There's much benefit now to building out this kind of stuff with green screen, as there is with you know. Say, a volume or pixel array, or however, you want to look at it.
Kon Wilms: In some cases there is actually
Kon Wilms: more use cases for a green screen. But anyway, hopefully, this will you know, give you a good understanding of that. If there's anything that you already know, or where I'm stating the obvious, please just interrupt me.
Kon Wilms: Or if you've got any questions you know, feel free to just jump in and ask as well.
Kon Wilms: So we started with, this is an internal series.
Kon Wilms: at Aws, there's a group called Media series. And they basically do these small documentary shows that get published to Youtube called, this is my architecture. So this particular one here is on toyota has or had a bunch of IoT based vehicles so like self-driving cars right? And then we're starting to play around with the technology for this.
Kon Wilms: And an easy way to demonstrate this and talk about like, what is the architecture when all of these devices run in the cloud is to actually have a practical set right? So you can see one of the cars and the and the robotics in the background of the set. So
Kon Wilms: we wanted to achieve this on green screen, where we could fluidly move the camera around, capture everything, do the lighting, etc. And this was, you have to bear in mind. This was like just before
Kon Wilms: Covid. This was maybe 2019 2020 timeframe. So you know, at that time the
Kon Wilms: there's a lot more expensive to build a volume. There wasn't really that many. They want that many options to build a volume for something.
Kon Wilms: That's almost like advertising, based right? So like the panels won't commodity like they are now. So the best way to do this was green screen at that time, and we sort of looked at like, how should we do this right? Where do we start? Do we start from
Kon Wilms: the right side of the equation, where you have full tracking. You have, you know, a large volume you have in camera visual effects.
Kon Wilms: Or maybe somewhere in the middle where we're using commodity like projectors. Right? We're not really caring about color balance, etc., cause we don't have to. Maybe it's just a shot like this is, for example, a shot through a car right? Or are we going for something where it's traditional green screen?
Kon Wilms: So we ended up going to the left side, because, as you could see in a previous frame, we wanted everything to seamlessly fit together. From, for example, the host sitting on the floor. That's also green screened, and then have that key into you know, like a a
Kon Wilms: hardwood floor, for example, so like that had to be key. So that meant that things we we needed to be able to color great, etc., accurately
Kon Wilms: and to move to the right at this point in time was complete. Overkill.
Kon Wilms: So just to give you an idea of you know, how did we construct this? What did we put together? What did it look like?
Kon Wilms: So we ended up. We initially started with Htc, so just 5 trackers cause it wasn't anything like Mars trackers right then. And then we quickly moved off after testing that. Just poc, that to using the Moses trackers.
Kon Wilms: You know, as you're probably aware, there, there's inside out, tracking outside, in tracking, etc. But most of were pretty much the only game in town at that point in time that were really efficient could work well on a green screen, could get configured quickly and worked with unreal
Kon Wilms: so we wanted to build all of the backgrounds in the game. Engine. We didn't want to use anything. Third party, although there are lots of options to do that like X symmetry, etc. Now, right? And there are lots of things that exist now, where a lot of the work is done for you. So at that point in time none of the more advanced tools are out. I don't think even disguise has released anything just as like a random example.
Kon Wilms: So our pipeline consisted of tracked camera rigs
Kon Wilms: unreal engine on a separate server and then on the right hand side. Here, you know, we're basically doing green screen chroma processing and then using the compositing system inside of unreal to comp the live foreground plate onto the background plates. So the live actor onto the unreal scene.
Kon Wilms: And then the output of that was just recorded locally to disk. And then we ship that up to frame I/O, and that's basically the pipeline.
Kon Wilms: if you look at where virtual production was then, and I'm sure. Ii assume everyone has seen us in some shape or form, hopefully. Yes.
Ethan Wilde: yeah.
Kon Wilms: Okay, so this is. This is like the you know.
Kon Wilms: I think there's a a handbook or guide set of guides that were were released around this. For in camera visual effects and virtual production.
Kon Wilms: that Usc, did a lot did a lot of work on but essentially, this is epics. View of virtual production, and like 2020, you know where you're going from all the way from source written treatment into various parts of pre-production
Kon Wilms: and then finally into production. Where you're looking at either we're doing green screen or in camera visual effects, or we're just doing live action. And we're doing, compositing later
Kon Wilms: those 3 facets. And I'm pushing that into editorial. So we were focused heavily on this number one here. So like just trying to do everything with green screen and do it in unreal engine and use a lot of commodity off the shelf stuff.
Kon Wilms: So I'll actually just skip ahead, and then I'll go back.
Kon Wilms: this is a list of what we were using. Or at least more of a list of things that were important for us.
Kon Wilms: right? So, like the stage configuration, we had to be able to bring all of the equipment to a green screen. Facility. Set it all up really quickly.
Kon Wilms: I think we used automatic studios and Van Nice.
Kon Wilms: They're somewhere around that area. But they're they at that time, or a traditional mocap stage, right? So you can think of that as large green screen facility with lighting. But almost no other hardware software servers anything like that. Just a bunch of hot disks to set up equipment on
Kon Wilms: so we needed to have everything pretty portable from one perspective. So we had a lot of live cases that we just configured. Everything in. Everything was racked and ready to go all the way from
Kon Wilms: the unreal servers
Kon Wilms: all the way down to capture and then upload to cloud or upload to frame. I/O
Kon Wilms: I'd say maybe the only important thing here was
Kon Wilms: because we had a crew, and you know, even though you look at this, and you go well, that's kind of like a simple production, right? There's still time and and expense to move all the equipment schedule a crew go through the Covid testing. Get on, set.
Kon Wilms: set it all up and shoot. So we had a lot of, I'd say backup, and that could have been like one or 2 extra machines. So our unreal system, for example, that consisted of
Kon Wilms: chroma keys. Sdi routers rack mounted servers, blackmagic capture and play out cards. All of that stuff was basically duplicated into racks so that we're completely identical. So you know, we were banking on. If we had anything fail we would just quickly switch to a duplicate and not have any problems there.
Kon Wilms: and the
Kon Wilms: signal chain, for that is this is pretty standard fare. Now, right where you have your tracking devices on your cameras on the top left.
Kon Wilms: This is specific to axemetry. But it's a lot the same if you look at this for unreal or anything else. And this is basically almost like a one-to-one of what we did for this particular project. So most is tracking devices on 2 cameras at that point. One single capture system for compositing and then obviously a sync generator for genloc
Kon Wilms: local switching to route any telemetry or data around. Right? So move the captured data off of the unreal systems onto a Mac for color grading and upload to frame I/O, and then external peers and routers to be able to go. I wanna take this Sdi feed from camera one and I wanna route it through. A chroma keyer.
Kon Wilms: You know, blackmagic device. Strip out the background will strip out the Co chroma key and garbage matted for me and then send it into unreal
Kon Wilms: so that that was basically the process that we have. And then, if you look at what that looks like, and so that the real world right, we captured this.
Kon Wilms: And this is just a small excerpt right where we have camera A, you can see camera B behind, and as you pan across, it's also an unreal scene represented there.
Kon Wilms: And then the the compositing was, Live here. These are just a bunch of red cameras with like, I think they had. One had a 35 mole, and one had a 50 mole just for different perspectives.
Kon Wilms: And then the tracking devices that Moses has.
Kon Wilms: There are couple ways to do the tracking.
Kon Wilms: You can probably guess that it's pretty hard to do tracking where you're coming into a location, and then you're not really able to stick a lot of markers up on the ceiling.
Kon Wilms: So the one novel thing you can do with something like Amosis that is not actually shown here is you can point it at the ground and put the tracking markers on the ground, or you can point to the wall to say the left, to the right, to the green screen that might not have any material on it and do the tracking off of a side wall. So it doesn't always have to be facing up and away from the camera.
Kon Wilms: so after that we got to this notion of we were capturing front plates. We're doing the backplate in unreal engine, and then we were doing this final graded. Comp. Using tools like premiere nuke da Vinci resolve, etc.
Kon Wilms: The interesting thing is, we did, I think maybe
Kon Wilms: 20 or so episodes or 25 episodes of this. So we did everything from the Rivian delivery truck that
Kon Wilms: prime users right now, you probably if you're in certain areas, you might see them delivering instead of like a traditional non-electric vehicle. But we basically had to get the schematics and all of the parts from Revien.
Kon Wilms: and then I had to bring that in and strip out all of the internal pieces, because all of these CAD drawings have millions of parts, and it doesn't really perform that well in unreal. And then basically put materials in the whole thing and import it into unreal. So
Kon Wilms: there is a set of automotive materials, for example, that epic publishers that give it away for free.
Kon Wilms: All of these vehicles are using that automotive library set. So these are actually really well done, and they have the advantage of you can pick like metallic non metallic
Kon Wilms: different paint styles that you would have on vehicles and just drop them on the cars and pretty much with hardly any work we're able to get this working within the scene.
Kon Wilms: The scene was another. you know, level of complexity, I'd say, maybe ran at like 10 frames a second. So what we would do is, and I might have another frame on this.
Kon Wilms: I'll actually show you what this final comp looks like.
Kon Wilms: So you can see, we have like smooth camera tracking and movements here. And this is like 2 years old now. But the problem here is that well, actually, there are a couple of things, and maybe you can see some of these. If you look towards the left side here
Kon Wilms: when the camera pans, you'll see that reflection disappear.
Kon Wilms: So that was like a 4.2 5 reflection map problem. But you don't. You can't really see it if you're not looking for it.
Kon Wilms: And it's relatively easy to solve this if you know how to position the lights. So this background scene, we did a lot of work to position the lights in a certain way where we knew if we panned the camera we wouldn't have any weird artifacts or occlusion effects.
Kon Wilms: You know, and that's just part of some of the things that you would do when you try and experiment and figure this out.
Kon Wilms: The other thing is the depth of field
Kon Wilms: so at this point in time, couple of years ago, there was nothing to basically map the camera depth of field into unreal engine. So a lot of this has to be eyeballed, and it's easier to do if you're on a tripod.
Kon Wilms: As you can see with this one, right? So like the camera has some movement to it, but it's staying relatively stationary so the depth of field doesn't really have this weird, uncanny valley effect to it, as if you were moving in, and nothing was changing.
Kon Wilms: I'd say the final thing or the problem we had at that point in time. And that was just because we had, I think, a 5,000 gp. Use, so nothing like an a 6,000 or an Ada board right now, that would easily render these. All of this stuff was getting rendered it maybe like 10 or 14 frames a second.
Kon Wilms: and that's because we had about a thousand different light sources in the background to light the cars in a certain way. Cause. I need a lot of reflections to capture the different contours on the cars, and just as one example, and also the highlights on the ground.
Kon Wilms: So what we would do is we would we would use Moses deferred, render to render those frames in the cloud and then bring them back and comp them.
Kon Wilms: So basically, the procedure there looked like when we're on set, we're running with a almost like a stripped down or close to Wireframe preview. So this preview right here is almost like a low resolution
Kon Wilms: of the scene. But it's still recording the camera positioning within that scene. So once that's recorded, you can re-render using, say, movie render queue or using the system that Moses has. But you can then do what's called a deferred render, and you're then rendering out those frames and bringing them back.
Kon Wilms: and then grading them and doing a final Comp. So it takes a bit more time, but it allows you to use cheaper hardware, and then take the time to slowly render those frames with all the ray tracing, turned on, and then come out at a you know a fairly good
Kon Wilms: you know. Final grade, if you will, or final master.
Kon Wilms: So that's what we did for the first I'd say iteration of this process. Any any questions
Ethan Wilde: since the virtual set is, you know, it's all being. It's real time. And you were talking about depth of field. Would you be able to change the depth of field
Ethan Wilde: after you've shot your footage and you're compositing it with the
Ethan Wilde: unreal engine. Yes.
Ethan Wilde: yes, honestly, there's nothing special happening here. It's it's the same like, if you use a vicon system or an optic track system or an Htc Mar system, which is what we're using now. You know everything that when you hit record on the take recorder depending on what kind of tools you have?
Kon Wilms: That camera motion in space is tracked into unreal sequencer. This is to say, if you're using
Kon Wilms: unreal engine out of the box
Kon Wilms: right? And you're not using some third party app that kind of wraps that and hides it from you. Or you know, application suite, or or whatever but essentially that then allows us. You know, we're basically recording the motion of a semic camera actor right?
Kon Wilms: And because we're doing that we can then come back and we can adjust the depth of field. So the nice thing here is that when you are doing green screen you can. You can do like a preview of the plates inside of unreal engine.
Kon Wilms: And you can adjust the depth of field you know, as as as you need to. I'll get to some of the other stuff for the volume based. Capture that we did in a bit. But
Kon Wilms: I'd say, without having the ability to being able to adjust that sunny camera actor. It it makes it. It would be almost impossible to do a lot of these shots. So we're doing a lot of things now where?
Kon Wilms: we will shoot plates and say 24 on a red out in the field somewhere. We'll bring that onto the volume, and then we'll shoot on 50 and then we'll basically calculate what that looks like in terms of a
Kon Wilms: tone, and where the talent should stand, where the screen should be, and where the camera should be, and then we can start playing with depth of field. But a lot of these things we wouldn't even be able to start doing if we couldn't just
Kon Wilms: cheat the system and adjust the
Kon Wilms: you know the aperture or the iris, etc., on the sunny cam actor in some cases to settings that it shouldn't even work at, but it looks it looks acceptable to the human eye if that makes sense
Kon Wilms: any other questions before I get to the next piece.
Kon Wilms: Nope.
Kon Wilms: okay, so where we are today, right? So
Kon Wilms: we experimented with that, we did it off site. And then we were thinking about, how would we actually build products on top of this? We would need facilities to do this because it's just cost prohibitive to always have to find a rental facility and then try to get a production going on it. And while the production is rolling, try to figure out, what data do we need to? You know, start, look at looking at for building products.
Kon Wilms: And a lot of these areas that we have been looking at and are still looking at.
Kon Wilms: They almost fall into these 4 categories right? So there's a notion of or at least an area of
Kon Wilms: if you're doing workflow and you're doing pipelines, and you're doing that on set. And it takes a potentially a couple of staff to do it. Things can go wrong. What if you could automate that and run that in the cloud? So, in other words, your workflow, and your your pipeline controls those run in the cloud, and you just turn them on and off for production.
Kon Wilms: The second part is, how do I manage all my assets so my textures, my materials. What if I shoot something? And it's kind of like a throwaway? Maybe the unreal project stays with it. But I just want to store that somewhere, because I might know.
Kon Wilms: you know, maybe 6 months down the line. I need a version of that texture that I had on a certain thing, and someone wants me to use that because they view the clip. And they have said II need that metallic texture that you have right there. So all of these things have to be
Kon Wilms: kept, versions backed up, etc., so moving that into, say, cloud based storage. It just lets you use it as like a dumping or a landing ground. For all of this content.
Kon Wilms: Which is cheaper than like, say, maintaining tape archives or backups, or, you know, administering a Nas server or no one wants to do that when they're working on this type of work.
Kon Wilms: we're also looking at. Could we do a lot of could we move some of the real time stuff into the cloud and defer it? So, in other words, like, I explained in this previous example.
Kon Wilms: What if you could just do all of this with a laptop right on set? But you're then moving all of these, you know, captured camera movements and unreal scenes into, say, cloud based infrastructure. You're turning on the Nvidia compute when you need to render the frames, and you're turning it off when you're done.
Kon Wilms: That is a lot cheaper than investing in a lot of hardware that's very quickly outdated, that has to run on set and needs to be maintained that needs to be patched and someone needs to know. If the drivers are working, etc., right? So that that's an option, too. That we looked at. And then we looked at some things where you know, one of the problems is okay. If I'm doing deferred processing. And I'm maybe putting some of the stuff in the cloud.
Kon Wilms: How would I simulate that? It's even working right? I can't just keep running tests. Maybe I want to see what the virtual set looks like before I build it.
Kon Wilms: So if I'm doing text or anything like that, maybe I just need an unreal instance in the cloud that I can load a project in. I can show someone that, and then I can delete it when I'm done. I don't have to cart a whole bunch of hardware around with me.
Kon Wilms: So those are sort of the areas, you know, that we've looked at. And then, because this group is creative tools, right? It's a combination of
Kon Wilms: traditional visual effects production. And then you add in these other all these other things that we're looking at. And you come to this equation of all of this comprises real time, game engines, real time rendering.
Kon Wilms: And because you're dealing with the game engine. Anyone operating this has the expectation that the in camera visual effects will be fairly fast and almost real time. So that has a bunch of like interesting challenges.
Kon Wilms: So we ended up building a stage in downtown San Francisco, on Market Street, in the financial district.
Kon Wilms: I'll get into some details about that but these are, you know, just some images here. It's about
Kon Wilms: 25 feet by 12 feet
Kon Wilms: for the the wall. That's in the in the background there and then there's maybe, like
Kon Wilms: 50 feet deep by 25 feet in terms of volume that you have to work with them if you will.
Kon Wilms: This is basically the technical specifications of what that what it comprises, right? So the volume is Aodo led panels.
Kon Wilms: So it's about 4 K
Kon Wilms: in that space that I described. We have a bunch of silver draft nodes those all run, and for the
Kon Wilms: and a 6,008 is. So it's either 2 and the reason that we have 2 is one is for the out of custom, and the other one is for the inner, for system. And that gives us like very good performance for capture.
Kon Wilms: It's not to say you need this. It really just depends on your complexity of your scene that you want to capture in camera.
Kon Wilms: Everything is 12 g, so it's like 2160 p. etc., 4 k. Right? So we capture in that. And then the red camera, for example, that we have the V-rap tool capturing 8 k. And the Re. Alexa Manilf will capture, you know. Around the 6 K. Mark interpolated.
Kon Wilms: We have a lot of storage onsets. There's like a single Mac pro. That's our edit station, if you will. And then we hand a lot of stuff off to the cloud. So that's like the footprint that makes us work. And then it sort of fans out into the cloud from there.
Kon Wilms: We've done a a number of productions on this since, I'd say, since we got going like
Kon Wilms: June, July of this year and launched we have, and I'll get into what those are. So we have an internal group called Fuzzy Pixel
Kon Wilms: and that's a lot of industry veterans. I would say, that are now at aws! It's a
Kon Wilms: it's almost like an internal visual effects team
and they get tasks to
Kon Wilms: do a production with the tools that we think might work. And it's almost like an experiment if you will. So a lot of them come from dreamworks or other types of visual effects companies, and that's some pretty deep experience in animation, visual effects.
Kon Wilms: green screen, etc.
Kon Wilms: On the other. You know other side of the equation, if you will. There's a team called Media Series, and that's what I went through earlier. So they're almost doing these one on one or one on 2 interviews, and then they might. Do you know some pan outs to show how things are working in terms of
Kon Wilms: let's say vehicle telemetry or self-driving cars, or you know, whatever you might think of
Kon Wilms: But it's all more like docus type series on Youtube
Kon Wilms: corporate type, add tech type videos, we've also done some stuff with another
Kon Wilms: partner, union movie works. Michael, one occur and that's being more around using the volume for
Kon Wilms: fashion type used cases. So you know, shooting short form ads, seeing how we could put that together, etc. And then, you know, there's a whole host of other folks that we're dealing with as well. And I'll just go through some of these quickly.
Kon Wilms: So fuzzy pixel used basically a lot of green screen. They also used on bottom, right? some in camera visual effects.
Kon Wilms: And then the visual effects that they did. For the most part. They were compping in animation on top of or visual effects on top of the scenes that had been shot. So you can see
Kon Wilms: bottom left here. This was. Their production was animating Rose, and it was almost an outtake, if you will. But this is, you know, Blender
Kon Wilms: Compton, animated on top of the green screen. And then, basically, that backplate gets ripped and replaced with the backplate unroll engine. Seen that you see over here. So live action combined with animation, combined with VR Xr type stuff combined with green screen and in camera visual effects.
Kon Wilms: media series fairly straightforward. You know, this gives you an idea of what that looks like. You already saw. Some of that. I'd say the interesting thing that we did here was
Kon Wilms: we ran out of backplates for this particular shoot.
Kon Wilms: and we used stable diffusion, which is the open source. Stability AI alternative to say, mid journey.
And we generated a lot of these background plates.
Kon Wilms: for some of these scenes. The thing you can see here is that it's actually very hard to get
Kon Wilms: tools like mid journey, etc., to generate the correct
Kon Wilms: you know, camera spec, like, say, a 50 camera, etc. So we had to do a lot of basically, I think I generated maybe 200 different variants of an office. And we picked one. And then we had to literally size it into the wall and and squish it into frame. In order to get the perspectives to look correct. This obviously doesn't track
Kon Wilms: but they're now. This was done almost
Kon Wilms: 6 months ago or 7 months ago. Now they're actually a lot of things where you can take these frames and give depth and animation, etc. Out of them as well.
Kon Wilms: This was Michael's Union movie work stuff. You know, we're looking at 4 different
Kon Wilms: let's say, high fashion type. Ad campaigns mock campaigns we did firestarter, and we did star. So this is fire starter. So we combined
Kon Wilms: unreal with actually animated fire that we control through Midi faders. So we could basically push a Midi fader up.
Kon Wilms: And then the lighting would dim and the fire would blow up in the background. And you can see a lot of that here. These are mostly flat plates in an unreal scene.
Kon Wilms: So it it is
Kon Wilms: actual particle fire. That just looks realistic. But if you were to move the camera too much you would see that they were flat planes. And then we had a panther, Dolly, that we're using for the effects like the push in and and pull out, etc.
Kon Wilms: this is the stars project. So a lot of these backgrounds in the right hand side were also generated in stable diffusion. So there's an actual. There's A AI model that will that has been trained, I think, on data sets from one of the large scale Cosmic Telescopes.
Kon Wilms: That's an open data set.
Kon Wilms: and someone trained a model on that. And because you have a model trained on that, you can have, it generates you random variants of star fields and extremely high resolution.
Kon Wilms: So we combine that with software called Mad mapper which is typically used for art installations. So things like the star fields were generated by mad mapper put on the volume, and then the talent was put in front of that.
Kon Wilms: I'd say the interesting thing with this is that we used quite a few commodity 60 inch just off the shelf.
Kon Wilms: Tvs. So we had no way to get the reflection that you can see here
Kon Wilms: in this helmet, because the field of view is is about almost 270 degrees
Kon Wilms: when she has the helmet on
Kon Wilms: so we basically had to bring down the lights enough and then bring a bunch of panels around. So the reflection you see here is from a 60 inch TV, that's probably
Kon Wilms: 5 inches from her helmet just stage left. And is like has been oriented portrait style. And then the thing that you can do with tools like mad mapper is you can synchronize the star field and the perspective on the star field across multiple displays
Kon Wilms: so that star field that's coming out here is correctly going across the display, but it's flipped the other direction. Anyway, it's it's a lot of like
Kon Wilms: trial and error.
Kon Wilms: But because you can do that, and it's real time. It's in this case much more powerful than we could do with. Say, unreal engine, and it's a good tool to have. If you're looking at this kind of work.
Kon Wilms: this is basically what that's done. What one of the star fields looks like.
Kon Wilms: coming out of unreal. This is also controlled with Midi faders.
Kon Wilms: So basically, as you push the Midi fader up, the star field moves faster and that allows us to control the speed at which it, you know, comes out of a screen. There are a lot of things here that are.
Kon Wilms: I am. Lots of people would say, do not do that. So you can see there's a lot of bloom on all these star field images. But it's for autistic uses right? That's the point of that. And that's why we're essentially doing things the wrong way to get them to look correct. If you wanna look at it that way.
Kon Wilms: this is a list of all of the companies that we use for the stage right now. So there, there are quite a few here. I'd say that the interesting ones here are, we use a company called iconic
Kon Wilms: for asset storage and
Kon Wilms: their solution is interesting, and that you can use a local Naz as a iconic store. And then you can also use multiple clouds or remote locations. And you can essentially go this this, you know, say, video, clip or image or media file that resides on this server somewhere else in the world. I would like you to pull it locally to me.
Kon Wilms: And I, iconic, will take care of moving all of the data to wherever you need it. Which is like super useful. If you have multiple stages of people working remote, etc.
Kon Wilms: Probably one of the other interesting ones is Dcs over here, digital camera systems. So they make a tracker that you can connect to
Kon Wilms: either the iris or the focal or the zoom rings and basically get your telemetry offer that into unreal engine. So this allows us to do dynamic depth of field it's not a hundred percent accurate.
Kon Wilms: but with a little bit of tweaking you can tweak the you know the extents of which you can turn, say, the zoom lens, or the focus rings the iris rings and then map that into unreal, so that allows you to do things like as you
change the focus and you wreck the focus from an object in the unreal scene on the volume
Kon Wilms: to a local like, say, talent. Standing in front of the volume. It will defocus the unreal scene as you rack the focus to someone that's standing in the foreground, and that is a game changer for a lot of these types of, you know, especially the the beauty shots or the other types of material that we were shooting there.
Ethan Wilde: Just on a last slide. Yeah. So I have a rococo smart suit. We haven't really used it for anything like any of these, yet
Ethan Wilde: wondering how reliable it wasn't a production thing but it sounds like you have yet to determine that.
Kon Wilms: Yeah, I we were thinking of using it real time for one or 2 of the productions we're looking at, where we would capture, say.
Kon Wilms: met a human and then inject the the real time mocap into that. And you know I have a have someone wearing the the mocap suit standing behind camera or out of frame in an area and then map that onto the wall.
Kon Wilms: But essentially, there's so many things that can go wrong with that trying to do that real time. You know. Yeah, I mean, we found that like as is with all production, right? Them the more that you can simplify things and remove any
Kon Wilms: barrier for error the better everything is. So, you know, we have a lot of, I'd say we even have, like, for example, a lot of render nodes that we can use right. But I think we have one attached to the wall. With one we can do full use for real time compositing, and then we have another one for doing testing. But what we'll typically do is we'll literally just put mad mapper on one, and that'll be the sole purpose of that machine.
Kon Wilms: We'll use one other machine with unreal for the back plates. That'll be the sole purpose of that. And then we'll literally just do
Kon Wilms: you know, switching on the Sdi fabric to just go switch from A to B put that on the volume, and there's no conflict that way. We can do setup beforehand, and no one is touching anything and breaking anything. So you know, even sometimes loading the software, this can be a problem, right? So I think this particular scene right when we had to go from
Kon Wilms: at least this on the left here. I don't have any other frames of this, but we did one on the planet in a hangar, and we had to go from talent in front of that in a let's scene to
Kon Wilms: star field and reposition everything it involved. I think 3 or 4, just, you know, regular Tvs on stands. But you know, for example, mad Mapper, if you disconnect those HDMI connections. It's gonna reconfigure all of the displays, and it tries to do it automatically. So you almost have to load the project and prep it and connect all your things and don't touch it.
Kon Wilms: and it's fine. That's what it does, right, you know, if you're doing it for, like laser light shows, or whatever
Kon Wilms: that that always things get turned on in a certain sequence, and their purpose built for that. So this is a slightly different use case. But you know one machine to do one thing is is always a good default to be at
Kon Wilms: just, you know, just to summarize. So our stage is called Wayland. I guess like reference to aliens.
Kon Wilms: But this is the, you know, if you looked at the previous slide with all these different components. And you know, random product names and partner names and company names. This is sort of the stack that they go into. For what is now the, you know, in camera visual effects type
Kon Wilms: solution to the previous iteration that we had.
Kon Wilms: So we have some things that are done completely on set. So you can see it. I'll just start from the left, right? So we've used assembly or assemble for a lot of the stage management. Stuff. So like production management, etc. There are plenty of systems to use for this.
Kon Wilms: That you could pick from but at least this startup is like very nimble and very willing to adapt their software for different use cases. The interfaces clean. They've been like good partners. They've listened to feedback and their stuff is like, honestly pretty easy to work with
Kon Wilms: and and pretty powerful and then so that's used mainly for production management planning etc., right storing storyboards, etc. And then, when we get into the actual on set to near set
Kon Wilms: led volume I already talked about that. That's iodo with Brompton controllers, you know. So it takes HDMI HDMI in, or it takes Sdi in off of like a blackmagic card, for example, or from a router. And you just put in basically a 2160 p. Or 4 K. Signal into it.
Kon Wilms: And you're off to the races. The grading that we have done.
Kon Wilms: we're using like Rex 709 across everything, and then we shoot in that fashion. I didn't get into the lighting. We have a bunch of re lights
Kon Wilms: so those are mostly in a pre-configured array. So there's one behind camera.
Kon Wilms: A sky panel. There are 2 sky panels, left and right of the actual led volume. There are 2 orbiters.
Kon Wilms: you know. I'd say, away from the stage, left and right, and then 2 smaller sky panels, and then we have a bunch of floor stuff as well. It's also sky panels, but that allows us to kind of set up any lighting, and we just use one of the re tools on an ipad to set it up. We also have a desk.
Kon Wilms: but we found that most people that want to come in and shoot something. They tend to find it much easier to set up the lighting. You know, using a using an ipad than they would configuring a make, a lighting console, for example. And that's just because we're dealing with that kind of production. If you will
Kon Wilms: the photography, etc.
Kon Wilms: So we have red. We have Ari
Kon Wilms: we use a Thomas for all of the time. Code, wireless time code to everything, and then all the genlock is wired. There are other ways that you could do this wirelessly.
Kon Wilms: You know, if you wanted to look at that route as well, and then we use solar draft for all of the you know the workstations, if you will.
Kon Wilms: That gets us out capture we then take the the Mags. We offload them onto the edit station. Obviously, we're using unreal. We're using Pomford, so we will use. We might use things like live grade and the reason that we'll do that is when people shoot material
Kon Wilms: like maybe that. You know, we had. We had some backplates that was shot as motion graphics.
Kon Wilms: And that was done in red red role format that we got those in, and at least. Pom, 4 gives us the gr the ability to do real time transcoding 4 k. On a fairly powerful mac and color great at the same time. So we can basically take that Rex 709. Output it to the volume and be ready to go and not have to worry about
Kon Wilms: transcoding all the clips and waiting for that, and then redoing that. So it's super useful for that
Kon Wilms: on the offload side. So we have to get the frames out right, and we put that in frame, I/O or iconic
Kon Wilms: a lot of the clients that we have will use. Frame I/O, it's really easy to get that in for review, and then they can pull it into whatever they want to use it on. Maybe they wanna use premiere or Davinci resolve or whatnot, and just pull it off locally onto their edit stations and do the final master. But we use hedge offshoots to basically take the Mag files
Kon Wilms: rename them semi intelligently to make sure that they're not duplicating anything that they're ordered by production. Company. You know the date of the production, the name of the production.
Kon Wilms: all of that is super easy to do with the scripting on hedge, and it just becomes a matter of popping the, you know the the portable drive, or the Mag into the machine dragging and dropping it, and then just waiting for it to upload and finish.
Kon Wilms: There are a couple of other things that I have in here.
Kon Wilms: you know. So for testing, etc., we run some things on aws, so we might do. Storage
Kon Wilms: and hedge will also push the storage into a long-term archive in like S. 3 and Glacier
Kon Wilms: as well as you know, we also have local archive as well.
Kon Wilms: So hopefully, that gives you an idea of how this all comes together.
Kon Wilms: yeah, any
Kon Wilms: any questions.
Ethan Wilde: That's that's really an amazing system that you've sorted out. Con. I think.
Ethan Wilde: what you mentioned is is trial and error us to represent
Ethan Wilde: tremendous number of person hours to get everything up and running the way it is. Now in the latest version.
Kon Wilms: Yeah, it was about. I think it took me maybe 4 months to build this whole thing.
Kon Wilms: that's with a team of like, maybe
Kon Wilms: 1.2 people myself, and like point 2 of another person. But I like. It's doable.
Kon Wilms: The hardest thing or the the stuff that takes the longest is the permitting right and the power. So those panels that. Let me see if I can get back up to this just to give you an idea.
Kon Wilms: Yeah. So you know, this is not a and you can see the you can actually see the re sky panels here, and what the layout is of the lighting
Kon Wilms: but outside of that this draws about 70 to 100 amps of power. So you know, you can see as you get up to larger volumes, and it's not even running at full power. So it's running at about
Kon Wilms: 3, 50 to 4, 50 min, which is, that's not even as bright as like your iphone. If you just crank the brightness up right?
Kon Wilms: So it's maybe at 50 of that you don't really need more. But you know it. It draws a lot of power is what I'm trying to say, and then you also need to cool the facility. Right? So
Kon Wilms: and you also need to, you know, dim all the lights, and you need to. There, there are a lot of things that go into this that are actually.
Kon Wilms: even though you can't see what is going on with the green screen. Right?
Kon Wilms: There are ways to do this. Where, if you're using green screen
Kon Wilms: I think one of the things you can't see here is. There's a confidence monitor like, right around confidence television right around here. So anyone that any of the talents here can actually see what they look like on camera.
Kon Wilms: and you can easily do that with an output from
Kon Wilms: the compositor. You know, that's built into unreal into a confidence monitor, so that anyone that is on set insets that doesn't really know what's going on around them outside of markers on the ground to figure out, you know, where should I go? Where should I stand? etc.? You can give a lot of visual feedback without it having to be on the wall per se, you know cause
Kon Wilms: as much as they wouldn't be looking back or focusing on the wall.
Kon Wilms: They might not also be focusing in front of them. So you know, sometimes this works for some use cases.
Ethan Wilde: questions, anybody for Con.
Ethan Wilde: It's done. Silence. We're we're speechless.
Ethan Wilde: Yeah, yes, yeah.
Kon Wilms: yeah. I mean, thanks. Thanks for having me on. III hope this was like somewhat informative.
Kon Wilms: it's it's pretty fun to to have worked on and built this kind of stuff there. There are a ton of use cases for it. You know. Other things that I see now are people are going. Oh, well, you know.
Kon Wilms: AI algorithms will just replace a lot of this.
Kon Wilms: I don't see that really happening. I almost see it in the same use cases that we did here right where it's like.
Kon Wilms: It's some. It's a tool that allows us to produce more, faster, with less effort.
Kon Wilms: But it still takes the creative people to figure out what that should look like
Kon Wilms: you know, just just as like some input on that cause that comes. That comes up quite a lot, at least from you know what what I see from our perspective.
Ethan Wilde: That's that's a great quote. Yeah, that's that's a really strong encouragement for everybody in the room here. People thinking, is this something I could get into, and and flex my technical and creative chops on it, and and the the looming
Ethan Wilde: concerns about AI. I think it is a relief to know that these are just more tools
Ethan Wilde: for a creative process that has fundamentally got a lot of humans involved in it.
Kon Wilms: Yeah, that's true. I mean, look on Set, you're still gonna need all the people operating this green screen or not green screen, because you're you're still shooting something, and they have to be lit right? So a lot of the things that could you almost have to look at it the same like with software development with AI right for
Kon Wilms: for it's re, it's replacing a lot of mundane tasks
Kon Wilms: that are not that interesting in this perspective for humans, right? So like this particular use case here. We needed more interior shots of office buildings.
Kon Wilms: That is not the most fun thing to work on. So you know, if you're an artist and you're going. I could generate this fast. I could still get paid for that, and I could spend my time working on interesting things instead.
Kon Wilms: And this allows me to be not like a me as like a times one, but like as a Me. Times 10, because I know which tools to use to produce things faster. I think that's where you know, we start to see
Kon Wilms: the advantages here. Right? It's and it's the same for software development for AI right where it's like
Kon Wilms: someone still has to architect and design and correctly code the system.
Kon Wilms: But if I can have essentially a couple of
Kon Wilms: you know algorithms doing the very junior stuff for me like that, and a that makes me like a A 10 times better than II am in terms of efficiency and productivity. And that means you don't have to waste your time doing the mundane things
Kon Wilms: or encouragement we like, we like all of your encouraging perspectives. Kahn. Yeah, sure.
Ethan Wilde: Anyway, anything anything else? I can answer any other questions.
Kon Wilms: Yeah, we do. But we can't get a car through the door.
Kon Wilms: That's the only problem. But I mean, you know, you could model it in unreal and just put it on the wall instead. I mean, we we've talked about just bringing a car door in, and then the rest of the car is just, you know, part of the unreal scene. So that's possible, too. There are a lot of like there are a lot you have to look at. This is like, it's interesting from the perspective of
Kon Wilms: how do I solve something challenging when I could replace components and think about things a different way.
Kon Wilms: And it's almost, you know, like
Kon Wilms: black magic to some, to some extent, you know, for someone that doesn't know how this works, for example. So that makes it interesting, too.
Ethan Wilde: even for somebody who knows how it works. It's really fun figuring out
nice.
Kon Wilms: Yes, totally
Kon Wilms: thank you. Okay.
Ethan Wilde: for taking your time to share your amazing journey and your your discovery and your expertise in this field. It's something a lot of us in the room, I think, are excited about the potential of, and
Ethan Wilde: I'm I'm sure we'll be in touch with you advice as as we try and get our our own technical chops
Ethan Wilde: sorted with with our own limited resources.
Kon Wilms: Yeah, anytime, let me know. Feel free to reach out. You know where to find me.
Ethan Wilde: Thanks everybody for being here. We've got more food to give a round of applause.
Ethan Wilde: The chapter.
Kon Wilms: Thank you. Happy holidays to you, too.
Kon Wilms: Thank you. I'll be in touch with you, Kahn. Okay, thanks. Take care. Well, have a great evening. You, too, bye.