Hey everyone! Hoopermation here.
So after a few weeks of contemplation, and many days of intial setup and experimentation, I'm finally ready to put out the ultimate guide on creating Mixed Reality videos for the HTC Vive!
This is gonna be a long process, and is NOT going to be cheap if you want to do it right and get the best looking results.
There are multiple methods to creating mixed reality videos. One is fixed camera, and the other is moving camera. I'll go more in-depth on those later.
First thing that you need to do is measure your space that you want to record in. Most of you will probably be doing stuff like this out of a bedroom or office, but the more space you have, the better. An ideal situation would be to have an actual studio space, but most people don't have access to something like that so I'll just stick to the small scale stuff.
Me personally, I cleared out half of my bedroom to do mixed reality in, and even that is just barely enough space to do it in. Once you've got a decent enough space, here's a laundry list of what you'll need in order to get started. This is assuming you're going to use half of a 12x12 room:
- 4 Green Screens
- An HTC Vive
- A 1080P Camera (DSLR or Camcorder works, 60FPS if you can)
- 3 Softbox lights (2 at the minimum)
- Lavalier microphone
- A BEAST computer
Once you got all that, you're set!
I know it's a lot, but this is for high-quality Mixed Reality. Feel free to mix and match with stuff you don't want/need/have/can't afford. I won't lie, this is an expensive process.
Let's start with making sure you're Vive's base stations are set up correctly. Wall Mounts are a must if you want this to look good. Make sure they're both high up and facing towards the center of your play-space.
Next, it's time to setup the greenscreen. This process took me 3 days just by itself. If you already have a way to set up greenscreen, great! If not and you're a total noob like me, follow along. There's lots of ways to setup a green screen, but you want to have the most coverage you can get for as cheap as you can. You can buy the actual stands they use for them, but you loose some space with those. What I did for mine was build a PVC Pipe rig to hold up the screen and stretch it across my walls. I've never worked with PVC before but it's pretty simple. And you don't have to use the PVC cement for it to stay in place either!
If you want to use PVC like I did, measure out your space and buy enough PVC to build a good rig that won't fall apart easily. Why PVC? It's sturdy and is easy enough to take down than using a more permanent solution. Plus, you can clamp down the screen and really pull it tight to get a good key.
Once you've got all of your walls and floor covered in a disgusting green color, you need to light it. The key to getting a good key with green screen is the lighting. You want it nice and even across the screen and you want to eliminate shadows. Since we're working with such a small space, you'll need to compensate in post for errors, but it can work really well if done right. If you can fit a good enough light overhead that won't get in the shot, please do. It will help make it look even better! You want to eliminate as much shadows as you can, but if you can't eliminate them at least make them lighter.
If you have only 2 lights, put each of them on opposite sides of the screen facing in towards the middle. This should light the screen pretty evenly, but if there's a lot of creases on the screen itself you'll need to tighten it, most likely with clamps. I use spring clamps that I bought at Lowes for like 20 bucks for a whole box of them to keep my screen pretty tight.
Great! You've now got all of the physical setup done. Now comes the most frustrating part of this whole ordeal. Seriously.
Get your Vive all set up in the space, and now we'll need to enable mixed reality on games! Currently, this only works on Unity based games unless you can get a specific build from the developer for other engines, or if it was disabled during development. Currently the only 2 games that have the functionality built into the game that is toggle-able from within menus is Fantastic Contraption and Job Simulator. The rest you'll have to use a much more manual method.
You'll need this:
Let's try a game that doesn't have it built in: Space Pirate Trainer. Go to the game in Steam, right click it, click properties, click the local files tab, then browse local files. This should bring up the game's root directory with the .exe and the [gamename]_Data folder. Right click in this folder, and create a text file. Copy and paste all that gobbledy-gook above into there, and save the file as "externalcamera.cfg". Without the quotes of course.
This is what makes the magic happen. If you launch the game with this here, then it will load up into 4 quadrants. Here's what they do: the top left shows everything in the game in the foreground, or in front of the headset. The top-right is an alpha channel (That we can use for keying!). The bottom-left is everything in the background (or behind the headset). The bottom-right is a first person view. We'll use at least 3 of these in conjunction with each other to get the effect we want, but I recommend you use the first person as well for anything the camera doesn't capture! Now, if you want to do 1080P video, you're gonna need a 4K monitor, because each quadrant is a quarter of the screen. You can use a 1440P monitor if you want to do 720P video.
This is where the BEAST PC comes in. Playing VR games in of itself is pretty costly, but recording them at the same time and at 4K is a different story. Let me put this into perspective: I have an I7 4790K processor and a GTX 980TI and I can barely scrape by at 90FPS with most games. A lot of times it dips below that, but ATW (Or whatever steam's equivalent of that is) helps. Just as long as you can record at 60FPS.
Here's the tricky part: You'll need to sync the in-game camera to the real world one. This is a bitch, flat out. It's easy with Fantastic Contraption and Job Simulator since they have the functionality built into the game, but anything else is gonna take a lot of time and patience.
If you're using a webcam, this is easy, but you're real world video quality isn't gonna be that great. If you use a DSLR, you'll need to use a capture card to see the camera on your PC. I'd recommend a good 1080P 60FPS camcorder for this process, but make sure you get one that you can see out of it on the PC!
From here, you'll need to use OBS to sync up the camera and game. Add a source, then use the "video source" and add either your webcam, capture card for DSLR, or your camcorder.
Next, add the game source. Right click the game, go to filters, and select crop. Set the bottom to 1080, and the right to 1920. Make sure you also set it to zoom so that it just shows the top left quadrant. Put the game capture on top of the video capture source, then go to filters. Add a color key, and select the black. Hopefully sometime we'll be able to change the color of the key so this is a little easier, but it's just for setup purposes now so it's fine.
Now, if you're using a static camera setup, you'll need to change the offset values in the externalcamera.cfg to where the real camera is. This is going to require a crap ton of trial and error, and a lot of hair tearing. DO NOT use wide angle lenses as it will mess up the sync a lot.
If you want to do a moving camera setup, you're going to have to have a 3rd controller Vive controller. HTC now sells them (which is awesome!) but they are pretty pricey. But it's necessary. You'll also have to have a long MicroUSB cable, as the Vive can only track 2 controllers wirelessly. When you start SteamVR, make sure the wireless controllers are on and that the wired controller is unplugged and off. Start it, wait for the base stations to find the controllers, then plug in the third controller. This will tell SteamVR that the 3rd controller is the camera controller. Now when you start SPT, the in-game camera will be wherever that 3rd controller is! This makes the actual placement of the camera much easier, but the other settings are not as easy. From here, you'll need to attach the controller to the real world camera somehow. I used a shit ton of rubber bands to keep it in place, and this does fine for the most part, but isn't very...elegant. What I would recommend is either taping the controller to the top of the camcorder/webcam, or buying a shotgun microphone clip.
The vive controller should fit in there, but if not then there must be one out there that will. If not, just tape it to the mount. I'm sure most of you wouldn't want to get some duct tape all over your DSLR itself, but that mount is fair game.
From here on out, you'll need to configure the different settings in the externalcamera.cfg file to sync up the footage to the in game camera. I'll explain what each setting does:
X, Y, and Z are the offset values for the actual camera placement within the game. These aren't as necessary to mess with if you're using a 3rd controller, but are entirely necessary to mess with without one. Use small numbers to start off when using this. X is left and right (positive and negative values), Y is up and down, and Z is forward and back.
RX, RY, and RZ are rotation values. These set the rotation of the camera in conjunction with the orientation of the 3rd controller. If your controller is laying flat on top of your camera, then you'll only need to change these a little bit to fine tune it. RX is rotation left or right based on the controller, RY is rotation up and down, and RZ is twisting rotation left or right. Configure this until it vaguely matches up with the real world camera.
The next setting is the FOV. It's difficult to calculate a real world camera's FOV in translation with In-game FOV, but most cameras would probably use around 35-40 FOV.
Adjust all these settings until the controllers in the real world match up with the in-game controllers. Also, set the "far=100" value to "far=1000". This sets how far the in-game camera can see, so you want that to be pretty far. It sucks, but guess what? Now that you've changed all that you don't really have to do it again, as you can copy and paste the file for the other games! Huzzah! Also, always always ALWAYS do your sync on a tripod that never moves. This will save you a lot of readjusting in the future.
Now, if you want to livestream like this, just add a window capture into OBS, crop it to the lower left quadrant, and put the source underneath the camera footage. If you want to make a video out of it, remove all the effects on the game capture, and remove the video capture. Now, just start recording the game footage and start the camera, and you're on your way!
But, if you want to do moving camera stuff, you'll need a camerman, and a stabilization rig. I recommend a fig rig for this as they are smaller and can get great results: https://www.amazon.com/Manfrotto-595B-Fig-Rig/dp/B000LRFURU
Or if you don't want to spend $300: https://www.youtube.com/watch?v=qT72hoYxESI
Also, remember that lavalier mic I mentioned earlier? Yeah you can just use your phone to record that. Just put the phone in your pocket while recording.
WOOT. Now that you've recorded all that awesome Mixed Reality footage, it's time to put it all together. I use Adobe Premiere Pro as my editing software, so I'm not sure about Sony Vegas, but the steps should be somewhat similar. All you're doing is using one of the quadrants as a luma key for the foreground to key out the elements correctly. It's kind of meh but it looks awesome!
Throw all that footage you got into Premiere and sync it up in the timeline. Add the game footage to track V1, then crop out to the video to show only the bottom-left quadrant. Then on V2, add the real world camera footage. Once that's done, go to effects and find Ultra Key. Use this to key out the greenscreen on your camera footage. Adjust the settings here to make sure you get an amazing looking key. If you did your screen and lighting right you shouldn't have to mess with this stuff too much, but do what you need to.
Now, with the foreground footage, you'll need to use the top-right quadrant as the "This is what I want to key out" thing for the top-left footage. Copy the game footage in V1 and add it to V4. Crop out the footage to only the top right quadrant, then add a "track matte' effect. Set the Composite to "Matte Alpha". Next, copy/paste this track, and place the copy underneath it in the timeline. Crop out the footage to only use the top-left quadrant, and on the track matte set the Matte to "Video 4" and set the Composite to "Matte Luma". You just super-imposed youself into a mixed reality game, congrats!
From here, you can add the mic recording and continue making your video, but if you want to use the first person footage as well and don't want to exacerbate the massive headache you already have, follow these steps.
With all that footage synced up, highlight everything in the timeline, right click and select "Nest..." This will smash all that footage down into a nice little sequence of it's own! Now, add the game footage again and crop it out to only show the bottom-right quadrant. Move that on top of the nested sequence and sync it up to fit with it exactly. In the bin with all your footage on the top left of premiere, select the original sequence and the nested sequence you created, right click and select "Create multi-camera source sequence".
This will put the original 2 sequences into a folder, and what will be left is our multi-cam sequence. Right click that and select "New sequence from clip". This should place that new multi-cam sequence into the timeline. From here, on the program monitor click the little wrench and click "Multi-Camera". Now you should see two videos on the left side: One is your composite mixed reality footage, and the other is the first person view. Here's the fun part: While editing your video, if you want to switch between the 2 camera views, hold Control and click on the other view in the program monitor. This will create a keyframe where the cursor was in the timeline and it will switch to the other footage for you.
And that's all! Now you can create all the mixed reality videos you could ever want! Have fun guys!