How To Live Stream Mixed Reality

How To Live Stream Mixed Reality

Introduction

With virtual reality rapidly gaining popularity, it’s becoming a great medium for generating live stream content. However, it can be hard translating the experience to an external viewer if you’re purely capturing the first person view.

The solution to this problem is Mixed Reality. Most people are familiar with Augmented Reality, which overlays computer graphics over real life. Mixed Reality is essentially the opposite, where you overlay real life over computer graphics.

The end result is a much more interesting point of view of what the player is experiencing in-game. You can see an animated example of this below for Google’s Tilt Brush.

Before we get started, I wanted to note that successfully getting everything working the first time will require a fair bit of tinkering and time. When I first tried mixed reality back in August 2016, there weren’t as many tools available. Luckily the VR and streaming community have released some handy tools since (January 2017 at the time of writing), but the whole process is still quite complicated.

1. Stage Setup

StreamShark Mixed Reality Green ScreenRequired Equipment

  1. Green screen, the larger the better
  2. 2x Box lights on stands

When setting up the stage for your mixed reality live stream you will need to consider how you want to shoot the subject. The more green screen space you have, the wider the shot you can take, which is essential if you plan on panning the camera. If the camera pans outside of the green screen, it will ruin the immersion for the viewer, as you will no longer see the in-game graphics.

The two box lights are essential for shooting with green screens as they greatly reduce shadows and creates an even green, which makes the chroma keying easier (removing the green background from the video).

*Update* I’ve written a blog post about the best way to setup a green screen for a live stream.

2. Hardware Setup

Required Equipment

  1. VR ready PC
  2. Monitor that supports at least 2K resolution
  3. Video camera
  4. Video capture card
  5. HTC Vive
  6. *Optional* 3rd Vive controller
  7. *Optional* If using a 3rd controller you will either require a USB extension cord or a steam controller dongle flashed with custom firmware (additional information here too)

It’s important you have a machine that is powerful enough to run VR content at 2K resolution and at a constant frame rate. The reason for 2K resolution is that due to the way mixed reality content is captured, the output resolution will be halved. With a standard 1080p monitor, the final output resolution would be 960 x 540, which is quite small. With 2K resolution (2560×1440), the final output resolution is 1280 x 720. If you want a 1080p stream, then you will need a 4K monitor.

You’ll also need a video camera and a capture card to get the video signal onto your computer. We’ve used the Magewell HDMI to USB dongle with success. Alternatively you can use a webcam, however the quality of the video will be much lower.

If you wish to pan the camera during the stream, you will need a 3rd controller mounted to the camera. Alternatively you can keep the camera stationary and use a virtual controller, which I’ll talk about below.

3. Software Setup

Required Software

  1. OBS (Open Broadcaster Software)
  2. Steam and Steam VR
  3. Most Unity based VR games
  4. Virtual controller driver (if you don’t have a 3rd controller)

If you don’t have a third controller, don’t worry! There’s a handy tool out there which lets you trick Steam VR into thinking you have a third controller. We call this a virtual controller. The only downside to a virtual controller is that the camera must remain stationary. If it moves at all during the stream, it will misalign the layers.

You can download the virtual controller driver here. Additionally you’ll need to follow this guide on how to correctly install the driver.

Once you’ve got the driver installed, and the externalcamera.cfg placed in the root directory of the Unity game you want to play, load up SteamVR. You should now notice the 3rd controller is being detected (hopefully). Once the 3rd controller is being detected, you’ll need to move onto calibrating it.

Virtual Controller/Camera CalibrationUsing cameraAlign for Mixed Reality

The purpose of the virtual camera calibration is to align the locations of both the virtual and physical cameras.

Method 1:

The drivers for the virtual controller include a tool called cameraAlign. When I first started streaming mixed reality content, this was the only tool available. It is extremely tedious to setup and requires a lot of trial and error. This step takes the most amount of time to get right.

You’ll also need to calculate the field of view (FOV) for your camera lens. It’s important to note that Unity measures field of view vertically. This tool can help you work out your lens field of view.

Method 2:

A YouTuber has created a tool called Mixed Reality Configurator, which makes it easier to generate the externalcamera.cfg value used for calibrating the virtual controller. You can download the tool here, and it also links to a video guide on how to use it.

4. OBS Setup

Now that you have a virtual controller setup, when starting a game make sure to hold shift and double click on the executable in the games directory. This will open up the Unity preferences screen. Make sure to select full screen and 2K resolution (2560×1440).

What you’ll see is the game screen split up into four quadrants. The top left is ‘foreground’, the top right is the ‘foreground alpha layer’, the bottom left is the ‘background’ and finally the bottom right is the first person view.

The guys over at Northway Games have an extensive guide for setting up OBS for mixed reality streaming, which you can read here.

A few things to note with their guide:

  1. They built their own in-game method of creating the four quadrant layers. As we’ve already generated the four quadrants, we only need to follow point 2 of their guide.
  2. Instead of adding a webcam as a source, you’ll need to add the capture card instead.
  3. Their guide is based on a 4K monitor, so you will need to base your crops on a 2K monitor.

One thing not mentioned in their guide is the use of an alpha layer (top right quadrant) as a mask. You will need to download and install a plugin for OBS in order to do this. Version 17.0.0+ of OBS is required for it to work well. Update: This plugin is no longer available. It has been superseded by a stand-alone mixed reality compositor available here.

The plugin uses the top-right quadrant of the layer it’s applied to as a mask for the top-left quadrant of the same source. So you’ll need to apply this to the ‘Foreground’ layer. To do this Select the “Image Mask/Blend” filter and set it to “Subtraction”. Then you’ll need to select any image, which isn’t used in the filter, but is required for it to work.

Once you’ve followed all the above steps you should have a composited scene of all the layers (see screenshot of layer order). At this point you might need to recalibrate your virtual controller to make sure everything lines up.

So now you’re good to go live! If you have any questions about mixed reality, please leave them in the comment section below.

TL;DR

  1. Make sure you’re a patient human being – mixed reality can be painfully frustrating to get correctly setup the first time.
  2. Film someone in front of a green screen playing a VR game.
  3. Define the location of the in-game 3rd person camera using a 3rd controller or with special drivers.
  4. When 3 controllers are detected and a special config file is placed in the game directory, Unity games will automatically output a source that is split into four quadrants. Foreground, transparency layer, background and first person view.
  5. Composite (combine) these layers with the raw video footage of the player in front of the green screen, using OBS. Essentially sandwiching the green screen footage between the background and the foreground.
  6. Live stream the results!

Further Reading:

59 Comments

  1. Paul Alvarez

    I was wondering if you had suggestions on where to purchase decent green screens for recording this type of video. Thanks in advance for the help!

    1. Cameron

      Hi Paul,

      We purchased the green screen, the stand for the green screen and the two box lights all on eBay. It was surprisingly cheap costing around $130 AUD in total.

      You could spend much more money on higher quality equipment, but if you’re just getting started the cheap stuff will work fine.

      Cheers,

      Cam

      1. imtiaz

        bro i have one problem.can you help me when i open video capture in obs handy cam attch in avermedia capture card c875 a not working . web cam working . i am trying form two mount please help me

        1. Cameron

          Hi imtiaz,

          I’m not familiar with that capture card, you might have more luck asking on the OBS forums?

          Cheers,

          Cameron

  2. Cyril Tuschi

    hey i just was starting to compile something alike – so, thanks a bunch!;) one thing i still dont 100% grasp: how do you aligne the real camera with the virtual cam? are you building calibration objects like “the unspoken” dudes?! https://developer.oculus.com/blog/sharing-vr-through-video-mixed-reality-in-the-unspoken/

    1. Cameron

      Hi Cyril,

      The alignment of the real camera and the virtual camera is the hardest part.

      In the virtual camera configuration section of this guide I’ve covered two methods for alignment and configuration. Method one uses a tool called cameraAlign which lets you move the virtual camera millimetre by millimetre until it lines up exactly with the real camera.

      For the second method I’d recommend you check out this video. He gives a good explanation on how the alignment tool works.

      Cheers,

      Cam

  3. Simon

    Struggling with the obs settings for crops I have a 55 inch tv it’s hd ready but not a 4 k so how what settings would I need to crop?

    1. Cameron

      Hi Simon,

      If your tv is 1080p then the resolution is 1920×1080. So each quadrant is 960×540.

      The crop settings in OBS would be:

      Foreground:
      Bottom – 540, Right – 960

      Foreground Alpha:
      Left – 960, Bottom – 540

      Background:
      Top – 540, Right – 960

      First Person View:
      Left – 960, Top – 540

      Cheers,

      Cameron

  4. Ken Boyce

    Hey Cameron,

    Great article!

    I am using a MSI VR Backpack. Can I do this without using any cables connected to the VRI backpack?

    Thanks
    Ken

    1. Cameron

      Hi Ken,

      You’ll need two additional cables connected to the VR backpack. A computer monitor, which you’re screen capturing in OBS, and the camera cable.

      In this instance, the VR backpack could be more of a hindrance and it could work better using a standard computer setup.

      Cheers,

      Cameron

      1. Simon

        Thankyou Cameron I’ve only just seen this reply, once I figured it out I realised what a stupid question it was lol

  5. Zara

    Hi Cameron,
    Great tutorial. Wanted to check — is this OK to run with just one PC? And what are the minimum specs recommended? I have a PC I use for the Vive but concerned this will take more power/graphics than I have.
    Thanks, Zara

    1. Cameron

      Hi Zara,

      It’s pretty much a requirement to use a single PC for everything, as you’ll need to capture the screen and in-game footage with OBS. If you casted the in-game footage to another computer for compositing/live streaming, it will likely add too much latency/delay.

      The computer we used to mixed reality had the following specs:
      CPU: Intel i7 6700K
      GPU: Nvidia Geforce GTX 980 TI
      RAM: 16GB DDR4
      Monitor: 2K resolution

      Cheers,

      Cam

  6. Simon

    Hi I’m making mixed reality videos for myself at the moment not streaming or recording, just doing it all so it’s on the screen as I’m doing it, it sort of works and looks ok, using obs to get it it all working, just struggling with camera align I want my fov larger so more of the game is shown on the green screen but when I do that all the other settings are way off (mixed reality configurator tool), also the game controllers will move about half second before I do, is there a way to delay the game movement so it matches my movement? And would using the alpha layer plugin help improve my results? Currently my obs set up is
    Foreground-colour key filter applied
    Camera- chroma key filter applied
    Background-no filters apart from crop

    Thankyou in advance

    1. Cameron

      Hi Simon,

      It sounds like you might need to use the cameraAlign tool to manually sync things up. In regards to video latency, in OBS it’s possible to add a delay to your video camera input which will allow you to sync up the in-game footage with the video footage.

      Yes, I’d recommend using the the alpha layer as it will improve the chroma keying on the foreground layer.

      Cheers,

      Cam

  7. Simon

    Thanks cam, with regards to the alpha layer, do I apply the alpha layer filter to the foreground? When it’s says add image can I just use any image I want or does it need to be specific to the game I’m using, thanks again

    1. Cameron

      Hi Simon,

      It can be any image, and it isn’t displayed anywhere.

      I personally used a gif of a cat getting scared by a cucumber.

      Cheers,

      Cam

  8. Simon

    Fantastic Thankyou, so that goes in the foreground then? Also gives me an option for a colour key, do I set that to black instead of added the chroma key to the foreground or do I need to do both?
    Thanks again

    1. Cameron

      Hi Simon,

      Yep, you will need to apply the alpha plugin to the uncropped foreground layer. From memory, I don’t believe you need to do a colour key when using the alpha plugin, only the subtraction filter.

      Cheers,
      Cam

      1. Sagar

        I want the alpha plugin to make a mixed reality video. Can you please send it. The link of the plugin mentioned above is error.

        1. Steve

          Hey,
          I’m also looking for the same alpha plugin. It’d be great if you could please send it!!

          1. Sagar

            Thank you Cameron 🙂 It worked!
            Can you make the mixed reality setup live in premiere pro like we do in OBS..?

  9. Simon

    Oh I cropped, I’ve been cropping them applying but didn’t notice anything different, can it be cropped after applying the alpha plugging? What does it actually do?
    Cheers for you help

    1. Cameron

      Hi Simon,

      The plugin automatically crops the source it’s applied to, so you don’t need to crop it again.

      Cheers,

      Cam

  10. Simon

    Also just seen your reply on the other site aswell, Thankyou again

  11. Alton Glass

    Was getting noise when using the Acer Media Capture Card. Anyone experience that?

    Also any notes to get rid of the ghosting of the headset?

    1. Cameron

      Hi Alton,

      You might need to mute the audio coming through the capture card, if you don’t require it.

      What headset are you using? I haven’t had any issue with ghosting on the Vive.

      Cheers,
      Cam

      1. ALTON

        sORRY I MEANT VIDEO NOISE BUT I THINK IT MAY HAVE BEEN MY CAPTURE CARD I UPDATED TO THE BLACKMAGI 4K INTENSITY FROM THE ACER

        HOW DO I GET RID OF THE HEADSET THAT MOVES IN THE SHOT?

        1. Cameron

          Hi Alton,

          Are you referring to the headset that is rendered in Tilt Brush? If so I’m not sure if it’s possible, I haven’t personally tried to remove it.

          There might be an option somewhere.

          Kind regards,

          Cameron

  12. Simon

    Does this have anything to do with alignment of the virtual and real camera? I used it last night and for the first time everything was lined up perfectly, I did t even have to mess around with the alignment tool, all I did was use the camera align tool, moved the lines to where my camera was and that’s it, loaded up a game and it was perfectly matched, that’s never happened before.

    1. Cameron

      Hi Simon,

      Do you have an existing camera alignment config file in the games directory? If you’ve previously aligned the cameras and all the physical equipment is in the exact same position, then it should be correctly aligned.

      If you’re starting from scratch, then you might have just lucked out haha!

      Cheers,
      Cam

  13. Simon

    Hi yeh I fluked it lol, I have however managed to get it pretty well aligned every time even if I move the camera I can do it, not perfect but pretty close, I do it with the camera align tool, I just can’t get it anywhere near with tribalinstincs configurator tool and that’s meant to be easier, weird!

  14. Gabriel Cheung

    Hi Simon,
    I was wondering how can i generate the 4 quadrant view in unity ? coz i m working on my own simple game and want to shoot it in mixed reality video.
    however, i tried many times that i only have the lower right view (view of HMD) come out correctly, the other three are stay in black.
    What’s more, i noticed that there is movement for third camera in the scene but not in the lower left view of the game view…
    it seem that the externalCamera script is missing the function to enable FG ,BG and the FG mask…
    Hope u can solve my problem.
    Thanks, Gabriel

    1. Cameron

      Hi Gabriel,

      I’m not sure how the game developers implemented support for the mixed reality component in Unity. You might be able to find the solution to your problem on Unity game developer forums.

      Kind regards,

      Cam

  15. Thong

    Thanks so much for this guide Cameron! I’m surprised that there aren’t more detailed tutorials like yours and I’ve been searching for a while.

    I was able to get this mostly working on Tilt Brush, but for some reason certain objects appear a bit transparent. For example, the moon when you change the background or imported 3d models. Have you come across this issue and/or know the fix?
    Thanks in advance

    1. Cameron

      Hi Thong,

      Glad my guide was helpful!

      I haven’t experienced this issue before. If it becomes semi transparent when the moon enters the foreground, perhaps it could be a chroma key issue, and you might need to test the settings?

      Cheers,

      Cam

  16. lys

    Hi,
    I use sony a7 r2 with 24-70mm lens. But cant get good results. I have tried with mixed reality configurator tool and some other technics but result not so good. I use blackmagic 4k declink. I got live video from camera. I setup OBS and I have third controller but FOV settings gonna make me crazy :).
    Pls give me advice. What am I missing and what should I do perfect result?

    Thanks

    Lys

    1. Cameron

      Hi Lys,

      Yeah working out the FOV can be a pain. Have you tried using this FOV calculator? Scroll down the the ‘Angular Field of View Calculator’ section. The values you’ll need are ‘Lens focal length (mm)’ and ‘Focal length multiplier’.

      It would be easiest starting at 24mm (based off your lens being 24-70mm) and making sure the lens is fully retracted (not zoomed out). From a quick Google it seems the focal length for your camera is 1. Once you’ve entered these values you’ll want the computed Vertical FOV, which in this case should be 53.1.

      This value should give you a good starting point for FOV but you’ll still likely need to fine tune it. Experiment by raising and lowering the FOV value.

      Cheers,

      Cam

  17. Ben Loveridge

    Is there an easy way to align the foreground and background layers in OBS? Since the background is cropped but the foreground is not, they are completely different sizes and I’m finding it tricky to get the two layers to match up for example – https://twitter.com/benloveridge/status/852512980243632129 (ignore the camera delay – I’m using a USB 2 capture card)

    1. Cameron

      Hi Ben,

      Is your output being divided into four quadrants like this?

      If so, it’s just a matter of cropping the top left and the bottom left quadrants and compositing them. They should be the exact same size. Alternatively if you’re using the Alpha mask filter plugin, it will crop the foreground layer automatically.

      Cheers,

      Cam

      1. Ben Loveridge

        Thanks for the response. Yes I’m getting the quadrant view. Some follow up qs:
        So the crop settings would be the same for the right side, then top and bottom just a reverse of each other?
        If I use the Alpha mask filter (which I’m doing I think) I don’t need to crop at all? hmmm but I am doing a crop and it’s not quite aligning….
        What is the exact name of the alpha mask plugin? Should it be different to the normal one? When I replace the old one in the plug in folder there doesn’t seem to be any difference in the name from the one previous so I’m not sure if the filter is actually been applied.

        1. Cameron

          Hi Ben,

          This is the plugin I’m referring to. There’s also a discussion thread about it on the OBS forums.

          What version of OBS are you using? It requires version 17+ to work correctly, and that version is currently only available on Windows. Also I don’t believe you need to crop the layer the alpha mask plugin is applied to, as it uses the top-right quadrant of the layer it’s applied to as a mask for the top-left quadrant. I can’t remember for sure as it’s been a little while since I’ve used it.

          The crop settings for each quadrant will be half of the input resolution e.g. 1920×1080 would be 960×540. The only difference for each quadrant would be wether you apply it to the left, top, right or bottom.

          The crop settings in OBS would be (if you input is 1920×1080):
          Foreground:
          Bottom – 540, Right – 960
          Foreground Alpha:
          Left – 960, Bottom – 540
          Background:
          Top – 540, Right – 960
          First Person View:
          Left – 960, Top – 540

          Cheers,

          Cam

          1. Ben Loveridge

            I’m on version 18. Weird thing is I don’t see any auto crop being applied to the foreground layer when I choose ‘Blend (Subtraction) as the image mask/blend type. Maybe this is an issue?

            Also any reason you’ve listed crop settings for four layers above? I thought all that was being composited was foreground, real life and background layer. Also if the foreground layer is automatically cropped then that wouldn’t need any crop settings? Sorry I’m still a bit confused. Do you still have a setup at the office, maybe I could drop in and get some expert tips!?

          2. Cameron

            Hi Ben,

            The reason my advice on this part is a bit fuzzy is we no longer have the mixed reality hardware setup, so I’m having to give advice based on memory. If we still had it set up, I’d just send through a quick gif of the process.

            I just had a look again at the alpha mask plugin, and it does appear like it doesn’t auto crop. In this case you’ll need to crop the foreground layer quadrant.

            So your initial issue was getting the foreground and background layers to line up. As I mentioned in my previous post, the crop for each quadrant uses the same values, but just changes wether you apply it to the left, top, right or bottom.

            Cheers,

            Cam

          3. Ben Loveridge

            Hi Cam, thanks for the help I was able get it working by adding the crop on the foreground and setting the order of the crop and image mask in the laters. Then I had to manually adjust the preview window size to align the layers but got there in the end!

  18. Adam

    Hi, we created free calibration tool named Mr Calibrator. Check this movie https://youtu.be/-p3Ave33GpU
    You can download it for free from our website studio.vrvisio.com

    1. Cameron

      Hi Adam,

      It’s great to see more people coming up with more calibration tools!

      Have you tried using the cameraAlign tool I mentioned in my post? It has some similarities to your tool, the main difference is cameraAlign is calibrated within a unity application with the Vive controllers. The benefit of this is that it only requires one person to calibrate.

      Can the values in your tool be updated with the Vive controllers, or is it restricted to the keyboard?

      Cheers,

      Cam

  19. Fabian

    Hey there. I am looking for a solution to sync the videos in obs. I have the same problem as discussed above. In game footage moves about half a second before webcam video does… You sait this can be fixed by a delay setting. But you did not describe it in detail and I have been looking everywhere without finding a solution…
    Thanks in advance.

    1. Cameron

      Hi Fabian,

      In the comment you’re referencing I believe I misread what they typed, and thought their in-game source was slower than their video camera source. The last time I checked it was not possible in OBS to add a video delay to a window/game capture source. The latency you’re seeing on the video source is caused by the capture card.

      A potential work around would be using a second computer to capture the in game footage, and then using a second capture card to pull this video feed into OBS, and it should have the same latency.

      Cheers,

      Cameron

  20. Cat

    Hi Cameron,

    Thanks for the sweet walk through! I’ve gotten everything working except for the alpha mask. I’ve downloaded the plugin, replaced it in the obs-filters folder, and applied the Image/Mask – Blend (Subtraction) to my un-cropped foreground layer but nothing is happening! I’ve even attached a random photo. Using OBS 19.0.3.

    Any ideas?

    1. Cameron

      Hi Cat,

      It sounds like you’ve done everything right, perhaps you can try asking the plugin creator directly here?

      Cheers,

      Cameron

  21. Rafael

    Hi!

    I’ve followed your tutorial about Mixed Reality, and it’s awesome!

    However, I’m seeking for help, on 2 issues I’m having.

    1 – Controllers goes to sleep every minute, making me have to restart everything. Is there a way to never let the controller goes to sleep?

    2 – Sometimes (even when it’s all green and not flashing) the game wont open into 4 quadrants, it opens normaly. Is there a way to ALWAYS open the game on quadrants (4 displays), even if there are no controllers
    conected at all?

    That would be great if you could help us!

    Kind regards,

    Rafael
    Microblast Games
    contact@microblastgames.com

    1. Cameron

      Hi Rafael,

      Glad to hear you’ve found my guide helpful 🙂

      1) I’m assuming you’re referring to your 3rd controller going to sleep? In the SteamVR overview settings there’s an option to change the energy settings for your controllers and HMD. In this area you can define the timeout period.
      2) I’m not aware of a way to do this unfortunately. Sorry!

      Cheers,

      Cameron

  22. Rafael

    Hi Cameron, thanks for the answer! 🙂

    The problem is that if any controller is ”flashing” on steam vr status, even if the HMD is flashing, if they are not all solid greenlight, the game wont start with that 4 split quadrants, it opens normaly.

    My solution is to keep moving both controlers and the head set in front of the stations to keep that solid green and than, very fast, click to open the game.
    Does that happen to you? Do you have any suggestions?

    Thanks!

  23. Ren

    Hello Cameron

    This tutorial is awesome but I’m trying to do something similar for a game running on Unreal Engine Version 4.16. Is there a way that you know of that could work with this version of Unreal Engine?

    Thanks, Ren

    1. Cameron

      Hi Ren,

      In order to do it with Unreal Engine the game developer would need to implement mixed reality support into their game.

      Cheers,

      Cameron

  24. JERRY KNAPP

    Hello,I downloaded the config file from Tribal on YouTube and had a question,Once I get the config right with Logitech Brio web cam with 3 controller over top of it on my monitor and save file.Can I relocate camera and 3rd controller at a different angle of room as to not interfere with me being at computer and turning monitor for other use.

    1. Thanks for the reply.

  25. samco

    hi. I’ve followed all the required steps to setup the mixed vr experience. everything works fine except for the fact that after I connect the third controller the 4 quadrants show up alright but my background layer is halved for some reason. has anyone come across this problem and if so what is the fix for it?

Leave A Comment

Your email address will not be published. Required fields are marked *