So in your Hololens app you can add UX buttons pretty easily. I’m going to do something super simple and set up 4 buttons that are going to change the colour of an interactive cube.




In your Content Window select Add/Import >> Blueprint Class >> Actor and call it something generic like CubeBaseClass. Double click to open it.

In the components panel select Add Component >> Cube



In the Functions panel select the + to Add Function and name that function funcChangeColor



Now we’re going to pass through the new material that we want to assign to the cube, so lets go to the FunctionChangeColor graph editor and add that in. So go to the Details panel and go down to the Inputs section and click on the + to Add Parameter.



Call the new param Material and in the search box type Material and select the blue object reference option.



Tick the Pass by Reference checkbox and check that your Inputs looks exactly the same as this pic.



Now it’s time to go into the function event graph. Left click off the pin and create a new Set Material(cube) node.



Now we need to create a reference to the cube so the function can assign the material to it. In the My Blueprint panel, scroll down to components and click and drag the Cube into the event graph and choose Get Cube



Now click and drag the cube pin to the set material Target Pin to assign the cube as the target to change the material of.



Now click and drag to link up the Func Change Color Material Pin to the Set Material Material Pin



VERY IMPORTANT – Compile and save




Since you have the UX Tools installed you can go to the Materials folder and duplicate M_SimpleLit_Color 4 times, rename them with the colour you want to change it to. eg MaterialBlue, MaterialOrange, MaterialPink, MaterialGreen.



Double click to open each material and then click on Base Color node to bring up the colour wheel to change the color. Save and shut each new material





Down in your Content Window select Add New > Blueprint Class, expand the All Classes section and search for UxtPressableButtonActor. Call it BtnUpdateCube and double click to open.



Make sure BtnUpdateCube(self) is selected in the Components panel and in the Details panel scroll down to the Uxt Pressable Button section and expand the Icon Brush section. You can scroll through and choose any icon you like.



Now go back to the Components panel and click ButtonComponent(UtxPressableButton)(inherited) and scroll down the Details panel to the Events section. Click the green + button next to On Button Pressed. This will bring up the event graph.



Now we need to set up the variables we will use which is a node to hold the reference to the cube, and a node to hold the material variable to send through to the cube.

Go to the MyBlueprint panel and scroll down to  the Variables section. Click the + to Add Variable. Name it Material and then go to the Details panel of your Material node and click the Variable Type dropdown and search for Material and select the blue Object Reference option. 

Check the Instance Editable and Expose on Spawn boxes.



Now create your cube variable – this is pretty similar. Back in the My Blueprint panel, Variables section click + to add a new variable and call it CubeBaseClass. Go to the details panel for the CubeBaseClass and click the dropdown next to Variable Type and search for CubeBaseClass and choose the blue object reference option. Check the Instance Editable and Expose on Spawn boxes.



Drag the Material Variable from the Variables section onto the EventGraph and choose Get Material, do the same with the CubeBaseClass.

Click on the Cube Base Class pin and drag and release to place a new node. Search for the Func Change Color (which is on our Cube BaseClass) 



Click on the Material pin and connect it to the FuncChangeColor Material pin



Now click and drag to connect the OnButtonPressed Pin to the Func Change Color pin. Compile and Save.





Drag and drop you CubeBaseClass to the scene and place it at x:50 y:00 z:-10 and change the scale to x:0.1 y:0.1 z:0.1.



Drag and drop you BtnUpdateCube to the scene and in the Details panel scroll to the Transform section and place it at x:50 y:-20 z:-20 with z rotation 180 and scale x:2 y:2 z:2.

Scroll to the Uxt Pressable Button section and update the label to be Pink. Scroll to the Default section and drag and drop your pink material onto the square next to Material and from the drop down next to Cube Base Class, choose CubeBaseClass.



Now duplicate your button 3 times and move them so you can see them all. Update the labels to be blue, green and orange and drag in the correct materials for each of them and assign the CubeBaseClass





Start up your Hololens 2 and open up the Holographic remoting app. I’m assuming here that you have it installed already so if you haven’t, go install it.



Use the IP address given by the app to connect to it from Unreal. Grab the number you see here in the hololens…



And go into Unreal Project Settings, search for holographic to bring up the connection panel. Put your IP in where I have shown you and hit connect.



You’ll see the green connected response when UE is connected to the Hololens. Now go to Play >> VR Preview and you’ll see it load in your headset.



Holy Moly Metahumans RUN on the Hololens 2!


It’s just a start, lighting needs fixing and the hair quality is low, but you can see the Metahuman in the Hololens 2 and it looks pretty ace.

I’m just streaming it to the HL2 headset straight from UE at the moment. I haven’t managed to successfully package it and copy it across to the headset yet. That’s proving difficult and I might have to come back to that later.

Next stop, let’s get it animating!




And with Live Link Face controlling the facial animation. The hair hasn’t shown up in the recording, but it is there when you are looking at the hologram.


STEP 1. Live Link Face


So before you do anything else, you need to go and download the Live Link Face app by Unreal Engine.



Make sure your computer and phone are connected to the same wireless network.

Open Live Link Face and go into the settings (cog icon top left) and then click on Live Link



Click on Add Target and put in your computer’s IP address



Step 2 Unreal Set Up


So back in UE go into your Plugins with Edit >> Plugins. Search for ARKit and enable both Apple ARKit and Apple ARKit Face Support. UE will want to restart to take effect so go ahead and do that.



Now open the Livelink tab with Window >> Live Link



And make sure your phone is selected.



Now select your character and go to the Default tab. Select the dropdown for LLink Face Subj and choose your phone from the dropdown. Then ensure the checkbox for LLink Face Head is ticked.



File >> Save All

Press Play >> Simulate and start talking into your phone to see it reproduced by your Metahuman!


Again I am following the Microsoft tutorial, just adding in extra screen grabs to help solidify the process in my mind. For the original tutorial, look here


Spawning Hand Interaction Actors


I’m not going to tell you about the kinds of hand interactions you can do in Hololens 2 because I’m assuming you know about that already. This is just to tell you how to set them up for Unreal.

Hand interaction with UX elements is done with Hand Interaction Actors, which create and drive the pointers and visuals for near and far interactions.

open the MRPawn Blueprint and go to the Event Graph.

  1. Drag and release the execution pin from Event BeginPlay to place a new node.
    • Select Spawn Actor from Class, click the dropdown next to the Class pin and search for Uxt Hand Interaction Actor.
  2. From the Right arrow at the top pf the Hand Interaction Actor Box, drag and release to spawn a second Uxt Hand Interaction Actor, this time setting the Hand to Right. When the event begins, a Uxt Hand Interaction Actor will be spawned on each hand.

Your Event Graph should match the following screenshot:

Spawn UXT Hand Interaction Actors


The SpawnActor function requires a Transform input to avoid a compiler error, so you’ll use the default values.

Drag and release the pin off one of the Spawn Transform pins to place a new node. Search for the Make Transform node.



Drag the Return Value to the other hand’s Spawn Transform so that both SpawnActor nodes are connected.



Select the down arrow at the bottom of both SpawnActor nodes to reveal the Owner pin. Drag the pin off one of the Owner pins and release to place a new node. Search for self and select the Get a reference to self variable.



Create a link between the Self object reference node and the other Hand Interaction Actor’s Owner pin.



check the Show Near Cursor on Grab Targets box for both Hand Interaction Actors. A cursor should appear on the grab target as your index finger gets close, so you can see where your finger is relative to the target.



Compilesave, and return to the Main window. – I’m not screen grabbing that. You know how to do that by now. But double check that your Event Graph looks the same as this last screen grab.


Create something to interact with


So let’s create some things to interact with. I’ll show you how to make a cube blueprint and then you can make however many objects you like to go in the scene.

Go to the Content Browser >> Add/import >> Blueprint Class


choose Actor. Name the actor InteractiveCube – or whatever name you like.



Double-click InteractiveCube to open it in the Blueprint Editor, select Add Component > Scene and name it Root.



Drag-and-drop Root onto DefaultSceneRoot to replace it.



Click Add Component >Cube



Compile and save your blueprint, then head back to the main window.



Drag InteractiveCube into the viewport



Attaching Manipulators


A Manipulator is a component that responds to articulated hand input and can be grabbed, rotated, and translated. Applying the Manipulator’s transform to an Actors transform allows direct Actor manipulation.

Open the InteractiveCube blueprint, click Add Component and search for Uxt Generic Manipulator in the Components panel.



Expand the Generic Manipulator section in the Details panel. You can set one-handed or two-handed manipulation, rotation mode, and smoothing from here. Feel free to select whichever modes you wish, then Compile and Save InteractiveCube.



Again I am following the Microsoft tutorial, just adding in extra screen grabs to help solidify the process in my mind. For the original tutorial, look here


Add The Session Asset

To use a session, you need an ARSessionConfig data asset to work with. To create one of these go to your Content Browser window. Make sure you are in the root Content folder by clicking on the Content folder.

Click Add/Import > Miscellaneous > Data Asset 


Select ARSessionConfig, click Select



name the asset ARSessionConfig.



Double-click ARSessionConfig to open it, leave all default settings and hit Save. Return to the Main window.



Start and Stop AR Session


To start the AR session you use a Level Blueprint. Connecting the ARSessionConfig asset in the Level Blueprint guarantees the AR session will fire right when the game starts playing.

To do this click Blueprints > Open Level Blueprint from the editor toolbar:



This will open up the Eventgraph window. Now this is node based work, so if you’ve ever used Shadergraph or Houdini then you’ll be familiar with this kind of coding. If not then just hang in there, this is a really easy entry example to set up. Don’t worry, you’ll get used to it.

Drag the execution node (left-facing arrow icon) off Event BeginPlay and release



then search for the Start AR Session node



and hit enter. Excellent, you just connected two nodes.



Click the Select Asset dropdown under Session Config and choose the ARSessionConfig asset.



Right-click anywhere in the EventGraph and create a new Event EndPlay node. 



Just like we did for the BeginPlay Event, we’re going to drag the execution pin and release, then search for a Stop AR Session node and hit enter.


Hit Compile, then Save and return to the Main window.



create a pawn

At this point, the project still needs a player object. In Unreal, a Pawn represents the user in the game, but in this case it’s going to be the HoloLens 2 experience.


Click Add New > Blueprint Class in the Content folder



Expand the All Classes section at the bottom. Search for DefaultPawn, click Select



Name it MRPawn, and double-click the asset to open.



Click Add Component > Camera from the Components panel and name it Camera.



Make sure that the Camera component is a direct child of the root (CollisionComponent). This allows the player camera to move with the HoloLens 2 device.



Select CollisionComponent from the Components panel and scroll down to the Collision section of the Details panel.

  • Click the Collision Presets dropdown and change the value to NoCollision.
  • Do the same for the MeshComponent



Compile and Save the Blueprint, then return to the Main Window



create a game mode


The last puzzle piece of the mixed reality setup is the Game Mode. The Game Mode determines a number of settings for the game or experience, including the default pawn to use.


Click Add New > Blueprint Class in the Content folder and select Game Mode Base as the parent class. 



Name it MRGameMode and double-click to open.



Go to the Classes section in the Details panel and change the Default Pawn Class to MRPawn



Hit Compile, then Save and return to the Main window.



Select Edit > Projects Settings and click Maps & Modes in the left-hand list.

  • Expand Default Modes and change Default Game Mode to MRGameMode.
  • Expand Default Maps and change both EditorStartupMap and GameDefaultMap to Main. When you close and reopen the editor or play the game, the Main map will now be selected by default.


<<<< STEP 4 – CREATE A LEVEL        STEP 6 – INTERACTIVity >>>>



Again  I am taking instructions straight from a section in Microsoft’s Chess Tutorial. But I am just adding in a few extra pictures because I am a visual learner and this really helps me solidify what I’m doing in my head. So some people might find this quite slow and hand holdy, so sorry, maybe the Microsoft pages will be better for you. The whole tutorial is great and recommend you read through it.


CREATE YOUR LEVEL – Select File > New Level 


and choose Empty Level. The default scene in the viewport should now be empty.



CREATE PLAYER START – Select Basic from the Modes tab and drag PlayerStart into the scene.



Set Location to X = 0Y = 0, and Z = 0 in the Details tab to set the user at the centre of the scene when the app starts up.


Viewport with PlayerStart


ADD A CUBE – Drag a Cube from the Basic tab into the scene.

  • Set Location to X = 50Y = 0, and Z = 0. to position the cube 50 cm away from the player at start time. (see picture above for where you’ll find Transform settings again)
  • Change Scale to X = 0.2Y = 0.2, and Z = 0.2 to shrink the cube down.



ADD A LIGHT – Switch to the Lights tab in the Modes panel and drag a Directional Light into the scene. Position the light above PlayerStart so you can see it



SAVE YOUR FILE – Go to File > Save Current As, name your level Main, and select Save.

With the scene set, press Play in the toolbar to see your cube in action! When you’re finished admiring your work, press Esc to stop the application.





Adding plugins to your project is pretty straight forward as long as you know what folders you need to copy and where to.

  1. Go to your root project folder and create a Plugins folder if there isn’t one already
  2. Unzip the zip you downloaded for the  UXTools
    1. Navigate to the Plugins folder in the unzipped files. It will be something like >> MixedReality-UXTools-Unreal-public-0.11.x >> UXToolsGame >> Plugins >>
    2. Copy the folders named UXTools  to your project Plugins folder


And then the final thing is you need to go into Unreal and enable your plugins. In UE to Edit >> Plugins to bring up the Plugins Editor.

Scroll down to HoloLens and check Enabled


Next, ensure All is selected in the top left, and in the top right grey search box, type mixed to bring up all the Mixed Reality Plugins. Click the Enabled checkbox in the Microsoft Mixed Reality Plugins. You will be prompted to restart so go ahead and press the Restart Now button in the bottom right corner.

When you restart Unreal will build the UXTools into the project



When Unreal restarts the UXTools plugin will be in your Content Browser. If you can’t see them there (and I couldn’t when I first set up my file), you can display them by adjusting the Content Browser View Options. Select >> View Options >> Show Plugin Content. You can also Show C++ Classes, but for now turn everything else off otherwise you’ll have a tonne of folders to sift through.


<<<< STEP 2 – FILE SET UP         STEP 4 – CREATE A LEVEL >>>>


YOU have everything downloaded, now what?

Launch the Unreal Engine 4.26 from the Epic Games Launcher.


In the Select or Create New Project Window, select Games in the New Project Category section and then hit Next



Next select a Blank Template and then hit Next



And in the Project Settings Select C++ Project, Scalable 2D or 3D and Mobile/Tablet

Choose a name and where you want to save it.

Then hit Create Project



<<<< Step 1 SET UP        STEP 3 – ADDING PLUGINS >>>>

I am following the microsoft Chess app tutorial as I do this – adding in screen grabs for extra information for proper nubies, so for the original instructions go to microsoft


Before starting


have the following PROGRAMS installed.


Download the following plugins so that you have them ready to copy to your Plugins folder once you have created your project. I’ll remind you later, but to find instructions on how to install plugins, see the Blog post titled Adding Plugins To Your Unreal Project


To do this,

>> go to the Epic Games Launcher

>> click on the Library button at the top of the page

>>click on the arrow button next to launch on the 4.26.1 icon

>> go to options


>> scroll down until you get to the Target Platform section and ensure that Hololens 2 is ticked.

>> if it isn’t ticked, tick it and hit the apply button. The download will start for you.


To use the cross platform input manager you need to make sure you have the Standard Assets Package included in your Unity project. To import it, use these instructions

Ensure that CrossPlatformInput is selected in both the “Editor” and “Standard Assets” folders.

Open up the Project Setting > Inputs to define all of the controls

To refer to the CPI from your class file you must include it at the top with

 using UnityStandardAssets.CrossPlatformInput;

Then you can reference it directly