So in your Hololens app you can add UX buttons pretty easily. I’m going to do something super simple and set up 4 buttons that are going to change the colour of an interactive cube.

 

STEP 1 CREATE CUBE BLUEPRINT WITH FUNCTIONS

 

In your Content Window select Add/Import >> Blueprint Class >> Actor and call it something generic like CubeBaseClass. Double click to open it.

In the components panel select Add Component >> Cube

 

 

In the Functions panel select the + to Add Function and name that function funcChangeColor

 

 

Now we’re going to pass through the new material that we want to assign to the cube, so lets go to the FunctionChangeColor graph editor and add that in. So go to the Details panel and go down to the Inputs section and click on the + to Add Parameter.

 

 

Call the new param Material and in the search box type Material and select the blue object reference option.

 

 

Tick the Pass by Reference checkbox and check that your Inputs looks exactly the same as this pic.

 

 

Now it’s time to go into the function event graph. Left click off the pin and create a new Set Material(cube) node.

 

 

Now we need to create a reference to the cube so the function can assign the material to it. In the My Blueprint panel, scroll down to components and click and drag the Cube into the event graph and choose Get Cube

 

 

Now click and drag the cube pin to the set material Target Pin to assign the cube as the target to change the material of.

 

 

Now click and drag to link up the Func Change Color Material Pin to the Set Material Material Pin

 

 

VERY IMPORTANT – Compile and save

 

STEP 2 CREATE 4 NEW MATERIALS

 

Since you have the UX Tools installed you can go to the Materials folder and duplicate M_SimpleLit_Color 4 times, rename them with the colour you want to change it to. eg MaterialBlue, MaterialOrange, MaterialPink, MaterialGreen.

 

 

Double click to open each material and then click on Base Color node to bring up the colour wheel to change the color. Save and shut each new material

 

 

STEP 3 CREATE UX BUTTON

 

Down in your Content Window select Add New > Blueprint Class, expand the All Classes section and search for UxtPressableButtonActor. Call it BtnUpdateCube and double click to open.

 

 

Make sure BtnUpdateCube(self) is selected in the Components panel and in the Details panel scroll down to the Uxt Pressable Button section and expand the Icon Brush section. You can scroll through and choose any icon you like.

 

 

Now go back to the Components panel and click ButtonComponent(UtxPressableButton)(inherited) and scroll down the Details panel to the Events section. Click the green + button next to On Button Pressed. This will bring up the event graph.

 

 

Now we need to set up the variables we will use which is a node to hold the reference to the cube, and a node to hold the material variable to send through to the cube.

Go to the MyBlueprint panel and scroll down to  the Variables section. Click the + to Add Variable. Name it Material and then go to the Details panel of your Material node and click the Variable Type dropdown and search for Material and select the blue Object Reference option. 

Check the Instance Editable and Expose on Spawn boxes.

 

 

Now create your cube variable – this is pretty similar. Back in the My Blueprint panel, Variables section click + to add a new variable and call it CubeBaseClass. Go to the details panel for the CubeBaseClass and click the dropdown next to Variable Type and search for CubeBaseClass and choose the blue object reference option. Check the Instance Editable and Expose on Spawn boxes.

 

 

Drag the Material Variable from the Variables section onto the EventGraph and choose Get Material, do the same with the CubeBaseClass.

Click on the Cube Base Class pin and drag and release to place a new node. Search for the Func Change Color (which is on our Cube BaseClass) 

 

 

Click on the Material pin and connect it to the FuncChangeColor Material pin

 

 

Now click and drag to connect the OnButtonPressed Pin to the Func Change Color pin. Compile and Save.

 

 

CREATE YOUR SCENE

 

Drag and drop you CubeBaseClass to the scene and place it at x:50 y:00 z:-10 and change the scale to x:0.1 y:0.1 z:0.1.

 

 

Drag and drop you BtnUpdateCube to the scene and in the Details panel scroll to the Transform section and place it at x:50 y:-20 z:-20 with z rotation 180 and scale x:2 y:2 z:2.

Scroll to the Uxt Pressable Button section and update the label to be Pink. Scroll to the Default section and drag and drop your pink material onto the square next to Material and from the drop down next to Cube Base Class, choose CubeBaseClass.

 

 

Now duplicate your button 3 times and move them so you can see them all. Update the labels to be blue, green and orange and drag in the correct materials for each of them and assign the CubeBaseClass

 

 

TEST ON THE HOLOLENS 2

 

Start up your Hololens 2 and open up the Holographic remoting app. I’m assuming here that you have it installed already so if you haven’t, go install it.

 

 

Use the IP address given by the app to connect to it from Unreal. Grab the number you see here in the hololens…

 

 

And go into Unreal Project Settings, search for holographic to bring up the connection panel. Put your IP in where I have shown you and hit connect.

 

 

You’ll see the green connected response when UE is connected to the Hololens. Now go to Play >> VR Preview and you’ll see it load in your headset.

 

 

Holy Moly Metahumans RUN on the Hololens 2!

 

It’s just a start, lighting needs fixing and the hair quality is low, but you can see the Metahuman in the Hololens 2 and it looks pretty ace.

I’m just streaming it to the HL2 headset straight from UE at the moment. I haven’t managed to successfully package it and copy it across to the headset yet. That’s proving difficult and I might have to come back to that later.

Next stop, let’s get it animating!

 

LIVE LINK FACE AND METAHUMANS ON HL2

 

And with Live Link Face controlling the facial animation. The hair hasn’t shown up in the recording, but it is there when you are looking at the hologram.

 

STEP 1. Live Link Face

 

So before you do anything else, you need to go and download the Live Link Face app by Unreal Engine.

 

 

Make sure your computer and phone are connected to the same wireless network.

Open Live Link Face and go into the settings (cog icon top left) and then click on Live Link

 

 

Click on Add Target and put in your computer’s IP address

 

 

Step 2 Unreal Set Up

 

So back in UE go into your Plugins with Edit >> Plugins. Search for ARKit and enable both Apple ARKit and Apple ARKit Face Support. UE will want to restart to take effect so go ahead and do that.

 

 

Now open the Livelink tab with Window >> Live Link

 

 

And make sure your phone is selected.

 

 

Now select your character and go to the Default tab. Select the dropdown for LLink Face Subj and choose your phone from the dropdown. Then ensure the checkbox for LLink Face Head is ticked.

 

 

File >> Save All

Press Play >> Simulate and start talking into your phone to see it reproduced by your Metahuman!

 

Again I am following the Microsoft tutorial, just adding in extra screen grabs to help solidify the process in my mind. For the original tutorial, look here https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unreal/tutorials/unreal-uxt-ch4

 

Spawning Hand Interaction Actors

 

I’m not going to tell you about the kinds of hand interactions you can do in Hololens 2 because I’m assuming you know about that already. This is just to tell you how to set them up for Unreal.

Hand interaction with UX elements is done with Hand Interaction Actors, which create and drive the pointers and visuals for near and far interactions.

open the MRPawn Blueprint and go to the Event Graph.

  1. Drag and release the execution pin from Event BeginPlay to place a new node.
    • Select Spawn Actor from Class, click the dropdown next to the Class pin and search for Uxt Hand Interaction Actor.
  2. From the Right arrow at the top pf the Hand Interaction Actor Box, drag and release to spawn a second Uxt Hand Interaction Actor, this time setting the Hand to Right. When the event begins, a Uxt Hand Interaction Actor will be spawned on each hand.

Your Event Graph should match the following screenshot:

Spawn UXT Hand Interaction Actors

 

The SpawnActor function requires a Transform input to avoid a compiler error, so you’ll use the default values.

Drag and release the pin off one of the Spawn Transform pins to place a new node. Search for the Make Transform node.

 

 

Drag the Return Value to the other hand’s Spawn Transform so that both SpawnActor nodes are connected.

 

 

Select the down arrow at the bottom of both SpawnActor nodes to reveal the Owner pin. Drag the pin off one of the Owner pins and release to place a new node. Search for self and select the Get a reference to self variable.

 

 

Create a link between the Self object reference node and the other Hand Interaction Actor’s Owner pin.

 

 

check the Show Near Cursor on Grab Targets box for both Hand Interaction Actors. A cursor should appear on the grab target as your index finger gets close, so you can see where your finger is relative to the target.

 

 

Compilesave, and return to the Main window. – I’m not screen grabbing that. You know how to do that by now. But double check that your Event Graph looks the same as this last screen grab.

 

Create something to interact with

 

So let’s create some things to interact with. I’ll show you how to make a cube blueprint and then you can make however many objects you like to go in the scene.

Go to the Content Browser >> Add/import >> Blueprint Class

 

choose Actor. Name the actor InteractiveCube – or whatever name you like.

 

 

Double-click InteractiveCube to open it in the Blueprint Editor, select Add Component > Scene and name it Root.

 

 

Drag-and-drop Root onto DefaultSceneRoot to replace it.

 

 

Click Add Component >Cube

 

 

Compile and save your blueprint, then head back to the main window.

 

 

Drag InteractiveCube into the viewport

 

 

Attaching Manipulators

 

A Manipulator is a component that responds to articulated hand input and can be grabbed, rotated, and translated. Applying the Manipulator’s transform to an Actors transform allows direct Actor manipulation.

Open the InteractiveCube blueprint, click Add Component and search for Uxt Generic Manipulator in the Components panel.

 

 

Expand the Generic Manipulator section in the Details panel. You can set one-handed or two-handed manipulation, rotation mode, and smoothing from here. Feel free to select whichever modes you wish, then Compile and Save InteractiveCube.

 

<<< STEP 5 – ADDING MIXED REALITY

Again I am following the Microsoft tutorial, just adding in extra screen grabs to help solidify the process in my mind. For the original tutorial, look here https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unreal/tutorials/unreal-uxt-ch3

 

Add The Session Asset

To use a session, you need an ARSessionConfig data asset to work with. To create one of these go to your Content Browser window. Make sure you are in the root Content folder by clicking on the Content folder.

Click Add/Import > Miscellaneous > Data Asset 

 

Select ARSessionConfig, click Select

 

 

name the asset ARSessionConfig.

 

 

Double-click ARSessionConfig to open it, leave all default settings and hit Save. Return to the Main window.

 

 

Start and Stop AR Session

 

To start the AR session you use a Level Blueprint. Connecting the ARSessionConfig asset in the Level Blueprint guarantees the AR session will fire right when the game starts playing.

To do this click Blueprints > Open Level Blueprint from the editor toolbar:

 

 

This will open up the Eventgraph window. Now this is node based work, so if you’ve ever used Shadergraph or Houdini then you’ll be familiar with this kind of coding. If not then just hang in there, this is a really easy entry example to set up. Don’t worry, you’ll get used to it.

Drag the execution node (left-facing arrow icon) off Event BeginPlay and release

 

 

then search for the Start AR Session node

 

 

and hit enter. Excellent, you just connected two nodes.

 

 

Click the Select Asset dropdown under Session Config and choose the ARSessionConfig asset.

 

 

Right-click anywhere in the EventGraph and create a new Event EndPlay node. 

 

 

Just like we did for the BeginPlay Event, we’re going to drag the execution pin and release, then search for a Stop AR Session node and hit enter.

 

Hit Compile, then Save and return to the Main window.

 

 

create a pawn

At this point, the project still needs a player object. In Unreal, a Pawn represents the user in the game, but in this case it’s going to be the HoloLens 2 experience.

 

Click Add New > Blueprint Class in the Content folder

 

 

Expand the All Classes section at the bottom. Search for DefaultPawn, click Select

 

 

Name it MRPawn, and double-click the asset to open.

 

 

Click Add Component > Camera from the Components panel and name it Camera.

 

 

Make sure that the Camera component is a direct child of the root (CollisionComponent). This allows the player camera to move with the HoloLens 2 device.

 

 

Select CollisionComponent from the Components panel and scroll down to the Collision section of the Details panel.

  • Click the Collision Presets dropdown and change the value to NoCollision.
  • Do the same for the MeshComponent

 

 

Compile and Save the Blueprint, then return to the Main Window

 

 

create a game mode

 

The last puzzle piece of the mixed reality setup is the Game Mode. The Game Mode determines a number of settings for the game or experience, including the default pawn to use.

 

Click Add New > Blueprint Class in the Content folder and select Game Mode Base as the parent class. 

 

 

Name it MRGameMode and double-click to open.

 

 

Go to the Classes section in the Details panel and change the Default Pawn Class to MRPawn

 

 

Hit Compile, then Save and return to the Main window.

 

 

Select Edit > Projects Settings and click Maps & Modes in the left-hand list.

  • Expand Default Modes and change Default Game Mode to MRGameMode.
  • Expand Default Maps and change both EditorStartupMap and GameDefaultMap to Main. When you close and reopen the editor or play the game, the Main map will now be selected by default.

 

<<<< STEP 4 – CREATE A LEVEL        STEP 6 – INTERACTIVity >>>>

 

 

Again  I am taking instructions straight from a section in Microsoft’s Chess Tutorial. But I am just adding in a few extra pictures because I am a visual learner and this really helps me solidify what I’m doing in my head. So some people might find this quite slow and hand holdy, so sorry, maybe the Microsoft pages will be better for you.

https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unreal/tutorials/unreal-uxt-ch2 The whole tutorial is great and recommend you read through it.

So,

CREATE YOUR LEVEL – Select File > New Level 

 

and choose Empty Level. The default scene in the viewport should now be empty.

 

 

CREATE PLAYER START – Select Basic from the Modes tab and drag PlayerStart into the scene.

 

 

Set Location to X = 0Y = 0, and Z = 0 in the Details tab to set the user at the centre of the scene when the app starts up.

 

Viewport with PlayerStart

 

ADD A CUBE – Drag a Cube from the Basic tab into the scene.

  • Set Location to X = 50Y = 0, and Z = 0. to position the cube 50 cm away from the player at start time. (see picture above for where you’ll find Transform settings again)
  • Change Scale to X = 0.2Y = 0.2, and Z = 0.2 to shrink the cube down.

 

 

ADD A LIGHT – Switch to the Lights tab in the Modes panel and drag a Directional Light into the scene. Position the light above PlayerStart so you can see it

 

 

SAVE YOUR FILE – Go to File > Save Current As, name your level Main, and select Save.

With the scene set, press Play in the toolbar to see your cube in action! When you’re finished admiring your work, press Esc to stop the application.

 

 

<<<< STEP 3 – ADDING PLUGINS         STEP 5 – SET UP FOR MIXED REALITY >>>>

 

Adding plugins to your project is pretty straight forward as long as you know what folders you need to copy and where to.

  1. Go to your root project folder and create a Plugins folder if there isn’t one already
  2. Unzip the zip you downloaded for the  UXTools
    1. Navigate to the Plugins folder in the unzipped files. It will be something like >> MixedReality-UXTools-Unreal-public-0.11.x >> UXToolsGame >> Plugins >>
    2. Copy the folders named UXTools  to your project Plugins folder

     

And then the final thing is you need to go into Unreal and enable your plugins. In UE to Edit >> Plugins to bring up the Plugins Editor.

Scroll down to HoloLens and check Enabled

 

Next, ensure All is selected in the top left, and in the top right grey search box, type mixed to bring up all the Mixed Reality Plugins. Click the Enabled checkbox in the Microsoft Mixed Reality Plugins. You will be prompted to restart so go ahead and press the Restart Now button in the bottom right corner.

When you restart Unreal will build the UXTools into the project

 

 

When Unreal restarts the UXTools plugin will be in your Content Browser. If you can’t see them there (and I couldn’t when I first set up my file), you can display them by adjusting the Content Browser View Options. Select >> View Options >> Show Plugin Content. You can also Show C++ Classes, but for now turn everything else off otherwise you’ll have a tonne of folders to sift through.

 

<<<< STEP 2 – FILE SET UP         STEP 4 – CREATE A LEVEL >>>>

 

YOU have everything downloaded, now what?

Launch the Unreal Engine 4.26 from the Epic Games Launcher.

 

In the Select or Create New Project Window, select Games in the New Project Category section and then hit Next

 

 

Next select a Blank Template and then hit Next

 

 

And in the Project Settings Select C++ Project, Scalable 2D or 3D and Mobile/Tablet

Choose a name and where you want to save it.

Then hit Create Project

 

 

<<<< Step 1 SET UP        STEP 3 – ADDING PLUGINS >>>>

I am following the microsoft Chess app tutorial as I do this – adding in screen grabs for extra information for proper nubies, so for the original instructions go to microsoft
https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unreal/tutorials/unreal-uxt-ch1

 

Before starting

 

have the following PROGRAMS installed.

HAVE THE FOLLOWING PLUGINS READY TO INSTALL

Download the following plugins so that you have them ready to copy to your Plugins folder once you have created your project. I’ll remind you later, but to find instructions on how to install plugins, see the Blog post titled Adding Plugins To Your Unreal Project

HAVE THE HOLOLENS 2 TARGET PLATFORM INSTALLED

To do this,

>> go to the Epic Games Launcher

>> click on the Library button at the top of the page

>>click on the arrow button next to launch on the 4.26.1 icon

>> go to options

 

>> scroll down until you get to the Target Platform section and ensure that Hololens 2 is ticked.

>> if it isn’t ticked, tick it and hit the apply button. The download will start for you.

STEP 2 – FILE SET UP >>>>

To use the cross platform input manager you need to make sure you have the Standard Assets Package included in your Unity project. To import it, use these instructions

https://docs.unity3d.com/560/Documentation/Manual/AssetPackages.html


Ensure that CrossPlatformInput is selected in both the “Editor” and “Standard Assets” folders.

Open up the Project Setting > Inputs to define all of the controls

To refer to the CPI from your class file you must include it at the top with

 using UnityStandardAssets.CrossPlatformInput;

Then you can reference it directly

 CrossPlatformInputManager.GetAxis("Horizontal");