Again I am following the Microsoft tutorial, just adding in extra screen grabs to help solidify the process in my mind. For the original tutorial, look here https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unreal/tutorials/unreal-uxt-ch4

 

Spawning Hand Interaction Actors

 

I’m not going to tell you about the kinds of hand interactions you can do in Hololens 2 because I’m assuming you know about that already. This is just to tell you how to set them up for Unreal.

Hand interaction with UX elements is done with Hand Interaction Actors, which create and drive the pointers and visuals for near and far interactions.

open the MRPawn Blueprint and go to the Event Graph.

  1. Drag and release the execution pin from Event BeginPlay to place a new node.
    • Select Spawn Actor from Class, click the dropdown next to the Class pin and search for Uxt Hand Interaction Actor.
  2. From the Right arrow at the top pf the Hand Interaction Actor Box, drag and release to spawn a second Uxt Hand Interaction Actor, this time setting the Hand to Right. When the event begins, a Uxt Hand Interaction Actor will be spawned on each hand.

Your Event Graph should match the following screenshot:

Spawn UXT Hand Interaction Actors

 

The SpawnActor function requires a Transform input to avoid a compiler error, so you’ll use the default values.

Drag and release the pin off one of the Spawn Transform pins to place a new node. Search for the Make Transform node.

 

 

Drag the Return Value to the other hand’s Spawn Transform so that both SpawnActor nodes are connected.

 

 

Select the down arrow at the bottom of both SpawnActor nodes to reveal the Owner pin. Drag the pin off one of the Owner pins and release to place a new node. Search for self and select the Get a reference to self variable.

 

 

Create a link between the Self object reference node and the other Hand Interaction Actor’s Owner pin.

 

 

check the Show Near Cursor on Grab Targets box for both Hand Interaction Actors. A cursor should appear on the grab target as your index finger gets close, so you can see where your finger is relative to the target.

 

 

Compilesave, and return to the Main window. – I’m not screen grabbing that. You know how to do that by now. But double check that your Event Graph looks the same as this last screen grab.

 

Create something to interact with

 

So let’s create some things to interact with. I’ll show you how to make a cube blueprint and then you can make however many objects you like to go in the scene.

Go to the Content Browser >> Add/import >> Blueprint Class

 

choose Actor. Name the actor InteractiveCube – or whatever name you like.

 

 

Double-click InteractiveCube to open it in the Blueprint Editor, select Add Component > Scene and name it Root.

 

 

Drag-and-drop Root onto DefaultSceneRoot to replace it.

 

 

Click Add Component >Cube

 

 

Compile and save your blueprint, then head back to the main window.

 

 

Drag InteractiveCube into the viewport

 

 

Attaching Manipulators

 

A Manipulator is a component that responds to articulated hand input and can be grabbed, rotated, and translated. Applying the Manipulator’s transform to an Actors transform allows direct Actor manipulation.

Open the InteractiveCube blueprint, click Add Component and search for Uxt Generic Manipulator in the Components panel.

 

 

Expand the Generic Manipulator section in the Details panel. You can set one-handed or two-handed manipulation, rotation mode, and smoothing from here. Feel free to select whichever modes you wish, then Compile and Save InteractiveCube.

 

<<< STEP 5 – ADDING MIXED REALITY

Again I am following the Microsoft tutorial, just adding in extra screen grabs to help solidify the process in my mind. For the original tutorial, look here https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unreal/tutorials/unreal-uxt-ch3

 

Add The Session Asset

To use a session, you need an ARSessionConfig data asset to work with. To create one of these go to your Content Browser window. Make sure you are in the root Content folder by clicking on the Content folder.

Click Add/Import > Miscellaneous > Data Asset 

 

Select ARSessionConfig, click Select

 

 

name the asset ARSessionConfig.

 

 

Double-click ARSessionConfig to open it, leave all default settings and hit Save. Return to the Main window.

 

 

Start and Stop AR Session

 

To start the AR session you use a Level Blueprint. Connecting the ARSessionConfig asset in the Level Blueprint guarantees the AR session will fire right when the game starts playing.

To do this click Blueprints > Open Level Blueprint from the editor toolbar:

 

 

This will open up the Eventgraph window. Now this is node based work, so if you’ve ever used Shadergraph or Houdini then you’ll be familiar with this kind of coding. If not then just hang in there, this is a really easy entry example to set up. Don’t worry, you’ll get used to it.

Drag the execution node (left-facing arrow icon) off Event BeginPlay and release

 

 

then search for the Start AR Session node

 

 

and hit enter. Excellent, you just connected two nodes.

 

 

Click the Select Asset dropdown under Session Config and choose the ARSessionConfig asset.

 

 

Right-click anywhere in the EventGraph and create a new Event EndPlay node. 

 

 

Just like we did for the BeginPlay Event, we’re going to drag the execution pin and release, then search for a Stop AR Session node and hit enter.

 

Hit Compile, then Save and return to the Main window.

 

 

create a pawn

At this point, the project still needs a player object. In Unreal, a Pawn represents the user in the game, but in this case it’s going to be the HoloLens 2 experience.

 

Click Add New > Blueprint Class in the Content folder

 

 

Expand the All Classes section at the bottom. Search for DefaultPawn, click Select

 

 

Name it MRPawn, and double-click the asset to open.

 

 

Click Add Component > Camera from the Components panel and name it Camera.

 

 

Make sure that the Camera component is a direct child of the root (CollisionComponent). This allows the player camera to move with the HoloLens 2 device.

 

 

Select CollisionComponent from the Components panel and scroll down to the Collision section of the Details panel.

  • Click the Collision Presets dropdown and change the value to NoCollision.
  • Do the same for the MeshComponent

 

 

Compile and Save the Blueprint, then return to the Main Window

 

 

create a game mode

 

The last puzzle piece of the mixed reality setup is the Game Mode. The Game Mode determines a number of settings for the game or experience, including the default pawn to use.

 

Click Add New > Blueprint Class in the Content folder and select Game Mode Base as the parent class. 

 

 

Name it MRGameMode and double-click to open.

 

 

Go to the Classes section in the Details panel and change the Default Pawn Class to MRPawn

 

 

Hit Compile, then Save and return to the Main window.

 

 

Select Edit > Projects Settings and click Maps & Modes in the left-hand list.

  • Expand Default Modes and change Default Game Mode to MRGameMode.
  • Expand Default Maps and change both EditorStartupMap and GameDefaultMap to Main. When you close and reopen the editor or play the game, the Main map will now be selected by default.

 

<<<< STEP 4 – CREATE A LEVEL        STEP 6 – INTERACTIVity >>>>

 

 

Again  I am taking instructions straight from a section in Microsoft’s Chess Tutorial. But I am just adding in a few extra pictures because I am a visual learner and this really helps me solidify what I’m doing in my head. So some people might find this quite slow and hand holdy, so sorry, maybe the Microsoft pages will be better for you.

https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unreal/tutorials/unreal-uxt-ch2 The whole tutorial is great and recommend you read through it.

So,

CREATE YOUR LEVEL – Select File > New Level 

 

and choose Empty Level. The default scene in the viewport should now be empty.

 

 

CREATE PLAYER START – Select Basic from the Modes tab and drag PlayerStart into the scene.

 

 

Set Location to X = 0Y = 0, and Z = 0 in the Details tab to set the user at the centre of the scene when the app starts up.

 

Viewport with PlayerStart

 

ADD A CUBE – Drag a Cube from the Basic tab into the scene.

  • Set Location to X = 50Y = 0, and Z = 0. to position the cube 50 cm away from the player at start time. (see picture above for where you’ll find Transform settings again)
  • Change Scale to X = 0.2Y = 0.2, and Z = 0.2 to shrink the cube down.

 

 

ADD A LIGHT – Switch to the Lights tab in the Modes panel and drag a Directional Light into the scene. Position the light above PlayerStart so you can see it

 

 

SAVE YOUR FILE – Go to File > Save Current As, name your level Main, and select Save.

With the scene set, press Play in the toolbar to see your cube in action! When you’re finished admiring your work, press Esc to stop the application.

 

 

<<<< STEP 3 – ADDING PLUGINS         STEP 5 – SET UP FOR MIXED REALITY >>>>

 

Adding plugins to your project is pretty straight forward as long as you know what folders you need to copy and where to.

  1. Go to your root project folder and create a Plugins folder if there isn’t one already
  2. Unzip the zip you downloaded for the  UXTools
    1. Navigate to the Plugins folder in the unzipped files. It will be something like >> MixedReality-UXTools-Unreal-public-0.11.x >> UXToolsGame >> Plugins >>
    2. Copy the folders named UXTools  to your project Plugins folder

     

And then the final thing is you need to go into Unreal and enable your plugins. In UE to Edit >> Plugins to bring up the Plugins Editor.

Scroll down to HoloLens and check Enabled

 

Next, ensure All is selected in the top left, and in the top right grey search box, type mixed to bring up all the Mixed Reality Plugins. Click the Enabled checkbox in the Microsoft Mixed Reality Plugins. You will be prompted to restart so go ahead and press the Restart Now button in the bottom right corner.

When you restart Unreal will build the UXTools into the project

 

 

When Unreal restarts the UXTools plugin will be in your Content Browser. If you can’t see them there (and I couldn’t when I first set up my file), you can display them by adjusting the Content Browser View Options. Select >> View Options >> Show Plugin Content. You can also Show C++ Classes, but for now turn everything else off otherwise you’ll have a tonne of folders to sift through.

 

<<<< STEP 2 – FILE SET UP         STEP 4 – CREATE A LEVEL >>>>

 

YOU have everything downloaded, now what?

Launch the Unreal Engine 4.26 from the Epic Games Launcher.

 

In the Select or Create New Project Window, select Games in the New Project Category section and then hit Next

 

 

Next select a Blank Template and then hit Next

 

 

And in the Project Settings Select C++ Project, Scalable 2D or 3D and Mobile/Tablet

Choose a name and where you want to save it.

Then hit Create Project

 

 

<<<< Step 1 SET UP        STEP 3 – ADDING PLUGINS >>>>

I am following the microsoft Chess app tutorial as I do this – adding in screen grabs for extra information for proper nubies, so for the original instructions go to microsoft
https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unreal/tutorials/unreal-uxt-ch1

 

Before starting

 

have the following PROGRAMS installed.

HAVE THE FOLLOWING PLUGINS READY TO INSTALL

Download the following plugins so that you have them ready to copy to your Plugins folder once you have created your project. I’ll remind you later, but to find instructions on how to install plugins, see the Blog post titled Adding Plugins To Your Unreal Project

HAVE THE HOLOLENS 2 TARGET PLATFORM INSTALLED

To do this,

>> go to the Epic Games Launcher

>> click on the Library button at the top of the page

>>click on the arrow button next to launch on the 4.26.1 icon

>> go to options

 

>> scroll down until you get to the Target Platform section and ensure that Hololens 2 is ticked.

>> if it isn’t ticked, tick it and hit the apply button. The download will start for you.

STEP 2 – FILE SET UP >>>>

Set up your HoloLens to use the Windows Device Portal

 

Before accessing the Windows Device Portal, you need to set it up on HoloLens:
1. Put on your HoloLens device and turn on the power.
2. Do the bloom gesture to open the Start menu.
3. Go into Settings
4. Select the Update & Security menu item.
5. Select the For developers menu item.
6. Turn on Developer Mode.
7. Scroll down and turn on Enable Device Portal.

 

Using the Windows Device Portal - Mixed Reality | Microsoft Docs

 

Connect with Wi-Fi

  1. Connect your HoloLens to Wi-Fi.
  2. Look up your device’s IP address. – To find the IP address on the device, go to Settings > Network & Internet > Wi-Fi > Advanced Options. The IP address is the set of numbers given for IPv4 Address. It will be something like 192.168.1.147
  3. From a web browser on your PC, go to https://<YOUR_HOLOLENS_IP_ADDRESS>

You will receive a warning that Your connection isn’t private

 

Ignore it and hit the Advanced button

 

You’ll receive more warnings, ignore them and hit the continue to xxx.xx.xx.xxx (unsafe) link

 

 

You’ll be prompted to sign in. Use the Username (email) and password you signed into the device with, and then press the blue sign in button.

 

 

This will now load the Windows Device Portal to your Hololens.

 

 

Click on Views and Apps to load the app page.

 

 

Then click on the Choose File button to select your app to install

 

 

Navigate to where you have saved the Visual Studio Master Folder, select the YourAppNameMaster.apx file and hit open

 

 

It will start to upload to the Hololens

 

 

It will display  the message Package Successfully Registered when complete. Hit the Done button.

 

 

The App is now installed and ready to play on the Hololens 2. On  the Hololens Headset go into apps and select your app to load it.

 

 

 

 

NowSo I needed to create a Hololens 2 App and deploy it to an actual Hololens 2 that I had with me for development, AND I needed to save a copy of the compiled app that could be copied onto a Hololens 2 that my client had in another office. It was a very convoluted process so I thought I would document it now so I wouldn’t forget it for next time round.

 

Before you start make sure you have the following

Unity 2019.4.4f1 

MRTK 2.4.0 

Visual Studio 2019

 

The Best Tutorial on Set Up

I could go through and tell you how to create a Unity file that will run on the Hololens2, but really, just follow this one that Microsoft put out. It is the best.

https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/tutorials/mr-learning-base-01

Just go through that page and you have set up your basic Unity MRTK file that compiles to the Hololens 2 through Visual Studio.

 

HOW TO CREATE A FILE TO SEND TO SOMEONE TO INSTALL

This is where is gets fiddly and you need to set up certificates and stuff. So…. let’s get started

 

In Unity, open your Player Settings File > Build Settings > Player Settings.

In the Player Settings Window Expand the Publishing Settings

Scroll down to  the Certificate section and click the Create… button

 

 

A pop up box will appear and ask you to fill in a Publisher (I usually put in either my client name or my Sole trader name), and a password. Do this and then click Create

 

 

Now go back to Visual Studio and in the Solution Explorer window expand your app and locate the Package.appxmanifest file. Click on it to open it.

 

 

Click on the Packaging heading and then click on the Choose Certificate button

 

 

Click on the Select from file… button

 

 

Navigate to  the Project Assets folder (make sure it’s not the Builds Asset folder!) and select the WSATestCertificate.pfx file

 

 

Now remember in Unity we gave the certificate a password, you’ll need to enter that now and press OK

 

Now click on View Full Certificate

 

 

And click on Install Certificate

 

Choose the Current User radio button and click Next

 

 

Choose the Place all certificates in the following store radio button and click Browse

 

 

Choose the Trusted Root Certification Authorities and click OK

 

 

Click Next

 

 

Click Finish

 

 

You’ll get a Security Warning. Click Yes

 

 

Then you’ll get a message that the Import Was Successful. Hit OK

 

 

And click OK

 

 

Now right click on your app in the Solution Explorer Your App > Publish > Create App Package

 

 

Select Distribution Method. Choose the sideloading radio button and click Next

 

 

Make sure the radio button Yes use the current certificate is selected and click Next

 

 

Hololens2 needs ARM64 so make sure that checkbox is selected also and hit Create

 

 

Navigate to the folder in Windows Explorer that you gave as your file path to save the packages, and you can send these to your tester to save onto their Hololens 2.

 

 

I have another blog post that shows you how to copy it onto your Hololens 2 here