Steamroll’s 3D User Interface

Steamroll’s 3D User Interface

 

Overview

 

Games normally display information such as score and inventories and interact with the player using traditional 2D UIs, where all the information is displayed on top of the 3D world using traditional 2D widgets painted with tools such as Photoshop or the Gimp.

This document shows how Steamroll renders 3D objects on the player’s HUD to make an animated and interactive 3D User Interface (UI) using Unreal Engine 4.

Introduction

 

Steamroll is a 3D Puzzle and Strategy game set in an industrial steampunk universe where the player assembles and shoots steamballs to solve problems and overcome obstacles to get to the next level. A steamball can be fitted with up to 4 items including walls, ramps and bombs which will be deployed according to how the player programmed them to. A good example would be a steamball containing a wall deployed two seconds after launch, a ramp attached to the first thing the steamball bounced off and a bomb activated when the ball comes to a halt.

In addition of the usual health and ammo (balls) displays, the game needs to show the user a fair amount of information regarding the ball’s contents and programming, the inventory of available items and the remaining steam pressure.

Early on the development of Steamroll we realized that it would be a challenge to present all this information in the simplest, most intuitive possible way so we decided to take advantage of the game’s steampunk industrial atmosphere and use geared moving machines to show and hide everything of interest to the player.

This decision forced us to go 3D, since 2D animation would severely restrict us to simple panning, rotating and zooming of 2D widgets due to the huge amount of combinations in the assembly of a steamball. We’re a small team and do not have the resources to produce an enormous amount of animations whereas some skeletal meshes and a bit of coding can produce stunning results in relatively short time.

Method description

 

This task is not as easy as setting up some actors attached in front of the camera since the 3D environment around these actors would affect them and break the illusion and even its usefulness.

For instance, when the player gets too close to other objects, they might poke through the UI and occlude the player’s view. Lighting is also problematic because the level’s own can and will bleed onto the 3D UI which is something the artists will complain about since they cannot control it. The opposite is also true, the UI’s lighting can also affect the level’s often resulting in undesirable artifacts. Look at how the UI is partially covered and strangely lit in the following screenshot.

Unreal Engine 4’s UMG has an experimental widget called Viewport that can render a scene independently of the main level, but after testing it for several versions and seeing it didn’t really work well enough for us and didn’t reach full stability we decided to look for an alternative way.

We use the old-fashioned chroma keying technique, where the objects of interest are rendered in front of a green background and then the captured scene is composited with the final background by removing the original shade of green.

Isolation and Instancing

 

The 3D UI actors and complementary setup have all to be grouped and then instanced in each of the game’s levels. That’s why we create a new level called “UI_Sublevel” for it and load it as a sublevel in each of the game’s real gameplay levels.

Just remember to place everything really far away from the origin so that when they are included in a real level level they don’t end up appearing in the middle of the action!

It is also important to remember to change the sublevel’s streaming method to “Always loaded” in the level’s tab, otherwise you won’t see anything.

Scene Capture to Render Target

 

Here is where we are going to capture whatever is in the 3D UI with a special type of camera called SceneCapture2D and render everything onto a RenderTarget which is really a texture that we will later use. Create the SceneCapture2D in front of the chroma background.

Create a Texture Render Target in the content browser:

Edit the new Render Target and set its desired on-screen size and addressing:

Edit the scene capture 2D details and set the FOV, the texture target to the one just created and the capture source to Final Color. Make sure the mark the “Capture every frame” checkbox.

It is also important to uncheck the Sky Lighting checkbox to prevent the skylight from affecting the UI. Take a few minutes to uncheck every rendering feature you are sure that you do not need for a little boost in performance.

Post-process

 

Now that the Scene Capture and the Render target are linked, we need to ensure that the post-process the Scene Capture is going to have doesn’t break the chroma keying. Unreal’s post-process includes a tone-mapping stage that subtly changes the render’s final color, including the keying color thus breaking the system. Fortunately, we can replace the default post-process with a custom one that essentially does nothing!

It is possible to defeat the complete post-process and tone-mapping pipeline in the scene capture details by selecting “Scene Color” instead of “Final Color” in the “Capture Source” property. It is certainly easier than what we are doing but it has the disadvantage of not being able to have any post-process effect such as bloom which artists tend to like.

Create a new material and change its Material Domain property to “Post Process”:

It is also necessary to specify that this post-process material replaces the Tonemapper:

Then set up the material. We need “Input 0” which is the scene’s final color plus “Input 1” to add post-process effects such as bloom. Wire up the addition to the emissive color.

 

Chroma Material

 

This is the key step of the whole process. The green background has to be “erased” so that the level can be seen behind the 3D UI. Fortunately Unreal’s incredibly powerful and flexible materials make this task a piece of cake.

Just create a new material which we called OverlayMaterial and set the following properties to make the material translucent and unlit:

And arrange the following nodes:

The UTexture node on the left is linked to the Render Target. If you have already played the level with everything set up as explained in the previous sections you should already see the UI over the chroma in this node.

The RGB color is passed directly to the material’s final emissive color, but the chroma background will be cut out using the material’s opacity or “alpha” channel. The chroma keying color is simply subtracted from the Scene Capture and the dot product with itself is computed to obtain a mask or stencil to use as alpha channel. The multiply node at the end is used to tweak the border.

UMG (Unreal Motion Graphics UI Designer)

 

Up to this point we have set up the UI scene in a sublevel located in a remote part of the level. Then we’ve captured it with a Scene Capture actor to a Texture Render target. The missing part is how to draw this texture to the screen.

Although we could draw it directly to the HUD mapped over a quad, we are going to use UMG because it gives us the ability to specify a resolution-independent adaptive layout in a simple and visual way.

First of all let’s create a new UMG Widget in the content browser:

In the newly created Widget Blueprint, drag and drop an image widget from the palette tab to the hierarchy tab to create an image. Here we call it ImageOverlay:

This image widget has to be sized and positioned on the viewport. The sizes have to be exactly equal to the ones of the Render Target so that there is no scaling. It is also a good idea to disable DPI scaling in the project’s settings setting all resolutions to scale 1.  Notice the anchors on the bottom-right corner of the viewport:

Now it’s time to finally link it to the texture target. The image’s brush can be linked to the chroma Overlay Material we created earlier:

Final set-up

 

The last step remaining is actually adding the UMG widget to the viewport. It would be impractical if we had to manually add blueprints in every level to create and add the UI. That’s why it is done in the UI_Sublevel blueprint. Open it by clicking on the gamepad icon next to UI_Sublevel in the levels tab:

And add the following nodes to the blueprint:

Notice the widget we are creating here is the UMG Widget the made a few sections ago and you’re all set! Hit play and check everything looks as it should.

Shortcomings

 

In its current state Steamroll’s 3D UI has a few shortcomings that are not critical but would be nice to improve:

-The tone-mapper has to be disabled in the SceneCapture to make chroma keying work. This has the unintentional side-effect of disabling anti-aliasing too because it is completely integrated in the post-process and tone-mapping pipeline. It makes the UI look a bit jagged around the edges.

-DPI scaling has to be disabled so that the UMG Image widget is rendered at exactly the same resolution as the Render Target. It would be nice to have DPI scaling for resolution-independence, but whenever it resizes the UI it partially breaks the chroma keying around the edges making the UI look outlined in chroma green.

-Having a UMG widget all over the viewport intercepts mouse click events, and click events to the 3D UI have to be intercepted, converted from screen space to world space, converted to UI_Sublevel space and then recast with line tracing.

Any doubts? Suggestions? Improvements? Please leave a comment below!

8 replies
  1. Vilarstudio
    Vilarstudio says:

    how do you place the chroma on the level ? i can only get the image from the camera to the “rendertarget 2d” ( im not used to speak/write english )
    Thnks for the tutorial seems to be the only one on the internet showing this 🙂

    Reply
  2. alexei
    alexei says:

    Hi Vilarstudio,

    I assume you mean how do you place the chroma scene in the viewport.If not, could you please explain where you are stuck.

    If you already have the UI scene in your rendertarget2D, you just have to use it in your ChromaMaterial, and use this material in a UMG Widget that you have to place in the viewport.

    Reply
  3. Vilarstudio
    Vilarstudio says:

    Hi Alexei¡

    Yes , our problem its placing the chroma in the viewport as you said , we put a widget in the scenecapture 2d blueprint but we cant see it in the viewport like this: ” http://anticto.com/wp-content/uploads/2015/06/image20.png ” .

    Also we tried to place a cube mesh with chroma material (0,1,0, witth Material Domain: Light Function) in front of the camera ( inside the scenecapture 2d blueprint ) but it is affected by the global ilumination so the result its a translucid effect.

    Thanks for you atention.

    Reply
    • alexei
      alexei says:

      Hola Vilarstudio,

      El Widget UMG no se pone en el Blueprint de SceneCapture2D. Se pone en el subnivel de UI. El SceneCapture2D simplemente se crea en el nivel y se editan unas cuantas propiedades.

      Resumiendo:

      -Tienes un subnivel con toda la UI, los objetos que deben salir en el HUD, el fondo de chroma i la SceneCapture2D camera, que o bien tiene la “capture source” cambiada a “Scene Color” o tiene el tonemapper sustituido por un nuevo como el descrito en el artículo para que no te salga el efecto translúcido.

      -Tienes un widget de UMG que simplemente pinta una imagen con el material de chroma que coge la captura de imagen y le resta el fondo chroma.

      Reply
  4. Taurus
    Taurus says:

    I was missing the step where you linked the Post Process Material to the Scene Capture 2D. I figured it out eventually, but just if anyone else is struggling:
    On the SceneCapture2D, in category ‘Post Process Volume’, open Post Process Settings -> Blendables. Add an element to the array and under ‘Choose’ select ‘Asset Reference’. Then click None and select the OverlayPostProcess material. The SceneCapture2D should now be using the post process material.

    Thank you for the nice tutorial! It helped me a lot getting that whole thing set up.

    Reply
    • Jordi Rovira i Bonet
      Jordi Rovira i Bonet says:

      Hi. We didn’t test it with mobile platforms, but at the time of writing of that post we believe it wouldn’t have worked very well. However, since then Unreal has improved and has added functionality that makes this kind of user interface much easier to create. For instance, there is no need to use the green chroma anymore, since it is possible to access the depth buffer of the User Interface rendertarget and use that to make transparency.

      Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply to Jordi Rovira i Bonet Cancel reply

Your email address will not be published. Required fields are marked *