Games normally display information such as score and inventories and interact with the player using traditional 2D UIs, where all the information is displayed on top of the 3D world using traditional 2D widgets painted with tools such as Photoshop or the Gimp.
This document shows how Steamroll renders 3D objects on the player’s HUD to make an animated and interactive 3D User Interface (UI) using Unreal Engine 4.
Steamroll is a 3D Puzzle and Strategy game set in an industrial steampunk universe where the player assembles and shoots steamballs to solve problems and overcome obstacles to get to the next level. A steamball can be fitted with up to 4 items including walls, ramps and bombs which will be deployed according to how the player programmed them to. A good example would be a steamball containing a wall deployed two seconds after launch, a ramp attached to the first thing the steamball bounced off and a bomb activated when the ball comes to a halt.
In addition of the usual health and ammo (balls) displays, the game needs to show the user a fair amount of information regarding the ball’s contents and programming, the inventory of available items and the remaining steam pressure.
Early on the development of Steamroll we realized that it would be a challenge to present all this information in the simplest, most intuitive possible way so we decided to take advantage of the game’s steampunk industrial atmosphere and use geared moving machines to show and hide everything of interest to the player.
This decision forced us to go 3D, since 2D animation would severely restrict us to simple panning, rotating and zooming of 2D widgets due to the huge amount of combinations in the assembly of a steamball. We’re a small team and do not have the resources to produce an enormous amount of animations whereas some skeletal meshes and a bit of coding can produce stunning results in relatively short time.
This task is not as easy as setting up some actors attached in front of the camera since the 3D environment around these actors would affect them and break the illusion and even its usefulness.
For instance, when the player gets too close to other objects, they might poke through the UI and occlude the player’s view. Lighting is also problematic because the level’s own can and will bleed onto the 3D UI which is something the artists will complain about since they cannot control it. The opposite is also true, the UI’s lighting can also affect the level’s often resulting in undesirable artifacts. Look at how the UI is partially covered and strangely lit in the following screenshot.
Unreal Engine 4’s UMG has an experimental widget called Viewport that can render a scene independently of the main level, but after testing it for several versions and seeing it didn’t really work well enough for us and didn’t reach full stability we decided to look for an alternative way.
We use the old-fashioned chroma keying technique, where the objects of interest are rendered in front of a green background and then the captured scene is composited with the final background by removing the original shade of green.
The 3D UI actors and complementary setup have all to be grouped and then instanced in each of the game’s levels. That’s why we create a new level called “UI_Sublevel” for it and load it as a sublevel in each of the game’s real gameplay levels.
Just remember to place everything really far away from the origin so that when they are included in a real level level they don’t end up appearing in the middle of the action!
It is also important to remember to change the sublevel’s streaming method to “Always loaded” in the level’s tab, otherwise you won’t see anything.
Here is where we are going to capture whatever is in the 3D UI with a special type of camera called SceneCapture2D and render everything onto a RenderTarget which is really a texture that we will later use. Create the SceneCapture2D in front of the chroma background.
Create a Texture Render Target in the content browser:
Edit the new Render Target and set its desired on-screen size and addressing:
Edit the scene capture 2D details and set the FOV, the texture target to the one just created and the capture source to Final Color. Make sure the mark the “Capture every frame” checkbox.
It is also important to uncheck the Sky Lighting checkbox to prevent the skylight from affecting the UI. Take a few minutes to uncheck every rendering feature you are sure that you do not need for a little boost in performance.
Now that the Scene Capture and the Render target are linked, we need to ensure that the post-process the Scene Capture is going to have doesn’t break the chroma keying. Unreal’s post-process includes a tone-mapping stage that subtly changes the render’s final color, including the keying color thus breaking the system. Fortunately, we can replace the default post-process with a custom one that essentially does nothing!
It is possible to defeat the complete post-process and tone-mapping pipeline in the scene capture details by selecting “Scene Color” instead of “Final Color” in the “Capture Source” property. It is certainly easier than what we are doing but it has the disadvantage of not being able to have any post-process effect such as bloom which artists tend to like.
Create a new material and change its Material Domain property to “Post Process”:
It is also necessary to specify that this post-process material replaces the Tonemapper:
Then set up the material. We need “Input 0” which is the scene’s final color plus “Input 1” to add post-process effects such as bloom. Wire up the addition to the emissive color.
This is the key step of the whole process. The green background has to be “erased” so that the level can be seen behind the 3D UI. Fortunately Unreal’s incredibly powerful and flexible materials make this task a piece of cake.
Just create a new material which we called OverlayMaterial and set the following properties to make the material translucent and unlit:
And arrange the following nodes:
The UTexture node on the left is linked to the Render Target. If you have already played the level with everything set up as explained in the previous sections you should already see the UI over the chroma in this node.
The RGB color is passed directly to the material’s final emissive color, but the chroma background will be cut out using the material’s opacity or “alpha” channel. The chroma keying color is simply subtracted from the Scene Capture and the dot product with itself is computed to obtain a mask or stencil to use as alpha channel. The multiply node at the end is used to tweak the border.
Up to this point we have set up the UI scene in a sublevel located in a remote part of the level. Then we’ve captured it with a Scene Capture actor to a Texture Render target. The missing part is how to draw this texture to the screen.
Although we could draw it directly to the HUD mapped over a quad, we are going to use UMG because it gives us the ability to specify a resolution-independent adaptive layout in a simple and visual way.
First of all let’s create a new UMG Widget in the content browser:
In the newly created Widget Blueprint, drag and drop an image widget from the palette tab to the hierarchy tab to create an image. Here we call it ImageOverlay:
This image widget has to be sized and positioned on the viewport. The sizes have to be exactly equal to the ones of the Render Target so that there is no scaling. It is also a good idea to disable DPI scaling in the project’s settings setting all resolutions to scale 1. Notice the anchors on the bottom-right corner of the viewport:
Now it’s time to finally link it to the texture target. The image’s brush can be linked to the chroma Overlay Material we created earlier:
The last step remaining is actually adding the UMG widget to the viewport. It would be impractical if we had to manually add blueprints in every level to create and add the UI. That’s why it is done in the UI_Sublevel blueprint. Open it by clicking on the gamepad icon next to UI_Sublevel in the levels tab:
And add the following nodes to the blueprint:
Notice the widget we are creating here is the UMG Widget the made a few sections ago and you’re all set! Hit play and check everything looks as it should.
In its current state Steamroll’s 3D UI has a few shortcomings that are not critical but would be nice to improve:
-The tone-mapper has to be disabled in the SceneCapture to make chroma keying work. This has the unintentional side-effect of disabling anti-aliasing too because it is completely integrated in the post-process and tone-mapping pipeline. It makes the UI look a bit jagged around the edges.
-DPI scaling has to be disabled so that the UMG Image widget is rendered at exactly the same resolution as the Render Target. It would be nice to have DPI scaling for resolution-independence, but whenever it resizes the UI it partially breaks the chroma keying around the edges making the UI look outlined in chroma green.
-Having a UMG widget all over the viewport intercepts mouse click events, and click events to the 3D UI have to be intercepted, converted from screen space to world space, converted to UI_Sublevel space and then recast with line tracing.
Any doubts? Suggestions? Improvements? Please leave a comment below!