Pixel Perfection: The Magic of Pixel Streaming in Unreal Engine
People typically experience your Unreal Engine application on the same device that runs the gameplay logic and renders the game world to the screen, regardless of whether you build for a desktop platform, mobile OS, or console. Multiplayer networked games may distribute parts of the gameplay logic among multiple instances of the application, but each instance still does the work of rendering the game world locally for its player. Even when you use the HTML5 deployment option to create a version of your Project that can run inside a Web browser, the game logic and rendering still happens locally within each user’s Web browser.
With Pixel Streaming, however, you run your Unreal Engine application remotely, on a computer that users probably never see. For example, this could be a physical desktop somewhere inside your organization or a virtual machine provided by a cloud hosting service. The Unreal Engine uses the resources available to that computer — CPU, GPU, memory, and so on — to run the game logic and render every frame. It continuously encodes this rendered output into a media stream, which passes through a lightweight stack of Web services. Users can then view that broadcast stream in standard Web browsers running on other computers and mobile devices.
The result for the user is just like watching a video stream from a service like YouTube or Netflix, except for two things:
- Instead of playing back a pre-recorded video clip, the stream is playing back the rendered frames and audio generated by the Unreal Engine in real-time.
- Users can control the experience from their browsers, sending keyboard, mouse, and touch events, plus custom events emitted from the player Web page, back to the Unreal Engine.
Benefits of Pixel Streaming
Using the Pixel Streaming system offers several possible benefits:
- It makes mobile devices and lightweight Web browsers capable of displaying better quality graphics than otherwise possible. They can show complex scenes at high resolutions, using rendering features that are only possible when rendering in a native desktop application with a powerful GPU.
- The user doesn’t need to download large executables or content files in advance and doesn’t need to install anything. The only thing the user needs to download is the media stream as it plays.
- You can support multiple platforms without creating and distributing multiple separate packages. You package your application once for Windows, and people can use any platform to experience your content. Users can view the stream in any modern browser that supports the WebRTC connection model, which includes Google Chrome and Mozilla Firefox on desktop, iOS, and Android platforms. (See the Pixel Streaming Reference.)
- The Pixel Streaming system contains a minimal number of components that are relatively easy for anyone to set up inside a local network. However, it’s powerful enough for teams with experience in deploying web services to use as a basis for creating custom cloud-hosted platforms.
- Pixel Streaming uses the WebRTC peer-to-peer communication framework for the lowest possible latency between the user and the Unreal Engine application.
Pixel Streaming architecture
The following image summarizes the components of a simple Pixel Streaming setup:
Components
- Pixel Streaming Plugin — This Plugin runs inside the Unreal Engine. It encodes the final results of every rendered frame using H.264 video compression, packs those video frames along with the game audio into a media stream, and sends that stream to one or more connected browsers over direct peer-to-peer connections.
- Signaling and Web Server — The Signaling and Web Server is responsible for negotiating connections between browsers and the Pixel Streaming Plugin and for providing browsers with the HTML and JavaScript environment that plays back the media stream.
Connection Process
- When you start up all the Pixel Streaming components, the Pixel Streaming Plugin running inside the Unreal Engine establishes a connection to the Signaling and Web Server.
- A client connects to the signaling server, which serves it an HTML page that contains a player widget and control code written in JavaScript.
- When the user starts the stream, the signaling server negotiates the establishment of a direct connection between the client browser and the Unreal Engine application.
In order for this connection to work, the browser and the Unreal Engine application need to know each other’s IP address. If both are running on the same network, they are typically visible to each other directly at their own IP addresses. However, network address translation (NAT) services running between the two endpoints may change the externally visible IP address of either party. Resolving this usually involves the use of a STUN or TURN server, which tells each component what its externally visible IP address is. For details, see the Hosting and Networking Guide.
4. As soon as the connection between the client and the Unreal Engine application is established, the Pixel Streaming Plugin starts streaming media directly to the browser. Input from the client is sent by the player page’s JavaScript environment directly back to the Unreal Engine application.
5. The Signaling and Web Server maintains its connection to the browser and the Unreal Engine application even after the media stream starts playing so that it can kick users out of the stream if needed, and handle disconnections initiated by the browser.
For more information on setting up the architecture shown above, you can read the following documentation.
Implementing PixelStreaming in your project
Installing Required Plugins
The Unreal Engine Marketplace has several plugins that are useful for creating metaverse content. You can find them here. The following plugins are required for the Unreal Metaverse project. You can add them from the places given below:
- Unreal Editor Plugins Manager
- Pixel Streaming
- Pixel Streaming Player
Enable all the plugins in the Unreal Editor Plugins Manager. You can find it in the Edit menu of the Unreal Editor. You can also enable them in the Plugins Settings in Unreal Editor.
Building the Project
- Open your Unreal Engine 5 project.
- In the main menu, click on “File.”
- Select “Package Project” from the dropdown menu. This will open the packaging settings.
- Configure the packaging settings according to your project’s requirements. This includes specifying the target platform (e.g., Windows, Android, iOS), packaging options, and other settings.
- Once you’ve configured the packaging settings, click the “Package” button. Unreal Engine 5 will begin the packaging process.
- Wait for the packaging process to complete. This may take some time depending on the complexity of your project and the selected platform.
- Once packaging is finished, you’ll find the packaged project files in the specified output directory.
These steps should help you package your Unreal Engine 5 project for distribution or testing on your chosen platform.
Steps to deploy your UE5 application using Pixel Streaming
- Use the UE5 template VM in GCP with the below configurations (Available in the GCP Marketplace)
- VM minimum configurations:
- Deployment zone: Anything
- Machine type: GPU
- GPU type: NVIDIA Tesla P4
- Number of GPU: Minimum 2
- Enable Virtual Workstation Grid
- Machine size: n1-standard-8 (8vcpus , 30GB RAM)
- Boot disk — Standard Persistent Disk
- Boot disk size — 250 GB (Minimum)
3. Set the login password and download the RDP file.
4. Connect via RDP and make sure the local microphone is forwarded.
5. Enable sound service in VM in the sound manager.
6. Enable the microphone privacy setting to allow microphone recording.
7. Go to the server manager and install a few dependencies to enable microphone forwarding on the server side.
Follow this tutorial if you need any clarification.
8. Upload your packaged game folder to the VM.
9. Navigate to the SignallingWebServer folder inside [GAME_NAME]/Samples/PixelStreaming.
10. [OPTIONAL] Create certificates to server your game over https using OpenSSL.
- Install OpenSSL using the link — https://slproweb.com/download/Win64OpenSSL_Light-3_1_0.exe
- Create a folder named certificates inside the SignallingWebServer folder and run the following commands inside it to generate the certificate and key.
openssl genrsa -out client-key.pem
openssl req -new -key client-key.pem -out csr.pem
openssl x509 -req -days 9999 -in csr.pem -signkey client-key.pem -out client-cert.pem
11. Go to the SignallingWebServer folder and change the following configurations:
{
…
“UseMatchmaker”: true,
“UseHTTPS”: true,
…
“MatchmakerAddress”: “”,
“MatchmakerPort”: 9999,
…
“HttpPort”: 81,
“HttpsPort”: 444,
“StreamerPort”: 8888,
“SFUPort”: 8889,
…
}
12. Copy the certificates folder to the Matchmaker folder
13. Edit config.json inside the Matchmaker folder to the following config:
{
“HttpPort”: 90,
“UseHTTPS”: true,
“MatchmakerPort”: 9999,
“LogToFile”: true
}
14. Install the dependencies for running Matchmaker by running platform_scripts/cmd/setup.bat.
15. Run the Matchmaker by running run.bat.
16. Install the dependencies for running Signalling server by running SignallingWebServer/platform_scripts/cmd/setup.bat.
17. Start the SignallingWebServer by running Start_SignallingServer.ps1.
18. Create a shortcut for your packaged game’s exe file
19. Right-click the shortcut and append the startup command to -AudioMixer -PixelStreamingIP=localhost -PixelStreamingPort=8888.
20. Double-click the shortcut and choose run anyway to make sure it runs.
21. [OPTIONAL] Run multiple instances of the UE5 app in the same VM
- Create another shortcut of the packaged game’s exe file and append the startup command to -AudioMixer -PixelStreamingIP=localhost -PixelStreamingPort=8890.
- Start the SignallingWebServer as shown below but change the following part of the config:
{
…
“HttpPort”: 82,
“HttpsPort”: 445,
“StreamerPort”: 8890,
“SFUPort”: 8891,
…
}
Follow the previous step as many times as you wish by changing the port configuration and startup command.
Multi VM
- Get the packaged game folder on the VM.
2. Redo all the steps 1–10 for the new VM.
3. Now starting step 11, go to the SignallingWebServer folder and change the following configurations:
{
…
“MatchmakerAddress”: “Public IP of VM running Matchmaker”,
…
}
4. Continue following the steps from step 16 of the previous instructions. We need only one instance of Matchmaker running and it's already set up in your first VM. You simply redirect the SignallingWebServers to the IP of the running Matchmaker.
Conclusion
Pixel Streaming in Unreal Engine revolutionizes the way we experience and interact with digital environments. By offloading the heavy rendering workload to powerful remote servers, it enables even the most complex Unreal Engine applications, including sophisticated metaverse projects, to be accessible on virtually any device with a web browser. This technology not only democratizes high-quality graphical experiences, making them available without the need for high-end hardware, but also simplifies cross-platform compatibility and deployment. With its streamlined architecture and interactive capabilities, Pixel Streaming is poised to become a game-changer in the realm of virtual experiences, offering unprecedented accessibility and quality to users worldwide.
