AeroCEO Net Bridge

While AeroCEO is primarily a single player game, I wanted to support multiplayer. It is more difficult to add multiplayer later in development, so I've been making sure it's integrated well from (almost) the start.

The first thing to do was chaning the game rules slightly to be asyncronous turn based. Instead of rotating through players taking one turn at a time, all turns happen in parallel, which makes the game run faster. In the end this does not change too much, but some things like multiple players bidding for a limited number of slots will have to resolve differently.

The second part to make multiplayer easier is that the game is always running a multiplayer game. So when running in single player, the game will still internally run a server that the client will connect to. I had thought about using a real loopback network connection for this, but that might not work on all platforms. Ie: Can you open a listen socket on Switch or iOS? And it would also be slightly slower having to throw all actions across a socket.

The alternative is what i've called a "bridge", basically it's a pretend socket. When playing in offline single player or online multiplayer, the players client will do all of its communication over the bridge. In true offline single player, the bridge will instantly pass data to the internal server via a function call. In multiplayer, the bridge will pass the data over the internet via GameStrut to be processed on a remote server.

To be a multilpayer game, we need ways to keep everyones game and the server in sync. We do that with two parts, serializable game state, and actions. Both of these can be packed or unpacked into blobs, which are a basic building block of GameStrut, like little virtual files that we can read and write to.

The game state includes the world, airports, companies, routes etc. So for AeroCEO, when a player joins a game, the server can pack the whole world stae into the blob and send it to the joining client, that can unpack it and have their game state match whatever is on the server.

Actions then make changes to this data, eg: purchsing planes, bidding on slots and creating routes. When a player makes an action, the client applies it locally, then sends the action to the server to be applied there and forwwaded to all the other players so everyone is up to date.

There is one little complication to that tho, we want to keep players actions secret from each other till the end of turn.
So for this the server will actually hold all queued actions and only finally applying and sending to others at the end of the turn.

Also, when sending the game state to a player, we only send their queued actions, not everyones. A neat feature of keeping the queued actions and applying at the end of turn, is that it's trivial to add undo functionality, we just drop the last queued action for a player.

C# On Linux ARM

Here's my MNT Reform. It's a laptop with a mechanical keyboard, and while it is physically robust, it's not a high performance monster. So while the performance is adequate, it cannot handle all the bloat of modern software.

I hope to turn this little laptop into a distraction free environment to do a lot of my development of GameStrut. It's still in the very early stages, but I know at the very least it will involve C# to support Unity3d. So I've been experimenting on how to build and debug C# on Linux ARM.

Runtimes

The two ways of running compiled code that I've looked into is dotnet and mono.

.NET Core aka dotnet is a successor to the original .NET Framework. It adds cross platform support, that the original was lacking, but also removes some things that were available in the previous incarnation. The things it cuts out are not a big deal for me as I'll be writing everything from scratch anyway, so I have no legacy dependencies or requirements, and my code is usually pretty vanilla so it will run anywhere.

I had some trouble installing dotnet via the Microsoft apt package server on my Reform as it is using Debian 12, and perhaps also because it was ARM. I was able to install easily using the dotnet-install.sh script and manually adding the ~/.dotnet directory to the PATH.

The other option is Mono. It has been around for a long time, initially developed as a open source Linux runtime for the .NET Framework. I'll probably lean towards Mono as I have used it previously for Puzzle Quest 3 and had no problems there. Also it is most similar to what Unity3d uses internally, so it would be good to stay somewhat in sync there as well.

While mono is available in the Debian apt repository, it's a bit old. So it's best to follow the install instructions on mono-project.com to get the latest version.

Building

C# code is usually built based on a project file generated by Visual Studio. It's just a simple XML file and there's enough info on the net if I need to modify or build it from scratch.

The msbuild command works the same was its Windows counterpart, building a project and outputting the results with little fuss. It has lots of options and is reliable, but it is a little slower than I'd like.

csc is the offical Microsoft C# compiler (and probably what msbuild uses internally). It does not know anything about projects, so you must supply all the files to be compiled as command line arguments, similar to something like gcc.

mcs is the Mono version of the C# compiler. As far as I can tell it's basically the same as the official counterpart, but a little faster.

Ive ran a very unscientific speed test on the ways of building a project:

time msbuild
real 0m12.497s

time csc -recurse:\*.cs -out:out.exe
real 0m5.476s

time mcs -recurse:\*.cs -out:out.exe
real 0m2.575s

This was a smallish project, but it did have about 70 tiny source files. Mono mcs is the clear winner here, but it remains to be seen if these differences will matter too much in a large project.

IDEs

While I can use a text editor and build and run with the command line. It would be nice to have a development environment to help.

There's a couple of features I'm specifically looking for:

• Omnibox - This is a feature in newer IDEs, similar to web browsers where there is a textbox that can accept many different commands. Tho primarily i'm interested in being able to jump to files by name, to be able to quickly move about the project.

• InteliSense - this is where the IDE scans your project and is able to use that information to make your life better. Things like auto complete for variable and function names, and being able to jump directly to where things are defined.

• DeDebugging - This is the ability to be able to set breakpoints directly in the IDE and being able to inspect a running app, things like being able to select a variable in the source file and seeing what value it is. This can be a lot easier than setting up a bunch of print statements then running the program to spit them out.

Visual Studio Code

Fully featured, but on the slow side. I think it will unfortunately end up being just too slow to keep using for the long run, especially as the project and VSC itself get bigger over the years and start demanding more resources.

✓ Omnibox
✓ InteliSense
✓ Debugging - Was having trouble with exceptions not being able to retrieve their info, but that's probably a weird config issue that could be fixed if i was committed to using VSC long term.

Monodevelop

A faster GUI than VS Code, but a teeny bit on the slow side. Thinks like opening a project or starting a build can stall for a little bit. I would probably use this for some deep debugging alongside a different editor.

x Omnibox
✓ InteliSense
✓ Debugging

Sublime Text

A fast enough GUI, it could be better with things like startup time, but I can live with it. I'm using it right now to write this blog post!
The problem is I was unable to get the C# plugin to actually work :(.

✓ Omnibox
x InteliSense - The Sublime Text 3 plugin for C# is abandoned and no longer working.
x Debugging - Even with the plugin I don't think this was supported.
x Jump to error

Geany

A very speedy editor that unfortunately does not have a lot of C# support.

✓ Omnibox - While it does not have an actual omnibox, Geany does have several plugins to jump to files by name. I choose Quick Open.
x InteliSense - Not available but it does have some typing completion from class names in the project.
x No debug

Vim

I have used vim with omnisharp in the past, but it felt a bit too clunky to me once several disparate packages were added together to create the functionality.

And it felt like a long process of tweaking and learning to vim better was ahead of me...

✓ Omnibox - Jump to file can be added by the "ctrl+p" plugin.
✓ Intelisense - There is plugin support for all the expected OmniSharp functionality
x Debugging - I have not had this setup, I assume it could, but may be a lot of work

Conclusion

At this point I'm not sure which IDE or combinations of IDEs I'll start out with. Definitely towards the faster end. The last worst/best option is for me to write my own IDE with the minimal feature set I want. But that's a very dumb idea. Stay tuned for my next post!

Dynamic Asset Downloading in Real Racing 2

In the beginning...

Starting with Real Racing 2 we were looking to add a special GUI for managing the players account and save games. We also intended (and did!) use this UI across other games, so it is built as HTML and JavaScript instead of C++ and OpenGL like the games. It would have been difficult to insert ourselves into the input and rendering pipeline, while also supporting other languages so it was a lot easier just to pop up a web browser and do everything there.

We also supported making server requests from this embedded JavaScript, they went through the same system as normal game requests so had no real limit on what could be done there. This allowed us to create a reusable interface for core functionality (account management, save game backup and restore) that can be themed to each game simply. And allowing for extra functionality to be added per game if needed, for example, Real Racing, Spy Mouse and Flight Control Rocket implemented their own leaderboards using this tech.

Special thanks to Touch Arcade for this screenshot. Since the servers are no longer online, the leaderboards are now lost to the ages.

All the assets for this UI (HTML, JavaScript, images) are shipped with the game and loaded off disk to reduce load times. During development of this feature, the edit and test loop was quite painful, we would have to edit the HTML, rebuild the game in XCode, redeploy the whole game to the device and launch it before we could see our changes.

So in an attempt to save time and restore my sanity, I developed a system to automatically refresh these files if the change, when launching the game or multitasking. This chopped a lot of steps from the process and saved a lot of time. I could now just edit the files, and refresh them quickly on the device.

Eventually to save the bandwidth of downloading every file every time, this was improved so that the device would be able to report what files it has, and the server can send back any updates needed.

The game itself also had a change, so that when a file is loaded, it would check the asset download folder to see if the file exists there, otherwise it will load the asset that shipped with the game. This simple change now allowed us to do a lot, by being able to substitute any asset or assets in the game, by simply serving new ones from the server.

Flipped BMW Logo

Just as we were about to publicly release the game, in fact after the final submission had been made to the App Store, we found out that the BMW badge on one of the cars in game was the wrong way around. The blue and white checker pattern was flipped, so we would not be able to get approval from BMW to release the game.

Here is the car in question, with a blue arrow pointing to the badge that was flipped. But note that the logo is actually correct here!

This was the first big success of the asset replacement system, as we were able to seamlessly fix the logo, by downloading a new asset when our players first launched the game, well before they even had a chance to see the car for the first time.

The most straightforward way to fix the issue would be to fix the texture that is drawn onto the car, but that was a large file, almost 1mb if I recall correctly. So having every player download it was going to be a lot of data. So we managed to find a more efficient fix by changing the 3D mesh of the car instead, flipping the texture coordinates over to flip the logo. This fix was about a tenth of the size.

Fixing Broken Save Games

A few months after release, some players reported bugs where the game would crash after completing a race. We eventually worked out that there was a fixed array of results that was being appended to after every race and these players had simply ran out of space.

The fix was straightforward enough, just keep the best result for every event, and remove the rest from the array. But this would require us to ship an update and wait for it to pass review, so the affected players would either have wait a week for the fix to arrive before they can continue playing, or delete their saved game and start over from the beginning.

But since we already had the tools to backup and restore a players save game, we had a third option to remotely fix the players save games.

Using asset replacement, we added a new button to the HTML UI that would do three things.

• Using the existing backup functionality, upload the save game to our servers.
• Send a new request to the server to fix the players most recent save. The game devs created a tiny command line program based off the C++ game code that would load a save, strip out the unnecessary results and save that file back to disk. So a new endpoint was written on the server to pull the players most recent save from the database, run the program to correct it, then update the database with the corrected save file.
• Again using existing functionality, the client then restores save game from the server to the client, so now the local save file is fixed!

This fix allowed us to have a fix out within the day for our players without having to rush out a new build of the game that would have been delayed up to a week with QA and App store reviews.

AB Testing

Once the game was up and running for a while we eventually extended it to support AB testing. It was somewhat rudimentary, we'd select a control and experiment group of players as they launched the game for the first time. Then for the experiment group we would send a different set of game assets that would change how the data driven game worked. For example, we could make cars cheaper or give more rewards. Then after some time, we could compare the two groups of users in our analytics database, and see what affect the changes had.