The idea of this post is to analyze in a superficial way the multiplayer solution behind Clash Royale. Before starting the analysis, you must know that I don’t have previous experience making multiplayer games, I am just learning, and all the analysis done here is just a theory.

So, you surely know about Clash Royale but in case you don’t, it is an online multiplayer 1v1 RTS game where the first player to destroy the other’s towers win. To do so, they play cards which transforms into units that advance and attack enemy towers, structures that spawn units or attack enemy units or special powers which can be used to perform damage, among other things. Cards cost energy that regenerates over time (to a maximum). I recommend it since it is a great game, and it is really really polished in every detail (including multiplayer).

Here is a video explaining and showing the game:

Analysis

Since the game is very competitive, they probably have an authoritative server to validate player actions to avoid cheating. For example, a Player could say “I played card X” to the server, the server has to validate the player had that card in hand to use it, and enough energy.

In each game there could be several units at the same time (like 30 units in the worst case), also, there are tons of games being played at the same time. In order to make the game run over mobile networks and to support all those games, with all those units at the same time, they have to reduce the bandwidth to the minimum.

One strategy could be to compress the data sent, other could be to send data not so frequent and to interpolate to be as smooth as possible. However, considering that each unit has a position, looking direction, target, health, animation frame, among other stuff, that could still be a lot of data. I am guessing here that they follow another approach like a synchronized simulation of the game in each client and, since it is not so CPU heavy, every mobile device nowadays (game was released March 2016) shouldn’t have problems running the game logic.

Another thing that made me think about synchronized simulation was that every player action is not performed instantly, it has a small delay or a cast time. That could be a design choice but I believe it considers the fact that actions must be synchronized between players and having a delay allows them to do that.

If they are simulating in the client, they have to control the simulation to be deterministic or they must have some way to fix the game state if it was desynchronized at some point.

If I remember correctly, they follow an approach that the game never stops, players could be disconnected for a while and then reconnect and continue playing, they just lose part of the game (couldn’t perform actions). That could be used also for player desynchronization. Don’t know the resynchronization strategy but maybe the server sends game state snapshot and the player continues from there.

They even have game replays, so I am guessing that simulating the game in each device could help in reducing the cost of watching a replay even though they could replay it in a server if they want since they probably have tons of servers :).

Conclusion

My guess is that the game has a client/server architecture where both the server and the client simulate the game synchronously. The server is in charge of validating player actions and responsible of deciding the real game state in case of desynchronization.

As I said at the beginning of the blog post, this is a superficial analysis based on my current knowledge. If I wanted to perform a deeper analysis I could have follow another approach like doing some reverse engineering over the game connections to validate some of my guesses but that wasn’t the blog post purpose.

And here is a really fun video of the game to finish the post:


In this Gamasutra article, Tundra developers explain their approach to minimize problems when developing a lockstep multiplayer platform used for their game Rapture - World Conquest. Based on that, my first approach was to create and understand a deterministic lockstep logic without even considering networking yet, to focus on one problem at a time.

My objective is to have a reproducible game that given the start state and all the players actions during time, the game can be played again and the final state will be the same.

Lockstep logic

The first step was start by creating a logic to encapsulate the fixed game state for physics and game logic, and a lockstep to perform player actions. As I started without considering networking yet, all the player actions are directly enqueued locally by the user input or by a saved replay.

A lockstep logic means that there are some conditions that, if not met, the game should pause the simulation and wait for them. In the case of multiplayer games this is when the game have to wait for other players actions that didn’t arrive in time (and where the waiting for other players dialog is shown in some games).

My code looks something like this pseudo code:

update(dt) {
  if (lockstepLogic.IsLockstepTurn()) {
    if (!lockstepLogic.IsReady())
      return;
    lockstepLogic.Process();
  }
  // normal accumulator logic for the fixed gamestep
}

Since I create all player actions locally, the lockstep never happens right now but it is a good practice to simulate and test it anyways.

Testing it

My first prototype using this logic is a box that moves over the screen. When the right mouse button is pressed, the game enqueues a player action to move the box to the specified position, then when the lockstep logic is processed the box receives the command and start moving to that position.

Interpolation

The moving logic was pretty simple, based on start and destination positions and a speed, the box moves itself to the destination in a straight line. Since that logic is performed in each fixed gamestep, the player see the box jumping between positions, it works but it doesn’t look so good. To improve that and make the movement smoother I created a simple interpolation code.

Interpolation depends a lot on what you are trying to interpolate, in some cases could be a simple as the code I used there but there are other cases like the bouncing ball which need more data to create a better interpolation.

Note: adding interpolation means we have now a delay of one fixed gamestep between the box position being rendered and the real position in the game since we are using previous and current positions info for the interpolation.

My first replays

By recording all the player actions with the fixed gamestep frame they were executed, I created a basic way to replay the “game” by just reseting the game state to the initial state and start enqueing all the recorded actions in each corresponding frame.

Here is a video of the progress so far:

Yeah, I know, it’s not a great thing, but at least I am starting to understand some of the challenges of making multiplayer games.

Validating simulation

To validate the game is always performing the same simulation given the same input, I added a checksum calculation based on the game state (for now just the moving box state) which is saved from time to time to use later as validation when simulating the game again. The idea was to start defining an API to get the important game state to consider when validating the simulation, and also to start testing game state validation. The code looks like this:

update(frame) {
  if (IsChecksumFrame(frame)) {
    if (recording) {
      checksumRecorder.RecordState(frame, CalculateChecksum());
    } else {
      if (!checksumValidator.IsValid(frame, 
          checksumRecorder.SavedChecksums, CalculateChecksum())) {
        throw Exception("current game state is not valid!");
      }
    }
  }
}

In my first tests I wasn’t reseting the game to the initial state properly so my game was producing different checksums when reproducing the saved player actions, checksums validation started to work after that was fixed.

How am I calculating the Checksum? Right now I am not sure exactly what algorithm to use nor which game state should I consider for the checksum, so what I did was to encapsulate that in some interfaces and implemented a basic way to get both. For the checksum I am using a simple MD5 over a string, and for the game state, well…, a string with all important values concatenated (the moving box position, destination, if it is moving or not, etc).

string CalculateChecksum() {
  string gameStateValue = game.GetGameState();
  return MD5.CalculateHash(gameStateValue);
}

string GetGameState() {
  string gameState = "";
  foreach (object in gameObjects)
      gameState += object.GetGameState();
  return gameState;
}

For a future implementation what I know is that the game state should be composed with important values that can affect the simulation, so I shouldn’t care about about audiovisual stuff like particles, effects and sounds.

Also, the game state concept could be used to reset to the initial game state or even for saved game states (to easily replicate some bug for example), and finally, it could be used to synchronize and validate state if for some reason I end up using a client/server architecture.

Next frontier: Determinism?

For now I didn’t explore determinism realm because the solution really depends on the game logic but at the same time I must have it clear before starting the game code. One of the next steps is probably start testing with fixed point math, not sure yet, the idea is try to follow an approach similar to the gamasutra article’s of reducing non determinism problem to the minimum before going multiplayer.

If you want to take a look all the code used for this blog post, here is the link.

Example of a dynamic lockstep implementation for Unity

Lockstep Framework for Unity

Reddit post about that Lockstep Framework

Another framework named lockstep.io

Developing your own Replay System


Some time ago I’ve started to prototype a multiplayer game, it is some kind of super simplified RTS game for mobile devices where macro decisions are encapsulated into one action through a button.

First versions were prototyped in a multiplayer hot seat fashion by using only one device (a tablet or a phone), that allowed me to quickly iterate between different ideas and find fun as soon as possible. That was a successful approach since the idea of the game proved that it was indeed fun to play with friends, so the next step was to go remote multiplayer with two or more devices.

Since networking is a whole new world for me, I started by reading lots of different articles about making multiplayer games.

The first thing to know is that the common solutions around depend on each kind of game, and knowing that beforehand helps you deciding the best solution for your game, and how your game must be adapted to support it.

In my case, I am developing a really simplified version of a RTS game for mobile, similar to Clash Royale. It is a 1v1 game where each player controls multiple units by giving high level orders like “everyone attack!” or “build new supply depot”.

The common networking approach for this kind of games, where synchronizing the whole world is a bit heavy given there are tons of units, is to do a synchronized simulation where only the main actions are transmitted (to save bandwidth) and each machine/device perform the same simulation. This implies the game should be deterministic in order to avoid desynchronization between players, so given one start state of the game and a list of actions over time, the final state should be always the same.

Making a game deterministic is a really hard problem, but it also has its rewards, like being able to replay the game given the actions of each player over time. This is a great debug tool for developers and at the same time a great freature for players since it is used to see other strategies or to share a great victory with your friends, everything with almost no storage cost.

The idea is to start writing my findings in the remote multiplayer journey to share all the lessons learned.

References

These are some of the articles I read:

What every programmer needs to know about game networking

1500 Archers on a 28.8: Network Programming in Age of Empires and Beyond

Fast-Paced Multiplayer

Why adding multiplayer makes game coding twice as much work

Core network structures for games

Making Fast-Paced Multiplayer Networked Games is Hard

How to create an online multiplayer game with Unity

Networking in Unity

Creating a Cross-Platform Multiplayer Game in Unity

Understanding Fighting Game Networking

Desyncs and FPU synchronization

Cross platform RTS synchronization and floating point indeterminism

Minimizing the Pain of Lockstep Multiplayer

The Tech of Planetary Annihilation: ChronoCam

Synchronous RTS Engines and a Tale of Desyncs

Don’t Store That in a Float

Developing your own Replay System


We want to talk a bit about how we migrated from our custom highscores system to use Google Play Game Services (GPGS) Leaderboards for Vampire Runner and the consequences of that change.

Before GPGS was released, we worked on our custom solution for highscores for Vampire Runner. That solution consisted in having scores for multiple time intervals (daily, weekly, monthly and all times scores) and letting guest users play and make scores without the need to log in. You could play the game, make scores and then, if you thought your scores were good, link those scores with your name, really simple and easy, it worked fine.

That custom solution was running on an amazon server and costed about 15 USD/month and since the game was making around that money with the ads, we thought it was acceptable.

A while after GPGS was released, we thought, why maintain and pay money for our own server when we can use GPGS instead? It was very logic, they have some of the features we had on our system and everybody was starting to use it. Also, we trust Google, so we changed Vampire Runner to use GPGS.

However, some time after we changed Vampire Runner, we started to see an important decrease in the number of people playing the game. The problem was something we were aware of but we thought it wouldn’t be so significant. We had a lot of players that didn’t have a Google account or even Google Play (they probably downloaded the game from other store), and they couldn’t log in to compete with other players so the game was less interesting for them since part of our game is based on that, competing to be better than other players.

The truth is the game is not a good game by itself and the main thing that kept our players was being able to compete with others (for a while at least), now that we are forcing them to log in they feel it isn’t worth it so they just go away or they play without logging in, hence without making scores.

Conclusion

We don’t think Google Play Game Services is the bad guy here but it is not good if you want to capture users from outside Google, same example would apply if we had forced Facebook or Twitter accounts. So if you have a game where you need players to catch other players, consider that. Of course, if you have a great game maybe with only Facebook, Google or Twitter you could keep your game alive but if you consider the three of them, you will get more players. In our case, we had all the world and then we restricted our game to only G+ accounts and that is why we lost players.

That was part of the story of the dying Vampire Runner, hope you like it.


Hi guys, long time no see.

It’s been a while since we came here and write, and we want to share why.

What happened

By the end of 2012, after we finished Clash of the Olympians to Android and iOS working hand in hand with Ironhide Game Studio, we had no ideas to work on. We weren’t really excited in working by ourselves anymore because we didn’t feel we could create something of the quality of Clash. Also, we really enjoyed working with Ironhide’s team. In addition, Ironhide was looking for programmers and they told us they enjoyed working with us as well, so they offered us a job and we couldn’t refuse (and we are still glad we did not).

Working at Ironhide

Two words: Indescribable awesomeness.

We are part of an excellent and diverse team composed by artists, developers, game designers and more. The workplace is comfy and the game development workflow is pretty dynamic. We can participate in a lot of game decisions and that rocks!

Working really close with artists allowed us to validate how much we needed one at Gemserk to define ideas and visualize, in an easy way, how our games should look like. It also works as motivation since you really want to see that art in the game.

Game design was another flaw at Gemserk, and we now see and understand it much better.

We learn new stuff each day and we also collaborate and share our knowledge to make the game and the team better. And at the end of the day, that feels really good. We hope it lasts long.

What about Gemserk

As company, Gemserk is kinda dead. We never reached what we wanted with our creations, and in those terms, we weren’t successful. However, we don’t see Gemserk as a failure either since we accomplished a lot in terms of technical knowledge and relationship with other game developers and tool makers and we are really happy with that.

We have no plans to work on anything Gemserk related for now but we believe we still have some stuff to share.

Thanks for reading our blog and hope we could still write interesting posts in the future.

The Gemserk team.