Replication

Introduction to Replication

Computer games have supported multiple players pretty well since their inception. At first, it was merely turn-based on the same PC. Soon, games controlled by multiple players using different input devices on the same computer evolved. Multiplayer games were, naturally, quite popular, especially since game AI at that time was relatively poor. It wasn't long before people figured out how they could play their games over a network.

Thus, network gaming was born. Each player could play using their own computer, had their own screen so the other players couldn't "peek," and there was a certain something about playing against other people who you couldn't see in real life at the time. Initially restricted to null modem cables, things quickly escalated to modems and IPX networks, and eventually to internet gaming through the use of a progam called Kali. (which was later followed by numerous clones)

However, there was an issue that needed to be addressed. Obviously, the actions of players needed to be communicated to the other players in the game. They also had to remain synchronized with the other computers to ensure that everything was happening at the same time on all computers involved, despite lag on the network. This problem was not hard to solve: it was called lock-step.

Lock-step reigned for a long time as the only effective method of keeping players on a network synchronized. Every "tick" on the network, information was passed around until there was a consensus among all computers about where everything was and what it was doing. This didn't take very long and was not particularly noticeable, however it had a big problem: if a person with a fast computer played with a person with a slow computer, the fast computer was always waiting around for the slow computer to keep up. The players on the slower computers thus seemed to "skip" around whenever their position was updated on the network. Obviously, this annoyed the fast computer users. Quake pioneered a technology which was only slightly better - one machine was designated the 'server' and was the absolute authority. All of the other machines just told the server what the player on that end was doing, and the server dictated what happened and told the clients what to render as a result. Eventually, the Unreal engine came around with a thing called "replication."

Replication was a much more advanced method of keeping computers in sync. It was not as intuitive, but allowed computers of varying speeds to play together seamlessly. Replication consisted of updating values on the clients only when they were updated. Every client computer running the game would simulate the game as if nothing had changed until told otherwise. This not only sharply decreased bandwidth, but cut down on skipping for the fast computers/fast connections.

How Replication Works

There are several parts that relate to replication, some of which are more obvious than others. Key concepts are those of relevance and the replication block. Role, RemoteRole, NetMode, simulated functions and replicated functions also come into play.

The Types of Replication

There are three 'types' of replication, though they are all in a way the same:

Actor Replication: This is called whenever an actor is 'created.' It instructs the client to spawn an actor of this class and initialize it with its (clientside) default properties. This is also called when an actor becomes relevant, as it needs to be created on the client. Actors are replicated as soon as the server decides to.
Variable Replication: Called whenever a variable changes, if the replication conditions are also met. The replication conditions are defined in the class script. This instructs the client to change a variable to the new value. Clients can also replicate some variables to the server, in the case of inputs, etc. Variables waiting to be replicated enter a queue and are replicated when bandwidth is availible.
Function Replication: Instructs the client to call a function locally. It can also be used to call a function on the server based on the client's input. See Replicated Function.

Relevance

To cut down on bandwidth usage, only actors that are deemed relevant are replicated. (updated) Often there are actors in the level that cannot interact or be seen by a given client. Any bandwidth used updating these actors is wasted. It is worth noting that actors deemed irrelevant may, on occasion, cease to exist on the client. You must keep this in mind when checking a reference to another object within script on a client if the possibility exists that that actor may be irrelevant. The object and the reference still persists on the server and will be linked back up automatically once the server decides it is relevant again. The Unreal engine uses a set of conditions to determine whether an actor is relevant or not:

If the actor has bAlwaysRelevant set to true, then this actor is always relevant.
ZoneInfo and it's subclasses are always relevant.
Actors with bStatic or bNoDelete set to true are always relevant.
If an actors owner is set to the local player, then that actor is relevant.
If an actor has bOnlyRelevantToOwner set to true, it is only relevant to it's owner. It will not be replicated to any other clients. It still exists on the server, but is not replicated to anyone but the actor's owner.
If an actor is a subclass of Weapon and it is owned by an actor who is visible, then it is relevant.
If the actor has bHidden set to true, bBlockPlayers is set to false, and AmbientSound is set to None, then the actor is not relevant. (barring the above conditions, which will override this.)
If the actor is visible on the clients screen, then it is relevant. In some cases this can be inaccurate for actors with very large meshes, as it goes by the center of the object, however, some leeway is provided.
If the actor is not visible on the clients screen but was recently visible, then it is still considered relevant. The amount of time it remains relevant varies based on bandwidth availible, and will vary from 2 to 10 seconds.
The linked lists Level.ControllerList (UT2003) and Level.PawnList (UT) do not exist on network clients. If you attempt to reference them the list will appear empty. Level.Game is also never replicated and does not exist on the client at all. If you need to access the game's GameReplicationInfo, it can still be referenced by the locally controlled player's PlayerController or PlayerPawn. See Netcode Idioms for an explanation of how to do this.

NetPriority: If there are more relevant actors that need updating than bandwidth is availible, this floating point variable is used to determine priority. The ratio of an actors NetPriority and another actors NetPriority is the same ratio of bandwidth allocated to each. As a result, actors with more NetPriority will be updated more frequently.
bNetOwner: A boolean that simply indicates if the client we are considering replicating this to is the owner of the thing we are considering replicating. This is only of value within the replication block.
bNetInitial: Similar to bNetOwner, this is used only inside the replication block. It is true if this is the first time this actor has been replicated to the client. It is false otherwise. This is useful, for instance, for things like projectiles that only need to have their location and velocity replicated once and their movement can then be predicted reliably.

Role and RemoteRole

The Role of an actor determines what it "is" on the local machine with regards to the network. The RemoteRole is similar but represents what it "is" on the remote machine. See Role for details.

NetMode

The NetMode is not directly connected to the concept of replication itself, but also very important for writing netcode. See NetMode for details.

The Replication Block

In many classes there is a block of code that looks like this:

replication
{
    reliable if( bNetOwner && (Role==ROLE_Authority) )
        NumCopies;
}

This is called the "replication block" or "replication statement", see Replication Block for details.

Simulated Functions

Another great mystery of UnrealScript's netcode is the keyword simulated for functions and states. It's not really replication, but only tells the engine whether a function may be executed on clients. See Simulated Function for details.

Miscellaneous Replication Information

Tearing Off Actors: If an actor has bTearOff set to true, it will, as soon as possible, notify all clients to whom the actor is relevant that the actor is no longer being replicated by the client and that they should take over all responsibility for the actor. This sets the actor to ROLE_Authority on the clients to whom this actor is relevant. The actor will no longer be replicated to any clients, even if it later becomes relevant to them. If the variable bNetTemporary is set to true, this actor will be 'torn off' on a given client as soon as it is initially replicated to that client. This saves bandwidth by not replicating physics changes to objects with predictable motion. (IE a projectile) Note that although the projectile is no longer being replicated, even if something happens to it on the client the takedamage calls will still reach the client when the projectile impacts on the server.
Dirty Replication: This is a hack built in to the replication system. It allows the replication to work much faster, as replication conditions are simply bypassed if no variables have been set within unrealscript since the last time the conditions were checked. This is only logical - if no variables have changed, nothing needs to be replicated. Unrealscript will natively set this to true and false for you whenever you assign a non-local variable. This applies even if the variable is not replicated. (not that it really matters, but I suppose it's a way of optimizing if you're desperate.) The variable bNetDirty, declared in actor, is true whenever the actor should get it's conditions checked. If it is false, no variables have changed since the replication conditions were checked. (not counting bNetDirty itself, obviously) This excludes natively changed variables, however. They will be replicated regardless. The variable bOnlyDirtyReplication, also defined in actor, can be used to override this behaviour. If it is set to true, natively changed variables will not be replicated until a variable is replicated in Unrealscript. This will run faster, but should only be used if no variables will be updated through native code, meaning basically that no physics should be performed on the actor and preferably it should be as inert as possible. (like a decoration)
Server Packages: The packages that the client will load in order to open the level on the server are defined by the ServerPackages in your UnrealTournament or UT2003 INI file. All of the packages specified in the list found in the INI file will be loaded, as will any packages that contain superclasses of the classes of any class in the packages specified. (Hope that's not confusing. )
PostBeginPlay versus PostNetBeginPlay: The difference between these two functions is small, but important. PostBeginPlay is called after the actor is created but BEFORE any values have been replicated to it. PostNetBeginPlay is called after all of the variables have been replicated to the actor. PostBeginPlay should be used for initialization sorts of things, while PostNetBeginPlay can be used for stuff like setting up variables that are not replicated but instead divined from the value of the variables that are replicated. In botplay or on the server of a network game, PostNetBeginPlay will be called immediately after PostBeginPlay. One caveat of PostNetBeginPlay() that may not be immediately obvious is that since PostNetBeginPlay() is not called until variables have been replicated, this means that it will not be called until any variable's value has actually been changed.

Hints and Tips from those who've been there

Network-compatible code isn't "code plus netcode": Try to avoid the approach "I'll write my code now and when it's free of bugs I'll add 'netcode'." While that might work for less complex cases, more often than not coding something in a way that makes it work in network games requires a completely different approach to start with, and keeping that in mind before you start coding is much easier and less error-prone than altering your code afterwards.
Test your Code under Network Conditions: For obvious reasons, you must test your code under network conditions. This is not the same as running a listen server. A listen server is halfway between a botmatch and a dedicated server. Nothing is being replicated to the local machine when you run a listen server - the values are read directly from the server values, since they are on the same machine. Debugging Techniques explains how to run a dedicated server and a client simultaneously.

External Links

Replication is a complex and sometimes difficult to understand part of writing code for the Unreal Engine. The following are some documents which describe replication in all of it's glory. They can be used as a guide or simply as supplements to the information provided here.

[Networking Architecture] by Tim Sweeney himself.
[Replication De-Obfuscation] by Mongo
[Guided Ripper Tutorial] – A UT-based tutorial describing the pitfalls of creating a weapon that works not only offline.
[Networking Sample] by GW
[Another Networking Sample] by GW

Discussion

Daid303: A strange bug shows up in replication when replicating Structures: (I've only tested this in UT2003) I was creating my own Vehicle (A SpaceShip) with it's own replication code, I had copied the Car replication code, and cut out everything about wheels and other not usefull stuff. It looked fine in code, it compiled nice. But it DIDN'T work. I was testing, trying and doing some other wierd stuff for about 4 days, without any result, the stuct seemt only to replicate when i wasn't in the vehicle. (I got very frustraded at a moment and almost deleted my code) I redid everything, and still it didn't work. Then i noticed, after the Struct in Car is the Gear replicated. I had tried everything, so i replicated a Var after my struct. And a mircle happend, I killed the UnrealEd goblin , no not really, but it worked. It seems that epic made an error somewhere in the replication code. What do we learn of this?

If you want to replicate a struct, and it doesn't work, try to replicate something else with it.

Dawn: In UT, I'm replicating structures reliably with no problems, so the above may only apply to UT2K3.

Foxpaw: The amount of control you have over what gets replicated when is somewhat minimal.. the variable after the struct may not actually get replicated every time that the struct does. At any rate, it's something to try if anyone is having problems replicating structs.

Haral: I think that an actor set to bStatic or bNoDelete is not relevant, contrary to what this page says. In fact, if you set an actor to bStatic, it won't even try to send replicated functions. On a side note, anyone know why you'd get 'unwanted function received' on the server end of a replicated function?

Foxpaw: I'm pretty sure they are relevant, like the page says. If they were not they would not appear in the map at all, unless strictly client side only. The lack of function replication may be a result of the objects role or remoterole, nettemporary, or others. I'm not clear on your other question though. I actually have a question of my own though: if multiple elements of a dynamic array are being replicated and the packets arrive out of order, will they end up out of order in the array, or is there a timestamp of sorts sent along with it to ensure that the elements are ordered properly when they arrive?

Wormbo: Dynamic arrays can't be replicated, this is mentioned at least on the Variable Syntax page.

Foxpaw: So it says. I was hoping they could since the compiler didn't complain, but I can write a different implementation. Are reliable functions guaranteed to reach the other side in the same order they are sent? Right now I'm using a "net leeway" variable and any elements recieved within that amount of time are assumed to be out of order and sorted, but this is rather inefficient, especially since it's being used to transfer player input and some things, like axis movements, occur very frequently.

Mysterial: No, they're not. Why not simply add a parameter to the replicated function indicating the index into the array where the data should go?

Mychaeel: Keep in mind that "replicating an array" this way will leave it completely empty for players joining mid-game. You could use a generously dimensioned static array instead and set it up as a round-robin buffer; for all clients in game only one value will be replicated at a time when it is added to the buffer (plus the updated number of elements in the buffer), while joining clients will automatically get the entire buffer.

Foxpaw: It's okay if newly joined players don't get the full array - it's simply a buffer of time-stamped inputs from the player. Information is only stored in it for a maximum of four seconds, to allow all of the things that want to read the player's input to have a chance to see it.

Haral: Well I see Mungo and Tim Sweeney seem to disagree on the point of bStatic/bNoDelete variables. In the Replication DeObfuscation article an actor is listed as passing the first stage of checks if it is not either of these. It makes sense as far as bStatic goes. Why in the world would something bStatic need to be relevant all the time? The client only needs to know if something changes. I think I'll test when I'm off of work and see what's up. I still am not quite sure what that error message (received in the server log) was about. However, I got my code working when I came to understand the difference between calling a function on the server and executing it on the server. Whoops!

Foxpaw: I believe it is because, depending on that actors Role, often things that are not relevant will simply cease to exist on the client. Often those bStatic things you still want to be around. Things that are relevant still don't get replicated unless something changes, with the exception of actor replication which occurs only when they first become relevant. So, in effect, a bStatic thing will probrably never actually get replicated because it generally doesn't change, even though it is always relevant. I had another question about replicating dynamic arrays/functions too. Does anyone know what the result will be if you replicate a function which takes a dynamic array as an argument? Will this simply replicate the entire array along with the function or will it arrive as an empty argument?

GW: Just an FYI, I havn't see this documented anywhere. It is not possible to call a function on a client from the server using any sort of 'player input' function. This includes exec functions and things like playerMove(). To get around this, have the input function set a boolean and then check against that boolen in a timer or tick function...which will then call your replicated function.

Foxpaw: I'm confused: you would have to replicate the bool to the server in order for it to replicate the function back.. wouldn't it be easier to just have the client call that function itself?

GW: I'll post another 'Sample' with an example so you can see exactly what I mean. You dont have to replicate the boolean because the server is the only box that needs to know its value. The server is the one that's calling a client only function via it's tick.

Foxpaw: Ooooh, I see what you mean. I thought you meant that it wasn't possible for the server to pick up on client inputs AT ALL without that workaround. I see now, but wouldn't it be easier in that instance to call the client function from the exec function itself? IE:

// I don't actually write my programs like this BTW. :P
exec function ExecuteThis( string ImAnArgument )
{
  MangleMyArguments( ImAnArgument );
  DoSomethingnow( ImAnArgument );

  CallThisOnTheClient( ImAnArgument );
}

simulated function CallThisOnTheClient( string ImAnArgument )
{
  if ( Role == ROLE_Authority ) return;

  ClientFunctionality( ImAnArgument );
  DoSomeOtherStuff();
  IHopeThisFunctionGotCalled( true );
}

GW: hehe, thats exactly what I was trying to say. You cannot call a function on a client from the server using an exec. Here's another sample. [Another Networking Sample] It contains a working example of the boolean method via tick(). There's a few commented sections, which if uncommented, show the function in question doesn't log on the client like it's supposed too.

Wormbo: Would all this fit onto a [/pitfalls and known bugs]? page? It clearly requires a Refactor Me tag.

MasterOfTheDark: In response to Haral's 'unwanted function': I have managed to duplicate this effect and discovered the problem: "This replicated function calls for FunctionName to the server if the actor is owned by this client's player." The object you are trying to replicate a function on isn't owned by the client you're replicating from. I had the same problem when trying to make a shopkeeper - I fixed it by having my client call a function on one of it's owned objects (the player's Pawn) which in turn called the shopkeeper's Purchase function.

Category Tutorial