We're excited to have the patch going up tonight - crossing fingers that the netcode improvements address many of the shortfalls in the previous version. While waiting for it to go live, I thought I'd share the last bit of info regarding tweaks to the ‘rollback' system.
As a reminder from previous posts: The HD Remix latency hiding solution works by allowing players to input moves immediately and deal with network mismatches later. When conflicts occur, the game is rewound to the last known good state and then corrected moves are played back until the game is re-synchronized.
In an ideal situation, this process of rewinding and catching up to the correct state would happen as quickly as possible. The longer it takes, the more likely players are to notice a timing difference. There is already an inherent visual flaw in the system as moves that were incorrectly issued have to be canceled and rolled back (appears a momentary ‘glitch'). Our goal is to at least try to finish corrected moves at the appropriate time.
Rather than play entire moves back during the correction phase, we actually attempt to recover faster by skipping frames of animation and blending between them. The first Beta release had an early version of this method that didn't account well for the relative length of time between frames. For example, if a character performed a jump and too many frames were skipped in the middle of the animation, the player would appear to be warping in air.
After some experimentation, Backbone arrived at a better mix of original and blended frames to smooth this out. There is still some tweaking to be done on this, but the algorithm appears much improved. One thing to note is that this is purely a cosmetic fix to the renderer - it does not actually involve a change to the network layer.
If you would like to see the difference between the old and new smoothing, follow Rey's advice on the Beta Patch announcement:
"Hold the Left Trigger, Right Trigger, RB, LB and while holding down the buttons, hit start. You should hear an audible tone."
Since there was a request from Michael [comment] for more information on what we can test via network emulation, I thought I'd give a brief summary (can't really go into great detail about Xbox 360 proprietary tools)
Options available for packet loss include:
* Periodic Loss: drop one packet every N packets
* Random Loss: drop packets randomly at a specified loss rate
* Burst Loss: drop multiple packets in bursts specified by a min/max size and a loss probability
* Gilbert-Elliot Loss: packet loss is simulated based on the Gilbert-Elliot model, which describes a system of two states: good and bad.
Gilbert-Elliot Model
The image below shows a diagram of good (G) and bad (B) states. We can specify the probability that the network will transition from one state to the other, the rate at which the simulation is evaluated, and the packet loss in both the good and the bad states.
It was a bit misleading to state that the problems in HD Remix were caused by packet loss. One of the issues was related to how we handled the re-transmission when this type of error occurs. This is where it gets a little tricky, because it ends up being a matter of tuning and tweaking.
What we discovered was that the reliable transfer protocol that Backbone developed was correctly identifying the situations where packets were received out-of-order, greatly delayed, or not at all. The problem is in how those are re-transmitted - One error was to tag re-transmission data as important, rather than correctly evaluate it based on priority. The rest of the problem was a matter of finding an optimal rate to resend.
The bad situation that we want to avoid is getting packet loss and then making the problem worse by continually re-sending data while trying to catch up.
The other half of the rollback problems with HD Remix has more to do with how images are rendered to the screen than a network issue.. which I will touch on later.
Before going too far down the road of explaining what we have done to hide latency in HD Remix, I want to point out that we are aware of similar work that is being done on various projects, such as Touhou SuimusouIaMP / Caster netplay, 2DF FreePlay and the related FBA / nFBA Kaillera/P2P client, as well as GGPO.
[However, we are not referencing any source code from these projects nor are we attempting to reverse engineer the work of others.]
All of these efforts attempt to address the issue of input response in a network environment where it typically takes 5 or more frames before joystick input can be transmitted and confirmed from other players. The traditional solution is to keep all clients in a multiplayer game completely synchronized, and to introduce artificial lag to the joystick response to cover latency.
As fans in the online fighting community have aptly identified, input lag is not good for reproducing a true arcade experience.
The solution we use for HD Remix is to eliminate input lag and let players issue moves immediately. We deal with de-synchronization after the fact by identifying conflicts and rewinding the game back to the point where the two clients diverged. This is commonly referred to as a ‘rollback' technique.
Here's a high level description:
* Players are allowed to input moves directly and the screen is updated within one frame. Game state is advanced. There is no input lag perceived by the local player.
* Joystick information is sent over the network.
* When a player receives joystick input from the other, game state is re-evaluated. If there is a conflict - for example, if one player had actually been blocking the other player's move, game state is corrected.
The method we use to correct game state is to rewind to the last known point in time when joystick information from both clients agreed. We then kick off the correct animation and try to catch up as fast as possible to the current game state.
Last post, I gave some background on how we test multiplayer network games under adverse conditions - simulating poor Internet connections. Today, I'll discuss one of the more significant things that we observed in the SFII Beta.
When the reports started rolling in on the Beta test forums regarding issues with large rollbacks / teleporting effects, we started an investigation. Many of these issues didn't appear during our internal pre-Beta QA period, so it was interesting to see them reported so widely.
The first thing we attempted to do was to reproduce the rollbacks in the Capcom QA environment. The cause wasn't entirely obvious at first, because the condition does not occur when two players have high latency connections alone - We can set up a match between two machines that have 200ms ping times and still not see the same problem.
After some experimentation, we discovered that large rollbacks could be reproduced when we combined 100+ms latency conditions WITH packet loss.
For some background information, it may be useful to explain how packet loss contributes to the overall network experience. Once again, I'll turn to the Wikipedia for a description of Packet Loss Packet Loss and Quality of Service (QoS)
If you look in the "Problems" section on QoS, you'll notice that any number of factors can contribute to packet loss in a network environment. It seems, based on this Beta test, that 5%-10% packet loss is fairly common in the real world, particularly over wireless Internet connections.
Since packet loss occurs randomly, there is no guarantee that it will show up at the time that a matchmaking connection is underway. As a result, our Beta testers can only make a determination of who to challenge based on ping rating. [Note: We show the Xbox LIVE Quality of Service latency value, which is close, but not quite the same as the ping from a PC command shell]
Next blog, I'll discuss some of the things that we've done to address these issues.
Microsoft provides tools as part of the Xbox 360 development environment that let engineers experiment with network performance in multiplayer games. The one we use most often is a network emulator called N.E.W. - see this Wikipedia entry for a high level overview of Network Emulation
There are several open source tools that perform a similar function. One of the easier ones to get set up is Dummynet :
Network emulation essentially requires a PC with two Ethernet cards to act as a gateway: One card connects to the outside world and the other connects to rest of your network via a switch. [Under Windows, this is equivalent to Internet Connection Sharing ]
Dummynet runs on the PC and controls the flow of packets to/from network devices. You need to have some knowledge of how to control a command-line application to get anything useful out of the tool.
The things we control for the testing of SSFII HD Remix include:
Bandwidth : We need to meet requirements that all applications run under 64 kbps bandwidth restriction - this is roughly the low-end limit of a DSL/Cable connection.
Latency : We can simulate poor latency connections by inserting a delay in the Internet uplink. The general worst case latency situation we are looking at is 250+ms roundtrip sustained.
Packet Loss : Games generally need to be able to withstand a 10% random packet loss.
There is a very good article on the use of Dummynet on the personal website of Márton Anka, codefromthe70s.org
Using Márton's step-by-step instructions, you can set up a Dummynet environment on a single PC using VMWare to simulate several virtual PC's.
