Since there was a request from Michael [comment] for more information on what we can test via network emulation, I thought I'd give a brief summary (can't really go into great detail about Xbox 360 proprietary tools)

Options available for packet loss include:

• * Periodic Loss: drop one packet every N packets
• * Random Loss: drop packets randomly at a specified loss rate
• * Burst Loss: drop multiple packets in bursts specified by a min/max size and a loss probability
• * Gilbert-Elliot Loss: packet loss is simulated based on the Gilbert-Elliot model, which describes a system of two states: good and bad.

Gilbert-Elliot Model

The image below shows a diagram of good (G) and bad (B) states.  We can specify the probability that the network will transition from one state to the other, the rate at which the simulation is evaluated, and the packet loss in both the good and the bad states.

Images from http://www.utdallas.edu/~aria/mcl/packet/ and http://www.tkn.tu-berlin.de/publications/papers/tkn_report02.pdf

HD Remix lessons:

It was a bit misleading to state that the problems in HD Remix were caused by packet loss.  One of the issues was related to how we handled the re-transmission when this type of error occurs.  This is where it gets a little tricky, because it ends up being a matter of tuning and tweaking.

What we discovered was that the reliable transfer protocol that Backbone developed was correctly identifying the situations where packets were received out-of-order, greatly delayed, or not at all.  The problem is in how those are re-transmitted - One error was to tag re-transmission data as important, rather than correctly evaluate it based on priority.  The rest of the problem was a matter of finding an optimal rate to resend.

The bad situation that we want to avoid is getting packet loss and then making the problem worse by continually re-sending data while trying to catch up.

The other half of the rollback problems with HD Remix has more to do with how images are rendered to the screen than a network issue.. which I will touch on later.

 - Gary

Before going too far down the road of explaining what we have done to hide latency in HD Remix, I want to point out that we are aware of similar work that is being done on various projects, such as Touhou Suimusou IaMP / Caster netplay, 2DF FreePlay and the related FBA / nFBA Kaillera/P2P client, as well as GGPO.

[However, we are not referencing any source code from these projects nor are we attempting to reverse engineer the work of others.]

All of these efforts attempt to address the issue of input response in a network environment where it typically takes 5 or more frames before joystick input can be transmitted and confirmed from other players.  The traditional solution is to keep all clients in a multiplayer game completely synchronized, and to introduce artificial lag to the joystick response to cover latency.

As fans in the online fighting community have aptly identified, input lag is not good for reproducing a true arcade experience.

The solution we use for HD Remix is to eliminate input lag and let players issue moves immediately.  We deal with de-synchronization after the fact by identifying conflicts and rewinding the game back to the point where the two clients diverged.  This is commonly referred to as a ‘rollback' technique.

Here's a high level description:

• * Players are allowed to input moves directly and the screen is updated within one frame.  Game state is advanced.  There is no input lag perceived by the local player.
• * Joystick information is sent over the network.
• * When a player receives joystick input from the other, game state is re-evaluated.  If there is a conflict - for example, if one player had actually been blocking the other player's move, game state is corrected.

The method we use to correct game state is to rewind to the last known point in time when joystick information from both clients agreed.  We then kick off the correct animation and try to catch up as fast as possible to the current game state.

  - Gary

Last post, I gave some background on how we test multiplayer network games under adverse conditions - simulating poor Internet connections.  Today, I'll discuss one of the more significant things that we observed in the SFII Beta.

When the reports started rolling in on the Beta test forums regarding issues with large rollbacks / teleporting effects, we started an investigation.  Many of these issues didn't appear during our internal pre-Beta QA period, so it was interesting to see them reported so widely.

The first thing we attempted to do was to reproduce the rollbacks in the Capcom QA environment.  The cause wasn't entirely obvious at first, because the condition does not occur when two players have high latency connections alone - We can set up a match between two machines that have 200ms ping times and still not see the same problem.

After some experimentation, we discovered that large rollbacks could be reproduced when we combined 100+ms latency conditions WITH packet loss.

For some background information, it may be useful to explain how packet loss contributes to the overall network experience.  Once again, I'll turn to the Wikipedia for a description of Packet Loss  Packet Loss and Quality of Service (QoS)

If you look in the "Problems" section on QoS, you'll notice that any number of factors can contribute to packet loss in a network environment.  It seems, based on this Beta test, that 5%-10% packet loss is fairly common in the real world, particularly over wireless Internet connections.

Since packet loss occurs randomly, there is no guarantee that it will show up at the time that a matchmaking connection is underway.  As a result, our Beta testers can only make a determination of who to challenge based on ping rating.  [Note: We show the Xbox LIVE Quality of Service latency value, which is close, but not quite the same as the ping from a PC command shell]

 Next blog, I'll discuss some of the things that we've done to address these issues.

Microsoft provides tools as part of the Xbox 360 development environment that let engineers experiment with network performance in multiplayer games.  The one we use most often is a network emulator called N.E.W. - see this Wikipedia entry for a high level overview of Network Emulation 

There are several open source tools that perform a similar function.  One of the easier ones to get set up is Dummynet :

Network emulation essentially requires a PC with two Ethernet cards to act as a gateway:  One card connects to the outside world and the other connects to rest of your network via a switch.  [Under Windows, this is equivalent to Internet Connection Sharing ]

Dummynet runs on the PC and controls the flow of packets to/from network devices.  You need to have some knowledge of how to control a command-line application to get anything useful out of the tool.

The things we control for the testing of SSFII HD Remix include: 

• Bandwidth : We need to meet requirements that all applications run under 64 kbps bandwidth restriction - this is roughly the low-end limit of a DSL/Cable connection.
•  
• Latency : We can simulate poor latency connections by inserting a delay in the Internet uplink.  The general worst case latency situation we are looking at is 250+ms roundtrip sustained.
•  
• Packet Loss : Games generally need to be able to withstand a 10% random packet loss.

There is a very good article on the use of Dummynet on the personal website of Márton Anka, codefromthe70s.org

Using Márton's step-by-step instructions, you can set up a Dummynet environment on a single PC using VMWare to simulate several virtual PC's.

  - Gary