Failover Testbed

Failover and Failback

One of the great properties of Diameter is the failover mechanism, which allows more reliable delivery of messages (especially important for accounting/billing).

This mechanism works as follow: after a request has been sent or relayed by a node, if the connection it was sent over is teared down before the corresponding answer is received, the request must be sent again through another connection (when possible, otherwise an error is returned to the client). The request will reach the final destination through a different route.

Coupled with the watchdog mechanism of Diameter, this failover mechanism ensures that all requests receive an answer (which can be an error). The main drawbacks are that the same request may be received several times by the server (duplicates), and that there is no limit of time for receiving the answer to a message, which can create congestion in the Diameter stack.

The failback term means that after a connection has been restored, new messages are again sent over it.

Failover and failback are natively supported by the freeDiameter framework.

Testbed Description

This testbed involves 4 nodes in two different realms: a client (nas in realm a), two relays (proxy in realm a, and proxy in realm b), and a server (serv in realm b).

The client and server are using the test_app.fdx application to exchange messages. The configuration allows the messages to be conveyed through one of the relays, but not directly between the client and server.

In the graph bellow, the blue route is the primary route, and green is secondary in case the primary route is down.

Error: Failed to load processor graphviz

No macro or processor named 'graphviz' found

Screencast

The demonstration is split across two videos, because of its length. It is recommended to watch the video in HD and fullscreen, with the subtitles.

First video: preparation of the testbed.

Here is what happens in this first video:

• (0:00 ~ 0:19) Short explanation of the testbed structure.
• (0:20 ~ 1:19) Start the two relays, check they are connected properly.
• (1:20 ~ 1:49) Add the server, it connects to the two relays.
• (1:50 ~ 2:44) Add the client, also connects to the two relays.
• (2:45 ~ 2:59) Confirm the dbg_monitor.fdx output.
• (3:00 ~ 3:27) Trig the test_app.fdx application.
• (3:28 ~ 4:35) Confirm the basic behavior of Test application (see Simple Testbed for detail).

Second video: Failover experiments.

This is the script of the second video:

• (0:08 ~ 1:53) First test: stopping freeDiameter on a relay (notified failover).
  • (0:08 ~ 0:21) Explanation and test (stop freeDiameter on proxy.b).
  • (0:22 ~ 0:44) Confirm the result is as expected.
  • (0:45 ~ 1:10) Start proxy.b again and confirm the failback.
  • (1:10 ~ 1:53) Stop proxy.a, no effect, start it again.
• (1:55 ~ 5:45) Second test: simulate a network failure (unnotified failover).
  • (1:55 ~ 2:45) Explanation
  • (2:46 ~ 2:54) Cut the (virtual) network links for proxy.b
  • (2:55 ~ 4:20) Observe and comment the reaction of the testbed.
  • (4:21 ~ 4:55) Reconnect the network, observe the REOPEN state and then failback.
  • (4:55 ~ 5:45) Confirm the failback by stopping proxy.a then restart it.
• (5:45 ~ 6:56) Stop the server, then the proxy, and check the error code received is consistent.

Configuration files

Please note that these files may have been updated since the time the video was created. However, the purpose of the testbed remains the same.

Scripts

There is no script demonstrated in this video. Anyway, you might find the scripts from the Simple Testbed useful also here.

nas.a.rt.testbed.aaa

These are the configuration files for freeDiameter on the client:

[source:VirtualTestbed/conf/nas.a.rt.testbed.aaa/freeDiameter/CMakeFlags CMakeFlags]

The flags passed to cmake (on the command-line) when preparing the source. The noticeable ones are:

• -DBUILD_TEST_APP:BOOL=ON : compile the test_app.fdx extension.
• -DBUILD_DBG_MONITOR:BOOL=ON : compile the dbg_monitor.fdx extension.
• -DDEFAULT_CONF_PATH:PATH=/root/conf/freeDiameter : Default configuration file path (avoids typing it on the command line).
• Note that the rt_default.fdx? extension is also built (by default).

[source:VirtualTestbed/conf/nas.a.rt.testbed.aaa/freeDiameter/freeDiameter.conf freeDiameter.conf]

The main freeDiameter configuration file. It simply indicates to load the extensions and to connect to the relays (not the server directly).

[source:VirtualTestbed/conf/nas.a.rt.testbed.aaa/freeDiameter/test_app.conf test_app.conf]

Configuration file for the test_app.fdx extension, indicating the routing AVPs for the Test message (Destination-Host and Destination-Realm).

[source:VirtualTestbed/conf/nas.a.rt.testbed.aaa/freeDiameter/rtd.conf rtd.conf]

Configuration file for the rt_default.fdx? extension. We indicate to the daemon that all messages can be sent to the local relay (proxy.a.rt.testbed.aaa) by default, if no better route is available.

proxy.a.rt.testbed.aaa

The following details of the configuration files should be highlighted:

[source:VirtualTestbed/conf/proxy.a.rt.testbed.aaa/freeDiameter/CMakeFlags CMakeFlags]

• -DBUILD_DBG_RT:BOOL=ON : compile the dbg_rt.fdx? extension.
• The acl_wl.fdx? extension is also built (by default).
• The test_app.fdx extension is not built on the relay: the node does not need to understand fully the message in order to relay it.

[source:VirtualTestbed/conf/proxy.a.rt.testbed.aaa/freeDiameter/freeDiameter.conf freeDiameter.conf]

• The NoRelay directive is commented out here.
• The only peer to which this relay actively tries to connect is the other proxy. It does not initiate connections to the client or server.
• It loads the acl_wl.fdx? extension, to enable incoming connections from client and server.

[source:VirtualTestbed/conf/proxy.a.rt.testbed.aaa/freeDiameter/acl_wl.conf acl_wl.conf]

The configuration file passed to the acl_wl.fdx? extension. The meaning here is that any node that claims a Diameter Identity that matches the expression "*.rt.testbed.aaa" is allowed to connect to this peer.

proxy.b.rt.testbed.aaa

The configuration files are very similar to the other relay, the same explanation apply.

The file [source:VirtualTestbed/conf/proxy.b.rt.testbed.aaa/freeDiameter/CMakeFlags-altlibs CMakeFlags-altlibs] is an informational example on how to link freeDiameter against a non-default GNUTLS library.

serv.b.rt.testbed.aaa

The configuration files for the server can be found [source:VirtualTestbed/conf/serv.b.rt.testbed.aaa/freeDiameter/ in this folder]. There is nothing special to add here.

---