Changed Fix Version(s) to 1.8-DR#3

There seems to be 2 issues so far with GlassFish V2 fail-over that I encountered so far.

The first issue is regarding the jvmRoute part of the JSESSIONID in the Set-Cookie header. I have set up a cluster of 2 using GlassFish V2.1-b60f with session replication and session affinity working correctly. The very first response received from the first hit node correctly contains a Set-Cookie header for the JSESSIONID correctly post-fixed with the correct jvmRoute. However, when failing over to the other node, the first response received from that node does not contain a new Set-Cookie header for the JSESSIONID with the new jvmRoute post-fixed. This is fine for a 2 node cluster, but when using a 3 or more node cluster, this effectively turns into a round-robin as long as the first node is down as the requests still contain the JSESSIONID with the jvmRoute value of the downed node post-fixed. As Apache can't proxy it to the correct node, it round-robins the following requests over the remaining nodes.

I debugged the addSessionCookieJvmRoute() method of org.apache.coyote.tomcat5.OutputBuffer:

private void addSessionCookieWithJvmRoute() {
CoyoteRequest req = (CoyoteRequest) coyoteResponse.getRequest();
if (req.isRequestedSessionIdFromURL())

if (response.containsHeader(SET_COOKIE_HEADER))

// Creating JSESSIONID cookie that includes jvmRoute
Cookie cookie = new Cookie(Globals.SESSION_COOKIE_NAME,
sess.getIdInternal() + "." +
ctx.getJvmRoute());
req.configureSessionCookie(cookie);
response.addHeader(SET_COOKIE_HEADER,
coyoteResponse.getCookieString(cookie));
}

So again I used a two node cluster with one instance running on each node (instance1 and instance2). I hooked the debugger up to instance2 as that will be the instance we want to fail-over to in this case.

Using a browser I opened up our little test application and from the HTTP traffic I can see that we're on instance1. The JSESSIONID is nicely set with the correct jvmRoute. Then I shutdown instance1 and do something in our application to ensure another request goes to the server. My breakpoint is picked up and I can start debugging the addSessionCookieWithJvmRoute() method.

Debugging shows the following:

1. ctx is not null;

2. ctx.getJvmRoute() returns the correct jvmRoute for instance2;

3. response.containsHeader(SET_COOKIE_HEADER) returns true! And thus exits the method without setting the new JSESSIONID with the correct jvmRoute.

Further debugging showed that the response indeed contains a Set-Cookie header, but not for JSESSIONID but for JSESSIONIDVERSION. It looks like the JSESSIONIDVERSION is set first, which seems correct as the JSESSIONID with the correct jvmRoute should only be added at the very last moment if necessary. I think the logic here should be expanded: first check to see if the Set-Cookie header is already added to the response and if so check to see the Set-Cookie header already contains a new value for the JSESSIONID. If this is so return, otherwise add a new Set-Cookie header for the new JSESSIONID with the correct jvmRoute.

We've contacted Sun for more information regarding this.

The second issue involves an actual asynchronous ICEfaces application and is as follows:

During most runs the AJAX Bridge receives nothing but 200 OK responses, even during failover. However, the last response contains the session expired message. On the server I see the following debug logs:

[#|2008-12-09T17:47:13.713-0700|INFO|sun-appserver2.1|com.icesoft.faces.webapp.http.core.RequestVerifier|_ThreadID=26;_ThreadName=httpSSLWorkerThread-38080-0;|Missmatch! Sending SessionExpiredResponse.|#]

[#|2008-12-09T17:47:13.715-0700|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=26;_ThreadName=httpSSLWorkerThread-38080-0;_RequestID=cd3511e5-a1ec-440d-a686-83473febe14d;|java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1158)
at com.icesoft.faces.webapp.http.core.RequestVerifier.service(RequestVerifier.java:30)
...

The request contains the correct ice.session parameter. To me this looks like the new secondary node might not have the session completely setup to receive new requests yet.

Changed Assignee Priority to P1

Sun supplied us with a patch that addresses the first mentioned issue. This patch seems to fix the issue. Sun will include the fix in an upcoming V2 and V3 release: https://glassfish.dev.java.net/issues/show_bug.cgi?id=7101

The second issue still remains. I brought this to Sun's attention.

I created another Issue on GlassFish Issue Tracker to address the second issue mentioned: https://glassfish.dev.java.net/issues/show_bug.cgi?id=7200

Changed Fix Version(s) to 'Unknown' as both mentioned issues need to be addressed in a future release of GlassFish. One fix is ready, but not yet included in an official release of GlassFish. The other issue remains open.