ICEfaces
  1. ICEfaces
  2. ICE-3953

EPS Fail-over support for GlassFish

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: 1.8DR#2
    • Fix Version/s: None
    • Component/s: Framework
    • Labels:
      None
    • Environment:
      GlassFish V2 in a cluster

      Description

      This is a follow-up to ICE-3359.

      Add fail-over support to GlassFish V2.

        Issue Links

          Activity

          Jack Van Ooststroom created issue -
          Jack Van Ooststroom made changes -
          Field Original Value New Value
          Link This issue depends on ICE-3359 [ ICE-3359 ]
          Jack Van Ooststroom made changes -
          Assignee Jack Van Ooststroom [ jack.van.ooststroom ]
          Jack Van Ooststroom made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          Jack Van Ooststroom added a comment -

          Changed Fix Version(s) to 1.8-DR#3

          Show
          Jack Van Ooststroom added a comment - Changed Fix Version(s) to 1.8-DR#3
          Jack Van Ooststroom made changes -
          Salesforce Case []
          Fix Version/s 1.8DR#3 [ 10143 ]
          Hide
          Jack Van Ooststroom added a comment - - edited

          There seems to be 2 issues so far with GlassFish V2 fail-over that I encountered so far.

          The first issue is regarding the jvmRoute part of the JSESSIONID in the Set-Cookie header. I have set up a cluster of 2 using GlassFish V2.1-b60f with session replication and session affinity working correctly. The very first response received from the first hit node correctly contains a Set-Cookie header for the JSESSIONID correctly post-fixed with the correct jvmRoute. However, when failing over to the other node, the first response received from that node does not contain a new Set-Cookie header for the JSESSIONID with the new jvmRoute post-fixed. This is fine for a 2 node cluster, but when using a 3 or more node cluster, this effectively turns into a round-robin as long as the first node is down as the requests still contain the JSESSIONID with the jvmRoute value of the downed node post-fixed. As Apache can't proxy it to the correct node, it round-robins the following requests over the remaining nodes.

          I debugged the addSessionCookieJvmRoute() method of org.apache.coyote.tomcat5.OutputBuffer:

          private void addSessionCookieWithJvmRoute() {
          CoyoteRequest req = (CoyoteRequest) coyoteResponse.getRequest();
          if (req.isRequestedSessionIdFromURL())

          { return; }

          // start of the part of interest

          StandardContext ctx = (StandardContext) coyoteResponse.getContext();
          if (ctx == null || ctx.getJvmRoute() == null) { return; }

          if (response.containsHeader(SET_COOKIE_HEADER))

          { return; }

          // end of the part of interest

          Session sess = req.getSessionInternal(false);
          if (sess == null) { return; }

          // Creating JSESSIONID cookie that includes jvmRoute
          Cookie cookie = new Cookie(Globals.SESSION_COOKIE_NAME,
          sess.getIdInternal() + "." +
          ctx.getJvmRoute());
          req.configureSessionCookie(cookie);
          response.addHeader(SET_COOKIE_HEADER,
          coyoteResponse.getCookieString(cookie));
          }

          So again I used a two node cluster with one instance running on each node (instance1 and instance2). I hooked the debugger up to instance2 as that will be the instance we want to fail-over to in this case.

          Using a browser I opened up our little test application and from the HTTP traffic I can see that we're on instance1. The JSESSIONID is nicely set with the correct jvmRoute. Then I shutdown instance1 and do something in our application to ensure another request goes to the server. My breakpoint is picked up and I can start debugging the addSessionCookieWithJvmRoute() method.

          Debugging shows the following:

          1. ctx is not null;

          2. ctx.getJvmRoute() returns the correct jvmRoute for instance2;

          3. response.containsHeader(SET_COOKIE_HEADER) returns true! And thus exits the method without setting the new JSESSIONID with the correct jvmRoute.

          Further debugging showed that the response indeed contains a Set-Cookie header, but not for JSESSIONID but for JSESSIONIDVERSION. It looks like the JSESSIONIDVERSION is set first, which seems correct as the JSESSIONID with the correct jvmRoute should only be added at the very last moment if necessary. I think the logic here should be expanded: first check to see if the Set-Cookie header is already added to the response and if so check to see the Set-Cookie header already contains a new value for the JSESSIONID. If this is so return, otherwise add a new Set-Cookie header for the new JSESSIONID with the correct jvmRoute.

          We've contacted Sun for more information regarding this.

          Show
          Jack Van Ooststroom added a comment - - edited There seems to be 2 issues so far with GlassFish V2 fail-over that I encountered so far. The first issue is regarding the jvmRoute part of the JSESSIONID in the Set-Cookie header. I have set up a cluster of 2 using GlassFish V2.1-b60f with session replication and session affinity working correctly. The very first response received from the first hit node correctly contains a Set-Cookie header for the JSESSIONID correctly post-fixed with the correct jvmRoute. However, when failing over to the other node, the first response received from that node does not contain a new Set-Cookie header for the JSESSIONID with the new jvmRoute post-fixed. This is fine for a 2 node cluster, but when using a 3 or more node cluster, this effectively turns into a round-robin as long as the first node is down as the requests still contain the JSESSIONID with the jvmRoute value of the downed node post-fixed. As Apache can't proxy it to the correct node, it round-robins the following requests over the remaining nodes. I debugged the addSessionCookieJvmRoute() method of org.apache.coyote.tomcat5.OutputBuffer: private void addSessionCookieWithJvmRoute() { CoyoteRequest req = (CoyoteRequest) coyoteResponse.getRequest(); if (req.isRequestedSessionIdFromURL()) { return; } // start of the part of interest StandardContext ctx = (StandardContext) coyoteResponse.getContext(); if (ctx == null || ctx.getJvmRoute() == null) { return; } if (response.containsHeader(SET_COOKIE_HEADER)) { return; } // end of the part of interest Session sess = req.getSessionInternal(false); if (sess == null) { return; } // Creating JSESSIONID cookie that includes jvmRoute Cookie cookie = new Cookie(Globals.SESSION_COOKIE_NAME, sess.getIdInternal() + "." + ctx.getJvmRoute()); req.configureSessionCookie(cookie); response.addHeader(SET_COOKIE_HEADER, coyoteResponse.getCookieString(cookie)); } So again I used a two node cluster with one instance running on each node (instance1 and instance2). I hooked the debugger up to instance2 as that will be the instance we want to fail-over to in this case. Using a browser I opened up our little test application and from the HTTP traffic I can see that we're on instance1. The JSESSIONID is nicely set with the correct jvmRoute. Then I shutdown instance1 and do something in our application to ensure another request goes to the server. My breakpoint is picked up and I can start debugging the addSessionCookieWithJvmRoute() method. Debugging shows the following: 1. ctx is not null; 2. ctx.getJvmRoute() returns the correct jvmRoute for instance2; 3. response.containsHeader(SET_COOKIE_HEADER) returns true! And thus exits the method without setting the new JSESSIONID with the correct jvmRoute. Further debugging showed that the response indeed contains a Set-Cookie header, but not for JSESSIONID but for JSESSIONIDVERSION. It looks like the JSESSIONIDVERSION is set first, which seems correct as the JSESSIONID with the correct jvmRoute should only be added at the very last moment if necessary. I think the logic here should be expanded: first check to see if the Set-Cookie header is already added to the response and if so check to see the Set-Cookie header already contains a new value for the JSESSIONID. If this is so return, otherwise add a new Set-Cookie header for the new JSESSIONID with the correct jvmRoute. We've contacted Sun for more information regarding this.
          Hide
          Jack Van Ooststroom added a comment -

          The second issue involves an actual asynchronous ICEfaces application and is as follows:

          During most runs the AJAX Bridge receives nothing but 200 OK responses, even during failover. However, the last response contains the session expired message. On the server I see the following debug logs:

          [#|2008-12-09T17:47:13.713-0700|INFO|sun-appserver2.1|com.icesoft.faces.webapp.http.core.RequestVerifier|_ThreadID=26;_ThreadName=httpSSLWorkerThread-38080-0;|Missmatch! Sending SessionExpiredResponse.|#]

          [#|2008-12-09T17:47:13.715-0700|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=26;_ThreadName=httpSSLWorkerThread-38080-0;_RequestID=cd3511e5-a1ec-440d-a686-83473febe14d;|java.lang.Exception: Stack trace
          at java.lang.Thread.dumpStack(Thread.java:1158)
          at com.icesoft.faces.webapp.http.core.RequestVerifier.service(RequestVerifier.java:30)
          ...

          The request contains the correct ice.session parameter. To me this looks like the new secondary node might not have the session completely setup to receive new requests yet.

          Show
          Jack Van Ooststroom added a comment - The second issue involves an actual asynchronous ICEfaces application and is as follows: During most runs the AJAX Bridge receives nothing but 200 OK responses, even during failover. However, the last response contains the session expired message. On the server I see the following debug logs: [#|2008-12-09T17:47:13.713-0700|INFO|sun-appserver2.1|com.icesoft.faces.webapp.http.core.RequestVerifier|_ThreadID=26;_ThreadName=httpSSLWorkerThread-38080-0;|Missmatch! Sending SessionExpiredResponse.|#] [#|2008-12-09T17:47:13.715-0700|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=26;_ThreadName=httpSSLWorkerThread-38080-0;_RequestID=cd3511e5-a1ec-440d-a686-83473febe14d;|java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1158) at com.icesoft.faces.webapp.http.core.RequestVerifier.service(RequestVerifier.java:30) ... The request contains the correct ice.session parameter. To me this looks like the new secondary node might not have the session completely setup to receive new requests yet.
          Hide
          Jack Van Ooststroom added a comment -

          Changed Assignee Priority to P1

          Show
          Jack Van Ooststroom added a comment - Changed Assignee Priority to P1
          Jack Van Ooststroom made changes -
          Salesforce Case []
          Assignee Priority P1
          Hide
          Jack Van Ooststroom added a comment -

          Sun supplied us with a patch that addresses the first mentioned issue. This patch seems to fix the issue. Sun will include the fix in an upcoming V2 and V3 release: https://glassfish.dev.java.net/issues/show_bug.cgi?id=7101

          The second issue still remains. I brought this to Sun's attention.

          Show
          Jack Van Ooststroom added a comment - Sun supplied us with a patch that addresses the first mentioned issue. This patch seems to fix the issue. Sun will include the fix in an upcoming V2 and V3 release: https://glassfish.dev.java.net/issues/show_bug.cgi?id=7101 The second issue still remains. I brought this to Sun's attention.
          Ken Fyten made changes -
          Fix Version/s 1.8 [ 10161 ]
          Fix Version/s 1.8RC1 [ 10143 ]
          Hide
          Jack Van Ooststroom added a comment -

          I created another Issue on GlassFish Issue Tracker to address the second issue mentioned: https://glassfish.dev.java.net/issues/show_bug.cgi?id=7200

          Show
          Jack Van Ooststroom added a comment - I created another Issue on GlassFish Issue Tracker to address the second issue mentioned: https://glassfish.dev.java.net/issues/show_bug.cgi?id=7200
          Hide
          Jack Van Ooststroom added a comment -

          Changed Fix Version(s) to 'Unknown' as both mentioned issues need to be addressed in a future release of GlassFish. One fix is ready, but not yet included in an official release of GlassFish. The other issue remains open.

          Show
          Jack Van Ooststroom added a comment - Changed Fix Version(s) to 'Unknown' as both mentioned issues need to be addressed in a future release of GlassFish. One fix is ready, but not yet included in an official release of GlassFish. The other issue remains open.
          Jack Van Ooststroom made changes -
          Salesforce Case []
          Fix Version/s 1.8 [ 10161 ]
          Ken Fyten made changes -
          Issue Type Improvement [ 4 ] New Feature [ 2 ]
          Salesforce Case []
          Assignee Priority P1
          Assignee Jack Van Ooststroom [ jack.van.ooststroom ]
          Ken Fyten made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Incomplete [ 4 ]
          Ken Fyten made changes -
          Summary Fail-over support for GlassFish EPS Fail-over support for GlassFish
          Salesforce Case []
          Ken Fyten made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Jack Van Ooststroom
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: