Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: EE-1.8.2.GA_P03
-
Fix Version/s: EE-1.8.2.GA_P08
-
Component/s: Framework
-
Labels:None
-
Environment:WebSphere
-
Assignee Priority:P1
-
Salesforce Case Reference:
Description
When our SessionDispatcher.Listener runs, it checks to see if it should shut the session down. This is done in a thread that runs every 10 seconds and when it checks, it synchonizes on the SessionMonitors while it does so:
try {
synchronized (SessionMonitors) {
// Iterate over the session monitors using a copying iterator
Iterator iterator = new
ArrayList(SessionMonitors.values()).iterator();
while (iterator.hasNext()) {
final Monitor sessionMonitor = (Monitor) iterator.next();
sessionMonitor.shutdownIfExpired();
ThreadLocalUtility.checkThreadLocals(ThreadLocalUtility.EXITING_SESSION_MONITOR);
}
}
Thread.sleep(10000);
} catch (InterruptedException e) {
//ignore interrupts
}
During the check to see if it should invalidate the session, it checks for a session attribute which, in WebSphere, appears to be a synchronized HashMap:
Object o = session.getAttribute(POSITIVE_SESSION_TIMEOUT);
The other thread is a request to logout which invalidates the current session and then sends out a SessionDestroyed event which we listen for. This, presumably, locks down that same HashMap that holds the session attributes *before* it sends the SessionDestroyed event. So while our own internal thread has grabbed the lock on the SessionMonitors the logout thread grabbed the lock on the session attribute map. Our SessionMonitor cannot continue to shutdown without getting the session attribute and the logout thread cannot continue with the SessionDestroyed call because the SessionMonitors lock is taken.
try {
synchronized (SessionMonitors) {
// Iterate over the session monitors using a copying iterator
Iterator iterator = new
ArrayList(SessionMonitors.values()).iterator();
while (iterator.hasNext()) {
final Monitor sessionMonitor = (Monitor) iterator.next();
sessionMonitor.shutdownIfExpired();
ThreadLocalUtility.checkThreadLocals(ThreadLocalUtility.EXITING_SESSION_MONITOR);
}
}
Thread.sleep(10000);
} catch (InterruptedException e) {
//ignore interrupts
}
During the check to see if it should invalidate the session, it checks for a session attribute which, in WebSphere, appears to be a synchronized HashMap:
Object o = session.getAttribute(POSITIVE_SESSION_TIMEOUT);
The other thread is a request to logout which invalidates the current session and then sends out a SessionDestroyed event which we listen for. This, presumably, locks down that same HashMap that holds the session attributes *before* it sends the SessionDestroyed event. So while our own internal thread has grabbed the lock on the SessionMonitors the logout thread grabbed the lock on the session attribute map. Our SessionMonitor cannot continue to shutdown without getting the session attribute and the logout thread cannot continue with the SessionDestroyed call because the SessionMonitors lock is taken.
-
-
Hide
-
-
META-INF/MANIFEST.MF
-
com/icesoft/.../D2DViewHandler.class
-
com/.../D2DViewHandlerDelegating.class
-
com/.../PartialSubmitPhaseListener.class
-
com/icesoft/.../ProductInfo.class
-
com/icesoft/.../SettableLocaleViewRoot.class
-
com/.../SingleCopyStateManagerImpl.class
-
com/icesoft/.../StartupTime.class
-
com/icesoft/.../TemplateNotFound.class
-
com/icesoft/.../ViewHandlerProxy.class
-
com/icesoft/.../ViewRootHolder.class
-
com/.../ViewRootStateManagerImpl.class
-
com/.../serializedTagToComponentMapFull.ser
-
com/icesoft/.../Util$CaptureArray.class
-
com/icesoft/.../Util$FacetCaptureNode.class
-
com/icesoft/.../Util$TreeCaptureNode.class
-
com/icesoft/faces/.../state/Util.class
-
com/.../PushServerAdaptingServlet$1.class
-
com/.../PushServerAdaptingServlet$2.class
-
com/.../PushServerAdaptingServlet.class
-
com/icesoft/.../render/AsyncRenderer.class
-
com/icesoft/.../BroadcastRenderer.class
-
com/.../ContextDestroyedListener.class
-
com/icesoft/.../render/DelayRenderer.class
-
com/icesoft/.../render/Disposable.class
-
com/icesoft/.../GroupAsyncRenderer$1.class
-
com/icesoft/.../GroupAsyncRenderer.class
-
com/icesoft/.../IntervalRenderer.class
-
com/icesoft/.../OnDemandRenderer.class
-
com/icesoft/.../RejectionHandler.class
-
-
Activity
- All
- Comments
- History
- Activity
- Remote Attachments
- Subversion
The fix I chose was to remove synchronised access on the "contexts" object. Instead a copying ArrayList is used when iterating over the list of contexts. Also, moved back into synchronised block the call to Monitor.addInSessionContext.
By reducing the number of instances where we synchronise access the chances of getting into a deadlock is drastically decreased.
Show
Mircea Toma
added a comment - The fix I chose was to remove synchronised access on the "contexts" object. Instead a copying ArrayList is used when iterating over the list of contexts. Also, moved back into synchronised block the call to Monitor.addInSessionContext.
By reducing the number of instances where we synchronise access the chances of getting into a deadlock is drastically decreased.
To Carsten:
The Monitor.shutdown method is already synchronising access to SessionMonitors object, but it does that inside the method's body.
I attached the patch and also the patched icefaces.jar to this JIRA case. Since these kind of issues are always difficult to reproduce on alternate setups, please try to use the jar and hopefully confirm the fix.
Show
Mircea Toma
added a comment - To Carsten:
The Monitor.shutdown method is already synchronising access to SessionMonitors object, but it does that inside the method's body.
I attached the patch and also the patched icefaces.jar to this JIRA case. Since these kind of issues are always difficult to reproduce on alternate setups, please try to use the jar and hopefully confirm the fix.
Resolving pending customer confirmation that this patch is working as expected.
Verified session expiry on WebLogic8.1 (facelets), JBoss5.1 (jsp), WebLogic12.1.3, glassfish 3.1.2, tomcat 7 (facelets-enh) with component-showcase in IE11, FF34, Chrome43.
Show
Liana Munroe
added a comment - Verified session expiry on WebLogic8.1 (facelets), JBoss5.1 (jsp), WebLogic12.1.3, glassfish 3.1.2, tomcat 7 (facelets-enh) with component-showcase in IE11, FF34, Chrome43.
With the latest Patch there is still a Deadlock-Situation possible.
We had this Situation several Times on our (WebSphere 7) Production Servers.
We are using ICEfaces Version EE-1.8.2.GA_P07
The SessionDispatcher uses two different Objects in synchronized Block ("contexts" and "SessionMonitors").
Now it's possible that both of the Objects are blocked at the same Time.
After that the 2 Threads try to access the Object that is blocked from the other Thread. --> Deadlock.
Here are the two Stacktraces of the blocked Threads:
Thread 1 (already blocking "contexts" and trying to access "SessionMonitors"):
00000053 AlarmThreadMo W UTLS0008W: Die Rückgabe das Alarmthreads "Non-deferrable Alarm : 1" (00000005) an den Alarm-Thread-Pool wurde um 40480 Millisekunden verzögert. Dies kann die normale Alarmfunktion im Anwendungsserver blockieren. Der Stack-Trace des Alarmlisteners ist wie folgt:
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.notifySessionShutdown(SessionDispatcher.java:271)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.access$400(SessionDispatcher.java:73)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Listener.sessionDestroyed(SessionDispatcher.java:361)
at com.icesoft.faces.util.event.servlet.ContextEventRepeater.sessionDestroyed(ContextEventRepeater.java:319)
at com.ibm.ws.session.http.HttpSessionObserver.sessionDestroyed(HttpSessionObserver.java:179)
at com.ibm.ws.session.SessionEventDispatcher.sessionDestroyed(SessionEventDispatcher.java:160)
at com.ibm.ws.session.StoreCallback.sessionInvalidated(StoreCallback.java:126)
Thread 2 (already blocking "SessionMonitors" and trying to access "contexts"):
0000001e ThreadMonitor W WSVR0605W: Der Thread "WebContainer : 8" (00000060) ist seit 232226 Millisekunden aktiv und möglicherweise blockiert. Es sind insgesamt 26 Threads im Server vorhanden, die möglicherweise blockiert sind:
at java.util.Collections$SynchronizedCollection.add(Collections.java:423)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Monitor.addInSessionContext(SessionDispatcher.java:459)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.checkSession(SessionDispatcher.java:142)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.service(SessionDispatcher.java:96)
at com.icesoft.faces.webapp.http.servlet.BlockExpiredSessionRequests.service(BlockExpiredSessionRequests.java:53)
at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:55)
at com.icesoft.faces.webapp.http.servlet.MainServlet.service(MainServlet.java:204)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:71)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1661)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:944)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:507)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)
Now EVERY new WebRequest hungs because it waits on "SesssionMonitors" in the checkSession Method of the SessionDispatcher.
I found a solution to avoid this Deadlock-Situation:
In the shutdownIfExpired() Method of the SessionDispatcher Monitor I added another synchronized Block for "SessionMonitors".
The Thread now only blocks contexts, if there's no other Thread blocking "SessionMonitors".
Here is the Patch for the shutdownIfExpired() Method:
public void shutdownIfExpired() {
{ shutdown(); }if (isExpired()) {
synchronized (SessionMonitors)
}
}