Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: EE-1.8.2.GA_P03
-
Fix Version/s: EE-1.8.2.GA_P08
-
Component/s: Framework
-
Labels:None
-
Environment:WebSphere
-
Assignee Priority:P1
-
Salesforce Case Reference:
Description
When our SessionDispatcher.Listener runs, it checks to see if it should shut the session down. This is done in a thread that runs every 10 seconds and when it checks, it synchonizes on the SessionMonitors while it does so:
try {
synchronized (SessionMonitors) {
// Iterate over the session monitors using a copying iterator
Iterator iterator = new
ArrayList(SessionMonitors.values()).iterator();
while (iterator.hasNext()) {
final Monitor sessionMonitor = (Monitor) iterator.next();
sessionMonitor.shutdownIfExpired();
ThreadLocalUtility.checkThreadLocals(ThreadLocalUtility.EXITING_SESSION_MONITOR);
}
}
Thread.sleep(10000);
} catch (InterruptedException e) {
//ignore interrupts
}
During the check to see if it should invalidate the session, it checks for a session attribute which, in WebSphere, appears to be a synchronized HashMap:
Object o = session.getAttribute(POSITIVE_SESSION_TIMEOUT);
The other thread is a request to logout which invalidates the current session and then sends out a SessionDestroyed event which we listen for. This, presumably, locks down that same HashMap that holds the session attributes *before* it sends the SessionDestroyed event. So while our own internal thread has grabbed the lock on the SessionMonitors the logout thread grabbed the lock on the session attribute map. Our SessionMonitor cannot continue to shutdown without getting the session attribute and the logout thread cannot continue with the SessionDestroyed call because the SessionMonitors lock is taken.
try {
synchronized (SessionMonitors) {
// Iterate over the session monitors using a copying iterator
Iterator iterator = new
ArrayList(SessionMonitors.values()).iterator();
while (iterator.hasNext()) {
final Monitor sessionMonitor = (Monitor) iterator.next();
sessionMonitor.shutdownIfExpired();
ThreadLocalUtility.checkThreadLocals(ThreadLocalUtility.EXITING_SESSION_MONITOR);
}
}
Thread.sleep(10000);
} catch (InterruptedException e) {
//ignore interrupts
}
During the check to see if it should invalidate the session, it checks for a session attribute which, in WebSphere, appears to be a synchronized HashMap:
Object o = session.getAttribute(POSITIVE_SESSION_TIMEOUT);
The other thread is a request to logout which invalidates the current session and then sends out a SessionDestroyed event which we listen for. This, presumably, locks down that same HashMap that holds the session attributes *before* it sends the SessionDestroyed event. So while our own internal thread has grabbed the lock on the SessionMonitors the logout thread grabbed the lock on the session attribute map. Our SessionMonitor cannot continue to shutdown without getting the session attribute and the logout thread cannot continue with the SessionDestroyed call because the SessionMonitors lock is taken.
Activity
Field | Original Value | New Value |
---|---|---|
Salesforce Case Reference | 5007000000Y4X8RAAV |
Fix Version/s | EE-1.8.2.GA_P08 [ 11178 ] | |
Fix Version/s | EE-1.8.2.GA_P04 [ 10280 ] |
Assignee Priority | P1 [ 10010 ] |
Repository | Revision | Date | User | Message |
ICEsoft Public SVN Repository | #40031 | Fri Feb 14 11:55:11 MST 2014 | mircea.toma | |
Files Changed | ||||
![]() |
Attachment | ICE-9804.patch [ 16685 ] | |
Attachment | icefaces.jar [ 16686 ] |
Status | Open [ 1 ] | Resolved [ 5 ] |
Resolution | Fixed [ 1 ] |
Status | Resolved [ 5 ] | Closed [ 6 ] |
With the latest Patch there is still a Deadlock-Situation possible.
We had this Situation several Times on our (WebSphere 7) Production Servers.
We are using ICEfaces Version EE-1.8.2.GA_P07
The SessionDispatcher uses two different Objects in synchronized Block ("contexts" and "SessionMonitors").
Now it's possible that both of the Objects are blocked at the same Time.
After that the 2 Threads try to access the Object that is blocked from the other Thread. --> Deadlock.
Here are the two Stacktraces of the blocked Threads:
Thread 1 (already blocking "contexts" and trying to access "SessionMonitors"):
00000053 AlarmThreadMo W UTLS0008W: Die Rückgabe das Alarmthreads "Non-deferrable Alarm : 1" (00000005) an den Alarm-Thread-Pool wurde um 40480 Millisekunden verzögert. Dies kann die normale Alarmfunktion im Anwendungsserver blockieren. Der Stack-Trace des Alarmlisteners ist wie folgt:
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.notifySessionShutdown(SessionDispatcher.java:271)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.access$400(SessionDispatcher.java:73)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Listener.sessionDestroyed(SessionDispatcher.java:361)
at com.icesoft.faces.util.event.servlet.ContextEventRepeater.sessionDestroyed(ContextEventRepeater.java:319)
at com.ibm.ws.session.http.HttpSessionObserver.sessionDestroyed(HttpSessionObserver.java:179)
at com.ibm.ws.session.SessionEventDispatcher.sessionDestroyed(SessionEventDispatcher.java:160)
at com.ibm.ws.session.StoreCallback.sessionInvalidated(StoreCallback.java:126)
Thread 2 (already blocking "SessionMonitors" and trying to access "contexts"):
0000001e ThreadMonitor W WSVR0605W: Der Thread "WebContainer : 8" (00000060) ist seit 232226 Millisekunden aktiv und möglicherweise blockiert. Es sind insgesamt 26 Threads im Server vorhanden, die möglicherweise blockiert sind:
at java.util.Collections$SynchronizedCollection.add(Collections.java:423)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Monitor.addInSessionContext(SessionDispatcher.java:459)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.checkSession(SessionDispatcher.java:142)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.service(SessionDispatcher.java:96)
at com.icesoft.faces.webapp.http.servlet.BlockExpiredSessionRequests.service(BlockExpiredSessionRequests.java:53)
at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:55)
at com.icesoft.faces.webapp.http.servlet.MainServlet.service(MainServlet.java:204)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:71)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1661)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:944)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:507)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)
Now EVERY new WebRequest hungs because it waits on "SesssionMonitors" in the checkSession Method of the SessionDispatcher.
I found a solution to avoid this Deadlock-Situation:
In the shutdownIfExpired() Method of the SessionDispatcher Monitor I added another synchronized Block for "SessionMonitors".
The Thread now only blocks contexts, if there's no other Thread blocking "SessionMonitors".
Here is the Patch for the shutdownIfExpired() Method:
public void shutdownIfExpired() {
{ shutdown(); }if (isExpired()) {
synchronized (SessionMonitors)
}
}