Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: EE-1.8.2.GA_P03
-
Fix Version/s: EE-1.8.2.GA_P04
-
Component/s: Framework
-
Labels:None
-
Environment:WebSphere
Description
try {
synchronized (SessionMonitors) {
// Iterate over the session monitors using a copying iterator
Iterator iterator = new
ArrayList(SessionMonitors.values()).iterator();
while (iterator.hasNext()) {
final Monitor sessionMonitor = (Monitor) iterator.next();
sessionMonitor.shutdownIfExpired();
ThreadLocalUtility.checkThreadLocals(ThreadLocalUtility.EXITING_SESSION_MONITOR);
}
}
Thread.sleep(10000);
} catch (InterruptedException e) {
//ignore interrupts
}
During the check to see if it should invalidate the session, it checks for a session attribute which, in WebSphere, appears to be a synchronized HashMap:
Object o = session.getAttribute(POSITIVE_SESSION_TIMEOUT);
The other thread is a request to logout which invalidates the current session and then sends out a SessionDestroyed event which we listen for. This, presumably, locks down that same HashMap that holds the session attributes *before* it sends the SessionDestroyed event. So while our own internal thread has grabbed the lock on the SessionMonitors the logout thread grabbed the lock on the session attribute map. Our SessionMonitor cannot continue to shutdown without getting the session attribute and the logout thread cannot continue with the SessionDestroyed call because the SessionMonitors lock is taken.
Activity
- All
- Comments
- History
- Activity
- Remote Attachments
- Subversion
The fix for ICE-6873 which was supposed to synchronize the access to SessionMonitors is synchronizing also code that does not require it.
Synchronize access only during the copying of the session monitor list to avoid including sessionMonitor.shutdownIfExpired() method call.
With the latest Patch there is still a Deadlock-Situation possible.
We had this Situation several Times on our (WebSphere 7) Production Servers.
The SessionDispatcher uses two different Objects in synchronized Block ("contexts" and "SessionMonitors").
Now it's possible that both of the Objects are blocked at the same Time.
After that the 2 Threads try to access the Object that is blocked from the other Thread. --> Deadlock.
Here are the two Stacktraces of the blocked Threads:
Thread 1 (already blocking "contexts" and trying to access "SessionMonitors"):
00000053 AlarmThreadMo W UTLS0008W: Die Rückgabe das Alarmthreads "Non-deferrable Alarm : 1" (00000005) an den Alarm-Thread-Pool wurde um 40480 Millisekunden verzögert. Dies kann die normale Alarmfunktion im Anwendungsserver blockieren. Der Stack-Trace des Alarmlisteners ist wie folgt:
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.notifySessionShutdown(SessionDispatcher.java:271)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.access$400(SessionDispatcher.java:73)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Listener.sessionDestroyed(SessionDispatcher.java:361)
at com.icesoft.faces.util.event.servlet.ContextEventRepeater.sessionDestroyed(ContextEventRepeater.java:319)
at com.ibm.ws.session.http.HttpSessionObserver.sessionDestroyed(HttpSessionObserver.java:179)
at com.ibm.ws.session.SessionEventDispatcher.sessionDestroyed(SessionEventDispatcher.java:160)
at com.ibm.ws.session.StoreCallback.sessionInvalidated(StoreCallback.java:126)
Thread 2 (already blocking "SessionMonitors" and trying to access "contexts"):
0000001e ThreadMonitor W WSVR0605W: Der Thread "WebContainer : 8" (00000060) ist seit 232226 Millisekunden aktiv und möglicherweise blockiert. Es sind insgesamt 26 Threads im Server vorhanden, die möglicherweise blockiert sind:
at java.util.Collections$SynchronizedCollection.add(Collections.java:423)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Monitor.addInSessionContext(SessionDispatcher.java:459)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.checkSession(SessionDispatcher.java:142)
at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.service(SessionDispatcher.java:96)
at com.icesoft.faces.webapp.http.servlet.BlockExpiredSessionRequests.service(BlockExpiredSessionRequests.java:53)
at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:55)
at com.icesoft.faces.webapp.http.servlet.MainServlet.service(MainServlet.java:204)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:71)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1661)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:944)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:507)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)
Now EVERY new WebRequest hungs because it waits on "SesssionMonitors" in the checkSession Method of the SessionDispatcher.
I found a solution to avoid this Deadlock-Situation:
In the shutdownIfExpired() Method of the SessionDispatcher Monitor I added another synchronized Block for "SessionMonitors".
The Thread now only blocks contexts, if there's no other Thread blocking "SessionMonitors".
Here is the Patch for the shutdownIfExpired() Method:
public void shutdownIfExpired() {
if (isExpired()) {
synchronized (SessionMonitors)
}
}
Comment from Ted:
Is it enough to just synchronize the following line?
Iterator iterator = new ArrayList(SessionMonitors.values()).iterator();
That might break the deadlock.