Details
Description
In order to apply this update to the 1.5 branch, I've broken this issue off from
it's original home inICE-992.
The problem typically only occurs under heavy load but can be generated
synthetically by putting a long (20 sec) sleep in the correct location of the
PersistentFacesServlet.
A live thread dump + stack trace showing the deadlock from a clustered JBoss
instance (jboss2) running the ICEfaces.org site.
Found one Java-level deadlock:
=============================
"TP-Processor2326":
waiting to lock monitor 0x089e5c34 (object 0x3565e8d8, a java.lang.String),
which is held by "TP-Processor2603"
"TP-Processor2603":
waiting to lock monitor 0x0839bd14 (object 0x332218e0, a
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager),
which is held by "TP-Processor2326"
Java stack information for the threads listed above:
===================================================
"TP-Processor2326":
at
com.icesoft.faces.webapp.xmlhttp.BlockingResponseState.<init>(BlockingResponseState.java:84)
- waiting to lock <0x3565e8d8> (a java.lang.String)
at
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager.createState(ResponseStateManager.java:120)
at
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager.getState(ResponseStateManager.java:136)
- locked <0x332218e0> (a
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager)
at
com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:175)
- locked <0x332f66a8> (a com.icesoft.faces.webapp.xmlhttp.BlockingServlet)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
at
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
"TP-Processor2603":
at
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager.getState(ResponseStateManager.java:131)
- waiting to lock <0x332218e0> (a
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager)
at
com.icesoft.faces.webapp.xmlhttp.PersistentFacesServlet.service(PersistentFacesServlet.java:402)
- locked <0x3565e8d8> (a java.lang.String)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
at
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Found 1 deadlock.
it's original home in
The problem typically only occurs under heavy load but can be generated
synthetically by putting a long (20 sec) sleep in the correct location of the
PersistentFacesServlet.
A live thread dump + stack trace showing the deadlock from a clustered JBoss
instance (jboss2) running the ICEfaces.org site.
Found one Java-level deadlock:
=============================
"TP-Processor2326":
waiting to lock monitor 0x089e5c34 (object 0x3565e8d8, a java.lang.String),
which is held by "TP-Processor2603"
"TP-Processor2603":
waiting to lock monitor 0x0839bd14 (object 0x332218e0, a
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager),
which is held by "TP-Processor2326"
Java stack information for the threads listed above:
===================================================
"TP-Processor2326":
at
com.icesoft.faces.webapp.xmlhttp.BlockingResponseState.<init>(BlockingResponseState.java:84)
- waiting to lock <0x3565e8d8> (a java.lang.String)
at
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager.createState(ResponseStateManager.java:120)
at
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager.getState(ResponseStateManager.java:136)
- locked <0x332218e0> (a
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager)
at
com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:175)
- locked <0x332f66a8> (a com.icesoft.faces.webapp.xmlhttp.BlockingServlet)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
at
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
"TP-Processor2603":
at
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager.getState(ResponseStateManager.java:131)
- waiting to lock <0x332218e0> (a
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager)
at
com.icesoft.faces.webapp.xmlhttp.PersistentFacesServlet.service(PersistentFacesServlet.java:402)
- locked <0x3565e8d8> (a java.lang.String)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
at
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Found 1 deadlock.
I've checked the 1.5 branch in the repository, the src build, and decompiled
PersistentFacesServlet in the icefaces.jar of the binary build and the fix is
there in all of them. For this release, this is what we are considering as
verified so I'm marking it as such.