A live thread dump + stack trace showing the deadlock from a clustered JBoss
instance (jboss2) running the ICEfaces.org site.

Found one Java-level deadlock:
=============================
"TP-Processor2326":
waiting to lock monitor 0x089e5c34 (object 0x3565e8d8, a java.lang.String),
which is held by "TP-Processor2603"
"TP-Processor2603":
waiting to lock monitor 0x0839bd14 (object 0x332218e0, a
com.icesoft.faces.webapp.xmlhttp.ResponseStateManager),
which is held by "TP-Processor2326"

Java stack information for the threads listed above:
===================================================
"TP-Processor2326":
at
com.icesoft.faces.webapp.xmlhttp.BlockingResponseState.<init>(BlockingResponseState.java:84)

• waiting to lock <0x3565e8d8> (a java.lang.String)
  at
  com.icesoft.faces.webapp.xmlhttp.ResponseStateManager.createState(ResponseStateManager.java:120)
  at
  com.icesoft.faces.webapp.xmlhttp.ResponseStateManager.getState(ResponseStateManager.java:136)
• locked <0x332218e0> (a
  com.icesoft.faces.webapp.xmlhttp.ResponseStateManager)
  at
  com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:175)
• locked <0x332f66a8> (a com.icesoft.faces.webapp.xmlhttp.BlockingServlet)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
  at
  org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
  at
  org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
  at
  org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
  at
  org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
  at
  org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
  at
  org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
  at
  org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
  at
  org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
  at
  org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
  at
  org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
  at
  org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
  at
  org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
  at
  org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
  at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
  at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
  at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
  at
  org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
  at
  org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
  at
  org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
  at java.lang.Thread.run(Thread.java:595)
  "TP-Processor2603":
  at
  com.icesoft.faces.webapp.xmlhttp.ResponseStateManager.getState(ResponseStateManager.java:131)
• waiting to lock <0x332218e0> (a
  com.icesoft.faces.webapp.xmlhttp.ResponseStateManager)
  at
  com.icesoft.faces.webapp.xmlhttp.PersistentFacesServlet.service(PersistentFacesServlet.java:402)
• locked <0x3565e8d8> (a java.lang.String)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
  at
  org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
  at
  org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
  at
  org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
  at
  org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
  at
  org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
  at
  org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
  at
  org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
  at
  org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
  at
  org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
  at
  org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
  at
  org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
  at
  org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
  at
  org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
  at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
  at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
  at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
  at
  org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
  at
  org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
  at
  org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
  at java.lang.Thread.run(Thread.java:595)

Found 1 deadlock.

Removed synchronized(this) from BlockingServlet and synchronized ResponseStateManager.getState() on
iceID rather than the class.

svn commit src/com/icesoft/faces/webapp/xmlhttp/ResponseStateManager.java src/com/icesoft/faces/
webapp/xmlhttp/BlockingServlet.java -m "synchronizing on iceID (ICE-992)"
Sending src/com/icesoft/faces/webapp/xmlhttp/BlockingServlet.java
Sending src/com/icesoft/faces/webapp/xmlhttp/ResponseStateManager.java
Committed revision 12475.

So was I finally able to reproduce the deadlock synthetically by putting a long
sleep in the PersistentFacesServlet between when the execute and render
lifecycles are fired and when the "state" is put into the session.

What was happening is that when the initial request was made to the
PersistentFacesServlet, it was writing back the complete response before
creating the new state and putting it into the session. Under certain high load
conditions, the BlockingServlet would get invoked and request the state before
the PersistentFacesServlet could get around to it.

To avoid this, I've moved the following chunk of code so that it is executed
before the lifecycle methods are called. That way the state is ready for the
BlockingServlet.

// Set the correct ResponseState as the IncrementalNodeWriter for this
// combination of session and view number, then store the state so that is
// accessible from the async server.
String viewNumberString = String.valueOf(viewNumber);
ResponseState state = stateManager.getState(session, viewNumberString);
state.setFocusID(request.getParameter("focus"));

//ICE-256: We need to include the view number as part of the key
session.setAttribute(viewNumberString + "/" + ResponseState.STATE, state);

Coupled with Ted's changes, this should help us with performance under load.

Marking as FIXED but only real world load testing will fully validate it.

I'm splitting off the change I applied into its own bug report: 1059.

Marking as CLOSED. Please see the more general synchronization ICE-1039.

The following change has been backported to the 1.5 branch:

svn commit . -m "backport synchronizing on iceID (ICE-992)"
Sending core/src/com/icesoft/faces/webapp/xmlhttp/BlockingServlet.java
Sending core/src/com/icesoft/faces/webapp/xmlhttp/ResponseStateManager.java
Transmitting file data ..
Committed revision 12765.

Ted, this should be Resolved Fixed status. I need you to verify that this fix is
in v1.5.2, then mark it verified.

Resolved, needs verification.

BlockingServlet no longer uses synchronized(this) in the head or 1.5 branch; however, the general area of
this bug (deadlock under load) needs further investigation. Marking this as verified with the intent that
future specific bugs in the general area will be opened as necessary.