Not quite certain of the full extent of this. I'll be investigating a bit more. But certainly a problem with asynchronous ICEFaces.

An initial problem was a deadlock between ICEfaces server initiated rendering threads, and HTTP threads with incoming requests under high loads. The initial problem was thought to be easy to diagnose. HTTP threads and server initiated rendering threads obtain a lock on BridgeFacesContext objects, but then Spring webflow locks the webflow context when executing the state machine for any webflow.

In our PersistentFacesContext object, we have execute() and render() methods that both perform synchronization. It's possible then to:


Thread 1 - lock FacesContext in execute()
Thread 1 - lock Spring webflow executor state object
Thread 1 - unlock FacesContext after execute()
Thread 2 - Lock FacesContext in execute()
Thread 2 - Wait for Webflow Lock
Thread 1 - Wait for FacesContext lock
Deadlock.


As an experiment, I created an executeAndRender method that does the whole combo lifecycle thing under one lock, which should hold both the FacesContext and the Executor lock for the duration of the lifecycle, and which should have worked.

But I then obtained a deadlock between two HTTP threads, and it's not clear what exactly is the issue. Deadlock traces are as follows:


"http-8080-2" daemon prio=6 tid=0x0befed30 nid=0x704 waiting for monitor entry [0x0cf4f000..0x0cf4fce8]
        at com.icesoft.faces.webapp.http.core.ReceiveSendUpdates.renderCyclePartial(ReceiveSendUpdates.java:65)
        - waiting to lock <0x03673fb8> (a com.icesoft.faces.context.BridgeFacesContext)
        at com.icesoft.faces.webapp.http.core.ReceiveSendUpdates.service(ReceiveSendUpdates.java:43)
        at com.icesoft.faces.webapp.http.core.IDVerifier.service(IDVerifier.java:20)
        at com.icesoft.faces.webapp.http.servlet.BasicAdaptingServlet.service(BasicAdaptingServlet.java:16)
        at com.icesoft.faces.webapp.http.servlet.ViewBoundAdaptingServlet.service(ViewBoundAdaptingServlet.java:28)
        at com.icesoft.faces.webapp.http.servlet.PathDispatcher$Matcher.serviceOnMatch(PathDispatcher.java:52)
        at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:29)
        at com.icesoft.faces.webapp.http.servlet.MainSessionBoundServlet.service(MainSessionBoundServlet.java:93)
        at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.service(SessionDispatcher.java:35)
        at com.icesoft.faces.webapp.http.servlet.PathDispatcher$Matcher.serviceOnMatch(PathDispatcher.java:52)
        at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:29)
        at com.icesoft.faces.webapp.http.servlet.MainServlet.service(MainServlet.java:65)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:343)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:292)
        at com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:54)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:595)


And this is the thread in the deadlock state:


"http-8080-1" daemon prio=6 tid=0x0b727fe8 nid=0x1a4 waiting on condition [0x0be8e000..0x0be8fa68]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:681)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:711)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1041)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
        at org.springframework.webflow.conversation.impl.JdkConcurrentConversationLock.lock(JdkConcurrentConversationLock.java:36)
        at org.springframework.webflow.conversation.impl.ContainedConversation.lock(ContainedConversation.java:72)
        at org.springframework.webflow.execution.repository.support.ConversationBackedFlowExecutionLock.lock(ConversationBackedFlowExecutionLock.java:54)
        at org.springframework.webflow.executor.jsf.FlowPhaseListener.restoreFlowExecution(FlowPhaseListener.java:269)
        at org.springframework.webflow.executor.jsf.FlowPhaseListener.beforePhase(FlowPhaseListener.java:227)
        at org.apache.myfaces.lifecycle.PhaseListenerManager.informPhaseListenersBefore(PhaseListenerManager.java:70)
        at org.apache.myfaces.lifecycle.LifecycleImpl.restoreView(LifecycleImpl.java:103)
        at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:66)
        at com.icesoft.faces.webapp.http.core.ReceiveSendUpdates.renderCyclePartial(ReceiveSendUpdates.java:68)
        - locked <0x03673fb8> (a com.icesoft.faces.context.BridgeFacesContext)
        at com.icesoft.faces.webapp.http.core.ReceiveSendUpdates.service(ReceiveSendUpdates.java:43)
        at com.icesoft.faces.webapp.http.core.IDVerifier.service(IDVerifier.java:20)
        at com.icesoft.faces.webapp.http.servlet.BasicAdaptingServlet.service(BasicAdaptingServlet.java:16)
        at com.icesoft.faces.webapp.http.servlet.ViewBoundAdaptingServlet.service(ViewBoundAdaptingServlet.java:28)
        at com.icesoft.faces.webapp.http.servlet.PathDispatcher$Matcher.serviceOnMatch(PathDispatcher.java:52)
        at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:29)
        at com.icesoft.faces.webapp.http.servlet.MainSessionBoundServlet.service(MainSessionBoundServlet.java:93)
        at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.service(SessionDispatcher.java:35)
        at com.icesoft.faces.webapp.http.servlet.PathDispatcher$Matcher.serviceOnMatch(PathDispatcher.java:52)
        at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:29)
        at com.icesoft.faces.webapp.http.servlet.MainServlet.service(MainServlet.java:65)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:343)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:292)
        at com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:54)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:595)

Thread HTTP-1 is holding the lock on FacesContext, but I don't know what it's waiting for. Note that it is 'waiting on condition', not 'waiting to lock'. It's almost as if there is something not quite correctly implemented in the Unsafe java class, for which, interestingly, source is not available.