ICEpush
  1. ICEpush
  2. PUSH-364

Stuck non-container thread using Tomcat 7/8 ARP

          Details

          • Assignee Priority:
            P3
          • Support Case References:
          • Workaround Description:
            Hide
            suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:-
            57779 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57779> : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt)
            57621 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57621> : When an async request completes, ensure that any remaining request body data is swallowed. (markt)
            OR
            disable using tomcat ARP with the following context-param:-
            <context-param>
            <param-name>org.icepush.useAsyncContext</param-name>
            <param-value>false</param-value>
            </context-param>
            Show
            suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:- 57779 < http://bz.apache.org/bugzilla/show_bug.cgi?id=57779 > : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt) 57621 < http://bz.apache.org/bugzilla/show_bug.cgi?id=57621 > : When an async request completes, ensure that any remaining request body data is swallowed. (markt) OR disable using tomcat ARP with the following context-param:- <context-param> <param-name>org.icepush.useAsyncContext</param-name> <param-value>false</param-value> </context-param>

            Description

             "Notification queue consumer." daemon prio=3 tid=0x00000001014f6800 nid=0x17 in Object.wait() [0xffffffff713fe000]
            java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            at java.lang.Object.wait(Object.java:503)
            at org.apache.coyote.AsyncStateMachine.pauseNonContainerThread(AsyncStateMachine.java:382)
            - eliminated <0xfffffffe5f76aae0> (a org.apache.coyote.AsyncStateMachine)
            at org.apache.coyote.AsyncStateMachine.asyncComplete(AsyncStateMachine.java:240)
            - locked <0xfffffffe5f76aae0> (a org.apache.coyote.AsyncStateMachine)
            at org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:356)
            at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:906)
            at org.apache.coyote.Request.action(Request.java:344)
            at org.apache.catalina.core.AsyncContextImpl.complete(AsyncContextImpl.java:93)
            at org.icepush.servlet.AsyncAdaptingServlet$AsyncRequestResponse.respondWith(AsyncAdaptingServlet.java:74)at org.icepush.SequenceTaggingServer$TaggingRequest.respondWith(SequenceTaggingServer.java:67)at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:177)
            at org.icepush.BlockingConnectionServer.respondIfNotificationsAvailable(BlockingConnectionServer.java:149)
            - locked <0xfffffffe69880ba0> (a org.icepush.BlockingConnectionServer)
            at org.icepush.BlockingConnectionServer.sendNotifications(BlockingConnectionServer.java:126)
            at org.icepush.BlockingConnectionServer.receive(BlockingConnectionServer.java:293)
            at org.icepush.LocalNotificationBroadcaster.broadcast(LocalNotificationBroadcaster.java:39)
            at org.icepush.LocalPushGroupManager$Notification.run(LocalPushGroupManager.java:394)
            at org.icepush.LocalPushGroupManager$QueueConsumerTask.run(LocalPushGroupManager.java:432)
            at java.util.TimerThread.mainLoop(Timer.java:555)
            at java.util.TimerThread.run(Timer.java:505)

            The attempt is being made for an incoming request to establish a new blocking connection but it just piles up waiting for the lock.

            What’s unclear is why the “Notification queue consumer.” thread is having trouble completing and/or releasing the lock.
            On our end, we should be looking into the current locking strategy being used by BlockConnectionServer's respondIfBackOffRequested() and respondIfNotificationsAvailable() and possible coming up with a different locking strategy that avoid any future pile-ups of Threads in State BLOCKED.

              Activity

              Repository Revision Date User Message
              ICEsoft Public SVN Repository #51331 Wed Mar 15 10:04:23 MDT 2017 jack.van.ooststroom Backported PUSH-364
              Files Changed
              Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/BlockingConnectionServer.java
              Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/PushGroupManager.java
              Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/ConfirmationTimeout.java
              Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/NoopPushGroupManager.java
              Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/NotificationBroadcaster.java
              Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/LocalPushGroupManager.java
              Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java
              Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/LocalNotificationBroadcaster.java
              Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/Browser.java
              Repository Revision Date User Message
              ICEsoft Public SVN Repository #47949 Thu Feb 18 09:43:47 MST 2016 jack.van.ooststroom Fixed JIRA PUSH-364 : Stuck non-container thread using Tomcat 7/8 ARP
              Files Changed
              Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/BlockingConnectionServer.java
              Repository Revision Date User Message
              ICEsoft Public SVN Repository #47948 Thu Feb 18 08:08:05 MST 2016 jack.van.ooststroom Fixed JIRA PUSH-364 : Stuck non-container thread using Tomcat 7/8 ARP
              Files Changed
              Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/Browser.java
              Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/PushGroupManager.java
              Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/LocalPushGroupManager.java
              Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/LocalNotificationBroadcaster.java
              Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java
              Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/NoopPushGroupManager.java
              Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/NotificationBroadcaster.java
              Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/BlockingConnectionServer.java
              Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/ConfirmationTimeout.java

                People

                • Assignee:
                  Jack Van Ooststroom
                  Reporter:
                  Judy Guglielmin
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: