ICEpush
  1. ICEpush
  2. PUSH-364

Stuck non-container thread using Tomcat 7/8 ARP

    Details

    • Assignee Priority:
      P3
    • Support Case References:
    • Workaround Description:
      Hide
      suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:-
      57779 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57779> : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt)
      57621 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57621> : When an async request completes, ensure that any remaining request body data is swallowed. (markt)
      OR
      disable using tomcat ARP with the following context-param:-
      <context-param>
      <param-name>org.icepush.useAsyncContext</param-name>
      <param-value>false</param-value>
      </context-param>
      Show
      suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:- 57779 < http://bz.apache.org/bugzilla/show_bug.cgi?id=57779 > : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt) 57621 < http://bz.apache.org/bugzilla/show_bug.cgi?id=57621 > : When an async request completes, ensure that any remaining request body data is swallowed. (markt) OR disable using tomcat ARP with the following context-param:- <context-param> <param-name>org.icepush.useAsyncContext</param-name> <param-value>false</param-value> </context-param>

      Description

       "Notification queue consumer." daemon prio=3 tid=0x00000001014f6800 nid=0x17 in Object.wait() [0xffffffff713fe000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:503)
      at org.apache.coyote.AsyncStateMachine.pauseNonContainerThread(AsyncStateMachine.java:382)
      - eliminated <0xfffffffe5f76aae0> (a org.apache.coyote.AsyncStateMachine)
      at org.apache.coyote.AsyncStateMachine.asyncComplete(AsyncStateMachine.java:240)
      - locked <0xfffffffe5f76aae0> (a org.apache.coyote.AsyncStateMachine)
      at org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:356)
      at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:906)
      at org.apache.coyote.Request.action(Request.java:344)
      at org.apache.catalina.core.AsyncContextImpl.complete(AsyncContextImpl.java:93)
      at org.icepush.servlet.AsyncAdaptingServlet$AsyncRequestResponse.respondWith(AsyncAdaptingServlet.java:74)at org.icepush.SequenceTaggingServer$TaggingRequest.respondWith(SequenceTaggingServer.java:67)at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:177)
      at org.icepush.BlockingConnectionServer.respondIfNotificationsAvailable(BlockingConnectionServer.java:149)
      - locked <0xfffffffe69880ba0> (a org.icepush.BlockingConnectionServer)
      at org.icepush.BlockingConnectionServer.sendNotifications(BlockingConnectionServer.java:126)
      at org.icepush.BlockingConnectionServer.receive(BlockingConnectionServer.java:293)
      at org.icepush.LocalNotificationBroadcaster.broadcast(LocalNotificationBroadcaster.java:39)
      at org.icepush.LocalPushGroupManager$Notification.run(LocalPushGroupManager.java:394)
      at org.icepush.LocalPushGroupManager$QueueConsumerTask.run(LocalPushGroupManager.java:432)
      at java.util.TimerThread.mainLoop(Timer.java:555)
      at java.util.TimerThread.run(Timer.java:505)

      The attempt is being made for an incoming request to establish a new blocking connection but it just piles up waiting for the lock.

      What’s unclear is why the “Notification queue consumer.” thread is having trouble completing and/or releasing the lock.
      On our end, we should be looking into the current locking strategy being used by BlockConnectionServer's respondIfBackOffRequested() and respondIfNotificationsAvailable() and possible coming up with a different locking strategy that avoid any future pile-ups of Threads in State BLOCKED.

        Activity

        Repository Revision Date User Message
        ICEsoft Public SVN Repository #51331 Wed Mar 15 10:04:23 MDT 2017 jack.van.ooststroom Backported PUSH-364
        Files Changed
        Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/BlockingConnectionServer.java
        Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/PushGroupManager.java
        Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/ConfirmationTimeout.java
        Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/NoopPushGroupManager.java
        Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/NotificationBroadcaster.java
        Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/LocalPushGroupManager.java
        Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java
        Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/LocalNotificationBroadcaster.java
        Commit graph MODIFY /icepush/branches/icepush-core-4.x-maintenance/icepush/core/src/main/java/org/icepush/Browser.java
        Repository Revision Date User Message
        ICEsoft Public SVN Repository #47949 Thu Feb 18 09:43:47 MST 2016 jack.van.ooststroom Fixed JIRA PUSH-364 : Stuck non-container thread using Tomcat 7/8 ARP
        Files Changed
        Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/BlockingConnectionServer.java
        Repository Revision Date User Message
        ICEsoft Public SVN Repository #47948 Thu Feb 18 08:08:05 MST 2016 jack.van.ooststroom Fixed JIRA PUSH-364 : Stuck non-container thread using Tomcat 7/8 ARP
        Files Changed
        Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/Browser.java
        Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/PushGroupManager.java
        Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/LocalPushGroupManager.java
        Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/LocalNotificationBroadcaster.java
        Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java
        Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/NoopPushGroupManager.java
        Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/NotificationBroadcaster.java
        Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/BlockingConnectionServer.java
        Commit graph MODIFY /icepush/trunk/icepush/core/src/main/java/org/icepush/ConfirmationTimeout.java

          People

          • Assignee:
            Jack Van Ooststroom
            Reporter:
            Judy Guglielmin
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: