ICEpush
  1. ICEpush
  2. PUSH-364

Stuck non-container thread using Tomcat 7/8 ARP

          Details

          • Assignee Priority:
            P3
          • Support Case References:
          • Workaround Description:
            Hide
            suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:-
            57779 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57779> : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt)
            57621 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57621> : When an async request completes, ensure that any remaining request body data is swallowed. (markt)
            OR
            disable using tomcat ARP with the following context-param:-
            <context-param>
            <param-name>org.icepush.useAsyncContext</param-name>
            <param-value>false</param-value>
            </context-param>
            Show
            suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:- 57779 < http://bz.apache.org/bugzilla/show_bug.cgi?id=57779 > : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt) 57621 < http://bz.apache.org/bugzilla/show_bug.cgi?id=57621 > : When an async request completes, ensure that any remaining request body data is swallowed. (markt) OR disable using tomcat ARP with the following context-param:- <context-param> <param-name>org.icepush.useAsyncContext</param-name> <param-value>false</param-value> </context-param>

            Description

             "Notification queue consumer." daemon prio=3 tid=0x00000001014f6800 nid=0x17 in Object.wait() [0xffffffff713fe000]
            java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            at java.lang.Object.wait(Object.java:503)
            at org.apache.coyote.AsyncStateMachine.pauseNonContainerThread(AsyncStateMachine.java:382)
            - eliminated <0xfffffffe5f76aae0> (a org.apache.coyote.AsyncStateMachine)
            at org.apache.coyote.AsyncStateMachine.asyncComplete(AsyncStateMachine.java:240)
            - locked <0xfffffffe5f76aae0> (a org.apache.coyote.AsyncStateMachine)
            at org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:356)
            at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:906)
            at org.apache.coyote.Request.action(Request.java:344)
            at org.apache.catalina.core.AsyncContextImpl.complete(AsyncContextImpl.java:93)
            at org.icepush.servlet.AsyncAdaptingServlet$AsyncRequestResponse.respondWith(AsyncAdaptingServlet.java:74)at org.icepush.SequenceTaggingServer$TaggingRequest.respondWith(SequenceTaggingServer.java:67)at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:177)
            at org.icepush.BlockingConnectionServer.respondIfNotificationsAvailable(BlockingConnectionServer.java:149)
            - locked <0xfffffffe69880ba0> (a org.icepush.BlockingConnectionServer)
            at org.icepush.BlockingConnectionServer.sendNotifications(BlockingConnectionServer.java:126)
            at org.icepush.BlockingConnectionServer.receive(BlockingConnectionServer.java:293)
            at org.icepush.LocalNotificationBroadcaster.broadcast(LocalNotificationBroadcaster.java:39)
            at org.icepush.LocalPushGroupManager$Notification.run(LocalPushGroupManager.java:394)
            at org.icepush.LocalPushGroupManager$QueueConsumerTask.run(LocalPushGroupManager.java:432)
            at java.util.TimerThread.mainLoop(Timer.java:555)
            at java.util.TimerThread.run(Timer.java:505)

            The attempt is being made for an incoming request to establish a new blocking connection but it just piles up waiting for the lock.

            What’s unclear is why the “Notification queue consumer.” thread is having trouble completing and/or releasing the lock.
            On our end, we should be looking into the current locking strategy being used by BlockConnectionServer's respondIfBackOffRequested() and respondIfNotificationsAvailable() and possible coming up with a different locking strategy that avoid any future pile-ups of Threads in State BLOCKED.

              Activity

              Ascending order - Click to sort in descending order
              4 older comments
              Hide
              Permalink
              Jack Van Ooststroom added a comment - - edited

              ICEfaces 3

              Sending core/src/main/java/org/icepush/BlockingConnectionServer.java
              Sending core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java
              Transmitting file data ..
              Committed revision 47624.

              Show
              Jack Van Ooststroom added a comment - - edited ICEfaces 3 Sending core/src/main/java/org/icepush/BlockingConnectionServer.java Sending core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java Transmitting file data .. Committed revision 47624.
              Hide
              Permalink
              Jack Van Ooststroom added a comment -

              ICEfaces 4

              Sending core/src/main/java/org/icepush/BlockingConnectionServer.java
              Sending core/src/main/java/org/icepush/Browser.java
              Sending core/src/main/java/org/icepush/ConfirmationTimeout.java
              Sending core/src/main/java/org/icepush/LocalNotificationBroadcaster.java
              Sending core/src/main/java/org/icepush/LocalPushGroupManager.java
              Sending core/src/main/java/org/icepush/NoopPushGroupManager.java
              Sending core/src/main/java/org/icepush/NotificationBroadcaster.java
              Sending core/src/main/java/org/icepush/PushGroupManager.java
              Sending core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java
              Transmitting file data .........
              Committed revision 47948.

              Sending eps/src/main/java/com/icesoft/push/DynamicPushGroupManager.java
              Sending eps/src/main/java/com/icesoft/push/RemotePushGroupManager.java
              Sending eps/src/main/java/com/icesoft/push/messaging/MessagePayload.java
              Transmitting file data ...
              Committed revision 44450.

              Show
              Jack Van Ooststroom added a comment - ICEfaces 4 Sending core/src/main/java/org/icepush/BlockingConnectionServer.java Sending core/src/main/java/org/icepush/Browser.java Sending core/src/main/java/org/icepush/ConfirmationTimeout.java Sending core/src/main/java/org/icepush/LocalNotificationBroadcaster.java Sending core/src/main/java/org/icepush/LocalPushGroupManager.java Sending core/src/main/java/org/icepush/NoopPushGroupManager.java Sending core/src/main/java/org/icepush/NotificationBroadcaster.java Sending core/src/main/java/org/icepush/PushGroupManager.java Sending core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java Transmitting file data ......... Committed revision 47948. Sending eps/src/main/java/com/icesoft/push/DynamicPushGroupManager.java Sending eps/src/main/java/com/icesoft/push/RemotePushGroupManager.java Sending eps/src/main/java/com/icesoft/push/messaging/MessagePayload.java Transmitting file data ... Committed revision 44450.
              Hide
              Permalink
              Jack Van Ooststroom added a comment -

              ICEfaces 4

              Sending icepush-ee/core/src/main/java/org/icepush/BlockingConnectionServer.java
              Transmitting file data .
              Committed revision 47949.

              Show
              Jack Van Ooststroom added a comment - ICEfaces 4 Sending icepush-ee/core/src/main/java/org/icepush/BlockingConnectionServer.java Transmitting file data . Committed revision 47949.
              Hide
              Permalink
              Carmen Cristurean added a comment - - edited

              Emporium app fails on ICEfaces4 trunk r.47948/ Jenkins ICEfaces4 trunk Build # 1819/ all browsers/ server: Tomcat 7.0.42/8.0.21; not an issue on previous Jenkins build: clocks do not work, they tick just once on page load, then stop. Also, the automated bids are not seen to occur. This is not an issue on Jenkins ICEfaces 4.1.1 Build #5.

              This failure can no longer be reproduced on ICEfaces4 trunk Build# 1820 (r. 47949); all tests pass (FF41).

              Show
              Carmen Cristurean added a comment - - edited Emporium app fails on ICEfaces4 trunk r.47948/ Jenkins ICEfaces4 trunk Build # 1819/ all browsers/ server: Tomcat 7.0.42/8.0.21; not an issue on previous Jenkins build: clocks do not work, they tick just once on page load, then stop. Also, the automated bids are not seen to occur. This is not an issue on Jenkins ICEfaces 4.1.1 Build #5. This failure can no longer be reproduced on ICEfaces4 trunk Build# 1820 (r. 47949); all tests pass (FF41).
              Hide
              Permalink
              Jack Van Ooststroom added a comment -

              The BlockingConnectionServer and associated classes have been re-factored to include the following:

              1. An incoming listen.icepush request handled by the BlockingConnectionServer is only added to the pendingPushRequestQueue if the handling container thread is unable to send a response in the end.
              2. The BlockingConnectionServer itself is not used as a lock anymore in order to operate on the backOff or the notifiedPushIDQueue. Instead 2 separate locks have been introduced: one for the backOff and one for the notifiedPushIDQueue access.

              Marking this one as FIXED.

              Show
              Jack Van Ooststroom added a comment - The BlockingConnectionServer and associated classes have been re-factored to include the following: An incoming listen.icepush request handled by the BlockingConnectionServer is only added to the pendingPushRequestQueue if the handling container thread is unable to send a response in the end. The BlockingConnectionServer itself is not used as a lock anymore in order to operate on the backOff or the notifiedPushIDQueue . Instead 2 separate locks have been introduced: one for the backOff and one for the notifiedPushIDQueue access. Marking this one as FIXED.

                People

                • Assignee:
                  Jack Van Ooststroom
                  Reporter:
                  Judy Guglielmin
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: