ICEpush
  1. ICEpush
  2. PUSH-364

Stuck non-container thread using Tomcat 7/8 ARP

    Details

    • Assignee Priority:
      P3
    • Support Case References:
    • Workaround Description:
      Hide
      suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:-
      57779 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57779> : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt)
      57621 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57621> : When an async request completes, ensure that any remaining request body data is swallowed. (markt)
      OR
      disable using tomcat ARP with the following context-param:-
      <context-param>
      <param-name>org.icepush.useAsyncContext</param-name>
      <param-value>false</param-value>
      </context-param>
      Show
      suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:- 57779 < http://bz.apache.org/bugzilla/show_bug.cgi?id=57779 > : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt) 57621 < http://bz.apache.org/bugzilla/show_bug.cgi?id=57621 > : When an async request completes, ensure that any remaining request body data is swallowed. (markt) OR disable using tomcat ARP with the following context-param:- <context-param> <param-name>org.icepush.useAsyncContext</param-name> <param-value>false</param-value> </context-param>

      Description

       "Notification queue consumer." daemon prio=3 tid=0x00000001014f6800 nid=0x17 in Object.wait() [0xffffffff713fe000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:503)
      at org.apache.coyote.AsyncStateMachine.pauseNonContainerThread(AsyncStateMachine.java:382)
      - eliminated <0xfffffffe5f76aae0> (a org.apache.coyote.AsyncStateMachine)
      at org.apache.coyote.AsyncStateMachine.asyncComplete(AsyncStateMachine.java:240)
      - locked <0xfffffffe5f76aae0> (a org.apache.coyote.AsyncStateMachine)
      at org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:356)
      at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:906)
      at org.apache.coyote.Request.action(Request.java:344)
      at org.apache.catalina.core.AsyncContextImpl.complete(AsyncContextImpl.java:93)
      at org.icepush.servlet.AsyncAdaptingServlet$AsyncRequestResponse.respondWith(AsyncAdaptingServlet.java:74)at org.icepush.SequenceTaggingServer$TaggingRequest.respondWith(SequenceTaggingServer.java:67)at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:177)
      at org.icepush.BlockingConnectionServer.respondIfNotificationsAvailable(BlockingConnectionServer.java:149)
      - locked <0xfffffffe69880ba0> (a org.icepush.BlockingConnectionServer)
      at org.icepush.BlockingConnectionServer.sendNotifications(BlockingConnectionServer.java:126)
      at org.icepush.BlockingConnectionServer.receive(BlockingConnectionServer.java:293)
      at org.icepush.LocalNotificationBroadcaster.broadcast(LocalNotificationBroadcaster.java:39)
      at org.icepush.LocalPushGroupManager$Notification.run(LocalPushGroupManager.java:394)
      at org.icepush.LocalPushGroupManager$QueueConsumerTask.run(LocalPushGroupManager.java:432)
      at java.util.TimerThread.mainLoop(Timer.java:555)
      at java.util.TimerThread.run(Timer.java:505)

      The attempt is being made for an incoming request to establish a new blocking connection but it just piles up waiting for the lock.

      What’s unclear is why the “Notification queue consumer.” thread is having trouble completing and/or releasing the lock.
      On our end, we should be looking into the current locking strategy being used by BlockConnectionServer's respondIfBackOffRequested() and respondIfNotificationsAvailable() and possible coming up with a different locking strategy that avoid any future pile-ups of Threads in State BLOCKED.

        Activity

        Hide
        Jack Van Ooststroom added a comment - - edited

        ICEfaces 3

        Sending core/src/main/java/org/icepush/BlockingConnectionServer.java
        Sending core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java
        Transmitting file data ..
        Committed revision 47624.

        Show
        Jack Van Ooststroom added a comment - - edited ICEfaces 3 Sending core/src/main/java/org/icepush/BlockingConnectionServer.java Sending core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java Transmitting file data .. Committed revision 47624.
        Hide
        Jack Van Ooststroom added a comment -

        ICEfaces 4

        Sending core/src/main/java/org/icepush/BlockingConnectionServer.java
        Sending core/src/main/java/org/icepush/Browser.java
        Sending core/src/main/java/org/icepush/ConfirmationTimeout.java
        Sending core/src/main/java/org/icepush/LocalNotificationBroadcaster.java
        Sending core/src/main/java/org/icepush/LocalPushGroupManager.java
        Sending core/src/main/java/org/icepush/NoopPushGroupManager.java
        Sending core/src/main/java/org/icepush/NotificationBroadcaster.java
        Sending core/src/main/java/org/icepush/PushGroupManager.java
        Sending core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java
        Transmitting file data .........
        Committed revision 47948.

        Sending eps/src/main/java/com/icesoft/push/DynamicPushGroupManager.java
        Sending eps/src/main/java/com/icesoft/push/RemotePushGroupManager.java
        Sending eps/src/main/java/com/icesoft/push/messaging/MessagePayload.java
        Transmitting file data ...
        Committed revision 44450.

        Show
        Jack Van Ooststroom added a comment - ICEfaces 4 Sending core/src/main/java/org/icepush/BlockingConnectionServer.java Sending core/src/main/java/org/icepush/Browser.java Sending core/src/main/java/org/icepush/ConfirmationTimeout.java Sending core/src/main/java/org/icepush/LocalNotificationBroadcaster.java Sending core/src/main/java/org/icepush/LocalPushGroupManager.java Sending core/src/main/java/org/icepush/NoopPushGroupManager.java Sending core/src/main/java/org/icepush/NotificationBroadcaster.java Sending core/src/main/java/org/icepush/PushGroupManager.java Sending core/src/main/java/org/icepush/servlet/AsyncAdaptingServlet.java Transmitting file data ......... Committed revision 47948. Sending eps/src/main/java/com/icesoft/push/DynamicPushGroupManager.java Sending eps/src/main/java/com/icesoft/push/RemotePushGroupManager.java Sending eps/src/main/java/com/icesoft/push/messaging/MessagePayload.java Transmitting file data ... Committed revision 44450.
        Hide
        Jack Van Ooststroom added a comment -

        ICEfaces 4

        Sending icepush-ee/core/src/main/java/org/icepush/BlockingConnectionServer.java
        Transmitting file data .
        Committed revision 47949.

        Show
        Jack Van Ooststroom added a comment - ICEfaces 4 Sending icepush-ee/core/src/main/java/org/icepush/BlockingConnectionServer.java Transmitting file data . Committed revision 47949.
        Hide
        Carmen Cristurean added a comment - - edited

        Emporium app fails on ICEfaces4 trunk r.47948/ Jenkins ICEfaces4 trunk Build # 1819/ all browsers/ server: Tomcat 7.0.42/8.0.21; not an issue on previous Jenkins build: clocks do not work, they tick just once on page load, then stop. Also, the automated bids are not seen to occur. This is not an issue on Jenkins ICEfaces 4.1.1 Build #5.

        This failure can no longer be reproduced on ICEfaces4 trunk Build# 1820 (r. 47949); all tests pass (FF41).

        Show
        Carmen Cristurean added a comment - - edited Emporium app fails on ICEfaces4 trunk r.47948/ Jenkins ICEfaces4 trunk Build # 1819/ all browsers/ server: Tomcat 7.0.42/8.0.21; not an issue on previous Jenkins build: clocks do not work, they tick just once on page load, then stop. Also, the automated bids are not seen to occur. This is not an issue on Jenkins ICEfaces 4.1.1 Build #5. This failure can no longer be reproduced on ICEfaces4 trunk Build# 1820 (r. 47949); all tests pass (FF41).
        Hide
        Jack Van Ooststroom added a comment -

        The BlockingConnectionServer and associated classes have been re-factored to include the following:

        1. An incoming listen.icepush request handled by the BlockingConnectionServer is only added to the pendingPushRequestQueue if the handling container thread is unable to send a response in the end.
        2. The BlockingConnectionServer itself is not used as a lock anymore in order to operate on the backOff or the notifiedPushIDQueue. Instead 2 separate locks have been introduced: one for the backOff and one for the notifiedPushIDQueue access.

        Marking this one as FIXED.

        Show
        Jack Van Ooststroom added a comment - The BlockingConnectionServer and associated classes have been re-factored to include the following: An incoming listen.icepush request handled by the BlockingConnectionServer is only added to the pendingPushRequestQueue if the handling container thread is unable to send a response in the end. The BlockingConnectionServer itself is not used as a lock anymore in order to operate on the backOff or the notifiedPushIDQueue . Instead 2 separate locks have been introduced: one for the backOff and one for the notifiedPushIDQueue access. Marking this one as FIXED.

          People

          • Assignee:
            Jack Van Ooststroom
            Reporter:
            Judy Guglielmin
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: