[PUSH-364] Stuck non-container thread using Tomcat 7/8 ARP - ICEsoft JIRA Issue Tracker

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: EE-4.0.0.GA, EE-3.3.0.GA_P03
Fix Version/s: EE-4.1.0.RC1, EE-4.1.0.GA, EE-3.3.0.GA_P04, 4.2.BETA, 4.2
Component/s: Push Library
Labels:
None
Environment:
Tomcat 7, pre 7.0.62 and Tomcat8 pre 8.0.22 releases

Assignee Priority:
P3
Support Case References:
Support Case 13470:- https://icesoft.my.salesforce.com/5007000001GBRTQ
Workaround Description:

Hide
suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:-
57779 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57779> : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt)
57621 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57621> : When an async request completes, ensure that any remaining request body data is swallowed. (markt)
OR
disable using tomcat ARP with the following context-param:-
<context-param>
<param-name>org.icepush.useAsyncContext</param-name>
<param-value>false</param-value>
</context-param>

Show
suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:- 57779 < http://bz.apache.org/bugzilla/show_bug.cgi?id=57779 > : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt) 57621 < http://bz.apache.org/bugzilla/show_bug.cgi?id=57621 > : When an async request completes, ensure that any remaining request body data is swallowed. (markt) OR disable using tomcat ARP with the following context-param:- <context-param> <param-name>org.icepush.useAsyncContext</param-name> <param-value>false</param-value> </context-param>

Description

"Notification queue consumer." daemon prio=3 tid=0x00000001014f6800 nid=0x17 in Object.wait() [0xffffffff713fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.coyote.AsyncStateMachine.pauseNonContainerThread(AsyncStateMachine.java:382)
- eliminated <0xfffffffe5f76aae0> (a org.apache.coyote.AsyncStateMachine)
at org.apache.coyote.AsyncStateMachine.asyncComplete(AsyncStateMachine.java:240)
- locked <0xfffffffe5f76aae0> (a org.apache.coyote.AsyncStateMachine)
at org.apache.coyote.http11.Http11Processor.actionInternal(Http11Processor.java:356)
at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:906)
at org.apache.coyote.Request.action(Request.java:344)
at org.apache.catalina.core.AsyncContextImpl.complete(AsyncContextImpl.java:93)
at org.icepush.servlet.AsyncAdaptingServlet$AsyncRequestResponse.respondWith(AsyncAdaptingServlet.java:74)at org.icepush.SequenceTaggingServer$TaggingRequest.respondWith(SequenceTaggingServer.java:67)at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:177)
at org.icepush.BlockingConnectionServer.respondIfNotificationsAvailable(BlockingConnectionServer.java:149)
- locked <0xfffffffe69880ba0> (a org.icepush.BlockingConnectionServer)
at org.icepush.BlockingConnectionServer.sendNotifications(BlockingConnectionServer.java:126)
at org.icepush.BlockingConnectionServer.receive(BlockingConnectionServer.java:293)
at org.icepush.LocalNotificationBroadcaster.broadcast(LocalNotificationBroadcaster.java:39)
at org.icepush.LocalPushGroupManager$Notification.run(LocalPushGroupManager.java:394)
at org.icepush.LocalPushGroupManager$QueueConsumerTask.run(LocalPushGroupManager.java:432)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

The attempt is being made for an incoming request to establish a new blocking connection but it just piles up waiting for the lock.

What’s unclear is why the “Notification queue consumer.” thread is having trouble completing and/or releasing the lock.
On our end, we should be looking into the current locking strategy being used by BlockConnectionServer's respondIfBackOffRequested() and respondIfNotificationsAvailable() and possible coming up with a different locking strategy that avoid any future pile-ups of Threads in State BLOCKED.

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Download All

Attachments

after-thread-dump.txt

01/Feb/16 5:13 PM

33 kB

Arran Mccullough
before-thread-dump.txt

01/Feb/16 5:13 PM

16 kB

Arran Mccullough
Hide
Case13690Example.war

01/Feb/16 5:13 PM

3.53 MB

Arran Mccullough
META-INF/MANIFEST.MF 0.1 kB

META-INF/context.xml 0.1 kB

WEB-INF/classes/MyPhaseListener.class 0.9 kB

WEB-INF/classes/PushView.class 2 kB

WEB-INF/classes/TestBean.class 0.8 kB

WEB-INF/faces-config.xml 0.4 kB

WEB-INF/lib/icefaces-ee.jar 633 kB

WEB-INF/lib/icepush-ee.jar 376 kB

WEB-INF/lib/javax.faces.jar 2.54 MB

WEB-INF/web.xml 2 kB

index.xhtml 0.8 kB

mainTemplate.xhtml 0.4 kB

Download Zip
Show

Case13690Example.war

01/Feb/16 5:13 PM

3.53 MB

Arran Mccullough
Case13690Example.zip

01/Feb/16 5:13 PM

21 kB

Arran Mccullough

Activity

Ascending order - Click to sort in descending order

Judy Guglielmin created issue - 21/Jul/15 10:25 AM

Judy Guglielmin made changes - 21/Jul/15 10:27 AM

Field	Original Value	New Value
Workaround Description		use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:- 57779 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57779> : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt) 57621 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57621> : When an async request completes, ensure that any remaining request body data is swallowed. (markt)

Judy Guglielmin made changes - 21/Jul/15 10:28 AM

Workaround Description

use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:-
57779 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57779> : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt)
57621 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57621> : When an async request completes, ensure that any remaining request body data is swallowed. (markt)

Judy Guglielmin made changes - 21/Jul/15 10:32 AM

Assignee

Jack Van Ooststroom [ jack.van.ooststroom ]

Judy Guglielmin made changes - 21/Jul/15 1:22 PM

Workaround Description

suggested workarounds- but as of yet unconfirmed--use newer tomcat versions 7.0.62 or newer or 8.0.22 or newer that include the following 2 apache-tomcat fixes:-
57779 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57779> : When an I/O error occurs on a non-container thread only dispatch to a container thread to handle the error if using Servlet 3+ asynchronous processing. This avoids potential deadlocks if an application is performing I/O on a non-container thread without using the Servlet 3+ asynchronous API. (markt)
57621 <http://bz.apache.org/bugzilla/show_bug.cgi?id=57621> : When an async request completes, ensure that any remaining request body data is swallowed. (markt)
OR
disable using tomcat ARP with the following context-param:-
<context-param>
<param-name>org.icepush.useAsyncContext</param-name>
<param-value>false</param-value>
</context-param>

Ken Fyten made changes - 04/Aug/15 11:09 AM

Fix Version/s		4.1 [ 11570 ]
Fix Version/s		EE-3.3.0.GA_P04 [ 12271 ]
Assignee Priority		P3 [ 10012 ]

Ken Fyten made changes - 05/Oct/15 2:07 PM

Fix Version/s		EE-4.1.0.GA [ 12172 ]
Fix Version/s	4.1 [ 11570 ]

Arran Mccullough made changes - 01/Feb/16 5:13 PM

Attachment		Case13690Example.war [ 22054 ]
Attachment		Case13690Example.zip [ 22055 ]
Attachment		after-thread-dump.txt [ 22056 ]
Attachment		before-thread-dump.txt [ 22057 ]

Jack Van Ooststroom made changes - 10/Feb/16 11:42 AM

Summary

Manage issue with older tomcat 7 & 8 ARP with blocking connection

Stuck non-container thread using Tomcat 7/8 ARP

Jack Van Ooststroom made changes - 02/Mar/16 8:34 AM

Status	Open [ 1 ]	Resolved [ 5 ]
Resolution		Fixed [ 1 ]

Ken Fyten made changes - 29/Apr/16 10:31 AM

Fix Version/s

EE-4.1.0.BETA [ 13073 ]

Ken Fyten made changes - 29/Apr/16 3:11 PM

Issue Type

Improvement [ 4 ]

Bug [ 1 ]

Ken Fyten made changes - 22/Dec/16 10:53 AM

Fix Version/s		4.2.BETA [ 13092 ]
Fix Version/s		4.2 [ 12871 ]

Ken Fyten made changes - 13/Mar/17 1:32 PM

Status

Resolved [ 5 ]

Closed [ 6 ]

People

Assignee:

Jack Van Ooststroom

Reporter:

Judy Guglielmin

Votes:

0 Vote for this issue

Watchers:

5 Start watching this issue

Dates

Created:

21/Jul/15 10:25 AM

Updated:

13/Mar/17 1:32 PM

Resolved:

02/Mar/16 8:34 AM