Details
-
Type: Bug
-
Status: Closed
-
Priority: Major
-
Resolution: Fixed
-
Affects Version/s: EE-3.3.0.GA_P04
-
Fix Version/s: EE-4.2.0.GA, EE-3.3.0.GA_P05, 4.3
-
Component/s: ACE-Components, Framework
-
Labels:None
-
Environment:Tomcat 8, ICEfaces EE 3.3.0 P04
-
Assignee Priority:P2
-
Support Case References:Support Case #14043 - https://icesoft.my.salesforce.com/5007000001iIX1I
Description
A customer has been doing some performance testing of their application using ICEfaces EE 3.3.0 P04. After uploading many files they are seeing a drastic reduction in performance of the application.
Here is their analysis:
In our performance tests with P04 we encountered heavy performance issues!
We have the <ace:fileEntry> component in our application to allow users to upload files.
Our performance tests are uploading about 180 different files within 8 hours. We've run 6 runs of those 8 hours period and the tomcat performance has become really bad.
Actions that usually took milliseconds do now need several seconds (just changes in the view without backend interaction).
We took a heapdump from the tomcat.
In the heapdump (see attached file FileEntryFormSubmit_ReEnableCaptureSubmit_all_ice_classes.PNG) you will find 146.876 "ReEnableCaptureSubmit.class" instances.
In comparison to other objects in the heap, this amount of objects is very large.
While the performance tests are continueing (with horrible respond times in all userinterfaces (xhtmls), not just at the part where the upload component is used) we took several threaddumps from tomcat. (see attached threaddump_3793_170301_0839.txt)
In those threaddumps you can find RUNNABLE threads that look like this:
"http-bio-13081-exec-2861" #13041 daemon prio=5 os_prio=64 tid=0x0000000000d9d800 nid=0x32f3 runnable [0xffff80ff9f1e7000]
java.lang.Thread.State: RUNNABLE
at javax.faces.component.UIComponentBase.getFacetsAndChildren(UIComponentBase.java:721)
at javax.faces.component.UIComponentBase.findComponent(UIComponentBase.java:641)
at javax.faces.component.UIComponentBase.findComponent(UIComponentBase.java:648)
at javax.faces.component.UIComponentBase.findComponent(UIComponentBase.java:648)
at javax.faces.component.UIComponentBase.findComponent(UIComponentBase.java:600)
at org.icefaces.ace.component.fileentry.FileEntryFormSubmit$ReEnableCaptureSubmit.processEvent(FileEntryFormSubmit.java:223)
at javax.faces.event.SystemEvent.processListener(SystemEvent.java:106)
...
...
- locked <0x0000000776d06110> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Each of those threads take about 2% of our CPU. So if we have 10 of those threads that is already 20% CPU usage.
Our questions:
1. The question is, how come there is such a massive amount of those "ReEnableCaptureSubmit.class" objects and these have so much impact on the performance? Maybe there is a context with http://jira.icesoft.org/browse/ICE-10866?
2. Why we have performance kill threads on org.icefaces.ace.component.fileentry.FileEntryFormSubmit$ReEnableCaptureSubmit.processEvent(FileEntryFormSubmit.java:223)? What is the cause? Why do they need so much CPU?
Here is their analysis:
In our performance tests with P04 we encountered heavy performance issues!
We have the <ace:fileEntry> component in our application to allow users to upload files.
Our performance tests are uploading about 180 different files within 8 hours. We've run 6 runs of those 8 hours period and the tomcat performance has become really bad.
Actions that usually took milliseconds do now need several seconds (just changes in the view without backend interaction).
We took a heapdump from the tomcat.
In the heapdump (see attached file FileEntryFormSubmit_ReEnableCaptureSubmit_all_ice_classes.PNG) you will find 146.876 "ReEnableCaptureSubmit.class" instances.
In comparison to other objects in the heap, this amount of objects is very large.
While the performance tests are continueing (with horrible respond times in all userinterfaces (xhtmls), not just at the part where the upload component is used) we took several threaddumps from tomcat. (see attached threaddump_3793_170301_0839.txt)
In those threaddumps you can find RUNNABLE threads that look like this:
"http-bio-13081-exec-2861" #13041 daemon prio=5 os_prio=64 tid=0x0000000000d9d800 nid=0x32f3 runnable [0xffff80ff9f1e7000]
java.lang.Thread.State: RUNNABLE
at javax.faces.component.UIComponentBase.getFacetsAndChildren(UIComponentBase.java:721)
at javax.faces.component.UIComponentBase.findComponent(UIComponentBase.java:641)
at javax.faces.component.UIComponentBase.findComponent(UIComponentBase.java:648)
at javax.faces.component.UIComponentBase.findComponent(UIComponentBase.java:648)
at javax.faces.component.UIComponentBase.findComponent(UIComponentBase.java:600)
at org.icefaces.ace.component.fileentry.FileEntryFormSubmit$ReEnableCaptureSubmit.processEvent(FileEntryFormSubmit.java:223)
at javax.faces.event.SystemEvent.processListener(SystemEvent.java:106)
...
...
- locked <0x0000000776d06110> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Each of those threads take about 2% of our CPU. So if we have 10 of those threads that is already 20% CPU usage.
Our questions:
1. The question is, how come there is such a massive amount of those "ReEnableCaptureSubmit.class" objects and these have so much impact on the performance? Maybe there is a context with http://jira.icesoft.org/browse/ICE-10866?
2. Why we have performance kill threads on org.icefaces.ace.component.fileentry.FileEntryFormSubmit$ReEnableCaptureSubmit.processEvent(FileEntryFormSubmit.java:223)? What is the cause? Why do they need so much CPU?
Indeed we had a glaring bug here. The ReEnableCaptureSubmit listener instances are added on each JSF lifecycle but its removal occurs only when the file is uploaded.