Our Presentation later is IceFaces. There is a SL1 WAS 7 Instance hung issue today, multiple times. We raised the issue to Websphere support and they confirmed that the code is
Waiting on condition
Java Stack
at java/lang/Object.wait(Native Method)
at edu/emory/mathcs/backport/java/util/concurrent/TimeUnit.timedWait(TimeUnit.java:301(Compiled Code))

The details of the error details are as below :

The javacores were taken 16th May 16:13-16:19
 
The systemout.log for this period shows a lot of application related errors (icesoft) -
 
[5/16/19 16:14:11:611 SGT] 00000194 lifecycle E JSF1054: (Phase ID: INVOKE_APPLICATION 5, View ID: /eposMain.xhtml) Exception thrown during phase execution: javax.faces.event.PhaseEvent[source=com.sun.faces.lifecycle.LifecycleImpl@c740c74]
[5/16/19 16:14:11:613 SGT] 00000194 ReceiveSendUp E com.icesoft.faces.webapp.http.core.ReceiveSendUpdates service Exception occured during rendering on http://epos.nhg.local/nhg-epos-web/block/send-receive-updates [/eposMain.xhtml]
                 javax.faces.FacesException: #{qBean.updatePreference}: java.lang.NullPointerException
 at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:90)
 at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
 at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
 at com.icesoft.faces.webapp.http.core.JsfLifecycleExecutor.apply(JsfLifecycleExecutor.java:18)
 at com.icesoft.faces.webapp.http.core.ReceiveSendUpdates.renderCycle(ReceiveSendUpdates.java:132)
 at com.icesoft.faces.webapp.http.core.ReceiveSendUpdates.service(ReceiveSendUpdates.java:74)
 at com.icesoft.faces.webapp.http.core.RequestVerifier.service(RequestVerifier.java:31)
 at com.icesoft.faces.webapp.http.common.standard.PathDispatcherServer.service(PathDispatcherServer.java:24)
 at com.icesoft.faces.webapp.http.servlet.BasicAdaptingServlet.service(BasicAdaptingServlet.java:16)
 at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:23)
 at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.service(SessionDispatcher.java:60)
 at com.icesoft.faces.webapp.http.servlet.SessionVerifier.service(SessionVerifier.java:26)
 at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:23)
 at com.icesoft.faces.webapp.http.servlet.MainServlet.service(MainServlet.java:148)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
 at sg.com.nhg.epos.ui.common.PersistentFacesServletWrapper.service(PersistentFacesServletWrapper.java:41)
 at com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:56)
 
Caused by: javax.faces.FacesException: #{qBean.updatePreference}: java.lang.NullPointerException
 at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:118)
 at javax.faces.component.UICommand.broadcast(UICommand.java:387)
 at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:458)
 at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:763)
 at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:82)
 ... 46 more
Caused by: javax.faces.el.EvaluationException: java.lang.NullPointerException
 at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:108)
 at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
 ... 50 more
Caused by: java.lang.NullPointerException
 at sg.com.nhg.epos.ui.queue.QueueBean.updatePreference(QueueBean.java:513)
...
[5/16/19 16:15:03:922 SGT] 00000276 D2DFaceletVie E com.icesoft.faces.facelets.D2DFaceletViewHandler renderResponse Problem in renderResponse: /eposMain.xhtml @7,49 src="#{navigation.currentPage}" /eposMain.xhtml @7,49 src="#{navigation.currentPage}": Error reading 'currentPage' on type sg.com.nhg.epos.ui.navigation.NavigationBean
                 com.sun.facelets.tag.TagAttributeException: /eposMain.xhtml @7,49 src="#{navigation.currentPage}" /eposMain.xhtml @7,49 src="#{navigation.currentPage}": Error reading 'currentPage' on type sg.com.nhg.epos.ui.navigation.NavigationBean
 at com.sun.facelets.tag.TagAttribute.getObject(TagAttribute.java:235)
 at com.sun.facelets.tag.TagAttribute.getValue(TagAttribute.java:200)
 at com.sun.facelets.tag.ui.IncludeHandler.apply(IncludeHandler.java:57)
...
Caused by: javax.el.ELException: /eposMain.xhtml @7,49 src="#{navigation.currentPage}": Error reading 'currentPage' on type sg.com.nhg.epos.ui.navigation.NavigationBean
 at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:76)
 at com.sun.facelets.tag.TagAttribute.getObject(TagAttribute.java:233)
 ... 84 more
Caused by: java.lang.NullPointerException
 at sun.reflect.GeneratedMethodAccessor929.invoke(Unknown Source)
 
[5/16/19 16:15:29:662 SGT] 00000214 ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: Uncaught exception created in one of the service methods of the servlet InterfaceController in application epos.01.001.1389.02244. Exception created : java.lang.NullPointerException
 at sg.com.nhg.epos.ui.interfaces.external.InterfaceController.doPost(InterfaceController.java:123)
 at sg.com.nhg.epos.ui.interfaces.external.InterfaceController.doGet(InterfaceController.java:80)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)



 have reviewed the data for instance 5 by which i assume you mean svr05.
 
The javacores were taken 16th May 16:13-16:19
 
The systemout.log for this period shows a lot of application related errors (icesoft) -
 
[5/16/19 16:14:11:611 SGT] 00000194 lifecycle E JSF1054: (Phase ID: INVOKE_APPLICATION 5, View ID: /eposMain.xhtml) Exception thrown during phase execution: javax.faces.event.PhaseEvent[source=com.sun.faces.lifecycle.LifecycleImpl@c740c74]
[5/16/19 16:14:11:613 SGT] 00000194 ReceiveSendUp E com.icesoft.faces.webapp.http.core.ReceiveSendUpdates service Exception occured during rendering on http://epos.nhg.local/nhg-epos-web/block/send-receive-updates [/eposMain.xhtml]
                 javax.faces.FacesException: #{qBean.updatePreference}: java.lang.NullPointerException
 at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:90)
 at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
 at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
 at com.icesoft.faces.webapp.http.core.JsfLifecycleExecutor.apply(JsfLifecycleExecutor.java:18)
 at com.icesoft.faces.webapp.http.core.ReceiveSendUpdates.renderCycle(ReceiveSendUpdates.java:132)
 at com.icesoft.faces.webapp.http.core.ReceiveSendUpdates.service(ReceiveSendUpdates.java:74)
 at com.icesoft.faces.webapp.http.core.RequestVerifier.service(RequestVerifier.java:31)
 at com.icesoft.faces.webapp.http.common.standard.PathDispatcherServer.service(PathDispatcherServer.java:24)
 at com.icesoft.faces.webapp.http.servlet.BasicAdaptingServlet.service(BasicAdaptingServlet.java:16)
 at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:23)
 at com.icesoft.faces.webapp.http.servlet.SessionDispatcher.service(SessionDispatcher.java:60)
 at com.icesoft.faces.webapp.http.servlet.SessionVerifier.service(SessionVerifier.java:26)
 at com.icesoft.faces.webapp.http.servlet.PathDispatcher.service(PathDispatcher.java:23)
 at com.icesoft.faces.webapp.http.servlet.MainServlet.service(MainServlet.java:148)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
 at sg.com.nhg.epos.ui.common.PersistentFacesServletWrapper.service(PersistentFacesServletWrapper.java:41)
 at com.icesoft.faces.webapp.xmlhttp.BlockingServlet.service(BlockingServlet.java:56)
 
Caused by: javax.faces.FacesException: #{qBean.updatePreference}: java.lang.NullPointerException
 at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:118)
 at javax.faces.component.UICommand.broadcast(UICommand.java:387)
 at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:458)
 at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:763)
 at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:82)
 ... 46 more
Caused by: javax.faces.el.EvaluationException: java.lang.NullPointerException
 at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:108)
 at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
 ... 50 more
Caused by: java.lang.NullPointerException
 at sg.com.nhg.epos.ui.queue.QueueBean.updatePreference(QueueBean.java:513)
...
[5/16/19 16:15:03:922 SGT] 00000276 D2DFaceletVie E com.icesoft.faces.facelets.D2DFaceletViewHandler renderResponse Problem in renderResponse: /eposMain.xhtml @7,49 src="#{navigation.currentPage}" /eposMain.xhtml @7,49 src="#{navigation.currentPage}": Error reading 'currentPage' on type sg.com.nhg.epos.ui.navigation.NavigationBean
                 com.sun.facelets.tag.TagAttributeException: /eposMain.xhtml @7,49 src="#{navigation.currentPage}" /eposMain.xhtml @7,49 src="#{navigation.currentPage}": Error reading 'currentPage' on type sg.com.nhg.epos.ui.navigation.NavigationBean
 at com.sun.facelets.tag.TagAttribute.getObject(TagAttribute.java:235)
 at com.sun.facelets.tag.TagAttribute.getValue(TagAttribute.java:200)
 at com.sun.facelets.tag.ui.IncludeHandler.apply(IncludeHandler.java:57)
...
Caused by: javax.el.ELException: /eposMain.xhtml @7,49 src="#{navigation.currentPage}": Error reading 'currentPage' on type sg.com.nhg.epos.ui.navigation.NavigationBean
 at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:76)
 at com.sun.facelets.tag.TagAttribute.getObject(TagAttribute.java:233)
 ... 84 more
Caused by: java.lang.NullPointerException
 at sun.reflect.GeneratedMethodAccessor929.invoke(Unknown Source)
 
[5/16/19 16:15:29:662 SGT] 00000214 ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: Uncaught exception created in one of the service methods of the servlet InterfaceController in application epos.01.001.1389.02244. Exception created : java.lang.NullPointerException
 at sg.com.nhg.epos.ui.interfaces.external.InterfaceController.doPost(InterfaceController.java:123)
 at sg.com.nhg.epos.ui.interfaces.external.InterfaceController.doGet(InterfaceController.java:80)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
 
 5/16/19 16:15:34:139 SGT] 00000360 SessionDispat I com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Monitor shutdown Session already invalidated.
[5/16/19 16:15:34:239 SGT] 00000361 SessionDispat I com.icesoft.faces.webapp.http.servlet.SessionDispatcher$Monitor shutdown Session already invalidated.
[5/16/19 16:15:34:241 SGT] 000001ef D2DFaceletVie E com.icesoft.faces.facelets.D2DFaceletViewHandler renderResponse Problem in renderResponse: com.icesoft.faces.webapp.http.core.SessionExpiredException: User session has expired or it was invalidated.
                 javax.faces.FacesException: com.icesoft.faces.webapp.http.core.SessionExpiredException: User session has expired or it was invalidated.
at com.sun.faces.application.ApplicationImpl.createComponent(ApplicationImpl.java:257)
at com.sun.facelets.tag.jsf.ComponentHandler.createComponent(ComponentHandler.java:224)
 
Caused by: com.icesoft.faces.webapp.http.core.SessionExpiredException: User session has expired or it was invalidated.
at com.icesoft.faces.webapp.http.servlet.ProxyHttpSession.getAttribute(ProxyHttpSession.java:77)
at com.icesoft.faces.webapp.http.servlet.ServletSessionAttributeMap.getAttribute(ServletSessionAttributeMap.java:16)
at com.icesoft.faces.context.AbstractAttributeMap.containsKey(AbstractAttributeMap.java:85)
at com.sun.faces.mgbean.BeanManager.isBeanInScope(BeanManager.java:177)
at com.sun.faces.el.ManagedBeanELResolver.getValue(ManagedBeanELResolver.java:83)
at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:143)
at com.sun.faces.el.FacesCompositeELResolver.getValue(FacesCompositeELResolver.java:73)
at org.apache.el.parser.AstIdentifier.getValue(AstIdentifier.java:45)
at org.apache.el.parser.AstValue.getValue(AstValue.java:91)
at org.apache.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:263)
at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:71)
at com.sun.faces.application.ApplicationImpl.createComponent(ApplicationImpl.java:246)
... 111 more
Caused by: java.lang.IllegalStateException: The following session is not valid! L0b3Mz6AnIdbuysUetYgRlH
at com.ibm.ws.session.http.HttpSessionImpl.getAttribute(HttpSessionImpl.java:191)
 
etc etc
 
Please have the application team investigate these errors in case they are implicated in the slow
webcontainer threads we see in the javacores.
 
The javacores are showing the same pattern in the webcontainer threads as picked out by my colleagues
David and Alva with slow movement of the threads in the following stack -
 
Waiting on condition
Java Stack
at java/lang/Object.wait(Native Method)
at edu/emory/mathcs/backport/java/util/concurrent/TimeUnit.timedWait(TimeUnit.java:301(Compiled Code))
at edu/emory/mathcs/backport/java/util/concurrent/Semaphore$NonfairSync.attempt(Semaphore.java:275(Compiled Code))
at edu/emory/mathcs/backport/java/util/concurrent/Semaphore.tryAcquire(Semaphore.java:571(Compiled Code))
at com/icesoft/faces/webapp/http/servlet/ThreadBlockingAdaptingServlet$ThreadBlockingRequestResponse.blockUntilRespond(ThreadBlockingAdaptingServlet.java:53(Compiled Code))
at com/icesoft/faces/webapp/http/servlet/ThreadBlockingAdaptingServlet.service(ThreadBlockingAdaptingServlet.java:26(Compiled Code))
at com/icesoft/faces/webapp/http/servlet/EnvironmentAdaptingServlet.service(EnvironmentAdaptingServlet.java:63(Compiled Code))
at com/icesoft/faces/webapp/http/servlet/PathDispatcher.service(PathDispatcher.java:23(Compiled Code))
at com/icesoft/faces/webapp/http/servlet/SessionDispatcher.service(SessionDispatcher.java:60(Compiled Code))
at com/icesoft/faces/webapp/http/servlet/SessionVerifier.service(SessionVerifier.java:26(Compiled Code))
at com/icesoft/faces/webapp/http/servlet/PathDispatcher.service(PathDispatcher.java:23(Compiled Code))
at com/icesoft/faces/webapp/http/servlet/MainServlet.service(MainServlet.java:148(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:831(Compiled Code))
at sg/com/nhg/epos/ui/common/PersistentFacesServletWrapper.service(PersistentFacesServletWrapper.java:41(Compiled Code))
at com/icesoft/faces/webapp/xmlhttp/BlockingServlet.service(BlockingServlet.java:56(Compiled Code))
 
I dont believe that this is IBM code both icesoft and the edu/ememory code look to be third party software.
We have therefore recommended you engage the teams responsible to determine why the threads are slow
and appear to show no movement across several javacores time period when they hit this code.
Also could this be related to the errors we are seeing in the systemout.log.
 
I have checked the verbosegc but there is very little overhead for this period and the overall overhead
is very low (i.e good) and therefore i dont suspect the gc to be implicated at this stage.
 
Therefore, please focus on the webcontainer thread stacks and also the repeated errors mentioned in the systemout.log.