Here is AIX supports input: (essentially, they are saying that we need input from ICEPDF team on what their code is waiting on, and that this does not appear to be an AIX specific issue)
ACTION TAKEN:
– Reviewed uploaded logs.
Reviewed javacores uploaded on oct 21st for Java PID 17236040.
All the javacores were generated manually.
Java being used is a 64-bit Java6 at SR8FP20. This is bundled with
Websphere and the Java is in:
/opt/WebSphere/8_5/AppServerBase1/java/bin/java
So, the Java used is not a stand-alone Java.
Java version shows;
1CIJAVAVERSION JRE 1.6.0 AIX ppc64-64 build
(pap6460_26sr8fp20-20160111_01(SR8 FP20))
1CIVMVERSION VM build R26_Java626_SR8_20151222_1616_B283040
1CIJITVERSION tr.r11_20151209_107111.01
1CIGCVERSION GC - R26_Java626_SR8_20151222_1616_B283040_CMPRSS
1CIJITMODES JIT enabled, AOT enabled, FSD disabled, HCR disabled
AIX OS level is at AIX 7.1
the javacores were generated at;
82710.227.000.javacore.20161021.152717.17236040.0001.txt:1TIDATETIME
Date: 2016/10/21 at 15:27:17
82710.227.000.javacore.20161021.152917.17236040.0002.txt:1TIDATETIME
Date: 2016/10/21 at 15:29:17
82710.227.000.javacore.20161021.153117.17236040.0003.txt:1TIDATETIME
Date: 2016/10/21 at 15:31:17
All the javacores show blocked threads.
Lots of threads are blocked on :
2LKMONINUSE sys_mon_t:0x00000100179A2018 infl_mon_t:
0x00000100179A2060:
3LKMONOBJECT java/lang/Object@0x000000024D167D68: Flat locked by
"APPROVAL_MCM_THREAD153130.pdf" (J9VMT
hread:0x0000000043CC9900), entry count 1
3LKWAITERQ Waiting to enter:
3LKWAITER "APPROVAL_MCM_THREAD153120.pdf"
(J9VMThread:0x0000000043CDBC00)
3LKWAITER "APPROVAL_MCM_THREAD153124.pdf"
(J9VMThread:0x0000000042009C00)
3LKWAITER "APPROVAL_MCM_THREAD153866.pdf"
(J9VMThread:0x0000000043E86C00)
3LKWAITER "APPROVAL_MCM_THREAD151747.pdf"
(J9VMThread:0x0000000043D15900)
.......
Lock owner thread in turn is waiting to be notified for object monitor
"org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38" :
notified of: 2LKMONINUSE sys_mon_t:0x0000010017A2BBA8 infl_mon_t:
0x0000010017A2BBF0:
3LKMONOBJECT
org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38: <unowned>
3LKNOTIFYQ Waiting to be notified:
3LKWAITNOTIFY "pool-3-thread-1"
(J9VMThread:0x00000000431CFC00)
3LKWAITNOTIFY "APPROVAL_MCM_THREAD153130.pdf"
(J9VMThread:0x0000000043CC9900)
3LKWAITNOTIFY "pool-3-thread-2"
(J9VMThread:0x0000000044C2D300)
3LKWAITNOTIFY "pool-3-thread-4"
(J9VMThread:0x000000004491A700)
3LKWAITNOTIFY "pool-3-thread-5"
(J9VMThread:0x0000000042314100)
Stack trace of the blocked threads shows
UAPMarkupHelper.convertFileToImages class of unicacorp application:
3XMTHREADBLOCK Blocked on: java/lang/Object@0x000000024D167D68 Owned
by: "APPROVAL_MCM_THREAD153130.pdf"
(J9VMThread:0x0000000043CC9900, java/lang/Thread:0x00000002450879C8)
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0
(0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at
com/unicacorp/uap/common/markup/mcm/web/helper/UAPMarkupHelper.convertFileToI
mages(UAPMarkupHelper.java:115)
4XESTACKTRACE at
com/unicacorp/uap/common/markup/mcm/web/helper/UAPMarkupHelper.run(UAPMa
rkupHelper.java:185)
3XMTHREADINFO3 Native callstack:
4XENATIVESTACK _event_wait+0x344 (0x0900000000538348
[libpthreads.a+0x17348])
4XENATIVESTACK _cond_wait_local+0x4dc (0x09000000005468A0
[libpthreads.a+0x258a0])
4XENATIVESTACK _cond_wait+0xc8 (0x0900000000546E8C
[libpthrea
.......
Lock owner stack trace shows after obtaining waiting to be notified of
"org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38":
3XMTHREADINFO "APPROVAL_MCM_THREAD153130.pdf"
J9VMThread:0x0000000043CC9900, j9thread_t:0x0000010016781A
60, java/lang/Thread:0x00000002450879C8, state:CW, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x141, isDaemon:true)
3XMTHREADINFO1 (native thread ID:0x73C00F1, native
priority:0x5, native policy:UNKNOWN, vmstate:CW
, vm thread flags:0x00000101)
3XMCPUTIME CPU usage total: 0.210140000 secs, user:
0.096320000 secs, system: 0.113820000 secs
3XMTHREADBLOCK Waiting on:
org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38 Owned by:
<unowned
>
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0
(0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/lang/Object.wait(Native Method)
4XESTACKTRACE at java/lang/Object.wait(Object.java:167)
4XESTACKTRACE at
org/icepdf/core/pobjects/fonts/nfont/b/i.a(Bytecode PC:293)
5XESTACKTRACE (entered lock:
org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38, e
ntry count: 1)
4XESTACKTRACE at
org/icepdf/core/pobjects/fonts/nfont/i.getRA(Bytecode PC:31)
4XESTACKTRACE at
org/icepdf/core/pobjects/fonts/nfont/i.<init>(Bytecode PC:6)
4XESTACKTRACE at
org/icepdf/core/pobjects/fonts/nfont/NFontTrueType.<init>(Bytecode PC:2)
4XESTACKTRACE at
sun/reflect/NativeConstructorAccessorImpl.newInstance0(Native Method)
.......
and as shown above, the lock owner is waiting to be notified of some
event on:
Waiting on: org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38
Owned by: <unowned>
So, the icepdf application developers need to check what is it the lock
owner thread is waiting for and take care of it to resolve the blocked
threads issue.
The fonts used belong to "org/icepdf/core/pobjects/fonts", so, this
doesn't look like AIX issue.
For us to look at this as an AIX issue, please provide data that proves
this as an AIX issue and also output of following commands from the AIX
system where the problem is being experienced:
errpt -a > errpt-a.out
oslevel -s > oslevel-s.out
lslpp -hac > lslpp-hac.out
prtconf > prtconf.out
emgr -lv3 > emgr-lv3.out 2>&1
Here is AIX supports input: (essentially, they are saying that we need input from ICEPDF team on what their code is waiting on, and that this does not appear to be an AIX specific issue)
ACTION TAKEN:
– Reviewed uploaded logs.
Reviewed javacores uploaded on oct 21st for Java PID 17236040.
All the javacores were generated manually.
Java being used is a 64-bit Java6 at SR8FP20. This is bundled with
Websphere and the Java is in:
/opt/WebSphere/8_5/AppServerBase1/java/bin/java
So, the Java used is not a stand-alone Java.
Java version shows;
1CIJAVAVERSION JRE 1.6.0 AIX ppc64-64 build
(pap6460_26sr8fp20-20160111_01(SR8 FP20))
1CIVMVERSION VM build R26_Java626_SR8_20151222_1616_B283040
1CIJITVERSION tr.r11_20151209_107111.01
1CIGCVERSION GC - R26_Java626_SR8_20151222_1616_B283040_CMPRSS
1CIJITMODES JIT enabled, AOT enabled, FSD disabled, HCR disabled
AIX OS level is at AIX 7.1
the javacores were generated at;
82710.227.000.javacore.20161021.152717.17236040.0001.txt:1TIDATETIME
Date: 2016/10/21 at 15:27:17
82710.227.000.javacore.20161021.152917.17236040.0002.txt:1TIDATETIME
Date: 2016/10/21 at 15:29:17
82710.227.000.javacore.20161021.153117.17236040.0003.txt:1TIDATETIME
Date: 2016/10/21 at 15:31:17
All the javacores show blocked threads.
Lots of threads are blocked on :
2LKMONINUSE sys_mon_t:0x00000100179A2018 infl_mon_t:
0x00000100179A2060:
3LKMONOBJECT java/lang/Object@0x000000024D167D68: Flat locked by
"APPROVAL_MCM_THREAD153130.pdf" (J9VMT
hread:0x0000000043CC9900), entry count 1
3LKWAITERQ Waiting to enter:
3LKWAITER "APPROVAL_MCM_THREAD153120.pdf"
(J9VMThread:0x0000000043CDBC00)
3LKWAITER "APPROVAL_MCM_THREAD153124.pdf"
(J9VMThread:0x0000000042009C00)
3LKWAITER "APPROVAL_MCM_THREAD153866.pdf"
(J9VMThread:0x0000000043E86C00)
3LKWAITER "APPROVAL_MCM_THREAD151747.pdf"
(J9VMThread:0x0000000043D15900)
.......
Lock owner thread in turn is waiting to be notified for object monitor
"org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38" :
notified of: 2LKMONINUSE sys_mon_t:0x0000010017A2BBA8 infl_mon_t:
0x0000010017A2BBF0:
3LKMONOBJECT
org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38: <unowned>
3LKNOTIFYQ Waiting to be notified:
3LKWAITNOTIFY "pool-3-thread-1"
(J9VMThread:0x00000000431CFC00)
3LKWAITNOTIFY "APPROVAL_MCM_THREAD153130.pdf"
(J9VMThread:0x0000000043CC9900)
3LKWAITNOTIFY "pool-3-thread-2"
(J9VMThread:0x0000000044C2D300)
3LKWAITNOTIFY "pool-3-thread-4"
(J9VMThread:0x000000004491A700)
3LKWAITNOTIFY "pool-3-thread-5"
(J9VMThread:0x0000000042314100)
Stack trace of the blocked threads shows
UAPMarkupHelper.convertFileToImages class of unicacorp application:
3XMTHREADBLOCK Blocked on: java/lang/Object@0x000000024D167D68 Owned
by: "APPROVAL_MCM_THREAD153130.pdf"
(J9VMThread:0x0000000043CC9900, java/lang/Thread:0x00000002450879C8)
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0
(0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at
com/unicacorp/uap/common/markup/mcm/web/helper/UAPMarkupHelper.convertFileToI
mages(UAPMarkupHelper.java:115)
4XESTACKTRACE at
com/unicacorp/uap/common/markup/mcm/web/helper/UAPMarkupHelper.run(UAPMa
rkupHelper.java:185)
3XMTHREADINFO3 Native callstack:
4XENATIVESTACK _event_wait+0x344 (0x0900000000538348
[libpthreads.a+0x17348])
4XENATIVESTACK _cond_wait_local+0x4dc (0x09000000005468A0
[libpthreads.a+0x258a0])
4XENATIVESTACK _cond_wait+0xc8 (0x0900000000546E8C
[libpthrea
.......
Lock owner stack trace shows after obtaining waiting to be notified of
"org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38":
3XMTHREADINFO "APPROVAL_MCM_THREAD153130.pdf"
J9VMThread:0x0000000043CC9900, j9thread_t:0x0000010016781A
60, java/lang/Thread:0x00000002450879C8, state:CW, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x141, isDaemon:true)
3XMTHREADINFO1 (native thread ID:0x73C00F1, native
priority:0x5, native policy:UNKNOWN, vmstate:CW
, vm thread flags:0x00000101)
3XMCPUTIME CPU usage total: 0.210140000 secs, user:
0.096320000 secs, system: 0.113820000 secs
3XMTHREADBLOCK Waiting on:
org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38 Owned by:
<unowned
>
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0
(0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/lang/Object.wait(Native Method)
4XESTACKTRACE at java/lang/Object.wait(Object.java:167)
4XESTACKTRACE at
org/icepdf/core/pobjects/fonts/nfont/b/i.a(Bytecode PC:293)
5XESTACKTRACE (entered lock:
org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38, e
ntry count: 1)
4XESTACKTRACE at
org/icepdf/core/pobjects/fonts/nfont/i.getRA(Bytecode PC:31)
4XESTACKTRACE at
org/icepdf/core/pobjects/fonts/nfont/i.<init>(Bytecode PC:6)
4XESTACKTRACE at
org/icepdf/core/pobjects/fonts/nfont/NFontTrueType.<init>(Bytecode PC:2)
4XESTACKTRACE at
sun/reflect/NativeConstructorAccessorImpl.newInstance0(Native Method)
.......
and as shown above, the lock owner is waiting to be notified of some
event on:
Waiting on: org/icepdf/core/pobjects/fonts/nfont/b/i@0x000000024D0C9C38
Owned by: <unowned>
So, the icepdf application developers need to check what is it the lock
owner thread is waiting for and take care of it to resolve the blocked
threads issue.
The fonts used belong to "org/icepdf/core/pobjects/fonts", so, this
doesn't look like AIX issue.
For us to look at this as an AIX issue, please provide data that proves
this as an AIX issue and also output of following commands from the AIX
system where the problem is being experienced:
errpt -a > errpt-a.out
oslevel -s > oslevel-s.out
lslpp -hac > lslpp-hac.out
prtconf > prtconf.out
emgr -lv3 > emgr-lv3.out 2>&1