Atomikos Forum

I am observing hung threads on calling my application concurrently (5-6 concurrent access). The occrence of hung thread is fairly regular and is a big issue stopping the adoption of Atomikos as a TM solution.

The following is the thread dump:

2012-05-07 17:43:46
Full thread dump Java HotSpot(TM) Client VM (17.0-b17 mixed mode, sharing):

"Atomikos:0" daemon prio=6 tid=0x03781000 nid=0x17ac in Object.wait() [0x039bf00
0]
  java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x13102b68> (a java.lang.Object)
        at com.atomikos.timing.PooledAlarmTimer.doWait(PooledAlarmTimer.java:122
)
        - locked <0x13102b68> (a java.lang.Object)
        at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:88)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source
)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"Low Memory Detector" daemon prio=6 tid=0x03376000 nid=0x828 runnable [0x0000000
0]
  java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x03367400 nid=0x1838 waiting on condition
[0x00000000]
  java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x03365c00 nid=0x1cb8 runnable [0x00000000]

  java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x03364400 nid=0xb0 waiting on condition
[0x00000000]
  java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x0335d400 nid=0x1870 in Object.wait() [0x0352f000
]
  java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x12990b08> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x12990b08> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" daemon prio=10 tid=0x0335c000 nid=0x1c1c in Object.wait() [0
x034df000]
  java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x12990a10> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
        - locked <0x12990a10> (a java.lang.ref.Reference$Lock)

"main" prio=6 tid=0x00577000 nid=0x1958 runnable [0x0105f000]
  java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at com.ibm.db2.jcc.t4.ab.b(ab.java:195)
        at com.ibm.db2.jcc.t4.ab.c(ab.java:243)
        at com.ibm.db2.jcc.t4.ab.c(ab.java:354)
        at com.ibm.db2.jcc.t4.ab.v(ab.java:1139)
        at com.ibm.db2.jcc.t4.fb.f(fb.java:231)
        at com.ibm.db2.jcc.t4.ac.b(ac.java:2373)
        at com.ibm.db2.jcc.t4.ac.b(ac.java:1145)
        at com.ibm.db2.jcc.t4.ac.rollback(ac.java:1039)
        - locked <0x13457cf0> (a com.ibm.db2.jcc.t4.c)
        at com.atomikos.datasource.xa.XATransactionalResource.endRecovery(XATran
sactionalResource.java:676)
        at com.atomikos.icatch.imp.TransactionServiceImp.recover(TransactionServ
iceImp.java:635)
        - locked <0x13044bd8> (a java.lang.Object)
        at com.atomikos.datasource.xa.XATransactionalResource.setRecoveryService
(XATransactionalResource.java:503)
        at com.atomikos.icatch.system.Configuration.addResource(Configuration.ja
va:404)
        - locked <0x32af4ab0> (a java.lang.Class for com.atomikos.icatch.system.
Configuration)
        at com.atomikos.jdbc.AtomikosDataSourceBean.doInit(AtomikosDataSourceBea
n.java:197)
        at com.atomikos.jdbc.AbstractDataSourceBean.init(AbstractDataSourceBean.
java:275)
        - locked <0x12ea54b8> (a com.atomikos.jdbc.AtomikosDataSourceBean)
        at com.atomikos.jdbc.AbstractDataSourceBean.getConnection(AbstractDataSo
urceBean.java:318)
        at com.atomikos.jdbc.AbstractDataSourceBean.getConnection(AbstractDataSo
urceBean.java:373)
        at com.BusinessEntities.Operations.operateAPI(Operations.java:47)
        at com.batch.BatchDriver.main(BatchDriver.java:14)

"VM Thread" prio=10 tid=0x0335a800 nid=0x1a0c runnable

"VM Periodic Task Thread" prio=10 tid=0x03380400 nid=0x178c waiting on condition


JNI global references: 914

Heap
 def new generation  total 157248K, used 16834K [0x12990000, 0x1d430000, 0x1d43
0000)
  eden space 139776K,  12% used [0x12990000, 0x13a008e8, 0x1b210000)
  from space 17472K,  0% used [0x1b210000, 0x1b210000, 0x1c320000)
  to  space 17472K,  0% used [0x1c320000, 0x1c320000, 0x1d430000)
 tenured generation  total 349568K, used 0K [0x1d430000, 0x32990000, 0x32990000
)
  the space 349568K,  0% used [0x1d430000, 0x1d430000, 0x1d430200, 0x32990000)

 compacting perm gen  total 12288K, used 3954K [0x32990000, 0x33590000, 0x369900
00)
  the space 12288K,  32% used [0x32990000, 0x32d6cb50, 0x32d6cc00, 0x33590000)
    ro space 10240K,  51% used [0x36990000, 0x36ebb700, 0x36ebb800, 0x37390000)
    rw space 12288K,  54% used [0x37390000, 0x37a276c0, 0x37a27800, 0x37f90000)


Is this a known Atomikos issue?

I don't remember seeing this before. There is no deadlock either it seems (or the dump would show it at the bottom).

What version of Atomikos do you use?

It looks like rollback in DB2 is hanging, BTW. That might be the reason.

I am using v3.7.1.
I could not make out if rollback was failing but might be a possible explanation as I do see inconsistencies of the updates in the databases involved in the transactions.

Any clue as to what might be causing rollback to hang?

Following is another thread dump:

Full thread dump Java HotSpot(TM) Client VM (17.0-b17 mixed mode, sharing):

"Atomikos:2" daemon prio=6 tid=0x03aa4400 nid=0x17b8 in Object.wait() [0x03c7f00
0]
  java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x13ab7378> (a java.lang.Object)
        at com.atomikos.timing.PooledAlarmTimer.doWait(PooledAlarmTimer.java:122
)
        - locked <0x13ab7378> (a java.lang.Object)
        at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:88)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source
)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"Atomikos:1" daemon prio=6 tid=0x03a95000 nid=0x8cc in Object.wait() [0x03c2f000
]
  java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x136b8d68> (a java.lang.Object)
        at com.atomikos.timing.PooledAlarmTimer.doWait(PooledAlarmTimer.java:122
)
        - locked <0x136b8d68> (a java.lang.Object)
        at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:88)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source
)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"Atomikos:0" daemon prio=6 tid=0x03a3f000 nid=0xfbc in Object.wait() [0x03bdf000
]
  java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x130c7518> (a java.lang.Object)
        at com.atomikos.timing.PooledAlarmTimer.doWait(PooledAlarmTimer.java:122
)
        - locked <0x130c7518> (a java.lang.Object)
        at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:88)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source
)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"Low Memory Detector" daemon prio=6 tid=0x01566000 nid=0x1708 runnable [0x000000
00]
  java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x01557400 nid=0x1058 waiting on condition
[0x00000000]
  java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x01555c00 nid=0x1550 runnable [0x00000000]

  java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x01554800 nid=0xa88 waiting on condition
 [0x00000000]
  java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x01550c00 nid=0xd7c in Object.wait() [0x0171f000]

  java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x12990b08> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x12990b08> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" daemon prio=10 tid=0x0154c000 nid=0x1780 in Object.wait() [0
x016cf000]
  java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x12990a10> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
        - locked <0x12990a10> (a java.lang.ref.Reference$Lock)

"main" prio=6 tid=0x00577000 nid=0x17d0 runnable [0x0124f000]
  java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at com.ibm.db2.jcc.t4.ab.b(ab.java:195)
        at com.ibm.db2.jcc.t4.ab.c(ab.java:243)
        at com.ibm.db2.jcc.t4.ab.c(ab.java:354)
        at com.ibm.db2.jcc.t4.ab.v(ab.java:1139)
        at com.ibm.db2.jcc.t4.fb.b(fb.java:135)
        at com.ibm.db2.jcc.t4.c.Rb(c.java:277)
        at com.ibm.db2.jcc.b.p.h(p.java:310)
        at com.ibm.db2.jcc.t4.a.h(a.java:506)
        at com.ibm.db2.jcc.b.p.a(p.java:205)
        at com.ibm.db2.jcc.b.gm.e(gm.java:3212)
        at com.ibm.db2.jcc.b.gm.Rb(gm.java:612)
        at com.ibm.db2.jcc.b.gm.executeUpdate(gm.java:595)
        - locked <0x136837c8> (a com.ibm.db2.jcc.t4.c)
        at com.Controller.QueryController.insertIntoDB(QueryController.java:64)
        - locked <0x32d96e08> (a java.lang.Class for com.Controller.QueryControl
ler)
        at com.BusinessEntities.Operations.run(Operations.java:62)
        at com.batch.BatchDriver.main(BatchDriver.java:13)

"VM Thread" prio=10 tid=0x0154a800 nid=0x1450 runnable

"VM Periodic Task Thread" prio=10 tid=0x01570400 nid=0x15f0 waiting on condition

Here it shows error while updating the DB. Does concurrency require some special handling?