Atomikos Forum

Hi folks,

This is my first Atomikos post.  Thanks in advance for any guidance.

I'm troubleshooting a Java app that runs on Tomcat.  This app is crashing with Permgen errors after several hot deploys.

I am browsing the logs (catalina.out) and see the following after a redeploy (or stop) command has been issued on the application:

"SEVERE: The web application [/api] appears to have started a thread named [Atomikos:0] but has failed to stop it. This is very likely to create a memory leak."

I'm not sure how to best troubleshoot this issue.  Rather than just dump all the config files in this post, I'll start with describing how the app is configured at a high level.  I'd be happy to add any specific config info if that will help.

This app is using:
Atomikos TransactionsEssentials for JTA/XA
Spring
HornetQ
MySQL
Tomcat
Maven

I didn't wire the transaction manager but from what I can see it seems to be a combo of what is explained here:
http://www.atomikos.com/Documentation/Tomcat7Integration35
and
http://www.atomikos.com/Documentation/SpringIntegration

The EnhancedTomcatAtomikosBeanFactory is defined as a container managed resource in the context.xml

Spring is using J2eeTransactionManager and J2eeUserTransaction.

The app seems to be functioning normally except during redeploy when the Atomikos thread fails to stop.

I can confirm the AtomikosLifecycleListener is getting called during Tomcat startup and shutdown.  However, this is not called when the application is stopped or redeployed (I think that's expected).

So I guess the question is why is this Atomikos thread getting started by the app and why isn't it getting stopped when the app is stopped?

Many thanks!

I ran a thread dump after stopping the web app and receiving the memory leak warning.

"Atomikos:0" daemon prio=6 tid=0x000000000ed9d000 nid=0x1ed8 in Object.wait() [0x000000000e0af000]
  java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000f1d6c570> (a java.lang.Object)
    at com.atomikos.timing.PooledAlarmTimer.doWait(PooledAlarmTimer.java:122)
    - locked <0x00000000f1d6c570> (a java.lang.Object)
    at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:88)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

  Locked ownable synchronizers:
    - <0x00000000f6867ed0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

Hi Justin,

the atomikos thread in general is not the problem because the thread is owned by a pool and not by a webapp. The thread is given back to the pool when the webapp is closed or removed. The webapp maybe get's the same thread when the webapp is deployed again.

But memory leaks can arise from unfortunate configuration.
The J2ee classes have to be configured as Spring beans to allow atomikos to tidy up on webapp close or undeploy. Also all session attributes for transaction have to be set.

Default session attributes for transactions and JNDI configured transactions named by Spring doc is only valid for application servers with integrated transaction managers. It does not work for your environment.

Pleas follow these instructions:

http://www.atomikos.com/Documentation/TwoPhaseCommitWithTomcatSpringJMSAndJDBC

Regards,
Rainer

Hi Rainer,

Thanks for the link.  I'll compare the config recommendations to what we are using.

With regard to your comment, "the atomikos thread in general is not the problem because the thread is owned by a pool and not by a webapp. The thread is given back to the pool when the webapp is closed or removed."

I was under the impression this really is a problem because "threads started by a web application will have the web application class loader as the context class loader.  Applications must stop threads they start.  Tomcat will log an error if applications forget."

Since the thread is not stopped during app shutdown, it maintains a ref to the class loader which in turn cannot be GC'd.  This causes the PermGen usage to increase each time the app is redeployed (our CI server hot deploys the app after each build).  Eventually Tomcat crashes due to exhausted PermGen space.  (Increasing PermGen size only delays the crash)

Please let me now what you think.  Many thanks!

Hi Justin,

the documented example uses a lifecycle listener to start the transaction manager. This will also create the pool and the atomikos threads.
So the webapp only uses the thread but tomcat assumes the thread is started by this webapp. This leads to the error message which can be ignored (but only this case!).

Regards,
Rainer