Atomikos Forum

The Case
--------
(1) Application A (producer) sends messages to the queue
(2) Application B (consumer) reads messages from the queue using:
  (a) Atomikos as XA transaction manager
  <bean id="atomikosTransactionManager" class="com.atomikos.icatch.jta.UserTransactionManager"
          destroy-method="close"
          init-method="init"/>
  <bean id="atomikosUserTransaction" class="com.atomikos.icatch.jta.UserTransactionImp"
          p:transactionTimeout="300"/>
  <bean id="transactionManager" class="org.springframework.transaction.jta.JtaTransactionManager"
          depends-on="amqConnection"
          p:transactionManager-ref="atomikosTransactionManager"
          p:userTransaction-ref="atomikosUserTransaction"/>
  (b) Atomikos as JMS connection pool manager
  <bean id="amq" class="org.apache.activemq.ActiveMQXAConnectionFactory"
      p:brokerURL="failover://(tcp://localhost:61616,tcp://localhost:61617)?jms.prefetchPolicy.all=0&randomize=false"/>
  <bean id="amqConnection" class="com.atomikos.jms.AtomikosConnectionFactoryBean"
      init-method="init"
      destroy-method="close"
      p:uniqueResourceName="amqConnectionResource"
      p:xaConnectionFactory-ref="amq"
      p:minPoolSize="250"
      p:maxPoolSize="400"/>
  (c) Spring DMLC as message listener container (there is more than one DMLC declared, jmsListenerA
      is just a sample name/placeholer)
  <bean id="jmsListenerBase" class="org.springframework.jms.listener.DefaultMessageListenerContainer"
      abstract="true"
      depends-on="transactionManager"
      p:autoStartup="false"
      p:acceptMessagesWhileStopping="false"
      p:transactionManager-ref="transactionManager"
      p:connectionFactory-ref="amqConnection"
      p:receiveTimeout="3000"
      p:sessionTransacted="true"
      p:concurrentConsumers="1"
      p:maxConcurrentConsumers="1"
      p:cacheLevelName="CACHE_CONSUMER">
  </bean>
  <bean id="jmsListenerA" class="org.springframework.jms.listener.DefaultMessageListenerContainer"
      parent="jmsListenerBase"
      p:messageListener-ref="listenerA"
      p:concurrentConsumers="20"
      p:maxConcurrentConsumers="20">
    <property name="destination">
        <bean class="org.apache.activemq.command.ActiveMQQueue"
                p:physicalName="queue/a"/>
    </property>
  </bean>
(3) Fuse Message Broker as queue manager running Master Slave Shared Nothing configuration.

Environment
-----------
(1) Suse Enterprise 11
(2) Sun/Oracle JDK 1.6 update 29b11 x86
(3) Fuse Message Broker 5.5.1
(4) Spring Framework 3.0.7.RELEASE
(5) Atomikos Transaction Essentials 3.7.0

Issue
-----
Now, when I kill both master and slave (Fuse Message Broker), start them again and send few messages using
Application A it causes deadlocks of threads in Application B as shown in following thread dump:

------ Start of thread dump fragment -------
Found one Java-level deadlock:
=============================
"jmsListenerZleceniaBrokerskieInt-23":
  waiting to lock monitor 0xa3fc3614 (object 0xb3fd5c40, a java.lang.Object),
  which is held by "jmsListenerZleceniaBrokerskieInt-14"
"jmsListenerZleceniaBrokerskieInt-14":
  waiting to lock monitor 0x085660c8 (object 0xb3d8a368, a com.atomikos.datasource.pool.ConnectionPool),
  which is held by "jmsListenerZdarzenia-15"
"jmsListenerZdarzenia-15":
  waiting to lock monitor 0xad39c198 (object 0xb42f1340, a java.util.ArrayList),
  which is held by "jmsListenerZleceniaKlienta-14"
"jmsListenerZleceniaKlienta-14":
  waiting to lock monitor 0xaddb5a28 (object 0xb3dd4450, a com.atomikos.jms.AtomikosPooledJmsConnection),
  which is held by "jmsListenerZdarzenia-15"

Java stack information for the threads listed above:
===================================================
"jmsListenerZleceniaBrokerskieInt-23":
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:967)
    - waiting to lock <0xb3fd5c40> (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:662)
"jmsListenerZleceniaBrokerskieInt-14":
    at com.atomikos.datasource.pool.ConnectionPool.onXPooledConnectionTerminated(ConnectionPool.java:318)
    - waiting to lock <0xb3d8a368> (a com.atomikos.datasource.pool.ConnectionPool)
    at com.atomikos.datasource.pool.AbstractXPooledConnection.fireOnXPooledConnectionTerminated(AbstractXPooledConnection.java:97)
    at com.atomikos.jms.AtomikosPooledJmsConnection.onTerminated(AtomikosPooledJmsConnection.java:142)
    at com.atomikos.jms.AtomikosJmsConnectionProxy.close(AtomikosJmsConnectionProxy.java:265)
    at com.atomikos.jms.AtomikosJmsConnectionProxy.invoke(AtomikosJmsConnectionProxy.java:143)
    at $Proxy31.close(Unknown Source)
    at org.springframework.jms.connection.ConnectionFactoryUtils.releaseConnection(ConnectionFactoryUtils.java:80)
    at org.springframework.jms.listener.AbstractJmsListeningContainer.refreshSharedConnection(AbstractJmsListeningContainer.java:385)
    - locked <0xb3fd5a80> (a java.lang.Object)
    at org.springframework.jms.listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful(DefaultMessageListenerContainer.java:856)
    at org.springframework.jms.listener.DefaultMessageListenerContainer.recoverAfterListenerSetupFailure(DefaultMessageListenerContainer.java:838)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:969)
    - locked <0xb3fd5c40> (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:662)
"jmsListenerZdarzenia-15":
    at com.atomikos.jms.AtomikosJmsConnectionProxy.isAvailable(AtomikosJmsConnectionProxy.java:317)
    - waiting to lock <0xb42f1340> (a java.util.ArrayList)
    at com.atomikos.jms.AtomikosPooledJmsConnection.isAvailable(AtomikosPooledJmsConnection.java:89)
    - locked <0xb3dd4450> (a com.atomikos.jms.AtomikosPooledJmsConnection)
    at com.atomikos.datasource.pool.ConnectionPool.availableSize(ConnectionPool.java:299)
    - locked <0xb3d8a368> (a com.atomikos.datasource.pool.ConnectionPool)
    at com.atomikos.datasource.pool.ConnectionPool.borrowConnection(ConnectionPool.java:142)
    - locked <0xb3d8a368> (a com.atomikos.datasource.pool.ConnectionPool)
    at com.atomikos.jms.AtomikosConnectionFactoryBean.createConnection(AtomikosConnectionFactoryBean.java:585)
    at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
    at org.springframework.jms.listener.AbstractJmsListeningContainer.createSharedConnection(AbstractJmsListeningContainer.java:403)
    at org.springframework.jms.listener.AbstractJmsListeningContainer.refreshSharedConnection(AbstractJmsListeningContainer.java:388)
    - locked <0xb3e6ea50> (a java.lang.Object)
    at org.springframework.jms.listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful(DefaultMessageListenerContainer.java:856)
    at org.springframework.jms.listener.DefaultMessageListenerContainer.recoverAfterListenerSetupFailure(DefaultMessageListenerContainer.java:838)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:969)
    - locked <0xb3e6eb48> (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:662)
"jmsListenerZleceniaKlienta-14":
    at com.atomikos.jms.AtomikosPooledJmsConnection.onTerminated(AtomikosPooledJmsConnection.java:119)
    - waiting to lock <0xb3dd4450> (a com.atomikos.jms.AtomikosPooledJmsConnection)
    at com.atomikos.datasource.xa.session.SessionHandleState.fireTerminatedEvent(SessionHandleState.java:269)
    at com.atomikos.datasource.xa.session.SessionHandleState.notifySessionClosed(SessionHandleState.java:134)
    at com.atomikos.jms.AtomikosJmsXaSessionProxy.invoke(AtomikosJmsXaSessionProxy.java:125)
    - locked <0xb4308a30> (a com.atomikos.jms.AtomikosJmsXaSessionProxy)
    at $Proxy32.close(Unknown Source)
    at com.atomikos.jms.AtomikosJmsConnectionProxy.close(AtomikosJmsConnectionProxy.java:256)
    - locked <0xb42f1340> (a java.util.ArrayList)
    at com.atomikos.jms.AtomikosJmsConnectionProxy.invoke(AtomikosJmsConnectionProxy.java:143)
    at $Proxy31.close(Unknown Source)
    at org.springframework.jms.connection.ConnectionFactoryUtils.releaseConnection(ConnectionFactoryUtils.java:80)
    at org.springframework.jms.listener.AbstractJmsListeningContainer.refreshSharedConnection(AbstractJmsListeningContainer.java:385)
    - locked <0xb3f7c580> (a java.lang.Object)
    at org.springframework.jms.listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful(DefaultMessageListenerContainer.java:856)
    at org.springframework.jms.listener.DefaultMessageListenerContainer.recoverAfterListenerSetupFailure(DefaultMessageListenerContainer.java:838)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:969)
    - locked <0xb3f7c770> (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:662)

Found 1 deadlock.
------ End of thread dump fragment -------

The Question
------------
Any idea what's the problem and how to solve it?

Thanks - will need to have a look. I filed a case for this to be fixed in a future release.

I don't know if it helps, but maybe switching to the Atomikos MessageDrivenContainer is a workaround for now. At least that would change the connection acquiring behavior.

HTH

We've made additional tests using Atomikos 3.7.2, cause it seems there were some fixes made possibly related to our problem (not sure about 2nd one in case of Spring DMLC usage):

73729: Connection recycling bug in JMS
Fixed a bug where two subsequent getConnection() requests (by the same thread) would get the same connection from the same AtomikosConnectionFactoryBean - leading to problems with the later close logic.

73756: Bug in MessageConsumerSession
Improved the retry/refresh logic when the JMS backed is unreachable.

So far we didn't notice the problem to appear again (though the test has been done only few times), but still - having you guys taking a look at the issue would be highly appreciated.