Atomikos Forum

"the XA resource has become unavailable" after crash recovery.

Hi!

We are using Atomikos (3.7.1) as transaction manager with 4 datasources: three on PostgreSQL (9.1-901.jdbc3) DB and one MySql (5.1.20). Also Tomcat (6.0.29), Hibernate(3.6.10), and Spring (3.1.1) are used in our project.
All datasources are XA.

Now we are testing project recovery after crashing. I'm killing my application during DB operations. I had put a breakpoint at CoordinatorStateHandler:592, just after coordinator_.setState ( TxState.COMMITTING ). Then I'm killing my application with kill -9 command.

Then, after project startup following exception is printed to console every ~10 seconds:

1498179 [Atomikos:4] WARN  atomikos  - jdbc/source1-db: resource no longer available - recovery might be at risk!
1498180 [Atomikos:4] WARN  atomikos  - jdbc/source1-db: resource no longer available - recovery might be at risk!
1498181 [Atomikos:4] WARN  atomikos  - XA resource 'jdbc/source1-db': commit for XID '3132372E302E312E312E746D30303030313030303035:3132372E302E312E312E746D31' raised -7: the XA resource has become unavailable
javax.transaction.xa.XAException: jdbc/source1-db: resource no longer available - recovery might be at risk!
        at com.atomikos.datasource.xa.XAResourceTransaction.throwXAExceptionForUnavailableResource(XAResourceTransaction.java:311)
        at com.atomikos.datasource.xa.XAResourceTransaction.testOrRefreshXAResourceFor2PC(XAResourceTransaction.java:283)
        at com.atomikos.datasource.xa.XAResourceTransaction.commit(XAResourceTransaction.java:780)
        at com.atomikos.icatch.imp.CommitMessage.send(CommitMessage.java:73)
        at com.atomikos.icatch.imp.PropagationMessage.submit(PropagationMessage.java:111)
        at com.atomikos.icatch.imp.Propagator$PropagatorThread.run(Propagator.java:87)
        at com.atomikos.icatch.imp.Propagator.submitPropagationMessage(Propagator.java:66)
        at com.atomikos.icatch.imp.HeurHazardStateHandler.onTimeout(HeurHazardStateHandler.java:120)
        at com.atomikos.icatch.imp.CoordinatorImp.alarm(CoordinatorImp.java:1105)
        at com.atomikos.timing.PooledAlarmTimer.notifyListeners(PooledAlarmTimer.java:112)
        at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:99)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
1498182 [Atomikos:4] WARN  atomikos  - Unexpected error in commit
com.atomikos.icatch.SysException: XA resource 'jdbc/source1-db': commit for XID '3132372E302E312E312E746D30303030313030303035:3132372E302E312E312E746D31' raised -7: the XA resource has become unavailable
        at com.atomikos.datasource.xa.XAResourceTransaction.commit(XAResourceTransaction.java:825)
        at com.atomikos.icatch.imp.CommitMessage.send(CommitMessage.java:73)
        at com.atomikos.icatch.imp.PropagationMessage.submit(PropagationMessage.java:111)
        at com.atomikos.icatch.imp.Propagator$PropagatorThread.run(Propagator.java:87)
        at com.atomikos.icatch.imp.Propagator.submitPropagationMessage(Propagator.java:66)
        at com.atomikos.icatch.imp.HeurHazardStateHandler.onTimeout(HeurHazardStateHandler.java:120)
        at com.atomikos.icatch.imp.CoordinatorImp.alarm(CoordinatorImp.java:1105)
        at com.atomikos.timing.PooledAlarmTimer.notifyListeners(PooledAlarmTimer.java:112)
        at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:99)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

I looked through XAResourceTransaction code, and noticed that the issue is that the resource_ fields is null. Also there is a reference to some bug (bug 67951):
        if ( resource_ == null ) {
            // cf bug 67951
            // happens on recovery without resource found
            throwXAExceptionForUnavailableResource();
        }
But seems that your bugtracker isn't available from outside.
Also I've noticed that current instance of XAResourceTransaction class was deserialized with XAResourceTransaction:readExternal(ObjectInput) method. And there is no tries of filling the resource_ field.
Is it a bug? Could you please check it, and publish the result? 

The configs, we are using:

Tomcat context:

    <Resource name="jdbc/source1-db" auth="Container" type="com.atomikos.jdbc.AtomikosDataSourceBean"
              factory="com.atomikos.tomcat.BeanFactory"
              xaDataSourceClassName="org.postgresql.xa.PGXADataSource"
              xaProperties.serverName="localhost"
              xaProperties.portNumber="5433"
              xaProperties.databaseName="source13"
              uniqueResourceName="jdbc/source1-db"
              xaProperties.user="xxx"
              xaProperties.password="xxx"
              minPoolSize="5"
              maxPoolSize="20"
              testQuery="SELECT 1" />

    <Resource name="jdbc/source2-db" auth="Container" type="com.atomikos.jdbc.AtomikosDataSourceBean"
              factory="com.atomikos.tomcat.BeanFactory"
              xaDataSourceClassName="org.postgresql.xa.PGXADataSource"
              xaProperties.serverName="localhost"
              xaProperties.portNumber="5433"
              xaProperties.databaseName="source2_replication3"
              uniqueResourceName="jdbc/source2-db"
              xaProperties.user="xxx"
              xaProperties.password="xxx"
              minPoolSize="5"
              maxPoolSize="20"
              testQuery="SELECT 1"  />

    <Resource name="jdbc/source4-db" auth="Container" type="com.atomikos.jdbc.AtomikosDataSourceBean"
              factory="com.atomikos.tomcat.BeanFactory"
              xaDataSourceClassName="com.mysql.jdbc.jdbc2.optional.MysqlXADataSource"
              xaProperties.url="jdbc:mysql://localhost:3306/source4_replication"
              uniqueResourceName="jdbc/source4-db"
              xaProperties.user="xxx"
              xaProperties.password="xxx"
              minPoolSize="5"
              maxPoolSize="20"
              testQuery="SELECT 1"  />

    <Resource name="jdbc/source3-db" auth="Container" type="com.atomikos.jdbc.AtomikosDataSourceBean"
              factory="com.atomikos.tomcat.BeanFactory"
              xaDataSourceClassName="org.postgresql.xa.PGXADataSource"
              xaProperties.serverName="localhost"
              xaProperties.portNumber="5433"
              xaProperties.databaseName="source3_replication3"
              uniqueResourceName="jdbc/source3-db"
              xaProperties.user="xxx"
              xaProperties.password="xxx"
              minPoolSize="5"
              maxPoolSize="20"
              testQuery="SELECT 1"  />

    <Transaction factory="com.atomikos.icatch.jta.UserTransactionFactory" transactionTimeout="3000"/>

transactions.properties:

    com.atomikos.icatch.service=com.atomikos.icatch.standalone.UserTransactionServiceFactory
    com.atomikos.icatch.console_file_limit=0
    com.atomikos.icatch.hide_init_file_path=true
    com.atomikos.icatch.console_log_level=DEBUG
    com.atomikos.icatch.log_base_dir=/var/admin/data/transactionsLogs
    com.atomikos.icatch.output_dir=/var/admin/data/transactionsLogs


Transaction manager in spring application context:

    <bean id="embeddedAtomikosTransactionManager" class="com.atomikos.icatch.jta.UserTransactionManager"
          init-method="init" destroy-method="close" lazy-init="true">
        <property name="forceShutdown" value="false"/>
    </bean>


BTW, There is workaround: we can simply remove all atomikos log files before application is started. But this could lead to serious problems in feature.

Looking for your feedback! :)
Innokentii Foss Send private email
Friday, November 23, 2012
 
 
Hi,

Thanks for reporting this, but it is not a bug I am sure. Your resource is simply missing after restart.

HTH
Guy Pardon Send private email
Friday, November 23, 2012
 
 
Hello, Guy.
Thank you for your reply!

In fact, after restore application works and resources are ok, I can load and modify entities from DB. But those exceptions in log appear every 10 seconds.

Guy,
1. Could you please assume what might be wrong? Why the XAResourceTransaction.resource_ is missing on restore?
2. Where and when atomikos should insert the resource into restored XAResourceTransaction?

Looking forward to your feedback!
Innokentii Foss Send private email
Monday, November 26, 2012
 
 
Hi,

Actually, I checked and it turns out that this is a bug fix available in the 3.7.3 release...

Best
Guy Pardon Send private email
Monday, December 10, 2012
 
 
Hello, Guy!

Thanks for your answers.

When you are going to release 3.7.3?

Maybe I'm missing something, but I also can't find 3.7.2 version :)
At http://search.maven.org/ there are only 3.8.0, 3.7.0, and 3.7.1 versions available. On your website the only 3.8.0 is available.
Could you please tell me where do you store them?
Innokentii Foss Send private email
Tuesday, December 11, 2012
 
 
I've just experienced this as well. Hate to be that person, but is there any update on this issue?
Nick Spacek Send private email
Friday, January 04, 2013
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics