Atomikos Forum

One of my application is encountering a lot of heuristic exception. With JmxTransactionServiceMBean, I could see a lot of heuristic transactions with state=committing and resource name=test2-xa-ds. Following is one of the heuristic transaction I captured from JMX console:
--------------------------------------------------------------
Object Name atomikos.heuristics:transactionId=x.x.x.x.tm148637271798313274
Participant details
ParticipantLogEntry [id=x.x.x.x.tm148637271798313274, uri=x.x.x.x.tm81258, expires=1486372733012, state=COMMITTING, resourceName=test2-xa-ds]
State COMMITTING
Tid x.x.x.x.tm148637271798313274
--------------------------------------------------------------

When I check the log file, i found log are full with the following error. This error is complaining 'test3-xa-ds' has internal error :

2017-02-06 21:54:56,369 ERROR [Atomikos:1271] com.atomikos.datasource.xa.XAResourceTransaction Heuristic termination of XAResourceTransaction: 31302E362E3135332E33362E746D313438363337393239353432343138323332:31302E362E3135332E33362E746D3836343235 with state HEUR_HAZARD
2017-02-06 21:54:56,369 ERROR [Atomikos:1271] com.atomikos.icatch.imp.CommitMessage Unexpected error in commit
org.postgresql.util.PSQLException: ERROR: prepared transaction with identifier "1096044365_MTAuNi4xNTMuMzYudG0xNDg2Mzc5Mjk1NDI0MTgyMzI=_MTAuNi4xNTMuMzYudG04NjQyNQ==" does not exist
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284) ~[postgresql-9.4.1208.jre7.jar:9.4.1208.jre7]
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003) ~[postgresql-9.4.1208.jre7.jar:9.4.1208.jre7]
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200) ~[postgresql-9.4.1208.jre7.jar:9.4.1208.jre7]
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424) ~[postgresql-9.4.1208.jre7.jar:9.4.1208.jre7]
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:321) ~[postgresql-9.4.1208.jre7.jar:9.4.1208.jre7]
        at org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:297) ~[postgresql-9.4.1208.jre7.jar:9.4.1208.jre7]
        at org.postgresql.xa.PGXAConnection.commitPrepared(PGXAConnection.java:507) ~[postgresql-9.4.1208.jre7.jar:9.4.1208.jre7]
        at org.postgresql.xa.PGXAConnection.commit(PGXAConnection.java:443) ~[postgresql-9.4.1208.jre7.jar:9.4.1208.jre7]
        at com.atomikos.datasource.xa.XAResourceTransaction.commit(XAResourceTransaction.java:555) ~[transactions-jta-4.0.4.jar:?]
        ... 7 more
Wrapped by: org.postgresql.xa.PGXAException: Error committing prepared transaction
        at org.postgresql.xa.PGXAConnection.commitPrepared(PGXAConnection.java:513) ~[postgresql-9.4.1208.jre7.jar:9.4.1208.jre7]
        at org.postgresql.xa.PGXAConnection.commit(PGXAConnection.java:443) ~[postgresql-9.4.1208.jre7.jar:9.4.1208.jre7]
        at com.atomikos.datasource.xa.XAResourceTransaction.commit(XAResourceTransaction.java:555) ~[transactions-jta-4.0.4.jar:?]
        ... 7 more
Wrapped by: com.atomikos.icatch.SysException: XA resource 'test3-xa-ds': commit for XID '31302E362E3135332E33362E746D313438363337393239353432343138323332:31302E362E3135332E33362E746D3836343235' raised -3: the XA resource detected an internal error
        at com.atomikos.datasource.xa.XAResourceTransaction.commit(XAResourceTransaction.java:594) ~[transactions-jta-4.0.4.jar:?]
        at com.atomikos.icatch.imp.CommitMessage.send(CommitMessage.java:52) [transactions-4.0.4.jar:?]
        at com.atomikos.icatch.imp.CommitMessage.send(CommitMessage.java:23) [transactions-4.0.4.jar:?]
        at com.atomikos.icatch.imp.PropagationMessage.submit(PropagationMessage.java:67) [transactions-4.0.4.jar:?]
        at com.atomikos.icatch.imp.Propagator$PropagatorThread.run(Propagator.java:63) [transactions-4.0.4.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]


The application has 3 XA data sources:
1. test1-xa-ds
2. test2-xa-ds
3. test3-xa-ds

All transactions (commit or rollback) in the application only involves test1-xa-ds and test2-xa-ds. test3-xa-ds is for data retrieving purpose.

Appreciate if anyone could answer the following questions as it really important for me to understand how atomikos handle transaction commit and rollback :
1. Is the exception thrown due to the test3-xa-ds's connectivity issue?

2. I wonder why atomikos perform a commit on the datasource (test3-xa-ds) that not involved in the transaction?

3. Since error shown commit is not possible on test3-xa-ds why the JmxTransactionServiceMBean heuristics transactions resource name is test2-xa-ds instead of test3-xa-ds?

4. I noticed every heuristic transaction has an expires timestamp in participant log entry section. What is the purpose of this timestamp? Will atomikos force forget or force rollback the heuristic transaction after the expires timestamp?

5. If test3-xa-ds connectivity issue solved, will restarting the server help to commit all committing state heuristic transactions?

Below are the connection pool setting for all three xa data sources:

<Resource name="jdbc/test1" auth="Container"
          type="com.atomikos.jdbc.AtomikosDataSourceBean"          factory="com.atomikos.tomcat.EnhancedTomcatAtomikosBeanFactory"
        uniqueResourceName="test1-xa-ds"
        minPoolSize="5"
        maxPoolSize="30"
        maxIdleTime="180"
        maxLifetime="1800"
        maintenanceInterval="120"
        testQuery="SELECT 1"
        xaDataSourceClassName="org.postgresql.xa.PGXADataSource"
        xaProperties.url="jdbc:postgresql://db:5444/test1"
        xaProperties.user="admin"
        xaProperties.password="password" />

<Resource name="jdbc/test2" auth="Container"
      type="com.atomikos.jdbc.AtomikosDataSourceBean"        factory="com.atomikos.tomcat.EnhancedTomcatAtomikosBeanFactory"
        uniqueResourceName="test2-xa-ds"
        minPoolSize="5"
        maxPoolSize="30"
        maxIdleTime="180"
        maxLifetime="1800"
        maintenanceInterval="120"
        testQuery="SELECT 1"
        xaDataSourceClassName="org.postgresql.xa.PGXADataSource"
        xaProperties.url="jdbc:postgresql://db:5444/test2"
        xaProperties.user="admin"
        xaProperties.password="password" />

<Resource name="jdbc/test3" auth="Container"
      type="com.atomikos.jdbc.AtomikosDataSourceBean"        factory="com.atomikos.tomcat.EnhancedTomcatAtomikosBeanFactory"
        uniqueResourceName="test3-xa-ds"
        minPoolSize="5"
        maxPoolSize="30"
        maxIdleTime="180"
        maxLifetime="1800"
        maintenanceInterval="120"
        testQuery="SELECT 1"
        xaDataSourceClassName="org.postgresql.xa.PGXADataSource"
        xaProperties.url="jdbc:postgresql://db:5444/test3"
        xaProperties.user="admin"
        xaProperties.password="password" />