Atomikos Forum

TX log corrupted if active TX in a thread pool being shutdown

We have found that, in version 4.0.2, if a thread is in a thread pool inside an active transaction and shutdownNow() is invoked on the thread pool the active thread is interrupted and the TX log gets corrupted.  Furthermore, no recovery is possible and the error logs indicate to restart the JVM and this also requires removing the TX logs.  This error is always able to be reproduced and results in the following stack trace:
2016-04-07 13:19:04,706 [CallCacheManager-executor-thread-1] DEBUG: exception from @Transactional method protected java.util.List com.clj2.business.cache.CallCacheManager.loadObjects() on attempt #1: org.springframework.transaction.TransactionSystemException: JTA failure on commit; nested exception is com.atomikos.icatch.jta.ExtendedSystemException: Error in commit: could not flush state image Log corrupted - restart JVM com.atomikos.recovery.LogReadException (translates to null)
2016-04-07 13:19:04,706 [CallCacheManager-executor-thread-1] ERROR: exception in CallCacheManager
org.springframework.transaction.TransactionSystemException: JTA failure on commit; nested exception is com.atomikos.icatch.jta.ExtendedSystemException: Error in commit: could not flush state image Log corrupted - restart JVM com.atomikos.recovery.LogReadException
    at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1037)
    at com.clj2.business.lib.grid.tx.GridTransactionManager.doCommit(GridTransactionManager.java:88)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:757)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:726)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:478)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:272)
    at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:63)
    at com.clj2.business.cache.CallCacheManager.loadObjects_aroundBody2(CallCacheManager.java:104)
    at com.clj2.business.cache.CallCacheManager$AjcClosure3.run(CallCacheManager.java:1)
    at org.dellroad.stuff.spring.RetryTransactionAspect.ajc$around$org_dellroad_stuff_spring_RetryTransactionAspect$1$878f3a34proceed(RetryTransactionAspect.aj:106)
    at org.dellroad.stuff.spring.RetryTransactionAspect$RetryTransactionAspect$1.run(RetryTransactionAspect.aj:179)
    at org.dellroad.stuff.java.ThreadLocalHolder.invoke(ThreadLocalHolder.java:111)
    at org.dellroad.stuff.spring.RetryTransactionAspect.ajc$around$org_dellroad_stuff_spring_RetryTransactionAspect$1$878f3a34(RetryTransactionAspect.aj:176)
    at com.clj2.business.cache.CallCacheManager.loadObjects(CallCacheManager.java:104)
    at com.clj2.business.cache.AbstractCacheManager.reloadAndNotify(AbstractCacheManager.java:398)
    at com.clj2.business.cache.AbstractCacheManager.access$100(AbstractCacheManager.java:56)
    at com.clj2.business.cache.AbstractCacheManager$4.run(AbstractCacheManager.java:663)
    at com.clj2.business.service.impl.AbstractThreadingService$1.run(AbstractThreadingService.java:96)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: com.atomikos.icatch.jta.ExtendedSystemException: Error in commit: could not flush state image Log corrupted - restart JVM com.atomikos.recovery.LogReadException
    at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:181)
    at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:414)
    at com.atomikos.icatch.jta.UserTransactionImp.commit(UserTransactionImp.java:86)
    at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1021)
    ... 24 more


We are attempting to downgrade to 4.0.0 to see if the error is resolved as 4.0.2 is unusable in its current form.  Has anyone else encountered this issue and possibly have a fix or workaround?
Mark Thomas Send private email
Thursday, April 07, 2016
 
 
We shall look at this, thank you!
Guy Pardon Send private email
Friday, April 15, 2016
 
 
Fixed in dev in the following way: we now allow the new recovery to tolerate corrupted log entries by treating them as the (unexpected) end of the log file and moving on as usual.

Can you confirm that this would be ok?
Guy Pardon Send private email
Friday, April 22, 2016
 
 
I'm using essentials 4.0.4 and encountered this exact same problem. Is this supposed to be fixed in 4.0.4?

Thanks!
Natalia Send private email
Friday, July 15, 2016
 
 
Here's the log just in case

DEBUG 2016-07-15 12:12:38,208 [pool-18-thread-4] : Sync2Sync - beforeCompletion() called on Synchronization: SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=-1, stateTransferFlag=null} org.infinispan.transaction.synchronization.SyncLocalTransaction@399d7} org.infinispan.transaction.synchronization.SynchronizationAdapter@399f6
...
ERROR 2016-07-15 12:12:38,336 [pool-18-thread-4] : CachedRepository - Corrupted log file - restart JVM
com.atomikos.recovery.LogWriteException: java.nio.channels.ClosedByInterruptException
    at com.atomikos.recovery.imp.FileSystemRepository.writeCheckpoint(FileSystemRepository.java:233)
    at com.atomikos.recovery.imp.CachedRepository.performCheckpoint(CachedRepository.java:84)
    at com.atomikos.recovery.imp.CachedRepository.put(CachedRepository.java:77)
    at com.atomikos.recovery.imp.OltpLogImp.write(OltpLogImp.java:31)
    at com.atomikos.persistence.imp.StateRecoveryManagerImp.preEnter(StateRecoveryManagerImp.java:51)
    at com.atomikos.finitestates.FSMImp.notifyListeners(FSMImp.java:164)
    at com.atomikos.finitestates.FSMImp.setState(FSMImp.java:251)
    at com.atomikos.icatch.imp.CoordinatorImp.setState(CoordinatorImp.java:312)
    at com.atomikos.icatch.imp.CoordinatorStateHandler.commitFromWithinCallback(CoordinatorStateHandler.java:346)
    at com.atomikos.icatch.imp.ActiveStateHandler$6.doCommit(ActiveStateHandler.java:267)
    at com.atomikos.icatch.imp.CoordinatorStateHandler.commitWithAfterCompletionNotification(CoordinatorStateHandler.java:581)
    at com.atomikos.icatch.imp.ActiveStateHandler.commit(ActiveStateHandler.java:262)
    at com.atomikos.icatch.imp.CoordinatorImp.commit(CoordinatorImp.java:548)
    at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:685)
    at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:282)
    at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:172)
    at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:414)
    at com.atomikos.icatch.jta.UserTransactionImp.commit(UserTransactionImp.java:86)
    at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1023)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291)
    at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70)
    at ar.com.eox.idlogistics.test.HandheldPickerEmulator.getStoreLocationFeature(HandheldPickerEmulator.java:479)
    at ar.com.eox.idlogistics.test.HandheldPickerEmulator.workPicker(HandheldPickerEmulator.java:192)
    at ar.com.eox.idlogistics.test.HandheldPickerEmulator.start(HandheldPickerEmulator.java:131)
    at ar.com.eox.idlogistics.test.EmulatorService.lambda$4(EmulatorService.java:126)
    at ar.com.eox.idlogistics.test.EmulatorService$$Lambda$63/853683399.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
    at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:200)
    at com.atomikos.recovery.imp.FileSystemRepository.writeToFile(FileSystemRepository.java:91)
    at com.atomikos.recovery.imp.FileSystemRepository.write(FileSystemRepository.java:86)
    at com.atomikos.recovery.imp.FileSystemRepository.writeCheckpoint(FileSystemRepository.java:224)
    ... 33 more
WARN  2016-07-15 12:12:38,338 [pool-18-thread-4] : CoordinatorStateHandler - Error in committing: could not flush state image java.nio.channels.ClosedByInterruptException com.atomikos.recovery.LogWriteException - recovery will clean up in the background
java.lang.IllegalStateException: could not flush state image java.nio.channels.ClosedByInterruptException com.atomikos.recovery.LogWriteException
    at com.atomikos.persistence.imp.StateRecoveryManagerImp.preEnter(StateRecoveryManagerImp.java:54)
    at com.atomikos.finitestates.FSMImp.notifyListeners(FSMImp.java:164)
    at com.atomikos.finitestates.FSMImp.setState(FSMImp.java:251)
    at com.atomikos.icatch.imp.CoordinatorImp.setState(CoordinatorImp.java:312)
    at com.atomikos.icatch.imp.CoordinatorStateHandler.commitFromWithinCallback(CoordinatorStateHandler.java:346)
    at com.atomikos.icatch.imp.ActiveStateHandler$6.doCommit(ActiveStateHandler.java:267)
    at com.atomikos.icatch.imp.CoordinatorStateHandler.commitWithAfterCompletionNotification(CoordinatorStateHandler.java:581)
    at com.atomikos.icatch.imp.ActiveStateHandler.commit(ActiveStateHandler.java:262)
    at com.atomikos.icatch.imp.CoordinatorImp.commit(CoordinatorImp.java:548)
    at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:685)
    at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:282)
    at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:172)
    at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:414)
    at com.atomikos.icatch.jta.UserTransactionImp.commit(UserTransactionImp.java:86)
    at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1023)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291)
    at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70)
    at ar.com.eox.idlogistics.test.HandheldPickerEmulator.getStoreLocationFeature(HandheldPickerEmulator.java:479)
    at ar.com.eox.idlogistics.test.HandheldPickerEmulator.workPicker(HandheldPickerEmulator.java:192)
    at ar.com.eox.idlogistics.test.HandheldPickerEmulator.start(HandheldPickerEmulator.java:131)
    at ar.com.eox.idlogistics.test.EmulatorService.lambda$4(EmulatorService.java:126)
    at ar.com.eox.idlogistics.test.EmulatorService$$Lambda$63/853683399.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.atomikos.recovery.LogWriteException: java.nio.channels.ClosedByInterruptException
    at com.atomikos.recovery.imp.FileSystemRepository.writeCheckpoint(FileSystemRepository.java:233)
    at com.atomikos.recovery.imp.CachedRepository.performCheckpoint(CachedRepository.java:84)
    at com.atomikos.recovery.imp.CachedRepository.put(CachedRepository.java:77)
    at com.atomikos.recovery.imp.OltpLogImp.write(OltpLogImp.java:31)
    at com.atomikos.persistence.imp.StateRecoveryManagerImp.preEnter(StateRecoveryManagerImp.java:51)
    ... 29 more
Caused by: java.nio.channels.ClosedByInterruptException
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
    at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:200)
    at com.atomikos.recovery.imp.FileSystemRepository.writeToFile(FileSystemRepository.java:91)
    at com.atomikos.recovery.imp.FileSystemRepository.write(FileSystemRepository.java:86)
    at com.atomikos.recovery.imp.FileSystemRepository.writeCheckpoint(FileSystemRepository.java:224)
    ... 33 more
ERROR 2016-07-15 12:12:38,348 [pool-18-thread-1] : TransactionImp - Error in rollback: could not flush state image Log corrupted - restart JVM com.atomikos.recovery.LogReadException
com.atomikos.icatch.SysException: Error in rollback: could not flush state image Log corrupted - restart JVM com.atomikos.recovery.LogReadException
    at com.atomikos.icatch.imp.CoordinatorStateHandler.rollbackFromWithinCallback(CoordinatorStateHandler.java:512)
    at com.atomikos.icatch.imp.ActiveStateHandler$7.doRollback(ActiveStateHandler.java:284)
    at com.atomikos.icatch.imp.CoordinatorStateHandler.rollbackWithAfterCompletionNotification(CoordinatorStateHandler.java:563)
    at com.atomikos.icatch.imp.ActiveStateHandler.rollback(ActiveStateHandler.java:279)
    at com.atomikos.icatch.imp.CoordinatorImp.rollback(CoordinatorImp.java:577)
    at com.atomikos.icatch.imp.TransactionStateHandler.rollback(TransactionStateHandler.java:165)
    at com.atomikos.icatch.imp.TransactionStateHandler.rollbackWithStateCheck(TransactionStateHandler.java:180)
    at com.atomikos.icatch.imp.CompositeTransactionImp.doRollback(CompositeTransactionImp.java:201)
    at com.atomikos.icatch.imp.CompositeTransactionImp.rollback(CompositeTransactionImp.java:314)
    at com.atomikos.icatch.jta.TransactionImp.rollback(TransactionImp.java:199)
    at com.atomikos.icatch.jta.TransactionManagerImp.rollback(TransactionManagerImp.java:426)
    at com.atomikos.icatch.jta.UserTransactionImp.rollback(UserTransactionImp.java:97)
    at org.springframework.transaction.jta.JtaTransactionManager.doRollback(JtaTransactionManager.java:1050)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:853)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:830)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:503)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:285)
    at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70)
    at ar.com.eox.idlogistics.test.HandheldPickerEmulator.pick(HandheldPickerEmulator.java:309)
    at ar.com.eox.idlogistics.test.HandheldPickerEmulator.workPicker(HandheldPickerEmulator.java:220)
    at ar.com.eox.idlogistics.test.HandheldPickerEmulator.start(HandheldPickerEmulator.java:131)
    at ar.com.eox.idlogistics.test.EmulatorService.lambda$4(EmulatorService.java:126)
    at ar.com.eox.idlogistics.test.EmulatorService$$Lambda$63/853683399.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: could not flush state image Log corrupted - restart JVM com.atomikos.recovery.LogReadException
    at com.atomikos.persistence.imp.StateRecoveryManagerImp.preEnter(StateRecoveryManagerImp.java:54)
    at com.atomikos.finitestates.FSMImp.notifyListeners(FSMImp.java:164)
    at com.atomikos.finitestates.FSMImp.setState(FSMImp.java:251)
    at com.atomikos.icatch.imp.CoordinatorImp.setState(CoordinatorImp.java:312)
    at com.atomikos.icatch.imp.CoordinatorImp.setStateHandler(CoordinatorImp.java:222)
    at com.atomikos.icatch.imp.CoordinatorStateHandler.rollbackFromWithinCallback(CoordinatorStateHandler.java:506)
    ... 27 more
Caused by: com.atomikos.recovery.LogReadException: Log corrupted - restart JVM
    at com.atomikos.recovery.imp.CachedRepository.assertNotCorrupted(CachedRepository.java:137)
    at com.atomikos.recovery.imp.CachedRepository.get(CachedRepository.java:131)
    at com.atomikos.recovery.imp.OltpLogImp.assertEntryIsAllowedInCurrentState(OltpLogImp.java:35)
    at com.atomikos.recovery.imp.OltpLogImp.write(OltpLogImp.java:30)
    at com.atomikos.persistence.imp.StateRecoveryManagerImp.preEnter(StateRecoveryManagerImp.java:51)
    ... 32 more
Natalia Send private email
Friday, July 15, 2016
 
 
Thanks,

Can you send an atomikos log in DEBUG mode to support@atomikos.com with "case 151740" in the subject line?

We'll have a look ASAP.
Guy Pardon Send private email
Saturday, July 16, 2016
 
 
I sent the email with the log this morning. could you confirm you got it?
Natalia Send private email
Sunday, July 17, 2016
 
 
Got it - thanks
Guy Pardon Send private email
Sunday, July 17, 2016
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics