Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Hi,

since 2 weeks our production enviroment gets down multiple times a day, that means both primary and backup terracotta server exit with:
2011-10-24 07:44:27,875 [CommonShutDownHook] INFO com.terracottatech.dso - L2 Exiting...
=====================
Looking at the logfile I noticed a "com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 0 LOG_CHECKSUM" exception entry and quickly after that the processes terminate.
2011-10-24 07:44:24,462 [WorkerThread(commit_changes_stage, 3, 0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(commit_changes_stage, 3, 0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
com.tc.objectserver.persistence.db.DBException: com.sleepycat.je.EnvironmentFailureException: (JE 4.1.10) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) /home/terracotta/tc-data/configs/../data/objectdb com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 0 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed.
at com.tc.objectserver.storage.berkeleydb.BerkeleyDBPersistenceTransactionProvider.newTransaction(BerkeleyDBPersistenceTransactionProvider.java:25)
at com.tc.objectserver.tx.ServerTransactionManagerImpl.commit(ServerTransactionManagerImpl.java:386)
at com.tc.objectserver.handler.CommitTransactionChangeHandler.handleEvent(CommitTransactionChangeHandler.java:31)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:135)
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 4.1.10) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) /home/terracotta/tc-data/configs/../data/objectdb com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 0 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1455)
at com.sleepycat.je.Environment.checkEnv(Environment.java:2118)
at com.sleepycat.je.Environment.beginTransactionInternal(Environment.java:1280)
at com.sleepycat.je.Environment.beginTransaction(Environment.java:1262)
at com.tc.objectserver.storage.berkeleydb.BerkeleyDBPersistenceTransactionProvider.newNativeTransaction(BerkeleyDBPersistenceTransactionProvider.java:30)
at com.tc.objectserver.storage.berkeleydb.BerkeleyDBPersistenceTransactionProvider.newTransaction(BerkeleyDBPersistenceTransactionProvider.java:23)
... 3 more
=====================

And we get variation of this exception like:

=====================
com.sleepycat.je.EnvironmentFailureException: (JE 4.1.10) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) /home/terracotta/tc-data/configs/../data/objectdb com.sleepycat.je.log.ChecksumException: Read invalid log entry type: -117 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid.

com.tc.objectserver.persistence.db.DBException: com.sleepycat.je.EnvironmentFailureException: (JE 4.1.10) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) /home/terracotta/tc-data/configs/../data/objectdb com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 122 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed.
at com.tc.objectserver.storage.berkeleydb.BerkeleyDBPersistenceTransactionProvider.newTransaction(BerkeleyDBPersistenceTransactionProvider.java:25)

com.sleepycat.je.EnvironmentFailureException: (JE 4.1.10) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) /home/terracotta/tc-data/configs/../data/objectdb com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 0 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x14/0x261bcb parent IN=8 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x14/0x1d2729 parent.getDirty()=true state=2
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)
=====================

Any clue why this happens now? Our enviroment is
- Red Hat Linux 10-163-192-11 2.6.18-238.12.1.el5 #1 SMP Sat May 7 20:18:50 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
- Java -> Java(™) SE Runtime Environment (build 1.6.0_27-b07) Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
- Terracotta 3.5.3 running in permanent-store, networked-active-passive each with a local harddrive on /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)

Any hint would be very appreciated as the production system is bascially unusable atm. We updated from terracotta 3.2.1 to 3.5.3 last week, but no improvement concerning the BerkeleyDB exceptions.
I also didn't find an explanation in the Oracle forums.

Any suggestions how to localize the problem?

Aloha
Robert

If I have to guess, it could be a Berkeley DB bug or a corrupt disk. With the upgrade of Terracotta, did you throw away the data ? If you did and started fresh and still hit this exception, it could be caused by a corrupt disk.

The last time we closely looked at a problem similar to this, we found bad sectors in the hard disk caused the corruption.