Terracotta Discussion Forums

Hi,

Im a newbie with the nonstop cache and rejoin behaviour.
below is a snippit of a change to our current configuration:
what have I misconfigured?

<terracotta>
<nonstop
timeoutMillis="10000">
<timeoutBehavior
type="noop">
</timeoutBehavior>
</nonstop>
</terracotta>
</cache>
<terracottaConfig
url="localhost:9532"
rejoin="true">
</terracottaConfig>

now I shutdown jboss (the t1), the server wont shutdown
the console makes reference to the following:

Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Trying to rejoin again in 5000 msecs...

a fuller snippit is as below
***************************************************************
09:55:19,718 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Trying to rejoin again in 5000 msecs...
09:55:21,492 INFO [BaseComponentLoader] BasecomponentLoader:dispose
09:55:21,492 INFO [BaseComponentLoader] BasecomponentLoader:dispose
09:55:24,719 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Starting Terracotta Rejoin (as client id: ClientID[2] left the cluster) [rejoin count = 2] ...
09:55:24,719 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Shutting down old ClusteredInstanceFactory...
09:55:24,720 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Creating new ClusteredInstanceFactory
09:55:24,792 ERROR [STDERR] 2011-12-13 09:55:24,792 INFO - Terracotta 3.5.2, as of 20110707-110722 (Revision 18007 by cruise@kong from 3.5.2)
565.366: [GC [PSYoungGen: 119648K->44943K(120640K)] 441682K->366977K(470208K), 0.0604860 secs] [Times: user=0.17 sys=0.00, real=0.06 secs]
09:55:25,279 ERROR [STDERR] 2011-12-13 09:55:25,279 INFO - Successfully loaded base configuration from server at 'localhost:9532'.
09:55:25,398 ERROR [STDERR] 2011-12-13 09:55:25,398 INFO - Successfully loaded base configuration from file at '/tmp/tc-config5304510746656742323.xml'.
09:55:25,421 ERROR [STDERR] 2011-12-13 09:55:25,421 INFO - Log file: '/home/anadkats/terracotta/client-logs/terracotta-client.log'.
09:55:25,837 WARN [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Unable to register node left listener for rejoin
09:55:25,837 WARN [TerracottaClient] Caught exception while trying to rejoin cluster
net.sf.ehcache.CacheException: Unable to load class net.sf.ehcache.terracotta.StandaloneTerracottaClusteredInstanceFactory. Initial cause was java.lang.IllegalStateException: Shutdown in progress
at net.sf.ehcache.util.ClassLoaderUtil.createNewInstance(ClassLoaderUtil.java:109)
at net.sf.ehcache.terracotta.TerracottaClusteredInstanceHelper.newClusteredInstanceFactory(TerracottaClusteredInstanceHelper.java:178)
at net.sf.ehcache.terracotta.TerracottaClient.createNewClusteredInstanceFactory(TerracottaClient.java:199)
at net.sf.ehcache.terracotta.TerracottaClient.access$1500(TerracottaClient.java:48)
at net.sf.ehcache.terracotta.TerracottaClient$RejoinWorker.doRejoin(TerracottaClient.java:397)
at net.sf.ehcache.terracotta.TerracottaClient$RejoinWorker.run(TerracottaClient.java:337)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Shutdown in progress
at org.terracotta.express.ClientImpl.<init>(ClientImpl.java:318)
at org.terracotta.express.ClientFactoryImpl.newClient(ClientFactoryImpl.java:194)
at org.terracotta.express.ClientFactoryImpl.createClient(ClientFactoryImpl.java:187)
at org.terracotta.express.ClientFactoryImpl.createDedicatedClient(ClientFactoryImpl.java:121)
at org.terracotta.express.ClientFactoryExtras.createDedicatedClient(ClientFactoryExtras.java:34)
at net.sf.ehcache.terracotta.StandaloneTerracottaClusteredInstanceFactory.<init>(StandaloneTerracottaClusteredInstanceFactory.java:35)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at net.sf.ehcache.util.ClassLoaderUtil.createNewInstance(ClassLoaderUtil.java:92)
... 6 more
Caused by: java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:39)
at java.lang.Runtime.addShutdownHook(Runtime.java:192)
at com.tc.object.bytecode.ManagerImpl.<init>(ManagerImpl.java:155)
at com.tc.object.bytecode.hook.impl.DSOContextImpl.createStandaloneContext(DSOContextImpl.java:166)
at org.terracotta.express.StandaloneL1Boot.call(StandaloneL1Boot.java:190)
at org.terracotta.express.ClientImpl.<init>(ClientImpl.java:309)
... 16 more
09:55:25,837 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Trying to rejoin again in 5000 msecs...
09:55:30,839 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Starting Terracotta Rejoin (as client id: ClientID[2] left the cluster) [rejoin count = 3] ...
09:55:30,839 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Shutting down old ClusteredInstanceFactory...
09:55:30,839 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Creating new ClusteredInstanceFactory
571.253: [GC [PSYoungGen: 120591K->41659K(116800K)] 442625K->369021K(466368K), 0.0569350 secs] [Times: user=0.20 sys=0.01, real=0.06 secs]
09:55:30,977 ERROR [STDERR] 2011-12-13 09:55:30,977 INFO - Terracotta 3.5.2, as of 20110707-110722 (Revision 18007 by cruise@kong from 3.5.2)
09:55:31,375 ERROR [STDERR] 2011-12-13 09:55:31,375 INFO - Successfully loaded base configuration from server at 'localhost:9532'.
09:55:31,459 ERROR [STDERR] 2011-12-13 09:55:31,459 INFO - Successfully loaded base configuration from file at '/tmp/tc-config2082536002787342455.xml'.
09:55:31,495 ERROR [STDERR] 2011-12-13 09:55:31,494 INFO - Log file: '/home/anadkats/terracotta/client-logs/terracotta-client.log'.
571.932: [GC [PSYoungGen: 104763K->41217K(104384K)] 432125K->371133K(453952K), 0.0470520 secs] [Times: user=0.17 sys=0.00, real=0.05 secs]
09:55:31,944 WARN [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Unable to register node left listener for rejoin
09:55:31,944 WARN [TerracottaClient] Caught exception while trying to rejoin cluster
net.sf.ehcache.CacheException: Unable to load class net.sf.ehcache.terracotta.StandaloneTerracottaClusteredInstanceFactory. Initial cause was java.lang.IllegalStateException: Shutdown in progress
at net.sf.ehcache.util.ClassLoaderUtil.createNewInstance(ClassLoaderUtil.java:109)
at net.sf.ehcache.terracotta.TerracottaClusteredInstanceHelper.newClusteredInstanceFactory(TerracottaClusteredInstanceHelper.java:178)
at net.sf.ehcache.terracotta.TerracottaClient.createNewClusteredInstanceFactory(TerracottaClient.java:199)
at net.sf.ehcache.terracotta.TerracottaClient.access$1500(TerracottaClient.java:48)
at net.sf.ehcache.terracotta.TerracottaClient$RejoinWorker.doRejoin(TerracottaClient.java:397)
at net.sf.ehcache.terracotta.TerracottaClient$RejoinWorker.run(TerracottaClient.java:337)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Shutdown in progress
at org.terracotta.express.ClientImpl.<init>(ClientImpl.java:318)
at org.terracotta.express.ClientFactoryImpl.newClient(ClientFactoryImpl.java:194)
at org.terracotta.express.ClientFactoryImpl.createClient(ClientFactoryImpl.java:187)
at org.terracotta.express.ClientFactoryImpl.createDedicatedClient(ClientFactoryImpl.java:121)
at org.terracotta.express.ClientFactoryExtras.createDedicatedClient(ClientFactoryExtras.java:34)
at net.sf.ehcache.terracotta.StandaloneTerracottaClusteredInstanceFactory.<init>(StandaloneTerracottaClusteredInstanceFactory.java:35)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at net.sf.ehcache.util.ClassLoaderUtil.createNewInstance(ClassLoaderUtil.java:92)
... 6 more
Caused by: java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:39)
at java.lang.Runtime.addShutdownHook(Runtime.java:192)
at com.tc.object.bytecode.ManagerImpl.<init>(ManagerImpl.java:155)
at com.tc.object.bytecode.hook.impl.DSOContextImpl.createStandaloneContext(DSOContextImpl.java:166)
at org.terracotta.express.StandaloneL1Boot.call(StandaloneL1Boot.java:190)
at org.terracotta.express.ClientImpl.<init>(ClientImpl.java:309)
... 16 more
09:55:31,944 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Trying to rejoin again in 5000 msecs...
573.060: [GC [PSYoungGen: 104321K->44871K(115456K)] 434237K->376588K(465024K), 0.0666690 secs] [Times: user=0.23 sys=0.01, real=0.07 secs]
09:55:36,946 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Starting Terracotta Rejoin (as client id: ClientID[2] left the cluster) [rejoin count = 4] ...
09:55:36,946 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Shutting down old ClusteredInstanceFactory...
09:55:36,946 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Creating new ClusteredInstanceFactory
09:55:37,288 ERROR [STDERR] 2011-12-13 09:55:37,288 INFO - Terracotta 3.5.2, as of 20110707-110722 (Revision 18007 by cruise@kong from 3.5.2)
578.054: [GC [PSYoungGen: 103111K->45221K(116480K)] 434828K->384018K(466048K), 0.0961150 secs] [Times: user=0.30 sys=0.02, real=0.10 secs]
09:55:38,446 ERROR [STDERR] 2011-12-13 09:55:38,446 INFO - Successfully loaded base configuration from server at 'localhost:9532'.
09:55:38,619 ERROR [STDERR] 2011-12-13 09:55:38,619 INFO - Successfully loaded base configuration from file at '/tmp/tc-config8647598618778360374.xml'.
09:55:38,658 ERROR [STDERR] 2011-12-13 09:55:38,658 INFO - Log file: '/home/anadkats/terracotta/client-logs/terracotta-client.log'.
579.523: [GC [PSYoungGen: 103461K->42715K(116480K)] 442258K->386188K(466048K), 0.0962080 secs] [Times: user=0.27 sys=0.00, real=0.10 secs]
579.627: [Full GC [PSYoungGen: 42715K->0K(116480K)] [PSOldGen: 343473K->310533K(349568K)] 386188K->310533K(466048K) [PSPermGen: 178231K->170978K(182528K)], 1.8628100 secs] [Times: user=1.85 sys=0.00, real=1.87 secs]
09:55:41,304 WARN [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Unable to register node left listener for rejoin
09:55:41,304 WARN [TerracottaClient] Caught exception while trying to rejoin cluster
net.sf.ehcache.CacheException: Unable to load class net.sf.ehcache.terracotta.StandaloneTerracottaClusteredInstanceFactory. Initial cause was java.lang.IllegalStateException: Shutdown in progress
at net.sf.ehcache.util.ClassLoaderUtil.createNewInstance(ClassLoaderUtil.java:109)
at net.sf.ehcache.terracotta.TerracottaClusteredInstanceHelper.newClusteredInstanceFactory(TerracottaClusteredInstanceHelper.java:178)
at net.sf.ehcache.terracotta.TerracottaClient.createNewClusteredInstanceFactory(TerracottaClient.java:199)
at net.sf.ehcache.terracotta.TerracottaClient.access$1500(TerracottaClient.java:48)
at net.sf.ehcache.terracotta.TerracottaClient$RejoinWorker.doRejoin(TerracottaClient.java:397)
at net.sf.ehcache.terracotta.TerracottaClient$RejoinWorker.run(TerracottaClient.java:337)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Shutdown in progress
at org.terracotta.express.ClientImpl.<init>(ClientImpl.java:318)
at org.terracotta.express.ClientFactoryImpl.newClient(ClientFactoryImpl.java:194)
at org.terracotta.express.ClientFactoryImpl.createClient(ClientFactoryImpl.java:187)
at org.terracotta.express.ClientFactoryImpl.createDedicatedClient(ClientFactoryImpl.java:121)
at org.terracotta.express.ClientFactoryExtras.createDedicatedClient(ClientFactoryExtras.java:34)
at net.sf.ehcache.terracotta.StandaloneTerracottaClusteredInstanceFactory.<init>(StandaloneTerracottaClusteredInstanceFactory.java:35)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at net.sf.ehcache.util.ClassLoaderUtil.createNewInstance(ClassLoaderUtil.java:92)
... 6 more
Caused by: java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:39)
at java.lang.Runtime.addShutdownHook(Runtime.java:192)
at com.tc.object.bytecode.ManagerImpl.<init>(ManagerImpl.java:155)
at com.tc.object.bytecode.hook.impl.DSOContextImpl.createStandaloneContext(DSOContextImpl.java:166)
at org.terracotta.express.StandaloneL1Boot.call(StandaloneL1Boot.java:190)
at org.terracotta.express.ClientImpl.<init>(ClientImpl.java:309)
... 16 more
09:55:41,304 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Trying to rejoin again in 5000 msecs...
09:55:41,499 INFO [BaseComponentLoader] BasecomponentLoader:dispose
09:55:41,499 INFO [BaseComponentLoader] BasecomponentLoader:dispose
09:55:46,306 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Starting Terracotta Rejoin (as client id: ClientID[2] left the cluster) [rejoin count = 5] ...
09:55:46,306 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Shutting down old ClusteredInstanceFactory...
09:55:46,307 INFO [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Creating new ClusteredInstanceFactory
09:55:46,378 ERROR [STDERR] 2011-12-13 09:55:46,378 INFO - Terracotta 3.5.2, as of 20110707-110722 (Revision 18007 by cruise@kong from 3.5.2)
586.906: [GC [PSYoungGen: 58240K->2174K(116480K)] 368773K->312708K(466048K), 0.0511420 secs] [Times: user=0.14 sys=0.01, real=0.06 secs]
09:55:46,878 ERROR [STDERR] 2011-12-13 09:55:46,878 INFO - Successfully loaded base configuration from server at 'localhost:9532'.
09:55:46,980 ERROR [STDERR] 2011-12-13 09:55:46,979 INFO - Successfully loaded base configuration from file at '/tmp/tc-config8987681862348710258.xml'.
09:55:47,002 ERROR [STDERR] 2011-12-13 09:55:47,002 INFO - Log file: '/home/anadkats/terracotta/client-logs/terracotta-client.log'.
587.615: [GC [PSYoungGen: 60414K->3989K(116480K)] 370948K->314523K(466048K), 0.0475420 secs] [Times: user=0.14 sys=0.00, real=0.05 secs]
09:55:47,426 WARN [TerracottaClient] Thread [Rejoin Worker Thread [cacheManager: webConfig]] [cacheManager: 'webConfig']: Unable to register node left listener for rejoin
09:55:47,426 WARN [TerracottaClient] Caught exception while trying to rejoin cluster

**************************************************************

Please provide us the ehcache.xml , tc-config.xml and sample code to replicate this issue.
BTW, you should move to 3.5.3 version as it has some critical bug fixes.

Thanks