Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

You can config the timeout attribute timeoutMillis at <nonstop> sub-element of <terracotta> sub-element.

refer http://ehcache.org/documentation/non_stop_cache.html

Once client disconnected from server, client won't be allowed to connect back to server UNLESS l1reconnect enabled.
Enabled by tc.properties l2.l1reconnect.enabled = true. The default time for reconnect is 5sec l2.l1reconnect.timeout.millis = 5000, you can tune it according to your network environment.

tc-config.xml
Code:

    <tc-properties>
          <property name="l2.l1reconnect.enabled" value="true" />
      </tc-properties>

Can you post terracotta server logs for both active and passive.

Did you use the same tc-config.xml for starting TC server?

The config error indicated server was started with only one specified.
ConnInfo from L2[150.0.149.103:9510:Tc-Group-0 ]

Try this link http://www.terracotta.org/documentation/3.1.x/product-documentation-11.html

You can't specify dso-boot.jar in express mode.

Caused by: java.lang.RuntimeException: The Terracotta dso-boot.jar is specified via -Xbootclasspath. This is not a correct configuration, please remove it
at org.terracotta.express.ClientFactoryImpl.testForBootJar(ClientFactoryImpl.java:216)

Have you set maxElementsInMemory?
You can refer more info at http://ehcache.org/documentation/configuration.html

Why don't you just wrap it in a shell script or batch file?

Not clear why your server was existing? Did you see any exception? Can you provide more info to know more about the problem, such as tc-config.xml, terracotta-*.log

It is internal design, probably no document for it. The L2 has house keeping to know about states of all connected clients so that to know who needs to apply and who doesn't. There is also an enterprise feature DCV2 which behaves differently without applying transactions on clients and no waiting on acks from clients.

The client B was trying to flush(complete) a transaction to L2. Then L2 applying that to client A and waiting for A's ack which didn't happen. If L2 could clean up client A then drop waiting for A. Client B could resume the operation.

The mtc.disconnect() is to info the tc to remove the dead client so that TC can do house clean up. One thing is to revoke locks held by dead client. The dead client may hold locks which blocking other clients.

Thanks for uploaded logs. Seems a close event missed but not able to identify the cause. It that does happen again, can you take thread dump on servers and all clients. Since they may not connected, take couple thread dumps manually on each process by "kill -3 <pid>".
Thanks.

Difficult to tell from the portion of server log posted.
Can you post full server log and client log. Do you know the cause of first instance of client disconnection, was there any long gc on either side? It will help if you have gc logging. Was there any networking abnormality?

At that 15sec l1reconnect period, L2 is on hold for all operations till L1 reconnects back or gives up that L1 and then resumes the operations.
That saying L1 is out of syc with cluster, so L2 rejects any further connection from that L1 unless you restart it.