Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Nothing obvious from the server log. There are a few things that I can think of here.

1) Is there a possibility that someone ran stop-server.sh to stop the server and you are not aware of it ? Or some one issued the "kill" command ?

2) If the server was started in a terminal without using nohup, then logging out or closing the terminal could cause the server to exit. Try starting the server in background using nohup

3) Is there an exception in the console that didnt go to the log ? Maybe some unhandled exception ?

Anything that happens in the clients should not cause the server to exit, so the client logs are not that interesting. But generally client logs go to a file call terracotta-client.log

Can you also add the client log ? From just the server log I couldnt tell but my guess is that the clients are disconnecting due to long GCs in the server.

Alternatively you could run with verbosegc and check the gc timing at the server. If it is too long, then you have to do GC tuning (increase heap, increase no of parallel gc threads, use different surviver ratio etc.) Increasing heap normally helps provided your machine has enough RAM.

Thats weird. Do you still get the exception if you start the cluster fresh ? ie. remove the object db, and restart the servers. I am wondering if the object db is somehow corrupted.

If you still get the exception, please post the entire client/server logs and we will look into it.

mman,

Your jstat output seems to indicate perm space is close to full, but other than that it seem to have a lot of memory. Also looking at the logs again, there seem to be enought free space according to the logging.

Also noticed that its the server that kicked the client out, because of our network healthcheck. It was not able to do a ping probe or network socket connect to the L1 process. I suspect that there was some intermittent network problem which lasted for 2 minutes or more and hence the disconnect.

If this happens again, you could increase the timeout even further if you want clients to be resilient of such network failures.

cheers,
Saravanan

I think manoj is talking about full GCs in the VM, not DGC. What are your heap settings, GC stats ? You could monitor GC using vmstat.

There seem to be a lot of responses already and seems like you made some progress. Somethings I would like to point out are

1) Initial disconnects seemed to be caused by long GC in the server. The tuning guides talk about how to identify and tune them. I think you already made some progress here.

2) Note that you may also have to tune the passive servers to avoid long GCs there. The same settings should work here too. There is health check between the active and the passive too. So tuning GC will help that, also you could tune health check parameters too.

3) We are working to reproduce the concurrent modification exception on transaction folding. The "TransactionSequencer - Sequencer shutdown" loggings are a result of that. You could in the mean time switch off folding by adding this tc.property to your settings. Note that this is a l1 property and you need to add it either in ur config or in every l1 installation for it take effect.

com.tc.l1.transactionmanager.folding.enabled=false

4) Reconnect is to protect again network interruptions, but if the nodes detects long GC, then it will disconnect. The printing you are seeing in the passive is a side effect of reconnect trying to connect when healthcheck closed the connection. The logging should just be a harmless side effect, I think. We fixed this in latest 2.7 and trunk.

Post the logs if you are still experiencing problems after tuning GC and disabling transaction folding.

cheers,
Saravanan

In 2.7 we have improved DGC performance significantly for certain usecases. So if you are not running 2.7, I would recommend trying that.

Also in 2.7 we introduced young generational garbage collection. It can be turned on by setting the following properties in tc.properties.

l2.objectmanager.dgc.young.enabled = true
l2.objectmanager.dgc.young.frequencyInMillis = 180000

By turning it on, we will run a modified/optimized DGC algorithm more often (3 mins by default) to collect objects that become garbage soon after they are shared. If your usecase creates a lot of such garbage, then this will be useful. Note that you may have to tune the L1 heap to get the best results.

Some of this is covered in our tuning guide that can be found here.

http://www.terracotta.org/web/display/orgsite/Tuning+Terracotta

Having said all that, does 10 million objects as garbage every hour sound right for your usecase ? Reducing the amount of shared objects / garbage in your application will of course help DGC. You can look at the shared graph in the admin console to see if that is possible in your case. Many times we have had users discover that they are sharing more than they intended.

Hope that helps !

Unfortunately in 2.7 the logging doesnt print the remote ip address. I believe that in trunk the logging is enhanced to print the remote Ip too.

Meanwhile you could use netstat to inspect who is connecting (and disconnecting) to the terracotta server.

We have had the discussion about how to flush just the right amount of objects from the L1 and it is a tough problem to solve.

Currently the cachemanager takes a conservative approach to start with. But it has a learning heuristics where after the first GC, if the heap fell to 45 %, then it knows that it can hold more objects and the next time when the heap reaches 75%, it doesnt immediately flush objects. It constantly monitors the heap and gc characteristic and adjusts its calculations. So the idea is that over time it will be able to do a good job at predicting how much to flush.

Like I said earlier, we are planning on improving this area of the product in the near future.

Using soft references, the cachemanager is basically pushing its job to the vm's gc. Seems like a neat idea but there are some drawbacks.

Currently the cachemanager has complete control over which ones to flush and which ones not to, based on user access pattern and such. We will lose this if we go with soft references. We will be at the total discretion of the VM.

Also had some not so good experiences with soft references where the VM clear a lot or just too little but this was long time ago, so it might have improved.

Like Alex mentioned we will be working on improving the algorithm around flushing in the near future.

Keep these ideas flowing in, we much appreciate it.

Reconnect is not enabled at the client but is enabled in the server.

In latest releases, we read the stack configuration from the server when the client first connects to the server. So there should be no misconfiguration.

But in an active-passive setup, if one server is setup with reconnect on and another with reconnect off, then when a client fails over, this stack mismatch can happen. In that case, make sure all the servers in your cluster are having the same value for reconnect.

This is configured in the tc.properties as

l2.l1reconnect.enabled = true

It is configured through tc-config.xml. The element name is called fault-count. The documentation for it is available at http://www.terracotta.org/web/display/docs/Configuration+Guide+and+Reference

eddie wrote:

But problem on client side still exists - server always try to send whole content of shared root to client.

Actually the server doesnt send all the shared objects in one shot. It prefetches some amount of reachable objects though and the amount it prefetches depends on the fault depth that you set in your config. This is done as an optimization so that the clients dont have to wait for the next lookups.

By default this number is 500. (and Interger or anyother literals dont count) You could set the fault depth to 0 or something low if you dont want this behavior.

I believe the Out Of Memory Exception in the server is could be caused by some throttling issues between the flushing and faulting stage when the load is high and the server is run in non-persistent mode.

Can you please try running the server in persistent mode and see if the server still OOMEs ? Running the server in persistent mode makes sure that all your data will survive across server crash/ power failure etc. This is how many of our users run us in their production environment. So we have spent a lot more time here and implemented better throttling algorithms.

Running it locally, we were not able to reproduce this OOME in persistent mode. Can you please confirm this ?

You can run the server in persistent mode by add the following to the dso section under the server section in your config.
Code:

 <persistence>
       <mode>permanent-store</mode>
 </persistence>

je.properties is properties used by berkeley db and can be used as explained here.

But a easy way to configure any je property in terracotta is to add the prefix "l2.berkeleydb." to it and add it to tc.properties file that is used to tune terracotta. More on tc.properties can be found here.

The properties mentioned about tuning cleaner thread will only help if DGC is fast but the cleaner thread is falling behind in cleaning up space. In your case you are creating a lot of garbage and hence dgc is taking a long time to delete those objects. I would suggest turning verbose gc setting on in the config the next time you have a chance to restart and it will print more details about how long it took for each stage.

BTW, 2.7 will have many optimizations around DGC which will make it run faster for various usecases.