Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Can you please try "@ManyToMany(fetch = FetchType.EAGER)" for groups property?

Can you please send us your hibernate config files and the ehcache.xml?

Can you please let us know about your use case in brief? I would like to understand why do you want to use DSO product and can you not meet the requirement using Express Ehcache.

Creating dso-boot jar is not risky and is running in production with lots of customers but there should be a strong reason to use DSO rather than Express.

Cam you try by setting this property to true:
Code:

org.quartz.scheduler.threadsInheritContextClassLoaderOfInitializer

If this does not work you can try tuning this property:
Code:

org.quartz.scheduler.classLoadHelper.class

You can refer this link for documentation on these properties:
http://www.quartz-scheduler.org/docs/configuration/ConfigMain.html

If that is the case then you should look at the total heap allocation and what are the other objects that are occupying the heap. Might be little GC tuning will help you. Also see when the Full GC cycle runs how much memory is it able to claim.

This means that as you are now caching more data your application is running into memory pressure and thus is running back to back full gcs to free the heap for the application. To fix this you can chose from these options:
a. increase the heap size of the java process but with larger heaps you can run into longer full gc cycles
b. use big memory - which will enable you to cache larger amount of data in memory and still run your java process with smaller heaps and thus have shorter gc cycles and predictive SLAs. You can read more on Big Memory at: http://www.terracotta.org/bigmemory

Can you send your ehcache.xml? Your analysis doesn't seem correct because even if the L2 has maxElementsOnDisk no of elements that will only trigger a background eviction thread to clear the elements from the cache but that does not in any way imply that new elements can not be added to the cache.

Which version of ehcache and terracotta are you on?

This would largely depend on how much time would it take for job completion and what is your misfire Threshold.

misfireThreshold by default is 1 min and if in 1 min no thread is able to schedule the job then your triggers would be considered misfired.

Now if in 1 min time you bring up all the nodes then in most likely hood the triggers should be fired and also if the jobs complete in some seconds (or ms) then the threads would be free again to schedule other jobs so you can not ascertain that all the nodes would be running 2 jobs. Also all this is when all your 16 triggers have same fire time, if your triggers have different fire times then again any node which has free threads can schedule the jobs.

No, there are no inbuilt interceptors, but you can write your own ehcache wrappers which can perform what you need

If you still see issues can you take heap histogram on the L1's and see what are the objects that are occupying the heap?

Looking at your ehcache.xml, you have not set maxElementsOnDisk so what that means is that the periodic evictor will not be able to evict any data. Based on your tti/ttl settings only inline evictors will evict data. So you should set some value for maxElementsOnDisk for all the caches and that will start evicting your caches on L2.

But from the post it looks like your L1's are running under memory pressure. Have you tried running with lesser critical threshold for l1's. By default the critical threshold is set to 90% and threshold to 70% you can reduce it to 70 and 50 respectively and give it a try.

And as Eric had pointed out running with such large heaps is not a good idea as you can run into really long gc cycles (14 sec is not very long for such big heaps), you should continuously monitor your GC cycles. I am pretty sure you will see much longer gc cycles with 12gb heap. you should consider using "Big Memory" solution from Terracotta.

I think you are looking for cache event listeners. If that is the case have a look at http://ehcache.org/documentation/cache_event_listeners.html

Can you elaborate on what your are looking for?

Yes if you are running distributed ehcache (i.e ehcache backed by terracotta server array) you can be sure that the changes made by one node will always be visible to other nodes in the cluster. In coherent mode it will be available immediately and in in-coherent mode you may see stale data for a few seconds, so you need to make choice on the modes based on your application correctness and tps requirements.

Ehcache does not provide any thing out of the box but you can easily write a CacheWrapper which would encrypt the values before putting in the cache and decrypt it while retrieving it.