Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Yes, that size warning you see is from some fairly large pieces of data that's being used.

We have these fairly large data files that are parsed then cached.

Not quite sure why there are so many of these warning.

But these shouldn't cause a Direct buffer leak.

Ok, the issue has not gone away.

One thing I do have as a JVM option is -XX:+DisableExplicitGC. I read somewhere that having that option may lead to NIO memory leaks so I removed the option and tried again.

My observations were that it was still leaking into direct memory AND ignoring the -XX:MaxDirectMemorySize option !!!

With the DisableExplicitGC option, the MaxDirectMemorySize option was honored and the process would die once the direct memory maximum size was reached/exceeded.

Without the DisableExplicitGC option, direct memory continues to grow beyond the bounds specified by MaxDirectMemorySize and my belief is that it will continue to grow until all system memory has been exhausted leading to machine crash.

I would have expected MaxDirectMemorySize to be honored with or without DisableExplicitGC specified.

The behavior is quite odd and unexpected.

Does anyone have any explanation for this behavior?

Thank you.

I came across this thread and wanted to add to it.

I am also seeing the exact issue with the tc server memory growing continuously and finally going non-responsive.

I see these statements
2015-03-11 10:01:58,096 [WorkerThread(respond_to_request_stage, 5, 0)] WARN com.tc.net.core.TCConnection - Warning: Attempting to send a message (com.tc.net.protocol.delivery.OOOProtocolMessageImpl) of size 18689321 bytes
a lot on the tc log.

My use case : I use a distributed queue to push "jobs" which are are executed by the worker nodes. the jobs have data in xml (size is not relevant) that is executed by the worker nodes by executing a jni call.

my process xmx is 12 G and the process easily eats that up and goes upto 15G (machine is 16G).

i have attached my tc-config.xml

I finally figured it out.

Do not use this JVM option to start Terracotta: -XX:+DisableExplicitGC

Using that JVM option prevented Terracota from garbage collecting the direct memory so it just grew without bounds !!!

Arrgggghhhh.

So much pain