Latest posts for the topic "Quartz Terracotta Locking Up"

Quartz Terracotta Locking Up

kizoom — GMT

Hi, We're using distributed Quartz under quite high load e.g. 5 jobs per second. We initially used the JDBC JobStore but hit problems with locking. We have switched to using Terracotta which has improved things but still leads to problems. Initially the system behaves well but gradually the schedulers lock up and stop working. The problem manifests with multiple threads (100) trying to delete jobs. By inspecting with YourKit I can see these threads waiting on the following call: Code:

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
com.tc.object.locks.ClientLockManagerImpl.waitUntilRunning()
com.tc.object.locks.ClientLockManagerImpl.lock(LockID, LockLevel)
com.tc.object.bytecode.ManagerImpl.lock(LockID, LockLevel)
org.terracotta.locking.TerracottaLock.lock()
org.terracotta.quartz.ClusteredJobStore.lock()
org.terracotta.quartz.ClusteredJobStore.getTriggersForJob(SchedulingContext, String, String)
org.terracotta.quartz.PlainTerracottaJobStore.getTriggersForJob(SchedulingContext, String, String)
org.terracotta.quartz.TerracottaJobStore.getTriggersForJob(SchedulingContext, String, String)
org.quartz.core.QuartzScheduler.getTriggersOfJob(SchedulingContext, String, String)
org.quartz.core.QuartzScheduler.deleteJob(SchedulingContext, String, String)
org.quartz.impl.StdScheduler.deleteJob(String, String)

Attached is a copy of our terracotta config and a snippet of our Spring Quartz Scheduler config. We followed some online advice and tried distributing our jobs over multiple scheduler (5) but the problem still occurs. Does anyone have any ideas? Cheers, Matthew

Re:Quartz Terracotta Locking Up

alexsnaps — GMT

Can you please provide what version of Quartz & Terracotta you are using ? Thanks!

Re:Quartz Terracotta Locking Up

kizoom — GMT

Teracotta 3.4.1 Quartz 1.8.4 Cheers, Matthew

Re:Quartz Terracotta Locking Up

kizoom — GMT

Oh and the quartz/terracotta integration stuff is 1.2.1 (which I think was what was bundled with the Teracotta 3.4.1 distribution).

Re:Quartz Terracotta Locking Up

kbhasin — GMT

Is the lockup permanent or does the system return to normal behavior after a while? It would really help if you can post some thread dumps (application JVMs and Terracotta Server JVMs) from when the lockup occurs. Thanks.

Re:Quartz Terracotta Locking Up

teck — GMT

That call to waitUntilRunning() means that you have been disconnected from the terracotta server for some reason. Having a look in the terracotta client and server log files should provide a few more clues. A common cause of disconnects (other than real network errors) are excessive GC tripping a health check ping

Re:Quartz Terracotta Locking Up

kizoom — GMT

kbhasim: The lock up seems to be permanent. I'll try to get some thread dumps for you. Thanks.

Re:Quartz Terracotta Locking Up

kizoom — GMT

teck: Thanks. I'll check this out.