Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Hi

We are having serious performance issues when using Quartz with the JDBCJobStore. The application exposes a web service and when receiving a web service request schedules a Quartz job to process the request asynchronously.

The application runs in JBoss 4.0.3. There are two JBoss instances behind a load balancer configured to use an Oracle 10.2.0.4 database as the JobStore. Performance is acceptable when using a single instance but when using both instances the performance degrades rapidly. Our DBAs identified significant deadlocking in the database and found that the database was spending over 80% of its time executing the statement "SELECT * FROM QRTZ_LOCKS WHERE LOCK_NAME = :1 FOR UPDATE". We have tried changing the selectWithLockSQL configuration property to remove the "FOR UPDATE" but that caused other locking issues.

When using RAMJobStore the performance is as expected.

The quartz.properties used by the application is listed below. We tried increasing the threadCount to 150 with a dataSource maxConnections increased to 170 but performance remained poor.

Currently we are using Quartz version 1.7.3

Code:

org.quartz.scheduler.instanceName = VOFScheduler
 org.quartz.scheduler.instanceId = AUTO
 org.quartz.scheduler.rmi.export = false
 org.quartz.scheduler.rmi.proxy = false
 org.quartz.scheduler.wrapJobExecutionInUserTransaction = false
 org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
 org.quartz.threadPool.threadCount = 40
 org.quartz.threadPool.threadPriority = 5
 org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread = true
 org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
 org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
 org.quartz.jobStore.dataSource = quartzDs
 org.quartz.jobStore.misfireThreshold = 120000
 org.quartz.jobStore.isClustered = true
 org.quartz.plugin.shutdownhook.class =  org.quartz.plugins.management.ShutdownHookPlugin
 org.quartz.plugin.shutdownhook.cleanShutdown = true
 org.quartz.dataSource.quartzDs.driver = oracle.jdbc.OracleDriver
 org.quartz.dataSource.quartzDs.URL = jdbc:oracle:thin:@localhost:1521:vof
 org.quartz.dataSource.quartzDs.user = ******
 org.quartz.dataSource.quartzDs.password = *****
 org.quartz.dataSource.quartzDs.maxConnections = 45

Any suggestions would be much appreciated

We have tried changing the selectWithLockSQL configuration property to remove the "FOR UPDATE" but that caused other locking issues.

That's to be expected. The lock (which "FOR UPDATE" creates) is being made specifically to avoid deadlocks that would arise if threads were allowed to concurrently and randomly obtain row-locks on the data in the other tables. Removing the explicit locking (by removing "FOR UPDATE") means that there is nothing to protect threads from randomly running into each other's row locks.

database was spending over 80% of its time executing the statement "SELECT * FROM QRTZ_LOCKS WHERE LOCK_NAME = :1 FOR UPDATE".

This would be threads blocking, waiting their turn to get past the coordinating lock, so that they can alter the trigger/job data (which, again, involves UPDATE commands, which create row-locks, which if allowed to happen concurrently would cause dead-locks).

With a single node the lock that controls access to the data is a RAM-based lock, rather than a table-based lock. That's why it performs better.

I would be interested to know how many jobs / minute you need to execute, and how many you are actually achieving. Also how long (in milliseconds) do these jobs take to run, on average?

On my home desktop computer, running 2 quartz nodes against PostgreSql (all on the same machine), I can achieve ~3700 jobs / minute.

Quoting from the online docs about clustering Quartz:

The clustering feature works best for scaling out long-running and/or cpu-intensive jobs (distributing the work-load over multiple nodes). If you need to scale out to support thousands of short-running (e.g 1 second) jobs, consider partitioning the set of jobs by using multiple distinct schedulers (and hence multiple sets of tables (with distinct prefixes)). Using one scheduler forces the use of a cluster-wide lock, a pattern that degrades performance as you add more clients.

jhouse wrote:

I would be interested to know how many jobs / minute you need to execute, and how many you are actually achieving. Also how long (in milliseconds) do these jobs take to run, on average?

On my home desktop computer, running 2 quartz nodes against PostgreSql (all on the same machine), I can achieve ~3700 jobs / minute.

In my situation I'm experiencing a strong performance degradation using the Oracle server placed in the intranet (~700 jobs/minute) and MySQL running in my laptop (~5700 jobs/minute). I cannot explain why Quartz does not increases the db connections (only 3 or 4) to improve the tps.
I've posted more info here
http://forums.terracotta.org/forums/posts/list/3693.page

I'm curious to know if you are actually using the scheduling features (e.g. that the job needs to run at a particular time) - or if you are using Quartz more as just a job queue (just run the jobs as quick as you can).

We have developed a custom scheduling service with our object model and some extra logic.
It might be used for other services to perform periodic checks and to schedule notifications.
We were performing load tests, and then see how many concurrent checks can be done in a high load scenario. It's not a blocking issue but it has to be well documented in our specifications to take care about how services are scheduled.

Now we might evaluate the use of multiple scheduler instances, the improvement of the net connection and/or the use of the Terracota JobStore.

Thanks for your help.
Discolo

P.D. Edited object model instead of data model. sry

It’s a fact that there are lots of online job providers working nowadays, but I recommend you to visit www.utubeonlinejobs.org as this is a first registered company of the Pakistan. I found its setup, working environment quite elegant, friendly and smooth.