Terracotta Discussion Forums

Hi,

I have seen in the documentation

If you need to scale out to support thousands of short-running (e.g 1 second) jobs, consider partitioning the set of jobs by using multiple distinct schedulers. Using one scheduler currently forces the use of a cluster-wide lock, a pattern that degrades performance as you add more clients.

In a clustered environment using TerracottaJobStore, what is a usual limit to avoid to scale out with partitioning (100 jobs per seconds) [my job will run less than 10 ms] for 2 or 4 clients?

As quartz is more adapted to long running, is it a good idea to use quartz in this case ?

Regards

This is highly dependent upon hardware, network and OS (OS's thread scheduler, number of CPU cores, all sorts of things).

Really it would be best to do a proof-of-concept on your system architecture to know for sure. Around 100/sec isn't out of the question.

Quartz is most adapted to scheduled jobs, as opposed to a job queuing type use case.

Hi,
thanks for the answer.

I don't really know the hardware/configuration as the software will be run under multiple environment (linux/windows different hardware ...).

Really it would be best to do a proof-of-concept on your system architecture to know for sure. Around 100/sec isn't out of the question.

Thanks for the answer i'll do a POC to check that it is working.

We did a use case where :
Quartz with MemoryJobStore
One Scheduler with 1 job triggered 2 000 times every 5ms. Working with the right scheduling.
One Scheduler with 2 000 jobs triggered once with a delay of 5ms between each. The scheduler is exhausted. Can't respect frequency.

That's why i was worried about multiple fast triggers.

Looking deeper in the documentation, I guess that org.quartz.scheduler.batchTriggerAcquisitionMaxCount can solve a part of a too much and fast trigger problem (I guess some refactoring in how i add job / trigger could be better).

Hi,
I have done another test by replacing the SimpleThreadPool by my own implementation using a ThreadPoolExecutor from java 1.5+.

It seems that for short running, it is more efficent than the SimpleThreadPool.

I guess that some more job is done in the SimpleThreadPool than simply running a thread. Is there some extra requirements (which are not documented in the interface) ?