Terracotta Discussion Forums

What are your thougths about this with Spring's ThreadPoolTaskExecutor which is a thread pool that can grow dynamically as needed.

Because theoretically the number of jobs will always be less than the number of workers since new workers will keep being spawn upto Integer.MAX_VALUE in our case. Of course, unless we have more jobs than Integer.MAX_VALUE :)

So, is it likely that I will end up with most jobs being run in one node as my pool of workers will keep growing on demand and jobs will most likely always be less than the number of workers?

Thanks.

Using Quartz 1.8.4 on cluster mode with 6 nodes. Could someone shine some light on Quartz load balancing strategy?

To be specific, we have a cron job that looks on DB and it creates SimpleTrigger jobs to run immediately (new Date()) for each entry retrieved from DB.

If we have 25 threads for each Quartz scheduler instance. How will, for example, 30 SimpleTriggers get balanced among the 6 nodes?

Will it be random and balanced where each node probably runs 5 jobs?
Or will node 1 take most of them leaving the rest of the scheduler instances idle?

NOTE: by node 1 I mean logical node 1. So, the first node to start running the first job.

We need to get these jobs balanced among the 6 nodes and we are concerned that, in cases like this, one node will end up taking most of the jobs while the other schedulers are idle.

We are using Spring's ThreadPoolTaskExecutor with a corePoolSize=25 that can grow dynamically to Integer.MAX_VALUE for each scheduler instance.

Following the example above what would be the outcome if we get 3,000, or 6,000, or 9,000 SimpleTrigger jobs to run immediately?

Will they be balanced among the 6 nodes?
Or will node 1 thread pool keep growing taking most of the jobs while the other scheduler instances are idle?

All comments are appreciated.

nicolas.loriente

Hi all,

We have a Production environment with 6 nodes (6 schedulers) and have Quartz configured for cluster.
I'm pretty sure all configuration is good as we see scheduler instances checking in every 20 seconds and sometimes a node recovering jobs for another node that failed

All that seems to be working fine but our issue is with Misfired jobs. For some reason we are seeing misfired jobs being run days latter.
Here is the log showing that:

Code:

 Nov 8, 2011 4:32:18 PM org.quartz.impl.jdbcjobstore.JobStoreSupport recoverMisfiredJobs
 INFO: Handling 3 trigger(s) that missed their scheduled fire-time.
 Nov 8, 2011 4:32:18 PM org.quartz.plugins.history.LoggingTriggerHistoryPlugin triggerMisfired
 INFO: Trigger groupName.jobName_1 misfired job groupName.jobName_1 at:  16:32:18 11/08/2011.  Should have fired at:  17:26:04 10/27/2011
 Nov 8, 2011 4:32:18 PM org.quartz.plugins.history.LoggingTriggerHistoryPlugin triggerMisfired
 INFO: Trigger groupName.jobName_2 misfired job groupName.jobName_2 at:  16:32:18 11/08/2011.  Should have fired at:  17:26:04 10/27/2011
 Nov 8, 2011 4:32:18 PM org.quartz.plugins.history.LoggingTriggerHistoryPlugin triggerMisfired
 INFO: Trigger groupName.jobName_3 misfired job groupName.jobName_3 at:  16:32:18 11/08/2011.  Should have fired at:  17:26:04 10/27/2011

I've changed the name of the group and job with "groupName" and "jobName_X" for privacy but these where 3 different jobs.

As you can see the jobs were schedule for October 27th but ended up firing on November 8th.

As I mentioned before, this is a production environment with 6 servers running 6 quartz schedulers in cluster mode. Most nodes, if not all, where up during the misfired time.

The misfire action in this jobs is to recover NOW (immediately) and I'm sure we have many, if not all, nodes up and available to recover the job immediately.

What could have caused these jobs to be recovered from a misfire 12 days later?

I need to check with our support team but is my understanding that we continue to see this type of behavior and this was not an isolated incident.

All comments are welcome.

Thanks,

nicolas.loriente

I just wanted to share some points and a link to a post on Spring forum for those using Spring's Quartz support.

Spring sets org.quartz.jobStore.dontSetAutoCommitFalse to TRUE which is the opposite of Quartz recommendation as stated by Quartz documentation:

I'm seeing triggers stuck in the ACQUIRED state, or other weird data problems.

Spring defaults the Quartz property "org.quartz.jobStore.dontSetAutoCommitFalse" to "true" - which means Quartz will not turn off auto-commit mode on the database connections that it uses. This is the opposite of Quartz's own default for this setting. If your connection is defaulting to have auto-commit on, then you'll run into all sorts of strange problems relating to data inconsistencies -- the most common symptom being triggers that are "stuck" in the "ACQUIRED" state. Fix this by explicitly setting the property to "false".

Basically, Spring is disabling the property by doing so and even though you might have "org.quartz.jobStore.dontSetAutoCommitFalse=false" in your Quartz properties it is still running with TRUE.

Because this is harcoded in one of Spring's components in order to re-enable this property you have to go through a frenzy of Spring class overriding. A total of 6 Spring components need to be touched to just re-enable this property and get dontSetAutoCommitFalse to be configurable again.

Why is Spring disabling the configuration of this value? Why do they set it to TRUE which is clearly the opposite from what Quartz recommends? and why is Spring harcoding dontSetAutoCommitFalse and JobStore class which should be configurable as is Quartz way? ....I don't know but hopefully I'll find out soon http://forum.springsource.org/showthread.php?114622-Spring-s-default-opposed-to-Quartz-recommendation&p=379816#post379816

In the mean time I thought it would be good to let those using Spring/Quartz know about it.

Please share your thoughts if you have work through the same.

Thanks,

nicolas.loriente

Hi light5,

Thanks for your answer.

So having a job with a cron trigger and then calling it on demand via "triggerJob" or "triggerJobWithVolatileTrigger" as needed would be the correct approach, right?

Are there any other considerations when manipulating a job in this manner?
Any possible foreseen issues?

Thanks,

nicolas.loriente

I have the following scenario:

- I have a "main" stateful job with a cron trigger to fire, let's say, every minute.
- This job is scheduled via Spring when the application context is loaded.
- This "main" cron job will schedule two other jobs programmatically with a SimpleTrigger set to fire immediatly.

Now, sometimes when certain criteria is met I don't want to wait for the next fire time (possibly a minute away). I want to fire this "main" job on demand.
For example, lets say the cron trigger fire at 1:00pm and next fire time is 1:01pm but at 1:00:10 (10 seconds after 1pm fired) I want to run the job without having to wait until 1:01pm.

Has anyone done something similar?

Currently I've tested calling the cron job on demand via scheduler's triggerJob methods

scheduler.triggerJob( "myCronJobThatFiresEveryMinute", "myGroup" );

and this seems to work just fine.

1. Do you see any issues with this approach?
2. Is there a better way to achieve what I've described?
3. Is there anything that needs consideration that I might be missing?

If you solved something similar let me know what was your approach. If you didn't but have an idea.... go ahead, your help is welcomed.

Thanks,

nicolas.loriente

OK, found the issue. It WAS Spring configuration issue. I'll share it here just in case it can save someone some time.

The problem was that Spring's JpaTransactionManager was not managing JPA + JDBC operations uniformly under one transaction.

So, in a case like this:

Code:

@Transactional
 public void doSomethingAndRollbackTransaction() {
 
     // do something through JPA
 
    // do something through JDBC (e.g. scheduling job with quartz)
 
   // throw RuntimeException
 
 }

only the JPA operations were rolling back for me.

Spring's JpaTransactionManager IS capable of managing both JPA/JDBC uniformly under a common transaction but a bit of configuration needs to take place.

To have JpaTransactionManager manage JPA/JDBC you have to set "jpaDialect" property to a vendor specific value in either LocalContainerEntityManagerFactoryBean or JpaTransactionManager.

NOTE: of course you need to use the same dataSource for JPA and JDBC and your JDBC code needs to get a connection from the dataSource using Spring's connection lookup pattern (e.g. DataSourceUtils.getConnection(dataSource) or going through a TransactionAwareDataSourceProxy, which is the case when using Spring transaction management support).
In Spring/Quartz this is achieved via Spring's ConnectionProvider implementation set in the LocalDataSourceJobStore initialize() method where the getConnection() method does DataSourceUtils.doGetConnection(dataSource).

For example, in my case I set up the following in my LocalContainerEntityManagerFactoryBean.
Code:

....
 <property name="jpaDialect">
     <bean class="org.springframework.orm.jpa.vendor.HibernateJpaDialect"/>
 </property>
 ....

If "jpaDialect" is not declared explicitly Spring will use an instance of DefaultJpaDialect which will not enable the management of JPA/JDBC uniformly under one transaction. In such a case JDBC executes on its own out of the context of the transaction.

This is very clearly explained in JpaTransactionManager javadoc.

This transaction manager also supports direct DataSource access within a transaction (i.e. plain JDBC code working with the same DataSource). This allows for mixing services which access JPA and services which use plain JDBC (without being aware of JPA)! Application code needs to stick to the same simple Connection lookup pattern as with DataSourceTransactionManager (i.e. DataSourceUtils.getConnection(javax.sql.DataSource) or going through a TransactionAwareDataSourceProxy). Note that this requires a vendor-specific JpaDialect to be configured.

Of course, this does not only applies for Spring/Quartz but for any Spring JPA + JDBC integration effort (e.g. using JPA EntityManager and JdbcTemplate).

Hope it helps!

nicolas.loriente

Ha :) Kai was the guy that created the thread. I picked the wrong name when addressing my reply. Sorry about that.

The RuntimeException I'm referring to is not coming from Quartz but from our code. Let me outline the logic flow:

1. Begin Transaction ->
2. Do something
3. Schedule Job/s ( No issues here. No exception from Scheduler. )
4. Do something else ( RuntimeException thrown. )
5. <- Commit Transaction.

We are getting an exception in step 4. Everything done in step 2 as well as step 4 (before getting the exception) IS rolling back. But the scheduled job done in step 2 under the umbrella of the same transaction is NOT.
The job stays scheduled when I think it should roll back with the surrounding transaction.

Does this makes sense?

Thanks light5!

nicolas.loriente

Hi Kai,

Thanks for your reply. I'll take a look at these points but I believe the exception is a RuntimeException.

I'll post back my findings.

Thanks,

nicolas.loriente

I'm using Spring/Quartz and that is exactly what I do. I retrieve Singleton beans from the container from withing my custom JobFactory.

I don't think you need a new instance of the job unless you need to keep state buy using instance members.

nicolas.loriente

I posted the question on Spring forum as well. I was trying to get some info about how quartz deals with persistence and transactions.

I believe the spring configuration (transaction, datasource, etc) is good. I've debug as much as I could and I see the spring transaction being applied but haven't figured out why quartz is not rolling back when everything else done within the transaction is.

I'll get deeper and take a look at the code.

Any comments are welcome.

Thanks,

nicolas.loriente

I am experiencing exactly the same thing. I've tested it and the job scheduled is not rolling back.

I'm doing a few other things and EVERYTHING but quartz is rolling back. Job stays scheduled and it fires on time as expected if no roll back.

I have Quartz Scheduler wrapped around Spring transaction.

I've debug it and the transaction advice is intercepting the scheduler.schedule() call and calls to scheduler method are being wrapped around a transaction. But quartz is still not rolling back.

Code:

         <!-- Quartz Scheduler Transaction Config -->
 	<aop:config>
 		<aop:pointcut id="quartzSchedulerPointcut"
 			expression="execution(* org.quartz.Scheduler.*(..))" />
 		<aop:advisor advice-ref="quartzSchedulerAdvice"
 			pointcut-ref="quartzSchedulerPointcut" />
 	</aop:config>	
 	
 	<!--  Quartz Scheduler Transaction Propagation -->
 	<tx:advice id="quartzSchedulerAdvice">
 		<tx:attributes>		 
 			<tx:method name="get*" read-only="true" propagation="SUPPORTS" />
 			<tx:method name="set*" read-only="true" propagation="SUPPORTS" />
 			<tx:method name="is*" read-only="true" propagation="SUPPORTS" />
 			<tx:method name="insert*" read-only="false" propagation="REQUIRED"/>
 			<tx:method name="update*" read-only="false" propagation="REQUIRED"/>
 			<tx:method name="delete*" read-only="false" propagation="REQUIRED"/>
 			<tx:method name="schedule*" read-only="false" propagation="REQUIRED"/>
 			<tx:method name="pause*" read-only="false" propagation="REQUIRED"/>
 			<tx:method name="resume*" read-only="false" propagation="REQUIRED"/>
 			<tx:method name="run*" read-only="false" propagation="REQUIRED"/>
 			<tx:method name="update*" read-only="false" propagation="REQUIRED"/>
 			<tx:method name="delete*" read-only="false" propagation="REQUIRED"/>
 			<tx:method name="toggle*" read-only="false" propagation="REQUIRED"/>
 			<tx:method name="clone*" read-only="false" propagation="REQUIRED"/>
 		</tx:attributes>
 	</tx:advice>

Any comments will be appreciated.

Thanks,

nicolas.loriente

You should be able to implement this having some type of DB flag or communication mean.

- Start scheduler one and schedule a job that runs lets say every 2 minutes and writes a node check to DB. Similar to what Quartz does in QRTZ_SCHEDULER_STATE table.
This node will be the active node processing the scheduled jobs.

- Have the rest of the nodes in standby mode and have a task running every 2 minutes checking DB for node 1 periodic checks. The other nodes can do the same (e.g. node 2 checks node 1, node 3 checks node 2, and so on). If node 2 sees node 1 didn't do its cluster check in the last 2 minutes your node 2 task that checks DB can call scheduler.start()

I don't think it is too involved... it should be some what easy to implement.

I don't know if I'm being clear but hopefully you get the idea.

I hope it helps.

nicolas.loriente

Yes, I do stand alone as well as web app both using Spring.

For programmatic scheduling I recommend creating Quartz Scheduler via Spring's SchedulerFactoryBean and then injecting in it in your component which does the scheduling. In your component you can use Quartz APIs to create your JobDetail and Trigger objects.

I would also recommend creating your own ApplicationContextAware JobFactory implementation and setting that on the SchedulerFactoryBean. That way at job instantiation time Quartz will delegate to your custom JobFactory which can pull job beans from the context. The good part about this is that it allows your jobs to be declared as beans which in turn allows you to leverage any Spring managed component (e.g. daos, services, etc) from within your jobs.

nicolas.loriente

All jobs can run concurrently as long as there is threads available.

What you are describing is behavior of stateful jobs. For that scenario to happen you need to schedule the same job with different triggers and the job must be stateful.

For example: JobOne.class is schedule to fire at 12:00 and 12:01. When 12:01 trigger goes off if previous (12:00) trigger didn't complete yet it will queue and wait until it finishes before starting the 12:01 run. That is if the job is stateful. If the job is stateless it will just create another instance and run it concurrently if there is an available thread to do so.

nicolas.loriente