Terracotta Discussion Forums

Hi,

I noticed the quoted issue in the schedule for 2.0.3. I could not find any changes in the manner of validation in the src (in http://svn.terracotta.org/svn/quartz/branches/quartz-2.0.x or trunk). However, I note James House's comments:

idleWaitTime is not intended to be set to values less than 15 or 20 seconds.

and

Possibly throw an exception during initialization if too small.

This worried me as I currently use a 1000 millisecond idleWaitTime with 2.0.2. I do this because a set of polling tasks we schedule will range between a 1-15 second wait time, and I want the accuracy to be down to second granularity (the 20% variableness is fine). Having a wait time of above 15 seconds can make tasks seem unresponsive due to the fact that polling tasks can fire 15 seconds late.

I was just wondering if I could get any feedback as to what I risk by setting the wait time down to 1 second, and if this will cause any exceptions in upcoming releases.

Thanks again for any help or feedback.

Louis

Smaller times lead to a lot of contention in the database.

Also, idleWaitTime should not be affecting things as you are stating.

It IS a polling interval - to check for updates to the DB that the job store was unaware of, but if you're scheduling your jobs via the API as you should be, then the job store will be aware of any scheduling changes (adding/removing a trigger) already, without the need to poll.

Whenever a trigger is added, removed, fired, paused or resumed, that in and of itself triggers a poll. So the idleWaitTime is very redundant, and should only affect two use cases: 1) the db is changed by something other than Quartz - which is considered a very BAD practice for a number of reasons, 2) the db is changed by a different Quartz node (clustered JDBC JobStore) -- in which case the node that made the change will itself find the new next trigger to fire as described above - it's just the other nodes that may take a short while to re-poll.

Hi James,

Thanks very much for the information and explanation. However I must still be misunderstanding something. I am using the API (org.quartz.Scheduler#scheduleJob) to schedule my jobs. Each of my jobs is distinct and I create a new job and a trigger for it and then schedule when the trigger is to be fired. I also understand that scheduling this job prompts an extra poll which will pickup any triggers that are due for firing.

However in my scenario I will be scheduling a job to be run in the next 1 to 15 seconds, and although this prompts a poll at the time of scheduling, the trigger is not ready to be fired at this point as it has just been added. I can't rely on another job to be scheduled to prompt a check. Subsequently the Scheduler thread will wait the idleWaitTime before checking again. This for me means that you can't schedule a job to be run within a smaller amount of time in the future, than the time duration configured by idleWaitTime. I've tried testing this again, putting the idleWaitTime up to 60 seconds. If I schedule a job to be run one second from now, it is likely to be executed closer to 50 seconds from now (<60 secs due to variableness).

I apologise if I've got the wrong end of the stick, or if there is an alternative to accomplishing what I'm trying to achieve. However this is why I'm currently using a 1 second wait for the idleWaitTime. I also would like to avoid DB contention if possible. Again, any feedback is much appreciated.

Are you using JobStoreCMT? And which version of Quartz?

org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreCMT
We are using Quartz 2.0.2.

Is there something I'm missing?
Here is an adapted example of what we're doing

JobBuilder jobBuilder = JobBuilder.newJob(MyJobImpl.class);
jobBuilder = jobBuilder.withIdentity(id);
jobBuilder = jobBuilder.requestRecovery(true);
JobDetail jobDetail = jobBuilder.build();
jobDetail.getJobDataMap().put("myRunnableImplementation", runnable);

TriggerBuilder triggerBuilder = TriggerBuilder.newTrigger();
triggerBuilder = triggerBuilder.withIdentity(id);
triggerBuilder = triggerBuilder.startAt(new Date(System.currentTimeMillis() + 1000));
Trigger trigger = triggerBuilder.build();

scheduler.scheduleJob(jobDetail, trigger);

I believe the poll occurs from calling scheduledJob. However the triggers isn't due to start for another second. Should this be triggered in 1 second without a poll to the database?

I think you are likely running into some form of race condition.

You are using JobStoreCMT which means your act of storing the new trigger does not commit to the db until your overall action commits. Which means that the kick-off of a poll at the time of storing the trigger cannot find the new data. There is plans to rework some of the signalling within JobStoreCMT to allow for a TransactionSynchronization to do the signalling once the commit actually occurs, but this has not yet been done. :(

You'll need to use a low idleWaitTime to resolve this for now. Make it as big as you can stand for it to be! 1000ms should work ok, other than causing lots of extra queries/lock contention in the db.

I'm sorry, I'm still confused.

I believe you are suggesting the trigger gets stored, and then a poll triggered immediately after. The poll should see the trigger, however in my case, it's not seeing it because the transaction has not been committed yet. I am confused as to the actions of the poll. I understood that the polling would only fire triggers which were due and that it had no awareness of triggers which would soon be due. Thus even if it did find the trigger, the trigger would not be due yet and thus it would not be acknowledged by the polling action (it would have to wait until the next poll). Please clarify if I misunderstood this.

I think you are suggesting that due to the transactional nature of the JobStoreCMT that it is not finding this trigger in the DB as part of the poll even though it has been added already, because the commit is done at the end. I'm unsure as to how the connections are managed in the JobStoreCMT to know if it shouldn't be able to see the transactional data, however my misunderstanding is in that I don't see the impact of being able to find this trigger that was not due yet.

Is the poll supposed to do some sort of caching of upcoming triggers?

The poll finds the expected fire time of the trigger that is expected to be fired next.

The main thread then waits either a) until the time has arrived to fire that trigger, or b) until the idle wait time and then polls again.

Because it can't find the new trigger (because it isn't committed) it does not find the fire time of the new trigger, but will find the next fire time of an existing trigger and wait for that (or the idle wait time).

That makes perfect sense, thanks for explaining the details. I'll keep an eye out for any changes to the JobStoreCMT to address this and will increase the idleWaitTime as soon as possible.

I have created this patch where the JobStoreCMT will used transaction synchronization to signal scheduling change. This will not affect anyone who is using JobStoreTX or RAMJobStore. I have tested this changes on JBOSS, WebSphere and Weblogic.

This change will basically signal scheduling changes via the JobStore, and depending upon the type of JobStore used it will either be done via transaction synchronization (JobStoreCMT) or direct (as is currently done). Ideally all scheduler thread notifications in QuartzScheduler (notifySchedulerThread method) can go via JobStore, but I am unable to ascertain the impact this may have, therefore I have only sent the scheduleJob notifications via the JobStore.

Any feedback is much appreciated.

This fix in combination with the batch acquisition fix related to the idle wait time will enable us to increase our idle wait time from 1 second across our multiple concurrent schedulers.

We would hope, if possible, to switch back to working with 2.1.1 of Quartz if this could make the next build. Thanks.

I'll try to work in the time to evaluate the patch.

Signalling in general is already scheduled for an even more significant overhaul in 2.2 (transaction synchronizations being part of the plan). https://jira.terracotta.org/jira/browse/QTZ-203

It may be workable to to do something with just JobStoreCMT before hand.

We will need a signed contributor's agreement to use the patch.

Thanks very much for considering.

'You do not have permission to access /download/attachments/786516/Individual+Contributor+Agreement.pdf?version=2'

Unfortunately I don't have privileges on your confluence link, could you possibly attach it here?

I've put in a request to TC folks to get that fixed (I don't have the document myself).

Much appreciated, I'll await the response. Thx.