Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

We have Quartz clustered on two servers with Oracle RAC on the backend. I have about 120 or so jobs scheduled at any one time basically a rolling set of jobs with one being created every 10 seconds and repeating once every 5 minutes for 20 minutes.

I am using the Batch Aquire of triggers and use the aquire triggers in lock. Periodically we get:

Code:

 [o.q.c.ErrorLogger] - An error occurred while scanning for the next triggers to fire.
 org.quartz.JobPersistenceException: Couldn't acquire next trigger: Couldn't retrieve trigger: No record found for selection of Trigger with key: 'DEFAULT.6da64b5bd2ee-894279e1-a6bd-4151-bb9d-77acf5647ade' and statement: SELECT * FROM QSMS_SIMPLE_TRIGGERS WHERE SCHED_NAME = 'xxx' AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ?
 	at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2840) ~[quartz-2.1.5.jar:na]
 	at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2746) ~[quartz-2.1.5.jar:na]
 	at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3811) ~[quartz-2.1.5.jar:na]
 	at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2742) ~[quartz-2.1.5.jar:na]
 	at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:264) ~[quartz-2.1.5.jar:na]
 Caused by: org.quartz.JobPersistenceException: Couldn't retrieve trigger: No record found for selection of Trigger with key: 'DEFAULT.6da64b5bd2ee-894279e1-a6bd-4151-bb9d-77acf5647ade' and statement: SELECT * FROM QSMS_SIMPLE_TRIGGERS WHERE SCHED_NAME = 'xxx' AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ?
 	at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1524) ~[quartz-2.1.5.jar:na]
 	at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2790) ~[quartz-2.1.5.jar:na]
 	... 4 common frames omitted
 Caused by: java.lang.IllegalStateException: No record found for selection of Trigger with key: 'DEFAULT.6da64b5bd2ee-894279e1-a6bd-4151-bb9d-77acf5647ade' and statement: SELECT * FROM QSMS_SIMPLE_TRIGGERS WHERE SCHED_NAME = 'xxx' AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ?
 	at org.quartz.impl.jdbcjobstore.SimpleTriggerPersistenceDelegate.loadExtendedTriggerProperties(SimpleTriggerPersistenceDelegate.java:95) ~[quartz-2.1.5.jar:na]
 	at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:1801) ~[quartz-2.1.5.jar:na]
 	at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1520) ~[quartz-2.1.5.jar:na]
 	... 5 common frames omitted

if there are a handful of jobs running we never see this. only when we got up to around 60 or so jobs in the sheduler.

Can you share your quartz.properties? Do you have the same config on both of the clustered nodes? What type of triggers you used and what's their settings?

here is the configuration. Yes it is inside my war so it is the same on both Jboss servers. I am using SimpleTrigger

Code:

 TriggerBuilder<Trigger> triggerBuilder = TriggerBuilder.newTrigger();
 			triggerBuilder.withIdentity( job.getKey().getName(), "MyBuilderJob.triggers" );
 			triggerBuilder.startAt( new Date( System.currentTimeMillis() + (1000 * 10) ) );
 			triggerBuilder.withSchedule( SimpleScheduleBuilder.simpleSchedule().withMisfireHandlingInstructionFireNow() );

Code:

 #============================================================================
 # Configure Main Scheduler Properties  
 #============================================================================
 
 org.quartz.scheduler.instanceName: myScheduler
 org.quartz.scheduler.instanceId: AUTO
 org.quartz.scheduler.skipUpdateCheck: true
 
 #============================================================================
 # Configure ThreadPool  
 #============================================================================
 
 org.quartz.threadPool.class: org.quartz.simpl.SimpleThreadPool
 org.quartz.threadPool.threadCount: 100
 org.quartz.threadPool.threadPriority: 5
 
 #============================================================================
 # Configure JobStore  
 #============================================================================
 
 org.quartz.jobStore.misfireThreshold: 60000
 
 org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreCMT
 org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
 org.quartz.jobStore.useProperties = false
 org.quartz.jobStore.dataSource = quartzXA
 org.quartz.jobStore.tablePrefix = QSMS_
 org.quartz.jobStore.isClustered = true
 org.quartz.jobStore.nonManagedTXDataSource = quartzNonXA
 org.quartz.jobStore.dontSetAutoCommitFalse = true
 
 org.quartz.scheduler.batchTriggerAcquisitionMaxCount = 40
 org.quartz.scheduler.batchTriggerAcquisitionFireAheadTimeWindow = 5000
 org.quartz.jobStore.acquireTriggersWithinLock = true
 
 
 #============================================================================
 # Configure Datasources  
 #============================================================================
 org.quartz.dataSource.quartzXA.jndiURL=java:/comp/env/jdbc/my-xa
 org.quartz.dataSource.quartzNonXA.jndiURL=java:/comp/env/jdbc/my-nonXa
 
 
 #============================================================================
 # Configure Plugins 
 #============================================================================
 
 #org.quartz.plugin.shutdownHook.class: org.quartz.plugins.management.ShutdownHookPlugin
 #org.quartz.plugin.shutdownHook.cleanShutdown: true
 #org.quartz.plugin.triggHistory.class: org.quartz.plugins.history.LoggingJobHistoryPlugin
 
 
 org.quartz.plugin.jobInitializer.class = org.quartz.plugins.xml.XMLSchedulingDataProcessorPlugin
 org.quartz.plugin.jobInitializer.fileNames = my-quartz.xml
 org.quartz.plugin.jobInitializer.failOnFileNotFound = true
 org.quartz.plugin.jobInitializer.scanInterval = 0
 org.quartz.plugin.jobInitializer.wrapInUserTransaction = false

Hi,

We are facing the same issue. Did you get any solution? In that case, please share.

Thanks
Harsha

nope. it does not seem to adversely affect anything, however it is disconcerting to see. I am guessing it has to do with batch acquiring triggers not being handled properly.

Hi All,

I have encountered too with this exception, after reading quartz 2.1.6 code, I got to the following conclusion: There is a "like race condition" between two threads.

The scenario is:
Suppose that we have two threads, T1 & T2. T1 is quartz thread pool thread that executing a job with a single run trigger, T2 tries at the same time to read the executed job data. Since we using a JDBC job storage and read actions (as T2 do) did not require a DB lock, so when T1 will finish the job execution he will delete job's trigger. If T2 will call retrieveTrigger method before T1 deleted the trigger and the trigger will be deleted before T2 call to loadExtendedTriggerProperties we will get the exception.

Code:

	at org.quartz.impl.jdbcjobstore.SimpleTriggerPersistenceDelegate.loadExtendedTriggerProperties(SimpleTriggerPersistenceDelegate.java:95)
 	at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:1801)
 	at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTriggersForJob(StdJDBCDelegate.java:1688)
 	at org.quartz.impl.jdbcjobstore.JobStoreSupport.getTriggersForJob(JobStoreSupport.java:2132)

I think that this is a bug.

There was indeed a race condition in this code.

It has been fixed as part of QTZ-386.

The fix will be part of the upcoming 2.2.1 Quartz release.

Issue still exists when using Cron Triggers. It seems that the fix for QTZ-386 only fixed the Simple Triggers and not Cron Triggers. Along with the scheduler factory property - overwriteExistingJobs set to true.