Terracotta Discussion Forums

Hi,

Quartz 1.6.6
Java 1.4.2_03
Spring 1.1.3

RedHat Entreprise Linux 5.0 32 bits
CPU : 2 x Intel Xeon 4 Core 3GHz
Disks : 2x 2 disques SAS 146GB Raid

I have a very strange problem which seems to happen only on particular computers (not sure) :
All quartz jobs (12) stopped during about 70 minutes ! It can take several days before it happens.

Our application code is OK (IN and OUT traces in all our methods).

QuartzScheduler_QuartzSchedulerThread seems to wait for an object while QuartzScheduler_Workers wait for another one ...

But after 70-75 minutes, all jobs restart correctly !

I really need some help.

Thanks !

Here is the stack :

Here is an example of our job definition :

Code:

   <!-- Quartz server notification trigger. Fires server notification job -->
     <bean id="notifyServerTrigger"
 		class="org.springframework.scheduling.quartz.SimpleTriggerBean"
 		lazy-init="false">
         <property name="jobDetail">
             <ref bean="notifyServerJob" />
         </property>
         <property name="startDelay">
             <!-- 5 seconds -->
             <value>5000</value>
         </property>
         <property name="repeatInterval">
             <!-- repeat every 1 second(s) -->
             <value>1000</value>
 		</property>
 	</bean>
 
 	<!-- Quartz Job for starting server notifications from central to locals -->
 	<bean id="notifyLDSServersJob"
 		class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean"
 		lazy-init="false">
 		<property name="targetObject">
 			<ref bean="serverManager" />
 		</property>
 		<property name="targetMethod">
 			<value>notifyLDSUpdates</value>
 		</property>
 	</bean>

Sounds very odd, however the thread dump you provided is not complete (not showing all threads) and what it does show is normal operations - in fact the QuartzSchedulerThread is in the act of finding the next trigger to fire.

Any way you can provide more info?

* Complete thread dump file
* Description of what your jobs do, and whether or not they are StatefulJobs
* Description of how many jobs you have
* Full quartz properties (config) - or at least info like what is the size of the thread pool, what is idle wait time set to, etc.

Thank you for your answer.

Here are some information :

Indeed, the dump is not complete but I can tell you that all workers are suspended (12) and there is no lock in my application methods (no job is working).

quartz.properties is very simple :
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 13

My jobs just sent JMS messages periodically ...

NB : some jobs are executed in the same millisecond, can it be a problem ?

12 jobs with following configuration (SimpleTriggerBean) :

job1 : repeatInterval=1s, startDelay=2s
job2 : repeatInterval=1s, startDelay=5s
job3 : repeatInterval=1s, startDelay=5s
job4 : repeatInterval=60s, startDelay=5s
job5 : repeatInterval=60s, startDelay=5s
job6 : repeatInterval=60s, startDelay=5s
job7 : repeatInterval=180s, startDelay=5s
job8 : repeatInterval=60s, startDelay=10s
job9 : repeatInterval=60s, startDelay=10s
job10 : repeatInterval=21600s, startDelay=1800s
job11 : repeatInterval=30s, startDelay=600s
job12 : repeatInterval=10s, startDelay=30s

Can you give a few more samples of what the QuartzSchedulerThread's stack looks like when it is in this state?

Again, there's nothing apparently wrong that I can see. But it would be interesting to see if anything further was revealed by a few more peeks at that thread's activity.

> NB : some jobs are executed in the same millisecond, can it be a problem ?

No, not unless they do something (in their own execute() method) that causes them to interact badly with each other. Even then, it ought to appear in the thread dumps (the stacks of some of the worker threads).

Thanks for reply.

Unfortunately, this problem occurs rarely and on a production environment.
It can be during the night and it lasts only 70 min.

It is very difficult to have some new dumps ...
It seems that something on the computer locks Quartz ...