How to prevent duplicate cron jobs from running

Duplicate job executions are a common problem when scheduling and managing cron jobs. This guide will show you how to identify why this is happening and what to do if it is.

Why are duplicate cron jobs running?

When you discover multiple copies of a cron job running, the first thing to know is that this is probably not a malfunction with cron itself where your job was accidentally run twice. Cron implementations used today are resilient to these kinds of problems, and something like an ntp system time adjustment or daylight savings time transition will not cause your job to be run twice. If you have duplicate cron jobs running, it's usually caused when a job is hung and will never exit, or when a job is just running longer than the scheduled interval between executions.

If your job is hanging, you can use a tool like strace to identify what line of code or system call the application is currently stuck at. Often this will be an application or system issue like a deadlock or threading problem. On the other hand, if your job does appear to finish but is running longer than the scheduled interval, the easiest thing to do is to run the job less frequently, giving more time for each run to complete. If this is not an option for your workload, you may be forced to change how the job works. Some common strategies are:

• Performance optimizations, especially with queries.
• Sharding the job so multiple copies run that each handle a chunk of the total workload.
• Identify tasks that could be slow and done asynchronously (e.g. sending emails) and extract them to a job queue.
• Extract any logic that can be run less frequently into a separate job.
  • Allow the job to continue to take longer, but ensure that only one instance can run by using a lock like flock. This option will be discussed in more detail in the next section.

If your job is not hanging, and not running longer than the interval, but you are certain it's running twice, the last thing to check is whether this job has been accidentally duplicated in your crontab file. A common practice when scheduling a job is to copy and paste a line from another working job and from time-to-time you can expect that a mistake will happen. If a command is scheduled twice it will be run twice. To find and fix this, it could be as simple as removing a line from the same crontab, or you may need to look at all the places cron jobs can be created in order to find and remove the accidental duplicate job.

How do you prevent duplicate cron jobs?

To prevent duplicate cron job executions without modifying the job or changing the schedule, add a lock around the job using the flock utility. This is not just another application: in Linux, flock is a system call and the flock utility uses this built-in functionality to guarantee that your job will only run if it can obtain the lock. As long as it's running, the lock will be held, and flock will not allow any additional jobs to start. Here is a simple before and after to illustrate how flock can be added to your crontab:

# Before
$ crontab -l
*/5 * * * * /home/crons/send-incremental-data.py
$

# After
$ crontab -l 
*/5 * * * * flock -n /home/crons/send-incremental-data.lock /home/crons/send-incremental-data.py
$

With flock in place, crontab will invoke the command every 5 minutes, the same as before, but flock will check the lockfile specified by the -n argument, /home/crons/send-incremental-data.lock. If this file does not exist, it will be created and the job will be allowed to run. When the job completes, flock will release the lock by deleting the file. If the lock file exists, the next time cron tries to start your job, the job will not be run. Manually "freeing" a lock to allow another instance of the job to be run is as simple as removing the lockfile with rm like any other file.

If you are a Cronitor user, and your job does not run because flock prevented it, you will be alerted to the missed execution, and a recovery notice will be sent later when the job is able to run again.

How do you see if duplicate cron jobs are running?

To see if duplicate cron jobs are running, use the ps command and grep for the job name itself. Head over to our guide on how to check if a cron job is running right now to cover this in-depth.