How to prevent duplicate cron jobs from running

Duplicate job executions are a common problem when scheduling and managing cron jobs. This guide will show you how to identify why this is happening and what to do if it is.

Why are duplicate cron jobs running?

When you discover multiple copies of a cron job running, cron itself probably did not fire the job twice. Implementations used today do not double-run a job because of an NTP step or a daylight-saving change. If you have duplicate cron jobs running, it's usually caused when a job is hung and will never exit, or when a job is just running longer than the scheduled interval between executions.

If your job is hanging, you can use a tool like strace to identify what line of code or system call the application is currently stuck at. Often this will be an application or system issue like a deadlock or threading problem. On the other hand, if your job does appear to finish but is running longer than the scheduled interval, the easiest thing to do is to run the job less frequently, giving more time for each run to complete. If this is not an option for your workload, you may be forced to change how the job works. Some common strategies are:

• Performance optimizations, especially with queries.
• Sharding the job so multiple copies run that each handle a chunk of the total workload.
• Identify tasks that could be slow and done asynchronously (e.g. sending emails) and extract them to a job queue.
• Extract any logic that can be run less frequently into a separate job.
  • Allow the job to continue to take longer, but ensure that only one instance can run by using a lock like flock. This option will be discussed in more detail in the next section.

If your job is not hanging, and not running longer than the interval, but you are certain it's running twice, the last thing to check is whether this job has been accidentally duplicated in your crontab file. A common practice when scheduling a job is to copy and paste a line from another working job and from time-to-time you can expect that a mistake will happen. If a command is scheduled twice it will be run twice. To find and fix this, it could be as simple as removing a line from the same crontab, or you may need to look at all the places cron jobs can be created in order to find and remove the accidental duplicate job.

How do you prevent duplicate cron jobs?

To prevent duplicate cron job executions without modifying the job or changing the schedule, add a lock around the job using the flock utility. flock takes an advisory lock with the flock(2) system call and runs your command only when it gets that lock. Here is a simple before and after to illustrate how flock can be added to your crontab:

# Before
$ crontab -l
*/5 * * * * /home/crons/send-incremental-data.py
$

# After
$ crontab -l
*/5 * * * * flock -n /home/crons/send-incremental-data.lock /home/crons/send-incremental-data.py
$

With flock in place, crontab still invokes the line every 5 minutes. -n means do not wait: if the lock is already held, flock exits and the Python script does not start. The path after -n is the lock file. flock creates that file if it is missing, and it leaves the file in place when the command exits. The lock itself is released when flock exits.

Removing the lock file with rm does not stop a process that still holds the lock. A later flock creates a new file and can start a second copy while the first is running. To clear a stuck run, stop that process. The next scheduled start can take the lock.

If Cronitor is watching the job, where you put flock changes what Cronitor records. cronitor exec reports the wrapped command's exit status. When flock is outside cronitor exec, a skipped run sends no ping, and Cronitor sees a missed execution. When cronitor exec is outside flock, exec still runs, flock exits 1, and Cronitor sees a failure.

cronitor sync writes cronitor exec in front of the whole command. The discover alias rewrites the line the same way. A crontab line that already uses flock therefore ends up with the lock inside cronitor exec, and a skipped run shows up as a failure with exit 1 rather than a missed run.

On the job's alert settings, schedule tolerance is how many missed executions to allow before an alert. Failure tolerance is how many failed executions to allow. Grace period is extra time before a late start counts as missed. Those settings only decide when you get an alert. They do not keep the second copy from starting, which is what flock is for. The fields are documented in job monitoring.

How do you see if duplicate cron jobs are running?

To see if duplicate cron jobs are running, use the ps command and grep for the job name itself. Head over to our guide on how to check if a cron job is running right now to cover this in depth. A one-line list of the cron jobs that are still running is in how to see every cron job that's running.