Cron has been the standard job scheduler on unix and unix-like systems since the 1970s. It is widely used and usually very reliable. This guide will walk you through locating cron jobs on your server, validating their schedules, and troubleshooting why they fail to start or error unexpectedly.
Even if you're setting up a cron job on a server where you've already deployed your application you might find that differences in the unix shell and execution context make it painful to get your job working.
Some schedule expressions, while widely accepted, are not standard and might not always work as expected.
The most stable cron job is no match for a disk that is full or an OS that can't spawn new threads.
When app configuration and code changes are deployed it can be easy to overlook the cron jobs on each server.
Cron jobs are often used for batch processing, event sourcing and other data intensive tasks that can reveal the constraints of your stack. Jobs may work fine until your data size grows to a point where a new bottleneck is reached.
Once created, cron jobs are out of sight and small or intermittent failures can go unnoticed for a long time. We created Cronitor to solve the observability problems of cron jobs and give you instant alerts when things go wrong.
Cron will not retry failed jobs, it will not care if jobs begin overlapping themselves, and it will not alert you any differently of job failure than it would job success.
Cron jobs are run by a system daemon called
crond that watches several locations for scheduled jobs. The first step to understanding why your job didn't start when expected
is to find where your job is scheduled.
dev01: ~ $ crontab -l # Edit this file to introduce tasks to be run by cron. # m h dom mon dow command 5 4 * * * /var/cronitor/bin/database-backup.sh
/etc/cron.d/(editors note: this is the best practice)
crontab -u username -l
Once you have found your job, verify that it's scheduled correctly. Cron schedules are commonly used because they are expressive and powerful, but like regular expressions can sometimes be difficult to read. We suggest using Crontab Guru to validate your schedule.
date, check your crontab file for
CRON_TZtimezone declarations that would override system settings. For example,
Invalid permissions can cause your cron jobs to fail in at least 3 ways:
/etc/cron.*/directory must be owned by root. Files owned by other users will be ignored and you may see a message similar to
WRONG FILE OWNERin your syslog.
ubuntuuser crontab invokes a script like
ubuntumust have permission to execute the script. The most direct way is to ensure that the
ubuntuuser owns the file and then ensure execute permissions are available using
chmod +x database-backup.sh.
/etc/cron.allowfile exists, the user must be listed. Separately, the user cannot be in a
Related to the permissions problem, ensure that if your command string contains a
% that it is escaped with a backslash.
When cron attempts to run a command, it logs it in syslog. By grepping syslog for the name of the command you found in a crontab file you can validate that your job is scheduled correctly and cron is running.
/var/log/syslog(You will probably need root or sudo access.)
dev01: ~ $ grep database-backup.sh /var/log/syslog Aug 5 4:05:01 dev01 CRON: (ubuntu) CMD (/var/cronitor/bin/database-backup.sh)
* * * * *.
crond. Rule this out quickly by verifying that cron is running by looking up its process ID. If cron is not running no process ID will be returned.
dev01: ~ $ pgrep cron 323
crondhas correctly loaded your crontab file. The easiest way to do this is to force a reparse of your crontab by running
EDITOR=true crontab -efrom your command prompt. If everything is up to date you will see a message like
No modification made. Any other message indicates that your crontab file was not reloaded after a previous update but has now been updated. This will also ensure that your crontab file is free of syntax errors.
If you can see in syslog that your job was scheduled and attempted to run correctly but still did not produce the expected result you can assume there is a problem with the command you are trying to run.
When cron runs your command the environment is different from your normal command prompt in subtle but important ways. The first step to troubleshooting is to simulate the cron environment and run your command in an interactive shell.
/etc/crontabeach line is allowed to have an effective "run as" username after the schedule and before the command itself. If this applies to your job, or if your job is in another user's crontab, begin by opening a bash prompt as that user
sudo -u username bash
/bin/sh, not the bash or zsh prompt you are familiar with. Double check your crontab file for an optional
SHELL=/bin/bashdeclaration. If using the default
/bin/shshell, certain features that work in bash like
[[command]]syntax will cause syntax errors under cron.
bash_profileso any environment variables defined there are unavailable in your cron jobs. This is true even if you have a
SHELL=/bin/bashdeclaration. To simulate this, create a command prompt with a clean environment.
dev01: ~ $ env -i /bin/sh $
cd ~at your prompt.
command >> /path/to/filewill redirect console log messages to the specified file and
command >> /path/to/file 2>&1will redirect both the console log and error messages. Determine if your command has a verbose output or debug log flag that can be added to see additional details at runtime.
At Cronitor our data shows that runtime durations increase over time for a large percentage of cron jobs. As your dataset and user base grows it's normal to find yourself in a situation where cron starts an instance of your job before the previous one has finished. Depending on the nature of your job this might not be a problem, but several undesired side effects are possible:
To verify if any instances of a job are running on your server presently, grep your process list. In this example, 3 job invocations are running simultaneously:
dev01: ~ $ ps aux | grep database-backup.sh ubuntu 1343 0.0 0.1 2585948 12004 ?? S 31Jul18 1:04.15 /var/cronitor/bin/database-backup.sh ubuntu 3659 0.0 0.1 2544664 952 ?? S 1Aug18 0:34.35 /var/cronitor/bin/database-backup.sh ubuntu 7309 0.0 0.1 2544664 8012 ?? S 2Aug18 0:18.01 /var/cronitor/bin/database-backup.sh
To quickly recover from this, first kill the overlapping jobs and then watch closely when your command is next scheduled to run. It's possible that a one time failure cascaded into several overlapping instances. If it becomes clear that the job often takes longer than the interval between job invocations you may need to take additional steps, e.g.:
flockto ensure that only a single instance of your command is running at any given time. Using flock is easy. After installing from
yumyou only need to prefix the command in your crontab:
dev01: ~ $ crontab -l # Edit this file to introduce tasks to be run by cron. # m h dom mon dow command * * * * * flock -w 0 /var/cronitor/bin/database-backup.sh
Read more about flock from the man page.
Here are a few things you can try if you've followed this guide and find that your job works flawlessly when run from the cron-like command prompt but fails to complete successful under crontab.
First get the most basic cron job working with a command like
date >> /tmp/cronlog. This command will simply echo the execution time to the log file each time it runs. Schedule this to run every minute and tail the logfile for results.
If your basic command works, replace it with your command. As a sanity check, verify if it works.
If your command works by invoking a runtime like
python some-command.py perform a few checks to determine that the runtime version and environment is correct. Each language runtime has quirks
that can cause unexpected behavior under crontab.
pythonyou might find that your web app is using a virtual environment you need to invoke in your crontab.
nodea common problem is falling back to a much older version bundled with the distribution.
phpyou might run into the issue that custom settings or extensions that work in your web app are not working under cron or commandline because a different
If you are using a runtime to invoke your command double check that it's pointing to the expected version from within the cron environment.
If nothing else works, restart the cron daemon and hope it helps, stranger things have happened. The way to do this varies from distro to distro so it's best to ask google.