How To Run Cron Jobs In Kubernetes Cluster

In this blog post , We will learn to Run Jobs / Cron Jobs in Kubernetes Cluster.

CronJobs are used to execute jobs on a time-based schedule.As like We are running Cron tasks on Linux systems.

Cron jobs are useful for creating periodic and recurring tasks.For example : Creating backup , Sending Emails etc.

A job creates one or more pods and also ensures that they are terminated after successful completion.

When a job is deleted It will also cleanup the pods created for them.

A simple use case is to create one job object in order to reliably run one pod to completion.

If the Job objects fails or deleted , It will be rescheduled in case of node reboot or hardware failures.

The main function of the job is to create one or more pod and tracks the pods for success and also ensures that the specified number of pods are completed successfully.

When the specified number of successful run of pods is completed , Then the job is considered complete.

Prerequisites

  • First We need to have kubectl installed on your system , Discussed below.

Types Of Jobs In Kubernetes

  • Run to Completion – It will create one or more pods and run them parallely for the successful completion of jobs. Using this approach we can process batches of work in parallel.
  • Schedulers (CronJob) – It is same as scheduling a cronjob in Linux.

Installing Kubectl

To install kubectl use the below commands.

curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.16.8/2020-04-16/bin/linux/amd64/kubectl
chmod +x ./kubectl

Then move it to the /usr/local/bin folder.By doing so , You dont need to type the full path to kubectl.

Now If you run the below command ,

kubectl version

You will get the following response,

Client Version: version.Info{Major:"1", Minor:"16+"

Now that we have configured kubectl to access the cluster.

Creating A Job (CronJob)

First , Let us deploy a job which runs every 5 mins (as we are scheduling in Crontab in Linux ) and Displays Hello Kubernetes Job.

In the below example , We are going to create object of Kind : CronJob .

Create a job.yaml file and paste the below contents.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
   name: sample-job
spec:
   schedule: "*/5 * * * *"
   jobTemplate:
     spec:
       template:
          spec:
            containers:
            - name: hello
              image: busybox
              command : ["echo", "Hello Kubernetes Job"]
            restartPolicy: OnFailure

Lets create a job using kubectl using job.yaml file.

kubectl create -f job.yaml

You should get he output as shown below.

Lets check the status of the cron job using the below command.

kubectl get cronjob -o wide

We can watch the jobs using the command,

kubectl get jobs --watch

For running the jobs , The jobs will create a Pod , To get the pod details , Run the below command.

kubectl get pods

Lets find the latest job that we created by the Job using the above command and get the output of the job.

kubectl logs podname

From the above output , We can see that the Cron Job is executing properly on the pods and displaying message as per the cron.yaml file.

Deleting Cron Job

If cron job is needed any more , We can delete the con jobs.

First get the name of the cron job using this command

kubectl get cronjob

Using the below command , We can delete the Cron jobs.

kubectl delete cronjob sample-job

Deleting the Cron job removes all the jobs and pods it created and stops it from creating additional jobs.

We can use other parameters with the above job.yaml file.

Here is the example of how the below parameters can be used for the Job.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
   name: test-job
spec:
   schedule: "*/5 * * * *"
   concurrencyPolicy: Forbid
   successfulJobsHistoryLimit: 5
   failedJobsHistoryLimit: 5
   jobTemplate:
     spec:
       template:
         spec:
           containers:
           - name: hello
             image: busybox
             command : ["echo", "Hello Kubernetes Job"]
           restartPolicy: OnFailure

Concurrency Policy – This is optional.This parameters is responsible for running the jobs in parallel with other jobs.We can use of the following concurrency policy such as ,

Allow (Default) , The cron job allows concurrently running jobs.

Forbid – Doesn’t allow concurrent jobs. If the previous job which was already running is not completed yet , But If the time for the new job run has started , Then in this case the new job will be skipped.

Replace – If It is the time for the new job run and the previous job run hasn’t finished yet , Then the old job will be replaced with the new job.

Job History Limits – such as successfulJobHistoryLimit and failedJobsHistoryLimit are optional fields.

By default , they are set to 3 and 1 respectively.

These fields specify how many completed and failed jobs should be kept in the cluster.

Keeping the limit to 0 , won’t keep anything history about the jobs.

Running Multiple Job Pods In Parallel

In this method , When a Job is deployed , We can make the job to run in multiple pods with parallelism.

Lets create a file multiple-jobs.yaml and add the below contents

apiVersion: batch/v1
kind: Job
metadata:
    generateName: kube-jobs-
    name: parallel-job
    labels:
      jobgroup: jobsexample
spec:
    completions: 3
    parallelism: 2
    template:
      metadata:
         name: parallel-job
         labels:
           jobgroup: jobsexample
      spec:
        containers:
        - name: busybox
          image: busybox
          command: ["echo" , "kubernetes cluster parallel"]
        restartPolicy: OnFailure

Lets understand the parameters used in the multiple-jobs.yaml file.

generateName – When running multple jobs with the same name It will generate an error saying that the job with the same name already exists.

To over come this issue , We should use generateName field in the metadata section.Hence when the Job is executed m It will create the pods with prefix kube-jobs– with numbers.

completions – The number of pods that need to run to completion for the job to be successful.In the above example , We have specified to run 3 pods.

parallelism – The number of pods that can be run in parallel.In the above example , We have specified parallelism as 2 , which means 2 pods runs in parallel.

restartPolicy – accepts always , never and onFailure. As the jobs are intended to run pods till completion , We should use never and onFailure for restartPolicy.

Lets go ahead and deploy the Cron job using the below command.

kubectl create -f multiple-jobs.yaml

Now check the jobs ,

kubectl get jobs -o wide

Lets check the pods created by the Job,

kubectl get po

You should get a similar output.

If you check logs for one of the pod , You should get the output as mentioned in the .yaml file.

kubectl logs -f parallel-job-jbfpb

Lets clean up the Job and the Pods created for it.

kubectl delete -f multiple-jobs.yaml

We have learnt to run Cron Jobs in Kubernetes Cluster.

Thanks for reading.Hope you found it helpful.

Please do check out my other articles.