How to Back Up and Restore a Kubernetes Cluster on DigitalOcean Using Heptio Ark

Introduction

Heptio Ark is a convenient backup tool for Kubernetes clusters that compresses and backs up Kubernetes objects to object storage. It also takes snapshots of your cluster’s Persistent Volumes using your cloud provider’s block storage snapshot features, and can then restore your cluster’s objects and Persistent Volumes to a previous state.

StackPointCloud’s DigitalOcean Ark Plugin allows you to use DigitalOcean block storage to snapshot your Persistent Volumes, and Spaces to back up your Kubernetes objects. When running a Kubernetes cluster on DigitalOcean, this allows you to quickly back up your cluster’s state and restore it should disaster strike.

In this tutorial we’ll set up and configure the Ark client on a local machine, and deploy the Ark server into our Kubernetes cluster. We’ll then deploy a sample Nginx app that uses a Persistent Volume for logging, and simulate a disaster recovery scenario.

Prerequisites

Before you begin this tutorial, you should have the following available to you:

On your local computer:

• The kubectl command-line tool, configured to connect to your cluster. You can read more about installing and configuring kubectl in the official Kubernetes documentation.
• The git command-line utility. You can learn how to install git in Getting Started with Git.

In your DigitalOcean account:

• A DigitalOcean Kubernetes cluster, or a Kubernetes cluster (version 1.7.5 or later) on DigitalOcean Droplets
• A DNS server running inside of your cluster. If you are using DigitalOcean Kubernetes, this is running by default. To learn more about configuring a Kubernetes DNS service, consult Customizing DNS Service from the official Kuberentes documentation.
• A DigitalOcean Space that will store your backed-up Kubernetes objects. To learn how to create a Space, consult the Spaces product documentation.
• An access key pair for your DigitalOcean Space. To learn how to create a set of access keys, consult How to Manage Administrative Access to Spaces.
• A personal access token for use with the DigitalOcean API. To learn how to create a personal access token, consult How to Create a Personal Access Token.

Once you have all of this set up, you’re ready to begin with this guide.

Step 1 — Installing the Ark Client

The Heptio Ark backup tool consists of a client installed on your local computer and a server that runs in your Kubernetes cluster. To begin, we’ll install the local Ark client.

In your web browser, navigate to the Ark GitHub repo releases page, find the latest release corresponding to your OS and system architecture, and copy the link address. For the purposes of this guide, we’ll use an Ubuntu 18.04 server on an x86-64 (or AMD64) processor as our local machine.

Then, from the command line on your local computer, navigate to the temporary /tmp directory and cd into it:

• cd /tmp

Use wget and the link you copied earlier to download the release tarball:

• wget https://link_copied_from_release_page

Once the download completes, extract the tarball using tar (note the filename may differ depending on the current release version and your OS):

• tar -xvzf ark-v0.9.6-linux-amd64.tar.gz

The /tmp directory should now contain the extracted ark binary as well as the tarball you just downloaded.

Verify that you can run the ark client by executing the binary:

• ./ark –help

You should see the following help output:

Heptio Ark is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.

If you're familiar with kubectl, Ark supports a similar model, allowing you to
execute commands such as 'ark get backup' and 'ark create schedule'. The same
operations can also be performed as 'ark backup get' and 'ark schedule create'.

Usage:
  ark [command]

Available Commands:
  backup      Work with backups
  client      Ark client related commands
  completion  Output shell completion code for the specified shell (bash or zsh)
  create      Create ark resources
  delete      Delete ark resources
  describe    Describe ark resources
  get         Get ark resources
  help        Help about any command
  plugin      Work with plugins
  restic      Work with restic
  restore     Work with restores
  schedule    Work with schedules
  server      Run the ark server
  version     Print the ark version and associated image

. . .

At this point you should move the ark executable out of the temporary /tmp directory and add it to your PATH. To add it to your PATH on an Ubuntu system, simply copy it to /usr/local/bin:

• sudo mv ark /usr/local/bin/ark

You’re now ready to configure the Ark server and deploy it to your Kubernetes cluster.

Step 2 — Installing and Configuring the Ark Server

Before we deploy Ark into our Kubernetes cluster, we’ll first create Ark’s prerequisite objects. Ark’s prerequisites consist of:

• A heptio-ark Namespace
• The ark Service Account
• Role-based access control (RBAC) rules to grant permissions to the ark Service Account
• Custom Resources (CRDs) for the Ark-specific resources: Backup, Schedule, Restore, Config

A YAML file containing the specs for the above Kubernetes objects can be found in the official Ark Git repository. While still in the /tmp directory, download the Ark repo using git:

• git clone https://github.com/heptio/ark.git

Once downloaded, navigate into the ark directory:

• cd ark

The prerequisite resources listed above can be found in the examples/common/00-prereqs.yaml YAML file. We’ll create these resources in our Kubernetes cluster by using kubectl apply and passing in the file:

• kubectl apply -f examples/common/00-prereqs.yaml

You should see the following output:

customresourcedefinition.apiextensions.k8s.io/backups.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/schedules.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/restores.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/configs.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/downloadrequests.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/deletebackuprequests.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/podvolumebackups.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/podvolumerestores.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/resticrepositories.ark.heptio.com created
customresourcedefinition.apiextensions.k8s.io/backupstoragelocations.ark.heptio.com created
namespace/heptio-ark created
serviceaccount/ark created
clusterrolebinding.rbac.authorization.k8s.io/ark created

Now that we’ve created the necessary Ark Kubernetes objects in our cluster, we can download and install the Ark DigitalOcean Plugin, which will allow us to use DigitalOcean Spaces as a backupStorageProvider (for Kubernetes objects), and DigitalOcean Block Storage as a persistentVolumeProvider (for Persistent Volume backups).

Move back out of the ark directory and fetch the plugin from StackPointCloud’s repo using git:

• cd ..
• git clone https://github.com/StackPointCloud/ark-plugin-digitalocean.git

Move into the plugin directory:

• cd ark-plugin-digitalocean

We’ll now save the access keys for our DigitalOcean Space as a Kubernetes Secret. First, open up the examples/credentials-ark file using your favorite editor:

• nano examples/credentials-ark

Replace <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with your Spaces access key and secret key:

[default]
aws_access_key_id=your_spaces_access_key_here
aws_secret_access_key=your_spaces_secret_key_here

Save and close the file.

Now, create the cloud-credentials Secret using kubectl, inserting your Personal Access Token using the digitalocean_token data item:

• kubectl create secret generic cloud-credentials \
• –namespace heptio-ark \
• –from-file cloud=examples/credentials-ark \
• –from-literal digitalocean_token=your_personal_access_token

You should see the following output:

secret/cloud-credentials created

To confirm that the cloud-credentials Secret was created successfully, you can describe it using kubectl:

• kubectl describe secrets/cloud-credentials –namespace heptio-ark

You should see the following output describing the cloud-credentials secret:

Name:         cloud-credentials
Namespace:    heptio-ark
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
cloud:               115 bytes
digitalocean_token:  64 bytes

We can now move on to creating an Ark Config object named default. To do this, we’ll edit a YAML configuration file and then create the object in our Kubernetes cluster.

Open examples/10-ark-config.yaml in your favorite editor:

• nano examples/10-ark-config.yaml

Insert your Space’s name and region in the highlighted fields:

---
apiVersion: ark.heptio.com/v1
kind: Config
metadata:
  namespace: heptio-ark
  name: default
persistentVolumeProvider:
  name: digitalocean
backupStorageProvider:
  name: aws
  bucket: space_name_here
  config:
    region: space_region_here
    s3ForcePathStyle: "true"
    s3Url: https://space_region_here.digitaloceanspaces.com
backupSyncPeriod: 30m
gcSyncPeriod: 30m
scheduleSyncPeriod: 1m
restoreOnlyMode: false

persistentVolumeProvider sets DigitalOcean Block Storage as the the provider for Persistent Volume backups. These will be Block Storage Volume Snapshots.

backupStorageProvider sets DigitalOcean Spaces as the provider for Kubernetes object backups. Ark will create a tarball of all your Kubernetes objects (or some, depending on how you execute it), and upload this tarball to Spaces.

When you’re done, save and close the file.

Create the object in your cluster using kubectl apply:

• kubectl apply -f examples/10-ark-config.yaml

You should see the following output:

config.ark.heptio.com/default created

At this point, we’ve finished configuring the Ark server and can create its Kubernetes deployment, found in the examples/20-deployment.yaml configuration file. Let’s take a quick look at this file:

• cat examples/20-deployment.yaml

You should see the following text:

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  namespace: heptio-ark
  name: ark
spec:
  replicas: 1
  template:
    metadata:
      labels:
        component: ark
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8085"
        prometheus.io/path: "/metrics"
    spec:
      restartPolicy: Always
      serviceAccountName: ark
      containers:
        - name: ark
          image: gcr.io/heptio-images/ark:latest
          command:
            - /ark
          args:
            - server
          volumeMounts:
            - name: cloud-credentials
              mountPath: /credentials
            - name: plugins
              mountPath: /plugins
            - name: scratch
              mountPath: /scratch
          env:
            - name: AWS_SHARED_CREDENTIALS_FILE
              value: /credentials/cloud
            - name: ARK_SCRATCH_DIR
              value: /scratch
            - name: DIGITALOCEAN_TOKEN
              valueFrom:
                secretKeyRef:
                  key: digitalocean_token
                  name: cloud-credentials
      volumes:
        - name: cloud-credentials
          secret:
            secretName: cloud-credentials
        - name: plugins
          emptyDir: {}
        - name: scratch
          emptyDir: {}

We observe here that we’re creating a Deployment called ark that consists of a single replica of the gcr.io/heptio-images/ark:latest container. The Pod is configured using the cloud-credentials secret we created earlier.

Create the Deployment using kubectl apply:

• kubectl apply -f examples/20-deployment.yaml

You should see the following output:

deployment.apps/ark created

We can double check that the Deployment has been successfully created using kubectl get on the heptio-ark Namespace :

• kubectl get deployments –namespace=heptio-ark

You should see the following output:

NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
ark       1         1         1            0           8m

The Ark server Pod may not start correctly until you install the Ark DigitalOcean plugin. To install the ark-blockstore-digitalocean plugin, use the ark client we installed earlier:

• ark plugin add quay.io/stackpoint/ark-blockstore-digitalocean:latest

You can specify the kubeconfig to use with the --kubeconfig flag. If you don’t use this flag, ark will check the KUBECONFIG environment variable and then fall back to the kubectl default (~/.kube/config).

At this point Ark is running and fully configured, and ready to back up and restore your Kubernetes cluster objects and Persistent Volumes to DigitalOcean Spaces and Block Storage.

In the next section, we’ll run a quick test to make sure that the backup and restore functionality works as expected.

Step 3 — Testing Backup and Restore Procedure

Now that we’ve successfully installed and configured Ark, we can create a test Nginx Deployment and Persistent Volume, and run through a backup and restore drill to ensure that everything is working properly.

The ark-plugin-digitalocean repository contains a sample Nginx deployment called nginx-pv.yaml.

Let’s take a quick look:

• cat examples/nginx-pv.yaml

You should see the following text:

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-logs
  namespace: nginx-example
  labels:
    app: nginx
spec:
  storageClassName: do-block-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: nginx-example
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
        - name: nginx-logs
          persistentVolumeClaim:
           claimName: nginx-logs
      containers:
      - image: nginx:1.7.9
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/var/log/nginx"
            name: nginx-logs
            readOnly: false

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: my-nginx
  namespace: nginx-example
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

In this file, we observe specs for:

• An Nginx Deployment consisting of a single replica of the nginx:1.7.9 container image
• A 5Gi Persistent Volume Claim (called nginx-logs), using the do-block-storage StorageClass
• A LoadBalancer Service that exposes port 80

Create the deployment using kubectl apply:

• kubectl apply -f examples/nginx-pv.yml

You should see the following output:

namespace/nginx-example created
persistentvolumeclaim/nginx-logs created
deployment.apps/nginx-deployment created
service/my-nginx created

Check that the Deployment succeeded:

• kubectl get deployments –namespace=nginx-example

You should see the following output:

NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   1         1         1            1           1h

Once Available reaches 1, fetch the Nginx load balancer’s external IP using kubectl get:

• kubectl get services –namespace=nginx-example

You should see both the internal CLUSTER-IP and EXTERNAL-IP for the my-nginx Service:

NAME       TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)        AGE
my-nginx   LoadBalancer   10.32.27.0      203.0.113.0   80:30754/TCP   3m

Note the EXTERNAL-IP and navigate to it using your web browser.

You should see the following NGINX welcome page:

This indicates that your Nginx Deployment and Service are up and running.

Before we simulate our disaster scenario, let’s first check the Nginx access logs (stored on a Persistent Volume attached to the Nginx Pod):

Fetch the Pod’s name using kubectl get:

• kubectl get pods –namespace nginx-example

NAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-77d8f78fcb-zt4wr   1/1       Running   0          29m

Now, exec into the running Nginx container to get a shell inside of it:

• kubectl exec -it nginx-deployment-77d8f78fcb-zt4wr –namespace nginx-example — /bin/bash

Once inside the Nginx container, cat the Nginx access logs:

• cat /var/log/nginx/access.log

You should see some Nginx access entries:

10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-"
10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET /favicon.ico HTTP/1.1" 404 570 "http://203.0.113.0/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-"

Note these down (especially the timestamps), as we will use them to confirm the success of the restore procedure.

We can now perform the backup procedure to copy all nginx Kubernetes objects to Spaces and take a Snapshot of the Persistent Volume we created when deploying Nginx.

We’ll create a backup called nginx-backup using the ark client:

• ark backup create nginx-backup –selector app=nginx

The --selector app=nginx instructs the Ark server to only back up Kubernetes objects with the app=nginx Label Selector.

You should see the following output:

Backup request "nginx-backup" submitted successfully.
Run `ark backup describe nginx-backup` for more details.

Running ark backup describe nginx-backup should provide the following output after a short delay:

Name:         nginx-backup
Namespace:    heptio-ark
Labels:       <none>
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  app=nginx

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2018-09-26 00:14:30 -0400 EDT
Completed:  2018-09-26 00:14:34 -0400 EDT

Expiration:  2018-10-26 00:14:30 -0400 EDT

Validation errors:  <none>

Persistent Volumes:
  pvc-e4862eac-c2d2-11e8-920b-92c754237aeb:
    Snapshot ID:        2eb66366-c2d3-11e8-963b-0a58ac14428b
    Type:               ext4
    Availability Zone:
    IOPS:               <N/A>

This output indicates that nginx-backup completed successfully.

From the DigitalOcean Cloud Control Panel, navigate to the Space containing your Kubernetes backup files.

You should see a new directory called nginx-backup containing the Ark backup files.

Using the left-hand navigation bar, go to Images and then Snapshots. Within Snapshots, navigate to Volumes. You should see a Snapshot corresponding to the PVC listed in the above output.

We can now test the restore procedure.

Let’s first delete the nginx-example Namespace. This will delete everything in the Namespace, including the Load Balancer and Persistent Volume:

• kubectl delete namespace nginx-example

Verify that you can no longer access Nginx at the Load Balancer endpoint, and that the nginx-example Deployment is no longer running:

• kubectl get deployments –namespace=nginx-example

No resources found.

We can now perform the restore procedure, once again using the ark client:

• ark restore create –from-backup nginx-backup

Here we use create to create an Ark Restore object from the nginx-backup object.

You should see the following output:

• Restore request “nginx-backup-20180926143828” submitted successfully.
• Run `ark restore describe nginx-backup-20180926143828` for more details.

Check the status of the restored Deployment:

• kubectl get deployments –namespace=nginx-example

NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   1         1         1            1           1m

Check for the creation of a Persistent Volume:

• kubectl get pvc –namespace=nginx-example

NAME         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
nginx-logs   Bound     pvc-e4862eac-c2d2-11e8-920b-92c754237aeb   5Gi        RWO            do-block-storage   3m

Navigate to the Nginx Service’s external IP once again to confirm that Nginx is up and running.

Finally, check the logs on the restored Persistent Volume to confirm that the log history has been preserved post-restore.

To do this, once again fetch the Pod’s name using kubectl get:

• kubectl get pods –namespace nginx-example

NAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-77d8f78fcb-zt4wr   1/1       Running   0          29m

Then exec into it:

• kubectl exec -it nginx-deployment-77d8f78fcb-zt4wr –namespace nginx-example — /bin/bash

Once inside the Nginx container, cat the Nginx access logs:

• cat /var/log/nginx/access.log

10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-"
10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET /favicon.ico HTTP/1.1" 404 570 "http://203.0.113.0/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-"

You should see the same pre-backup access attempts (note the timestamps), confirming that the Persistent Volume restore was successful. Note that there may be additional attempts in the logs if you visited the Nginx landing page after you performed the restore.

At this point, we’ve successfully backed up our Kubernetes objects to DigitalOcean Spaces, and our Persistent Volumes using Block Storage Volume Snapshots. We simulated a disaster scenario, and restored service to the test Nginx application.

Conclusion

In this guide we installed and configured the Ark Kubernetes backup tool on a DigitalOcean-based Kubernetes cluster. We configured the tool to back up Kubernetes objects to DigitalOcean Spaces, and back up Persistent Volumes using Block Storage Volume Snapshots.

Ark can also be used to schedule regular backups of your Kubernetes cluster. To do this, you can use the ark schedule command. It can also be used to migrate resources from one cluster to another. To learn more about these two use cases, consult the official Ark documentation.

To learn more about DigitalOcean Spaces, consult the official Spaces documentation. To learn more about Block Storage Volumes, consult the Block Storage Volume documentation.

This tutorial builds on the README found in StackPointCloud’s ark-plugin-digitalocean GitHub repo.