open-nomad/demo/csi/kadalu-csi/README.md

# Kadalu CSI Plugin

Author: @leelavg and the [Kadalu][kadalu_org] team.

The configuration here is for using external [Gluster] volumes as persistent
storage in Nomad using [Kadalu CSI][kadalu].

Refer to the actual job files before proceeding with this demo and change the
configuration as required. Follow along with the commands according to your
configuration.

Locally tested against Nomad v1.3.1

## Local Development

This section can be skipped if you already have a Nomad cluster setup.

```console
# Clone configuration repository used to create local Nomad cluster in Docker
$ git clone https://github.com/leelavg/kadalu-nomad && cd kadalu-nomad

# Install Shipyard following the instructions in https://shipyard.run
# Create local cluster
$ shipyard run
[...]
$ eval $(shipyard env)
$ export job_dir="$(pwd)/kadalu"
```

## Demo

### Pre-requisites
- Configure varisables mentioned in `cluster.vars` to reflect your external
  Gluster details.
- For convenience the necessary variables are set from the CLI when running the
  job.

```console
$ export volname="sample-pool" gluster_hosts="10.x.x.x" gluster_volname="sample-vol" job_dir="${job_dir:-$(pwd)}"

# Make sure external gluster volume is started and quota is set
$ ssh $gluster_hosts "gluster volume info $gluster_volname | grep Status"
Status: Started

$ ssh $gluster_hosts "gluster volume quota $gluster_volname enable"
volume quota : success
```

### CSI Deployment

Deploy the CSI plugin controller.

```console
$ nomad run -var="volname=$volname" -var="gluster_hosts=$gluster_hosts" -var="gluster_volname=$gluster_volname" $job_dir/controller.nomad
==> 2021-09-20T18:23:07+05:30: Monitoring evaluation "19317b74"
    2021-09-20T18:23:07+05:30: Evaluation triggered by job "kadalu-csi-controller"
==> 2021-09-20T18:23:08+05:30: Monitoring evaluation "19317b74"
    2021-09-20T18:23:08+05:30: Evaluation within deployment: "d9ee4dd7"
    2021-09-20T18:23:08+05:30: Allocation "d55e314d" created: node "4e105698", group "controller"
    2021-09-20T18:23:08+05:30: Evaluation status changed: "pending" -> "complete"
==> 2021-09-20T18:23:08+05:30: Evaluation "19317b74" finished with status "complete"
==> 2021-09-20T18:23:08+05:30: Monitoring deployment "d9ee4dd7"
  ✓ Deployment "d9ee4dd7" successful

    2021-09-20T18:23:28+05:30
    ID          = d9ee4dd7
    Job ID      = kadalu-csi-controller
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully

    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    controller  1        1       1        0          2021-09-20T13:03:27Z
```

Deploy the CSI node plugin.

```console
$ nomad run -var="volname=$volname" -var="gluster_hosts=$gluster_hosts" -var="gluster_volname=$gluster_volname" $job_dir/nodeplugin.nomad
==> 2021-09-20T18:23:53+05:30: Monitoring evaluation "bd4d95d1"
    2021-09-20T18:23:53+05:30: Evaluation triggered by job "kadalu-csi-nodeplugin"
==> 2021-09-20T18:23:54+05:30: Monitoring evaluation "bd4d95d1"
    2021-09-20T18:23:54+05:30: Allocation "4c05ab5a" created: node "4e105698", group "nodeplugin"
    2021-09-20T18:23:54+05:30: Evaluation status changed: "pending" -> "complete"
==> 2021-09-20T18:23:54+05:30: Evaluation "bd4d95d1" finished with status "complete"
```

Verify the CSI plugin status.

```console
$ nomad plugin status kadalu-csi
ID                   = kadalu-csi
Provider             = kadalu
Version              = 0.8.15
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 1
Nodes Expected       = 1

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
d55e314d  4e105698  controller  0        run      running  1m20s ago  1m ago
4c05ab5a  4e105698  nodeplugin  0        run      running  35s ago    20s ago
```

### Volume Management

Next, you will go through volume creation, attachment and deletion operations,
covering a typical volume life-cycle.

#### Creating a Volume

```console
# Create Nomad volume
$ sed -e "s/POOL/$volname/" -e "s/GHOST/$gluster_hosts/" -e "s/GVOL/$gluster_volname/" $job_dir/volume.hcl | nomad volume create -
Created external volume csi-test with ID csi-test
```

#### Attaching and Using a Volume

```console
# Attach the volume to a sample app
$ nomad run $job_dir/app.nomad
==> 2021-09-20T18:28:28+05:30: Monitoring evaluation "e6dd3129"
    2021-09-20T18:28:28+05:30: Evaluation triggered by job "sample-pv-check"
==> 2021-09-20T18:28:29+05:30: Monitoring evaluation "e6dd3129"
    2021-09-20T18:28:29+05:30: Evaluation within deployment: "814e328c"
    2021-09-20T18:28:29+05:30: Allocation "64745b25" created: node "4e105698", group "apps"
    2021-09-20T18:28:29+05:30: Evaluation status changed: "pending" -> "complete"
==> 2021-09-20T18:28:29+05:30: Evaluation "e6dd3129" finished with status "complete"
==> 2021-09-20T18:28:29+05:30: Monitoring deployment "814e328c"
  ✓ Deployment "814e328c" successful

    2021-09-20T18:28:58+05:30
    ID          = 814e328c
    Job ID      = sample-pv-check
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully

    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    apps        1        1       1        0          2021-09-20T13:08:56Z

# Export allocation ID (64745b25) from the previous command output
$ export app=64745b25

# Verify that the CSI Volume is accessible
$ nomad alloc exec $app bash /kadalu/script.sh
This is a sample application

# df -h
Filesystem                               Size      Used Available Use% Mounted on
<gluster_hosts>:<gluster_volname>      181.2M         0    181.2M   0% /mnt/pv

# mount
Write/Read test on PV mount Mon
Sep 20 12:59:34 UTC 2021
SUCCESS

# Write some data on the volume
$ nomad alloc exec $app bash -c 'cd /mnt/pv; for i in {1..10}; do cat /dev/urandom | tr -dc [:space:][:print:] | head -c 1m > file$i; done;'

# Checksum the written data
$ nomad alloc exec $app bash -c 'ls /mnt/pv; find /mnt/pv -type f -exec md5sum {} + | cut -f1 -d" " | sort | md5sum'
file1   file2   file4   file6   file8
file10  file3   file5   file7   file9
6776dd355c0f2ba5a1781b9831e5c174  -

# Stop sample app and run it again to check data persistence
$ nomad status
ID                         Type     Priority  Status   Submit Date
kadalu-csi-controller      service  50        running  2021-09-20T18:23:07+05:30
kadalu-csi-nodeplugin      system   50        running  2021-09-20T18:23:53+05:30
sample-pv-check            service  50        running  2021-09-20T18:28:28+05:30

$ nomad stop sample-pv-check
==> 2021-09-20T18:36:47+05:30: Monitoring evaluation "eecc0c00"
    2021-09-20T18:36:47+05:30: Evaluation triggered by job "sample-pv-check"
==> 2021-09-20T18:36:48+05:30: Monitoring evaluation "eecc0c00"
    2021-09-20T18:36:48+05:30: Evaluation within deployment: "814e328c"
    2021-09-20T18:36:48+05:30: Evaluation status changed: "pending" -> "complete"
==> 2021-09-20T18:36:48+05:30: Evaluation "eecc0c00" finished with status "complete"
==> 2021-09-20T18:36:48+05:30: Monitoring deployment "814e328c"
  ✓ Deployment "814e328c" successful

    2021-09-20T18:36:48+05:30
    ID          = 814e328c
    Job ID      = sample-pv-check
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully

    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    apps        1        1       1        0          2021-09-20T13:08:56Z

$ nomad run $job_dir/app.nomad
==> 2021-09-20T18:37:49+05:30: Monitoring evaluation "e04b4549"
    2021-09-20T18:37:49+05:30: Evaluation triggered by job "sample-pv-check"
==> 2021-09-20T18:37:50+05:30: Monitoring evaluation "e04b4549"
    2021-09-20T18:37:50+05:30: Evaluation within deployment: "66d246ee"
    2021-09-20T18:37:50+05:30: Allocation "526d5543" created: node "4e105698", group "apps"
    2021-09-20T18:37:50+05:30: Evaluation status changed: "pending" -> "complete"
==> 2021-09-20T18:37:50+05:30: Evaluation "e04b4549" finished with status "complete"
==> 2021-09-20T18:37:50+05:30: Monitoring deployment "66d246ee"
  ✓ Deployment "66d246ee" successful

    2021-09-20T18:38:10+05:30
    ID          = 66d246ee
    Job ID      = sample-pv-check
    Job Version = 2
    Status      = successful
    Description = Deployment completed successfully

    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    apps        1        1       1        0          2021-09-20T13:18:08Z

# Export the new allocation ID and verify that md5sum matches after stopping and
# running the same job
$ export app=526d5543
$ nomad alloc exec $app bash -c 'ls /mnt/pv; find /mnt/pv -type f -exec md5sum {} + | cut -f1 -d" " | sort | md5sum'
file1   file10  file2 file3   file4   file5   file6   file7   file8   file9
6776dd355c0f2ba5a1781b9831e5c174  -
```

#### Cleanup
```console
# Stop sample app, delete the volume and stop the CSI plugin components
$ nomad stop sample-pv-check
$ nomad volume delete csi-test
$ nomad stop kadalu-csi-nodeplugin
$ nomad stop kadalu-csi-controller

# Destroy local Shipyard cluster
$ shipyard destroy
```

## Contact

- For any extra information/feature with regards to the Kadalu CSI plugin,
  please raise an issue against the [`kadalu` repo][kadalu].
- For any extra information with regards to the local Nomad dev setup for CSI,
  please raise an issue against the [`kadalu-nomad` repo][kadalu_nomad].
- Based on ask/feature request, we may work on supporting internal Gluster
  deployed and managed by Nomad itself (feature parity with current Kubernetes
  deployments).
- If this folder isn't updated frequently you can find updated jobs at the
  [`nomad` folder][nomad_folder] in the `kadalu` repository.

[Gluster]: https://www.gluster.org/
[kadalu]: https://github.com/kadalu/kadalu
[kadalu_org]: https://github.com/kadalu
[kadalu_nomad]: https://github.com/leelavg/kadalu-nomad
[nomad_folder]: https://github.com/kadalu/kadalu/tree/devel/nomad