open-nomad/demo/csi/ceph-csi-plugin

Openstack Ceph-CSI Plugin

The configuration here is for the Ceph RBD driver, migrated from the k8s config documentation. It can be easily modified for the CephFS Driver, as used here.

Requirements

The example plugin job creates a file at local/cloud.conf using a template stanza which pulls the necessary credentials from a Vault kv-v2 secrets store.

Docker Privileged Mode

The Ceph CSI Node task requires that privileged = true be set. This is not needed for the Controller task.

Container Arguments

Refer to the official plugin guide.

• --type=rbd

  • Driver type rbd (or alternately cephfs)

• --endpoint=unix:///csi/csi.sock

  • This option must match the mount_dir specified in the csi_plugin stanza for the task.

• --nodeid=${node.unique.name}

  • A unique ID for the node the task is running on. Recommend using ${node.unique.name}

• --cluster=${NOMAD_DC}

  • The cluster the Controller/Node is a part of. Recommend using ${NOMAD_DC}

• --instanceid=${attr.unique.platform.aws.instance-id}

  • Unique ID distinguishing this instance of Ceph CSI among other instances, when sharing Ceph clusters across CSI instances for provisioning. Used for topology-aware deployments.

Deployment

Plugin

export NOMAD_ADDR=https://nomad.example.com:4646
export NOMAD_TOKEN=34534-3sdf3-szfdsafsdf3423-zxdfsd3
nomad job run ceph-csi-plugin.hcl

Volume Registration

The external_id value for the volume must be strictly formatted, see ceph_csi.tf. Based on Ceph-CSI ID Format, see examples.

The secrets block will be populated with values pulled from /etc/ceph/ceph.client.<user>.keyring, e.g.

userid = "<user>"
userkey = "AWBg/BtfJInSFBATOrrnCh6UGE3QB3nYakdF+g=="

export NOMAD_ADDR=https://nomad.example.com:4646
export NOMAD_TOKEN=34534-3sdf3-szfdsafsdf3423-zxdfsd3
nomad volume register example_volume.hcl

Ceph CSI Driver Source

• https://github.com/ceph/ceph-csi