open-nomad/website/source/guides/stateful-workloads/portworx.html.md

layout	page_title	sidebar_current	description
guides	Stateful Workloads with Portworx	guides-stateful-workloads-portworx	There are multiple approaches to deploying stateful applications in Nomad. This guide uses Portworx deploy a MySQL database.

Stateful Workloads with Portworx

Portworx integrates with Nomad and can manage storage for stateful workloads running inside your Nomad cluster. This guide walks you through deploying an HA MySQL workload.

Reference Material

• Portworx on Nomad

Estimated Time to Complete

20 minutes

Challenge

Deploy an HA MySQL database with a replication factor of 3, ensuring the data will be replicated on 3 different client nodes.

Solution

Configure Portworx on each Nomad client node in order to create a storage pool that the MySQL task can use for storage and replication.

Prerequisites

To perform the tasks described in this guide, you need to have a Nomad environment with Consul installed. You can use this repo to easily provision a sandbox environment. This guide will assume a cluster with one server node and three client nodes.

~> Please Note: This guide is for demo purposes and is only using a single server node. In a production cluster, 3 or 5 server nodes are recommended.

Steps

Step 1: Ensure Block Device Requirements

• Portworx needs an unformatted and unmounted block device that it can fully manage. If you have provisioned a Nomad cluster in AWS using the environment provided in this guide, you already have an external block device ready to use (/dev/xvdd) with a capacity of 50 GB.

• Ensure your root volume's size is at least 20 GB. If you are using the environment provided in this guide, add the following line to your terraform.tfvars file:

  root_block_device_size = 20
  

Step 2: Install the MySQL client

We will use the MySQL client to connect to our MySQL database and verify our data. Ensure it is installed on each client node:

$ sudo apt install mysql-client

Step 3: Set up the PX-OCI Bundle

Run the following command on each client node to set up the PX-OCI bundle:

sudo docker run --entrypoint /runc-entry-point.sh \
    --rm -i --privileged=true \
    -v /opt/pwx:/opt/pwx -v /etc/pwx:/etc/pwx \
    portworx/px-enterprise:2.0.2.3

If the command is successful, you will see output similar to the output show below (the output has been abbreviated):

Unable to find image 'portworx/px-enterprise:2.0.2.3' locally
2.0.2.3: Pulling from portworx/px-enterprise
...
Status: Downloaded newer image for portworx/px-enterprise:2.0.2.3
Executing with arguments: 
INFO: Copying binaries...
INFO: Copying rootfs...
[###############################################################################[.....................................................................................................Total bytes written: 2303375360 (2.2GiB, 48MiB/s)
INFO: Done copying OCI content.
You can now run the Portworx OCI bundle by executing one of the following:

    # sudo /opt/pwx/bin/px-runc run [options]
    # sudo /opt/pwx/bin/px-runc install [options]
...

Step 4: Configure Portworx OCI Bundle

Configure the Portworx OCI bundle on each client node by running the following command (the values provided to the options will be different for your environment):

$ sudo /opt/pwx/bin/px-runc install -k consul://172.31.49.111:8500 \
    -c my_test_cluster -s /dev/xvdd

• You can use client node you are on with the -k option since Consul is installed alongside Nomad

• Be sure to provide the -s option with your external block device path

If the configuration is successful, you will see the following output (abbreviated):

INFO[0000] Rootfs found at /opt/pwx/oci/rootfs          
INFO[0000] PX binaries found at /opt/pwx/bin/px-runc    
INFO[0000] Initializing as version 2.0.2.3-c186a87 (OCI) 
...
INFO[0000] Successfully written /etc/systemd/system/portworx.socket 
INFO[0000] Successfully written /etc/systemd/system/portworx-output.service 
INFO[0000] Successfully written /etc/systemd/system/portworx.service 

Since we have created new unit files, please run the following command to reload the systemd manager configuration:

sudo systemctl daemon-reload

Step 5: Start Portworx and Check Status

Run the following command to start Portworx:

$ sudo systemctl start portworx

Verify the service:

$ sudo systemctl status portworx
● portworx.service - Portworx OCI Container
   Loaded: loaded (/etc/systemd/system/portworx.service; disabled; vendor preset
   Active: active (running) since Wed 2019-03-06 15:16:51 UTC; 1h 47min ago
     Docs: https://docs.portworx.com/runc
  Process: 28230 ExecStartPre=/bin/sh -c /opt/pwx/bin/runc delete -f portworx ||
 Main PID: 28238 (runc)
...

Wait a few moments (Portworx may still be initializing) and then check the status of Portworx using the pxctl command.

$ pxctl status

If everything is working properly, you should see the following output:

Status: PX is operational
License: Trial (expires in 31 days)
Node ID: 07113eef-0533-4de8-b1cf-4471c18a7cda
	IP: 172.31.53.231 
 	Local Storage Pool: 1 pool
	POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE	REGION
	0	LOW		raid0		50 GiB	4.4 GiB	Online	us-east-1c	us-east-1
	Local Storage Devices: 1 device

Once all nodes are configured, you should see a cluster summary with the total capacity of the storage pool (if you're using the environment provided in this guide, the total capacity will be 150 GB since the external block device attached to each client nodes has a capacity of 50 GB):

Cluster Summary
	Cluster ID: my_test_cluster
	Cluster UUID: 705a1cbd-4d58-4a0e-a970-1e6b28375590
	Scheduler: none
	Nodes: 3 node(s) with storage (3 online)
...
Global Storage Pool
	Total Used    	:  13 GiB
	Total Capacity	:  150 GiB

Step 6: Create a Portworx Volume

Run the following command to create a Portworx volume that our job will be able to use:

$ pxctl volume create -s 10 -r 3 mysql

You should see output similar to what is shown below:

Volume successfully created: 693373920899724151

• Please note from the options provided that the name of the volume we created is mysql and the size is 10 GB.

• We have configured a replication factor of 3 which ensures our data is available on all 3 client nodes.

Run pxctl volume inspect mysql to verify the status of the volume:

$ pxctl volume inspect mysql
Volume	:  693373920899724151
	Name            	 :  mysql
	Size            	 :  10 GiB
	Format          	 :  ext4
	HA              	 :  3
...
	Replica sets on nodes:
		Set 0
		  Node 		 : 172.31.58.210 (Pool 0)
		  Node 		 : 172.31.51.110 (Pool 0)
		  Node 		 : 172.31.48.98 (Pool 0)
	Replication Status	 :  Up

Step 7: Create the mysql.nomad Job File

We are now ready to deploy a MySQL database that can use Portworx for storage. Create a file called mysql.nomad and provide it the following contents:

job "mysql-server" {
  datacenters = ["dc1"]
  type        = "service"

  group "mysql-server" {
    count = 1

    restart {
      attempts = 10
      interval = "5m"
      delay    = "25s"
      mode     = "delay"
    }

    task "mysql-server" {
      driver = "docker"

      env = {
        "MYSQL_ROOT_PASSWORD" = "password"
      }

      config {
        image = "hashicorp/mysql-portworx-demo:latest"

        port_map {
          db = 3306
        }

        volumes = [
          "mysql:/var/lib/mysql"
        ]

        volume_driver = "pxd"
      }

      resources {
        cpu    = 500
        memory = 1024

        network {
          port "db" {
            static = 3306
          }
        }
      }

      service {
        name = "mysql-server"
        port = "db"

        check {
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

• Please note from the job file that we are using the pxd volume driver that has been configured from the previous steps.

• The service name is mysql-server which we will use later to connect to the database.

Step 8: Deploy the MySQL Database

Register the job file you created in the previous step with the following command:

$ nomad run mysql.nomad 
==> Monitoring evaluation "aa478d82"
    Evaluation triggered by job "mysql-server"
    Allocation "6c3b3703" created: node "be8aad4e", group "mysql-server"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "aa478d82" finished with status "complete"

Check the status of the allocation and ensure the task is running:

$ nomad status mysql-server
ID            = mysql-server
...
Summary
Task Group    Queued  Starting  Running  Failed  Complete  Lost
mysql-server  0       0         1        0       0         0

Step 9: Connect to MySQL

Using the mysql client (installed in Step 2), connect to the database and access the information:

mysql -h mysql-server.service.consul -u web -p -D itemcollection

The password for this demo database is password.

~> Please Note: This guide is for demo purposes and does not follow best practices for securing database passwords. See Keeping Passwords Secure for more information.

Consul is installed alongside Nomad in this cluster so we were able to connect using the mysql-server service name we registered with our task in our job file.

Step 10: Add Data to MySQL

Once you are connected to the database, verify the table items exists:

mysql> show tables;
+--------------------------+
| Tables_in_itemcollection |
+--------------------------+
| items                    |
+--------------------------+
1 row in set (0.00 sec)

Display the contents of this table with the following command:

mysql> select * from items;
+----+----------+
| id | name     |
+----+----------+
|  1 | bike     |
|  2 | baseball |
|  3 | chair    |
+----+----------+
3 rows in set (0.00 sec)

Now add some data to this table (after we terminate our database in Nomad and bring it back up, this data should still be intact):

mysql> INSERT INTO items (name) VALUES ('glove');

Run the INSERT INTO command as many times as you like with different values.

mysql> INSERT INTO items (name) VALUES ('hat');
mysql> INSERT INTO items (name) VALUES ('keyboard');

Once you you are done, type exit and return back to the Nomad client command line:

mysql> exit
Bye

Step 11: Stop and Purge the Database Job

Run the following command to stop and purge the MySQL job from the cluster:

$ nomad stop -purge mysql-server
==> Monitoring evaluation "6b784149"
    Evaluation triggered by job "mysql-server"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "6b784149" finished with status "complete"

Verify no jobs are running in the cluster:

$ nomad status
No running jobs

You can optionally stop the nomad service on whichever node you are on and move to another node to simulate a node failure.

Step 12: Re-deploy the Database

Using the mysql.nomad job file from Step 6, re-deploy the database to the Nomad cluster.

==> Monitoring evaluation "61b4f648"
    Evaluation triggered by job "mysql-server"
    Allocation "8e1324d2" created: node "be8aad4e", group "mysql-server"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "61b4f648" finished with status "complete"

Step 13: Verify Data

Once you re-connect to MySQL, you should be able to see that the information you added prior to destroying the database is still present:

mysql> select * from items;
+----+----------+
| id | name     |
+----+----------+
|  1 | bike     |
|  2 | baseball |
|  3 | chair    |
|  4 | glove    |
|  5 | hat      |
|  6 | keyboard |
+----+----------+
6 rows in set (0.00 sec)