Merge pull request #2738 from hashicorp/f-terraform-config
Add Packer, Terraform configs to spin up an integrated Nomad, Consul, Vault cluster in AWS
This commit is contained in:
Rob Genova
GitHub
commit
5f55f5a5ab
|
|
@ -0,0 +1,134 @@
|
|||
# Provision a Nomad cluster on AWS with Packer & Terraform
|
||||
|
||||
Use this to easily provision a Nomad sandbox environment on AWS with
|
||||
[Packer](https://packer.io) and [Terraform](https://terraform.io).
|
||||
[Consul](https://www.consul.io/intro/index.html) and
|
||||
[Vault](https://www.vaultproject.io/intro/index.html) are also installed
|
||||
(colocated for convenience). The intention is to allow easy exploration of
|
||||
Nomad and its integrations with the HashiCorp stack. This is *not* meant to be
|
||||
a production ready environment. A demonstration of [Nomad's Apache Spark
|
||||
integration](examples/spark/README.md) is included.
|
||||
|
||||
## Setup
|
||||
|
||||
Clone this repo and (optionally) use [Vagrant](https://www.vagrantup.com/intro/index.html)
|
||||
to bootstrap a local staging environment:
|
||||
|
||||
```bash
|
||||
$ git clone git@github.com:hashicorp/nomad.git
|
||||
$ cd terraform/aws
|
||||
$ vagrant up && vagrant ssh
|
||||
```
|
||||
|
||||
The Vagrant staging environment pre-installs Packer, Terraform, and Docker.
|
||||
|
||||
### Pre-requisites
|
||||
|
||||
You will need the following:
|
||||
|
||||
- AWS account
|
||||
- [API access keys](http://aws.amazon.com/developers/access-keys/)
|
||||
- [SSH key pair](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html)
|
||||
|
||||
Set environment variables for your AWS credentials:
|
||||
|
||||
```bash
|
||||
$ export AWS_ACCESS_KEY_ID=[ACCESS_KEY_ID]
|
||||
$ export AWS_SECRET_ACCESS_KEY=[SECRET_ACCESS_KEY]
|
||||
```
|
||||
|
||||
## Provision a cluster
|
||||
|
||||
`cd` to an environment subdirectory:
|
||||
|
||||
```bash
|
||||
$ cd env/us-east
|
||||
```
|
||||
|
||||
Update terraform.tfvars with your SSH key name:
|
||||
|
||||
```bash
|
||||
region = "us-east-1"
|
||||
ami = "ami-76787e60"
|
||||
instance_type = "t2.medium"
|
||||
key_name = "KEY_NAME"
|
||||
server_count = "3"
|
||||
client_count = "4"
|
||||
```
|
||||
|
||||
Note that a pre-provisioned, publicly available AMI is used by default
|
||||
(for the `us-east-1` region). To provision your own customized AMI with
|
||||
[Packer](https://www.packer.io/intro/index.html), follow the instructions
|
||||
[here](aws/packer/README.md). You will need to replace the AMI ID in
|
||||
`terraform.tfvars` with your own. You can also modify the `region`,
|
||||
`instance_type`, `server_count`, and `client_count`. At least one client and
|
||||
one server are required.
|
||||
|
||||
Provision the cluster:
|
||||
|
||||
```bash
|
||||
$ terraform get
|
||||
$ terraform plan
|
||||
$ terraform apply
|
||||
```
|
||||
|
||||
## Access the cluster
|
||||
|
||||
SSH to one of the servers using its public IP:
|
||||
|
||||
```bash
|
||||
$ ssh -i /path/to/key ubuntu@PUBLIC_IP
|
||||
```
|
||||
|
||||
Note that the AWS security group is configured by default to allow all traffic
|
||||
over port 22. This is *not* recommended for production deployments.
|
||||
|
||||
Run a few basic commands to verify that Consul and Nomad are up and running
|
||||
properly:
|
||||
|
||||
```bash
|
||||
$ consul members
|
||||
$ nomad server-members
|
||||
$ nomad node-status
|
||||
```
|
||||
|
||||
Optionally, initialize and unseal Vault:
|
||||
|
||||
```bash
|
||||
$ vault init -key-shares=1 -key-threshold=1
|
||||
$ vault unseal
|
||||
$ export VAULT_TOKEN=[INITIAL_ROOT_TOKEN]
|
||||
```
|
||||
|
||||
The `vault init` command above creates a single
|
||||
[Vault unseal key](https://www.vaultproject.io/docs/concepts/seal.html) for
|
||||
convenience. For a production environment, it is recommended that you create at
|
||||
least five unseal key shares and securely distribute them to independent
|
||||
operators. The `vault init` command defaults to five key shares and a key
|
||||
threshold of three. If you provisioned more than one server, the others will
|
||||
become standby nodes (but should still be unsealed). You can query the active
|
||||
and standby nodes independently:
|
||||
|
||||
```bash
|
||||
$ dig active.vault.service.consul
|
||||
$ dig active.vault.service.consul SRV
|
||||
$ dig standby.vault.service.consul
|
||||
```
|
||||
|
||||
## Getting started with Nomad & the HashiCorp stack
|
||||
|
||||
See:
|
||||
|
||||
* [Getting Started with Nomad](https://www.nomadproject.io/intro/getting-started/jobs.html)
|
||||
* [Consul integration](https://www.nomadproject.io/docs/service-discovery/index.html)
|
||||
* [Vault integration](https://www.nomadproject.io/docs/vault-integration/index.html)
|
||||
* [consul-template integration](https://www.nomadproject.io/docs/job-specification/template.html)
|
||||
|
||||
## Apache Spark integration
|
||||
|
||||
Nomad is well-suited for analytical workloads, given its performance
|
||||
characteristics and first-class support for batch scheduling. Apache Spark is a
|
||||
popular data processing engine/framework that has been architected to use
|
||||
third-party schedulers. The Nomad ecosystem includes a [fork that natively
|
||||
integrates Nomad with Spark](https://github.com/hashicorp/nomad-spark). A
|
||||
detailed walkthrough of the integration is included [here](examples/spark/README.md).
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
# -*- mode: ruby -*-
|
||||
# vi: set ft=ruby :
|
||||
|
||||
Vagrant.configure(2) do |config|
|
||||
|
||||
config.vm.box = "ubuntu/trusty64"
|
||||
config.vm.provision "shell", inline: <<-SHELL
|
||||
|
||||
cd /tmp
|
||||
|
||||
PACKERVERSION=1.0.0
|
||||
PACKERDOWNLOAD=https://releases.hashicorp.com/packer/${PACKERVERSION}/packer_${PACKERVERSION}_linux_amd64.zip
|
||||
TERRAFORMVERSION=0.9.8
|
||||
TERRAFORMDOWNLOAD=https://releases.hashicorp.com/terraform/${TERRAFORMVERSION}/terraform_${TERRAFORMVERSION}_linux_amd64.zip
|
||||
|
||||
echo "Dependencies..."
|
||||
sudo apt-get install -y unzip tree
|
||||
|
||||
# Disable the firewall
|
||||
sudo ufw disable
|
||||
|
||||
## Packer
|
||||
echo Fetching Packer...
|
||||
curl -L $PACKERDOWNLOAD > packer.zip
|
||||
echo Installing Packer...
|
||||
unzip packer.zip -d /usr/local/bin
|
||||
chmod 0755 /usr/local/bin/packer
|
||||
chown root:root /usr/local/bin/packer
|
||||
|
||||
## Terraform
|
||||
echo Fetching Terraform...
|
||||
curl -L $TERRAFORMDOWNLOAD > terraform.zip
|
||||
echo Installing Terraform...
|
||||
unzip terraform.zip -d /usr/local/bin
|
||||
chmod 0755 /usr/local/bin/terraform
|
||||
chown root:root /usr/local/bin/terraform
|
||||
|
||||
## Docker
|
||||
echo deb https://apt.dockerproject.org/repo ubuntu-`lsb_release -c | awk '{print $2}'` main | sudo tee /etc/apt/sources.list.d/docker.list
|
||||
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y docker-engine
|
||||
|
||||
SHELL
|
||||
|
||||
config.vm.synced_folder "../aws/", "/home/vagrant/aws", owner: "vagrant", group: "vagrant"
|
||||
config.vm.synced_folder "../shared/", "/home/vagrant/shared", owner: "vagrant", group: "vagrant"
|
||||
config.vm.synced_folder "../examples/", "/home/vagrant/examples", owner: "vagrant", group: "vagrant"
|
||||
|
||||
config.vm.provider "virtualbox" do |vb|
|
||||
vb.memory = "2048"
|
||||
vb.cpus = 2
|
||||
end
|
||||
|
||||
end
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
variable "region" {
|
||||
description = "The AWS region to deploy to."
|
||||
default = "us-east-1"
|
||||
}
|
||||
|
||||
variable "ami" {}
|
||||
|
||||
variable "instance_type" {
|
||||
description = "The AWS instance type to use for both clients and servers."
|
||||
default = "t2.medium"
|
||||
}
|
||||
|
||||
variable "key_name" {}
|
||||
|
||||
variable "server_count" {
|
||||
description = "The number of servers to provision."
|
||||
default = "3"
|
||||
}
|
||||
|
||||
variable "client_count" {
|
||||
description = "The number of clients to provision."
|
||||
default = "4"
|
||||
}
|
||||
|
||||
variable "cluster_tag_value" {
|
||||
description = "Used by Consul to automatically form a cluster."
|
||||
default = "auto-join"
|
||||
}
|
||||
|
||||
provider "aws" {
|
||||
region = "${var.region}"
|
||||
}
|
||||
|
||||
module "hashistack" {
|
||||
source = "../../modules/hashistack"
|
||||
|
||||
region = "${var.region}"
|
||||
ami = "${var.ami}"
|
||||
instance_type = "${var.instance_type}"
|
||||
key_name = "${var.key_name}"
|
||||
server_count = "${var.server_count}"
|
||||
client_count = "${var.client_count}"
|
||||
cluster_tag_value = "${var.cluster_tag_value}"
|
||||
}
|
||||
|
||||
output "primary_server_private_ips" {
|
||||
value = "${module.hashistack.primary_server_private_ips}"
|
||||
}
|
||||
|
||||
output "primary_server_public_ips" {
|
||||
value = "${module.hashistack.primary_server_public_ips}"
|
||||
}
|
||||
|
||||
output "client_private_ips" {
|
||||
value = "${module.hashistack.client_private_ips}"
|
||||
}
|
||||
|
||||
output "client_public_ips" {
|
||||
value = "${module.hashistack.client_public_ips}"
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
region = "us-east-1"
|
||||
ami = "ami-76787e60"
|
||||
instance_type = "t2.medium"
|
||||
key_name = "KEY_NAME"
|
||||
server_count = "1"
|
||||
client_count = "4"
|
||||
cluster_tag_value = "auto-join"
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
exec > >(sudo tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
|
||||
sudo bash /ops/shared/scripts/client.sh "${region}" "${cluster_tag_value}"
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
exec > >(sudo tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
|
||||
sudo bash /ops/shared/scripts/server.sh "${server_count}" "${region}" "${cluster_tag_value}"
|
||||
|
|
@ -0,0 +1,173 @@
|
|||
variable "region" {}
|
||||
variable "ami" {}
|
||||
variable "instance_type" {}
|
||||
variable "key_name" {}
|
||||
variable "server_count" {}
|
||||
variable "client_count" {}
|
||||
variable "cluster_tag_value" {}
|
||||
|
||||
data "aws_vpc" "default" {
|
||||
default = true
|
||||
}
|
||||
|
||||
resource "aws_security_group" "primary" {
|
||||
name = "hashistack"
|
||||
vpc_id = "${data.aws_vpc.default.id}"
|
||||
|
||||
ingress {
|
||||
from_port = 22
|
||||
to_port = 22
|
||||
protocol = "tcp"
|
||||
cidr_blocks = ["0.0.0.0/0"]
|
||||
}
|
||||
|
||||
# HDFS NameNode UI
|
||||
ingress {
|
||||
from_port = 50070
|
||||
to_port = 50070
|
||||
protocol = "tcp"
|
||||
cidr_blocks = ["0.0.0.0/0"]
|
||||
}
|
||||
|
||||
# HDFS DataNode UI
|
||||
ingress {
|
||||
from_port = 50075
|
||||
to_port = 50075
|
||||
protocol = "tcp"
|
||||
cidr_blocks = ["0.0.0.0/0"]
|
||||
}
|
||||
|
||||
# Spark history server UI
|
||||
ingress {
|
||||
from_port = 18080
|
||||
to_port = 18080
|
||||
protocol = "tcp"
|
||||
cidr_blocks = ["0.0.0.0/0"]
|
||||
}
|
||||
|
||||
ingress {
|
||||
from_port = 0
|
||||
to_port = 0
|
||||
protocol = "-1"
|
||||
self = true
|
||||
}
|
||||
|
||||
egress {
|
||||
from_port = 0
|
||||
to_port = 0
|
||||
protocol = "-1"
|
||||
cidr_blocks = ["0.0.0.0/0"]
|
||||
}
|
||||
}
|
||||
|
||||
data "template_file" "user_data_server_primary" {
|
||||
template = "${file("${path.root}/user-data-server.sh")}"
|
||||
|
||||
vars {
|
||||
server_count = "${var.server_count}"
|
||||
region = "${var.region}"
|
||||
cluster_tag_value = "${var.cluster_tag_value}"
|
||||
}
|
||||
}
|
||||
|
||||
data "template_file" "user_data_client" {
|
||||
template = "${file("${path.root}/user-data-client.sh")}"
|
||||
|
||||
vars {
|
||||
region = "${var.region}"
|
||||
cluster_tag_value = "${var.cluster_tag_value}"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_instance" "primary" {
|
||||
ami = "${var.ami}"
|
||||
instance_type = "${var.instance_type}"
|
||||
key_name = "${var.key_name}"
|
||||
vpc_security_group_ids = ["${aws_security_group.primary.id}"]
|
||||
count = "${var.server_count}"
|
||||
|
||||
#Instance tags
|
||||
tags {
|
||||
Name = "hashistack-server-${count.index}"
|
||||
ConsulAutoJoin = "${var.cluster_tag_value}"
|
||||
}
|
||||
|
||||
user_data = "${data.template_file.user_data_server_primary.rendered}"
|
||||
iam_instance_profile = "${aws_iam_instance_profile.instance_profile.name}"
|
||||
}
|
||||
|
||||
resource "aws_instance" "client" {
|
||||
ami = "${var.ami}"
|
||||
instance_type = "${var.instance_type}"
|
||||
key_name = "${var.key_name}"
|
||||
vpc_security_group_ids = ["${aws_security_group.primary.id}"]
|
||||
count = "${var.client_count}"
|
||||
depends_on = ["aws_instance.primary"]
|
||||
|
||||
#Instance tags
|
||||
tags {
|
||||
Name = "hashistack-client-${count.index}"
|
||||
ConsulAutoJoin = "${var.cluster_tag_value}"
|
||||
}
|
||||
|
||||
user_data = "${data.template_file.user_data_client.rendered}"
|
||||
iam_instance_profile = "${aws_iam_instance_profile.instance_profile.name}"
|
||||
}
|
||||
|
||||
resource "aws_iam_instance_profile" "instance_profile" {
|
||||
name_prefix = "hashistack"
|
||||
role = "${aws_iam_role.instance_role.name}"
|
||||
}
|
||||
|
||||
resource "aws_iam_role" "instance_role" {
|
||||
name_prefix = "hashistack"
|
||||
assume_role_policy = "${data.aws_iam_policy_document.instance_role.json}"
|
||||
}
|
||||
|
||||
data "aws_iam_policy_document" "instance_role" {
|
||||
statement {
|
||||
effect = "Allow"
|
||||
actions = ["sts:AssumeRole"]
|
||||
|
||||
principals {
|
||||
type = "Service"
|
||||
identifiers = ["ec2.amazonaws.com"]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_iam_role_policy" "auto_discover_cluster" {
|
||||
name = "auto-discover-cluster"
|
||||
role = "${aws_iam_role.instance_role.id}"
|
||||
policy = "${data.aws_iam_policy_document.auto_discover_cluster.json}"
|
||||
}
|
||||
|
||||
data "aws_iam_policy_document" "auto_discover_cluster" {
|
||||
statement {
|
||||
effect = "Allow"
|
||||
|
||||
actions = [
|
||||
"ec2:DescribeInstances",
|
||||
"ec2:DescribeTags",
|
||||
"autoscaling:DescribeAutoScalingGroups",
|
||||
]
|
||||
|
||||
resources = ["*"]
|
||||
}
|
||||
}
|
||||
|
||||
output "primary_server_private_ips" {
|
||||
value = ["${aws_instance.primary.*.private_ip}"]
|
||||
}
|
||||
|
||||
output "primary_server_public_ips" {
|
||||
value = ["${aws_instance.primary.*.public_ip}"]
|
||||
}
|
||||
|
||||
output "client_private_ips" {
|
||||
value = ["${aws_instance.client.*.private_ip}"]
|
||||
}
|
||||
|
||||
output "client_public_ips" {
|
||||
value = ["${aws_instance.client.*.public_ip}"]
|
||||
}
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
# Build an Amazon machine image with Packer
|
||||
|
||||
[Packer](https://www.packer.io/intro/index.html) is HashiCorp's open source tool
|
||||
for creating identical machine images for multiple platforms from a single
|
||||
source configuration. The Terraform templates included in this repo reference a
|
||||
publicly avaialble Amazon machine image (AMI) by default. The Packer build
|
||||
configuration used to create the public AMI is included [here](./packer.json).
|
||||
If you wish to customize it and build your own private AMI, follow the
|
||||
instructions below.
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
See the pre-requisites listed [here](../../README.md). If you did not use the
|
||||
included `Vagrantfile` to bootstrap a staging environment, you will need to
|
||||
[install Packer](https://www.packer.io/intro/getting-started/install.html).
|
||||
|
||||
Set environment variables for your AWS credentials if you haven't already:
|
||||
|
||||
```bash
|
||||
$ export AWS_ACCESS_KEY_ID=[ACCESS_KEY_ID]
|
||||
$ export AWS_SECRET_ACCESS_KEY=[SECRET_ACCESS_KEY]
|
||||
```
|
||||
|
||||
After you make your modifications to `packer.json`, execute the following
|
||||
command to build the AMI:
|
||||
|
||||
```bash
|
||||
$ packer build packer.json
|
||||
```
|
||||
|
||||
Don't forget to copy the AMI ID to your [terraform.tfvars file](../env/us-east/terraform.tfvars).
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
{
|
||||
"builders": [{
|
||||
"type": "amazon-ebs",
|
||||
"region": "us-east-1",
|
||||
"source_ami": "ami-80861296",
|
||||
"instance_type": "t2.medium",
|
||||
"ssh_username": "ubuntu",
|
||||
"ami_name": "nomad-packer {{timestamp}}",
|
||||
"ami_groups": ["all"]
|
||||
}],
|
||||
"provisioners": [
|
||||
{
|
||||
"type": "shell",
|
||||
"inline": [
|
||||
"sudo mkdir /ops",
|
||||
"sudo chmod 777 /ops"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "file",
|
||||
"source": "../../shared",
|
||||
"destination": "/ops"
|
||||
},
|
||||
{
|
||||
"type": "file",
|
||||
"source": "../../examples",
|
||||
"destination": "/ops"
|
||||
},
|
||||
{
|
||||
"type": "shell",
|
||||
"script": "../../shared/scripts/setup.sh"
|
||||
}]
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
# Examples
|
||||
|
||||
The examples included here are designed to introduce specific features and
|
||||
provide a basic learning experience. The examples subdirectory is automatically
|
||||
provisioned into the home directory of the VMs in your cloud environment.
|
||||
|
||||
- [Spark Integration](spark/README.md)
|
||||
|
|
@ -0,0 +1,193 @@
|
|||
# Nomad / Spark integration
|
||||
|
||||
The Nomad ecosystem includes a fork of Apache Spark that natively supports using
|
||||
a Nomad cluster to run Spark applications. When running on Nomad, the Spark
|
||||
executors that run Spark tasks for your application, and optionally the
|
||||
application driver itself, run as Nomad tasks in a Nomad job. See the
|
||||
[usage guide](./RunningSparkOnNomad.pdf) for more details.
|
||||
|
||||
Clusters provisioned with Nomad's Terraform templates are automatically
|
||||
configured to run the Spark integration. The sample job files found here are
|
||||
also provisioned onto every client and server.
|
||||
|
||||
## Setup
|
||||
|
||||
To give the Spark integration a test drive, provision a cluster and SSH to any
|
||||
one of the clients or servers (the public IPs are displayed when the Terraform
|
||||
provisioning process completes):
|
||||
|
||||
```bash
|
||||
$ ssh -i /path/to/key ubuntu@PUBLIC_IP
|
||||
```
|
||||
|
||||
The Spark history server and several of the sample Spark jobs below require
|
||||
HDFS. Using the included job file, deploy an HDFS cluster on Nomad:
|
||||
|
||||
```bash
|
||||
$ cd $HOME/examples/spark
|
||||
$ nomad run hdfs.nomad
|
||||
$ nomad status hdfs
|
||||
```
|
||||
|
||||
When the allocations are all in the `running` state (as shown by `nomad status
|
||||
hdfs`), query Consul to verify that the HDFS service has been registered:
|
||||
|
||||
```bash
|
||||
$ dig hdfs.service.consul
|
||||
```
|
||||
|
||||
Next, create directories and files in HDFS for use by the history server and the
|
||||
sample Spark jobs:
|
||||
|
||||
```bash
|
||||
$ hdfs dfs -mkdir /foo
|
||||
$ hdfs dfs -put /var/log/apt/history.log /foo
|
||||
$ hdfs dfs -mkdir /spark-events
|
||||
$ hdfs dfs -ls /
|
||||
```
|
||||
|
||||
Finally, deploy the Spark history server:
|
||||
|
||||
```bash
|
||||
$ nomad run spark-history-server-hdfs.nomad
|
||||
```
|
||||
|
||||
You can get the private IP for the history server with a Consul DNS lookup:
|
||||
|
||||
```bash
|
||||
$ dig spark-history.service.consul
|
||||
```
|
||||
|
||||
Cross-reference the private IP with the `terraforom apply` output to get the
|
||||
corresponding public IP. You can access the history server at
|
||||
`http://PUBLIC_IP:18080`.
|
||||
|
||||
## Sample Spark jobs
|
||||
|
||||
The sample `spark-submit` commands listed below demonstrate several of the
|
||||
official Spark examples. Features like `spark-sql`, `spark-shell` and `pyspark`
|
||||
are included. The commands can be executed from any client or server.
|
||||
|
||||
You can monitor the status of a Spark job in a second terminal session with:
|
||||
|
||||
```bash
|
||||
$ nomad status
|
||||
$ nomad status JOB_ID
|
||||
$ nomad alloc-status DRIVER_ALLOC_ID
|
||||
$ nomad logs DRIVER_ALLOC_ID
|
||||
```
|
||||
|
||||
To view the output of the job, run `nomad logs` for the driver's Allocation ID.
|
||||
|
||||
### SparkPi (Java)
|
||||
|
||||
```bash
|
||||
spark-submit \
|
||||
--class org.apache.spark.examples.JavaSparkPi \
|
||||
--master nomad \
|
||||
--deploy-mode cluster \
|
||||
--conf spark.executor.instances=4 \
|
||||
--conf spark.nomad.cluster.monitorUntil=complete \
|
||||
--conf spark.eventLog.enabled=true \
|
||||
--conf spark.eventLog.dir=hdfs://hdfs.service.consul/spark-events \
|
||||
--conf spark.nomad.sparkDistribution=https://s3.amazonaws.com/nomad-spark/spark-2.1.0-bin-nomad.tgz \
|
||||
https://s3.amazonaws.com/nomad-spark/spark-examples_2.11-2.1.0-SNAPSHOT.jar 100
|
||||
```
|
||||
|
||||
### Word count (Java)
|
||||
|
||||
```bash
|
||||
spark-submit \
|
||||
--class org.apache.spark.examples.JavaWordCount \
|
||||
--master nomad \
|
||||
--deploy-mode cluster \
|
||||
--conf spark.executor.instances=4 \
|
||||
--conf spark.nomad.cluster.monitorUntil=complete \
|
||||
--conf spark.eventLog.enabled=true \
|
||||
--conf spark.eventLog.dir=hdfs://hdfs.service.consul/spark-events \
|
||||
--conf spark.nomad.sparkDistribution=https://s3.amazonaws.com/nomad-spark/spark-2.1.0-bin-nomad.tgz \
|
||||
https://s3.amazonaws.com/nomad-spark/spark-examples_2.11-2.1.0-SNAPSHOT.jar \
|
||||
hdfs://hdfs.service.consul/foo/history.log
|
||||
```
|
||||
|
||||
### DFSReadWriteTest (Scala)
|
||||
|
||||
```bash
|
||||
spark-submit \
|
||||
--class org.apache.spark.examples.DFSReadWriteTest \
|
||||
--master nomad \
|
||||
--deploy-mode cluster \
|
||||
--conf spark.executor.instances=4 \
|
||||
--conf spark.nomad.cluster.monitorUntil=complete \
|
||||
--conf spark.eventLog.enabled=true \
|
||||
--conf spark.eventLog.dir=hdfs://hdfs.service.consul/spark-events \
|
||||
--conf spark.nomad.sparkDistribution=https://s3.amazonaws.com/nomad-spark/spark-2.1.0-bin-nomad.tgz \
|
||||
https://s3.amazonaws.com/nomad-spark/spark-examples_2.11-2.1.0-SNAPSHOT.jar \
|
||||
/home/ubuntu/.bashrc hdfs://hdfs.service.consul/foo
|
||||
```
|
||||
|
||||
### spark-shell
|
||||
|
||||
Start the shell:
|
||||
|
||||
```bash
|
||||
spark-shell \
|
||||
--master nomad \
|
||||
--conf spark.executor.instances=4 \
|
||||
--conf spark.nomad.sparkDistribution=https://s3.amazonaws.com/nomad-spark/spark-2.1.0-bin-nomad.tgz
|
||||
```
|
||||
|
||||
Run a few commands:
|
||||
|
||||
```bash
|
||||
$ spark.version
|
||||
|
||||
$ val data = 1 to 10000
|
||||
$ val distData = sc.parallelize(data)
|
||||
$ distData.filter(_ < 10).collect()
|
||||
```
|
||||
|
||||
### sql-shell
|
||||
|
||||
Start the shell:
|
||||
|
||||
```bash
|
||||
spark-sql \
|
||||
--master nomad \
|
||||
--conf spark.executor.instances=4 \
|
||||
--conf spark.nomad.sparkDistribution=https://s3.amazonaws.com/nomad-spark/spark-2.1.0-bin-nomad.tgz jars/spark-sql_2.11-2.1.0-SNAPSHOT.jar
|
||||
```
|
||||
|
||||
Run a few commands:
|
||||
|
||||
```bash
|
||||
$ CREATE TEMPORARY VIEW usersTable
|
||||
USING org.apache.spark.sql.parquet
|
||||
OPTIONS (
|
||||
path "/usr/local/bin/spark/examples/src/main/resources/users.parquet"
|
||||
);
|
||||
|
||||
$ SELECT * FROM usersTable;
|
||||
```
|
||||
|
||||
### pyspark
|
||||
|
||||
Start the shell:
|
||||
|
||||
```bash
|
||||
pyspark \
|
||||
--master nomad \
|
||||
--conf spark.executor.instances=4 \
|
||||
--conf spark.nomad.sparkDistribution=https://s3.amazonaws.com/nomad-spark/spark-2.1.0-bin-nomad.tgz
|
||||
```
|
||||
|
||||
Run a few commands:
|
||||
|
||||
```bash
|
||||
$ df = spark.read.json("/usr/local/bin/spark/examples/src/main/resources/people.json")
|
||||
$ df.show()
|
||||
$ df.printSchema()
|
||||
$ df.createOrReplaceTempView("people")
|
||||
$ sqlDF = spark.sql("SELECT * FROM people")
|
||||
$ sqlDF.show()
|
||||
```
|
||||
Binary file not shown.
|
|
@ -0,0 +1,9 @@
|
|||
FROM openjdk:7
|
||||
|
||||
ENV HADOOP_VERSION 2.7.3
|
||||
|
||||
RUN wget -O - http://apache.mirror.iphh.net/hadoop/common/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz | tar xz -C /usr/local/
|
||||
ENV HADOOP_PREFIX /usr/local/hadoop-$HADOOP_VERSION
|
||||
ENV PATH $PATH:$HADOOP_PREFIX/bin
|
||||
|
||||
COPY core-site.xml $HADOOP_PREFIX/etc/hadoop/
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||
<configuration>
|
||||
<property>
|
||||
<name>fs.defaultFS</name>
|
||||
<value>hdfs://hdfs.service.consul/</value>
|
||||
</property>
|
||||
</configuration>
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
FROM openjdk:7-jre
|
||||
|
||||
RUN curl https://spark-nomad.s3.amazonaws.com/spark-2.1.1-bin-nomad.tgz | tar -xzC /tmp
|
||||
RUN mv /tmp/spark* /opt/spark
|
||||
|
||||
ENV SPARK_HOME /opt/spark
|
||||
ENV PATH $PATH:$SPARK_HOME/bin
|
||||
|
|
@ -0,0 +1,91 @@
|
|||
job "hdfs" {
|
||||
|
||||
datacenters = [ "dc1" ]
|
||||
|
||||
group "NameNode" {
|
||||
|
||||
constraint {
|
||||
operator = "distinct_hosts"
|
||||
value = "true"
|
||||
}
|
||||
|
||||
task "NameNode" {
|
||||
|
||||
driver = "docker"
|
||||
|
||||
config {
|
||||
image = "rcgenova/hadoop-2.7.3"
|
||||
command = "bash"
|
||||
args = [ "-c", "hdfs namenode -format && exec hdfs namenode -D fs.defaultFS=hdfs://${NOMAD_ADDR_ipc}/ -D dfs.permissions.enabled=false" ]
|
||||
network_mode = "host"
|
||||
port_map {
|
||||
ipc = 8020
|
||||
ui = 50070
|
||||
}
|
||||
}
|
||||
|
||||
resources {
|
||||
memory = 500
|
||||
network {
|
||||
port "ipc" {
|
||||
static = "8020"
|
||||
}
|
||||
port "ui" {
|
||||
static = "50070"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
service {
|
||||
name = "hdfs"
|
||||
port = "ipc"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
group "DataNode" {
|
||||
|
||||
count = 3
|
||||
|
||||
constraint {
|
||||
operator = "distinct_hosts"
|
||||
value = "true"
|
||||
}
|
||||
|
||||
task "DataNode" {
|
||||
|
||||
driver = "docker"
|
||||
|
||||
config {
|
||||
network_mode = "host"
|
||||
image = "rcgenova/hadoop-2.7.3"
|
||||
args = [ "hdfs", "datanode"
|
||||
, "-D", "fs.defaultFS=hdfs://hdfs.service.consul/"
|
||||
, "-D", "dfs.permissions.enabled=false"
|
||||
]
|
||||
port_map {
|
||||
data = 50010
|
||||
ipc = 50020
|
||||
ui = 50075
|
||||
}
|
||||
}
|
||||
|
||||
resources {
|
||||
memory = 500
|
||||
network {
|
||||
port "data" {
|
||||
static = "50010"
|
||||
}
|
||||
port "ipc" {
|
||||
static = "50020"
|
||||
}
|
||||
port "ui" {
|
||||
static = "50075"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
job "spark-history-server" {
|
||||
datacenters = ["dc1"]
|
||||
type = "service"
|
||||
|
||||
group "server" {
|
||||
count = 1
|
||||
|
||||
task "history-server" {
|
||||
driver = "docker"
|
||||
|
||||
config {
|
||||
image = "barnardb/spark"
|
||||
command = "/spark/spark-2.1.0-bin-nomad/bin/spark-class"
|
||||
args = [ "org.apache.spark.deploy.history.HistoryServer" ]
|
||||
port_map {
|
||||
ui = 18080
|
||||
}
|
||||
network_mode = "host"
|
||||
}
|
||||
|
||||
env {
|
||||
"SPARK_HISTORY_OPTS" = "-Dspark.history.fs.logDirectory=hdfs://hdfs.service.consul/spark-events/"
|
||||
"SPARK_PUBLIC_DNS" = "spark-history.service.consul"
|
||||
}
|
||||
|
||||
resources {
|
||||
cpu = 500
|
||||
memory = 500
|
||||
network {
|
||||
mbits = 250
|
||||
port "ui" {
|
||||
static = 18080
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
service {
|
||||
name = "spark-history"
|
||||
tags = ["spark", "ui"]
|
||||
port = "ui"
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
{
|
||||
"log_level": "INFO",
|
||||
"server": true,
|
||||
"data_dir": "/opt/consul/data",
|
||||
"bind_addr": "0.0.0.0",
|
||||
"client_addr": "0.0.0.0",
|
||||
"advertise_addr": "IP_ADDRESS",
|
||||
"bootstrap_expect": SERVER_COUNT,
|
||||
"service": {
|
||||
"name": "consul"
|
||||
},
|
||||
"retry_join_ec2": {
|
||||
"tag_key": "ConsulAutoJoin",
|
||||
"tag_value": "CLUSTER_TAG_VALUE",
|
||||
"region": "REGION"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
{
|
||||
"log_level": "INFO",
|
||||
"data_dir": "/opt/consul/data",
|
||||
"bind_addr": "0.0.0.0",
|
||||
"client_addr": "0.0.0.0",
|
||||
"advertise_addr": "IP_ADDRESS",
|
||||
"retry_join_ec2": {
|
||||
"tag_key": "ConsulAutoJoin",
|
||||
"tag_value": "CLUSTER_TAG_VALUE",
|
||||
"region": "REGION"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
description "Consul"
|
||||
|
||||
start on runlevel [2345]
|
||||
stop on runlevel [!2345]
|
||||
|
||||
respawn
|
||||
|
||||
console log
|
||||
|
||||
script
|
||||
if [ -f "/etc/service/consul" ]; then
|
||||
. /etc/service/consul
|
||||
fi
|
||||
|
||||
# Allow Consul to use privileged ports
|
||||
export CONSUL_ALLOW_PRIVILEGED_PORTS=true
|
||||
|
||||
exec /usr/local/bin/consul agent \
|
||||
-config-dir="/etc/consul.d" \
|
||||
-dns-port="53" \
|
||||
-recursor="172.31.0.2" \
|
||||
\$${CONSUL_FLAGS} \
|
||||
>>/var/log/consul.log 2>&1
|
||||
end script
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||
<configuration>
|
||||
<property>
|
||||
<name>fs.defaultFS</name>
|
||||
<value>hdfs://hdfs.service.consul/</value>
|
||||
</property>
|
||||
</configuration>
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
data_dir = "/opt/nomad/data"
|
||||
bind_addr = "IP_ADDRESS"
|
||||
|
||||
# Enable the server
|
||||
server {
|
||||
enabled = true
|
||||
bootstrap_expect = SERVER_COUNT
|
||||
}
|
||||
|
||||
name = "nomad@IP_ADDRESS"
|
||||
|
||||
consul {
|
||||
address = "IP_ADDRESS:8500"
|
||||
}
|
||||
|
||||
vault {
|
||||
enabled = false
|
||||
address = "vault.service.consul"
|
||||
task_token_ttl = "1h"
|
||||
create_from_role = "nomad-cluster"
|
||||
token = ""
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
data_dir = "/opt/nomad/data"
|
||||
bind_addr = "IP_ADDRESS"
|
||||
name = "nomad@IP_ADDRESS"
|
||||
|
||||
# Enable the client
|
||||
client {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
consul {
|
||||
address = "127.0.0.1:8500"
|
||||
}
|
||||
|
||||
vault {
|
||||
enabled = true
|
||||
address = "vault.service.consul"
|
||||
}
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
description "Nomad"
|
||||
|
||||
start on runlevel [2345]
|
||||
stop on runlevel [!2345]
|
||||
|
||||
respawn
|
||||
|
||||
console log
|
||||
|
||||
script
|
||||
if [ -f "/etc/service/nomad" ]; then
|
||||
. /etc/service/nomad
|
||||
fi
|
||||
|
||||
exec /usr/local/bin/nomad agent \
|
||||
-config="/etc/nomad.d/nomad.hcl" \
|
||||
\$${NOMAD_FLAGS} \
|
||||
>>/var/log/nomad.log 2>&1
|
||||
end script
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
backend "consul" {
|
||||
path = "vault/"
|
||||
address = "IP_ADDRESS:8500"
|
||||
cluster_addr = "https://IP_ADDRESS:8201"
|
||||
redirect_addr = "http://IP_ADDRESS:8200"
|
||||
}
|
||||
|
||||
listener "tcp" {
|
||||
address = "IP_ADDRESS:8200"
|
||||
cluster_address = "IP_ADDRESS:8201"
|
||||
tls_disable = 1
|
||||
}
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
description "Vault"
|
||||
|
||||
start on runlevel [2345]
|
||||
stop on runlevel [!2345]
|
||||
|
||||
respawn
|
||||
|
||||
console log
|
||||
|
||||
script
|
||||
if [ -f "/etc/service/vault" ]; then
|
||||
. /etc/service/vault
|
||||
fi
|
||||
|
||||
# Make sure to use all our CPUs, because Vault can block a scheduler thread
|
||||
export GOMAXPROCS=`nproc`
|
||||
|
||||
exec /usr/local/bin/vault server \
|
||||
-config="/etc/vault.d/vault.hcl" \
|
||||
\$${VAULT_FLAGS} \
|
||||
>>/var/log/vault.log 2>&1
|
||||
end script
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
CONFIGDIR=/ops/shared/config
|
||||
|
||||
CONSULCONFIGDIR=/etc/consul.d
|
||||
NOMADCONFIGDIR=/etc/nomad.d
|
||||
HADOOP_VERSION=hadoop-2.7.3
|
||||
HADOOPCONFIGDIR=/usr/local/$HADOOP_VERSION/etc/hadoop
|
||||
HOME_DIR=ubuntu
|
||||
|
||||
# Wait for network
|
||||
sleep 15
|
||||
|
||||
IP_ADDRESS=$(curl http://instance-data/latest/meta-data/local-ipv4)
|
||||
DOCKER_BRIDGE_IP_ADDRESS=(`ifconfig docker0 2>/dev/null|awk '/inet addr:/ {print $2}'|sed 's/addr://'`)
|
||||
REGION=$1
|
||||
CLUSTER_TAG_VALUE=$2
|
||||
|
||||
# Consul
|
||||
sed -i "s/IP_ADDRESS/$IP_ADDRESS/g" $CONFIGDIR/consul_client.json
|
||||
sed -i "s/REGION/$REGION/g" $CONFIGDIR/consul_client.json
|
||||
sed -i "s/CLUSTER_TAG_VALUE/$CLUSTER_TAG_VALUE/g" $CONFIGDIR/consul_client.json
|
||||
sudo cp $CONFIGDIR/consul_client.json $CONSULCONFIGDIR/consul.json
|
||||
sudo cp $CONFIGDIR/consul_upstart.conf /etc/init/consul.conf
|
||||
|
||||
sudo service consul start
|
||||
sleep 10
|
||||
|
||||
# Nomad
|
||||
sed -i "s/IP_ADDRESS/$IP_ADDRESS/g" $CONFIGDIR/nomad_client.hcl
|
||||
sudo cp $CONFIGDIR/nomad_client.hcl $NOMADCONFIGDIR/nomad.hcl
|
||||
sudo cp $CONFIGDIR/nomad_upstart.conf /etc/init/nomad.conf
|
||||
|
||||
sudo service nomad start
|
||||
sleep 10
|
||||
export NOMAD_ADDR=http://$IP_ADDRESS:4646
|
||||
|
||||
# Add hostname to /etc/hosts
|
||||
echo "127.0.0.1 $(hostname)" | sudo tee --append /etc/hosts
|
||||
|
||||
# Add Docker bridge network IP to /etc/resolv.conf (at the top)
|
||||
echo "nameserver $DOCKER_BRIDGE_IP_ADDRESS" | sudo tee /etc/resolv.conf.new
|
||||
cat /etc/resolv.conf | sudo tee --append /etc/resolv.conf.new
|
||||
sudo mv /etc/resolv.conf.new /etc/resolv.conf
|
||||
|
||||
# Hadoop config file to enable HDFS CLI
|
||||
sudo cp $CONFIGDIR/core-site.xml $HADOOPCONFIGDIR
|
||||
|
||||
# Move examples directory to $HOME
|
||||
sudo mv /ops/examples /home/$HOME_DIR
|
||||
sudo chown -R $HOME_DIR:$HOME_DIR /home/$HOME_DIR/examples
|
||||
sudo chmod -R 775 /home/$HOME_DIR/examples
|
||||
|
||||
# Set env vars for tool CLIs
|
||||
echo "export VAULT_ADDR=http://$IP_ADDRESS:8200" | sudo tee --append /home/$HOME_DIR/.bashrc
|
||||
echo "export NOMAD_ADDR=http://$IP_ADDRESS:4646" | sudo tee --append /home/$HOME_DIR/.bashrc
|
||||
echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre" | sudo tee --append /home/$HOME_DIR/.bashrc
|
||||
|
||||
# Update PATH
|
||||
echo "export PATH=$PATH:/usr/local/bin/spark/bin:/usr/local/$HADOOP_VERSION/bin" | sudo tee --append /home/$HOME_DIR/.bashrc
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
CONFIGDIR=/ops/shared/config
|
||||
|
||||
CONSULCONFIGDIR=/etc/consul.d
|
||||
VAULTCONFIGDIR=/etc/vault.d
|
||||
NOMADCONFIGDIR=/etc/nomad.d
|
||||
HADOOP_VERSION=hadoop-2.7.3
|
||||
HADOOPCONFIGDIR=/usr/local/$HADOOP_VERSION/etc/hadoop
|
||||
HOME_DIR=ubuntu
|
||||
|
||||
sleep 15
|
||||
|
||||
IP_ADDRESS=$(curl http://instance-data/latest/meta-data/local-ipv4)
|
||||
DOCKER_BRIDGE_IP_ADDRESS=(`ifconfig docker0 2>/dev/null|awk '/inet addr:/ {print $2}'|sed 's/addr://'`)
|
||||
SERVER_COUNT=$1
|
||||
REGION=$2
|
||||
CLUSTER_TAG_VALUE=$3
|
||||
|
||||
# Consul
|
||||
sed -i "s/IP_ADDRESS/$IP_ADDRESS/g" $CONFIGDIR/consul.json
|
||||
sed -i "s/SERVER_COUNT/$SERVER_COUNT/g" $CONFIGDIR/consul.json
|
||||
sed -i "s/REGION/$REGION/g" $CONFIGDIR/consul.json
|
||||
sed -i "s/CLUSTER_TAG_VALUE/$CLUSTER_TAG_VALUE/g" $CONFIGDIR/consul.json
|
||||
sudo cp $CONFIGDIR/consul.json $CONSULCONFIGDIR
|
||||
sudo cp $CONFIGDIR/consul_upstart.conf /etc/init/consul.conf
|
||||
|
||||
sudo service consul start
|
||||
sleep 20
|
||||
export CONSUL_HTTP_ADDR=$IP_ADDRESS:8500
|
||||
export CONSUL_RPC_ADDR=$IP_ADDRESS:8400
|
||||
|
||||
# Vault
|
||||
sed -i "s/IP_ADDRESS/$IP_ADDRESS/g" $CONFIGDIR/vault.hcl
|
||||
sudo cp $CONFIGDIR/vault.hcl $VAULTCONFIGDIR
|
||||
sudo cp $CONFIGDIR/vault_upstart.conf /etc/init/vault.conf
|
||||
|
||||
sudo service vault start
|
||||
|
||||
# Nomad
|
||||
sed -i "s/IP_ADDRESS/$IP_ADDRESS/g" $CONFIGDIR/nomad.hcl
|
||||
sed -i "s/SERVER_COUNT/$SERVER_COUNT/g" $CONFIGDIR/nomad.hcl
|
||||
sudo cp $CONFIGDIR/nomad.hcl $NOMADCONFIGDIR
|
||||
sudo cp $CONFIGDIR/nomad_upstart.conf /etc/init/nomad.conf
|
||||
|
||||
sudo service nomad start
|
||||
sleep 10
|
||||
export NOMAD_ADDR=http://$IP_ADDRESS:4646
|
||||
|
||||
# Add hostname to /etc/hosts
|
||||
|
||||
echo "127.0.0.1 $(hostname)" | sudo tee --append /etc/hosts
|
||||
|
||||
# Add Docker bridge network IP to /etc/resolv.conf (at the top)
|
||||
|
||||
echo "nameserver $DOCKER_BRIDGE_IP_ADDRESS" | sudo tee /etc/resolv.conf.new
|
||||
cat /etc/resolv.conf | sudo tee --append /etc/resolv.conf.new
|
||||
sudo mv /etc/resolv.conf.new /etc/resolv.conf
|
||||
|
||||
# Hadoop
|
||||
sudo cp $CONFIGDIR/core-site.xml $HADOOPCONFIGDIR
|
||||
|
||||
# Move examples directory to $HOME
|
||||
sudo mv /ops/examples /home/$HOME_DIR
|
||||
sudo chown -R $HOME_DIR:$HOME_DIR /home/$HOME_DIR/examples
|
||||
sudo chmod -R 775 /home/$HOME_DIR/examples
|
||||
|
||||
# Set env vars for tool CLIs
|
||||
echo "export CONSUL_RPC_ADDR=$IP_ADDRESS:8400" | sudo tee --append /home/$HOME_DIR/.bashrc
|
||||
echo "export CONSUL_HTTP_ADDR=$IP_ADDRESS:8500" | sudo tee --append /home/$HOME_DIR/.bashrc
|
||||
echo "export VAULT_ADDR=http://$IP_ADDRESS:8200" | sudo tee --append /home/$HOME_DIR/.bashrc
|
||||
echo "export NOMAD_ADDR=http://$IP_ADDRESS:4646" | sudo tee --append /home/$HOME_DIR/.bashrc
|
||||
echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre" | sudo tee --append /home/$HOME_DIR/.bashrc
|
||||
|
||||
# Update PATH
|
||||
echo "export PATH=$PATH:/usr/local/bin/spark/bin:/usr/local/$HADOOP_VERSION/bin" | sudo tee --append /home/$HOME_DIR/.bashrc
|
||||
|
|
@ -0,0 +1,106 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
cd /ops
|
||||
|
||||
CONFIGDIR=/ops/shared/config
|
||||
|
||||
CONSULVERSION=0.8.4
|
||||
CONSULDOWNLOAD=https://releases.hashicorp.com/consul/${CONSULVERSION}/consul_${CONSULVERSION}_linux_amd64.zip
|
||||
CONSULCONFIGDIR=/etc/consul.d
|
||||
CONSULDIR=/opt/consul
|
||||
|
||||
VAULTVERSION=0.7.3
|
||||
VAULTDOWNLOAD=https://releases.hashicorp.com/vault/${VAULTVERSION}/vault_${VAULTVERSION}_linux_amd64.zip
|
||||
VAULTCONFIGDIR=/etc/vault.d
|
||||
VAULTDIR=/opt/vault
|
||||
|
||||
NOMADVERSION=0.5.6
|
||||
NOMADDOWNLOAD=https://releases.hashicorp.com/nomad/${NOMADVERSION}/nomad_${NOMADVERSION}_linux_amd64.zip
|
||||
NOMADCONFIGDIR=/etc/nomad.d
|
||||
NOMADDIR=/opt/nomad
|
||||
|
||||
HADOOP_VERSION=2.7.3
|
||||
|
||||
# Dependencies
|
||||
sudo apt-get install -y software-properties-common
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y unzip tree redis-tools jq
|
||||
sudo apt-get install -y upstart-sysv
|
||||
sudo update-initramfs -u
|
||||
|
||||
# Numpy (for Spark)
|
||||
sudo apt-get install -y python-setuptools
|
||||
sudo easy_install pip
|
||||
sudo pip install numpy
|
||||
|
||||
# Disable the firewall
|
||||
|
||||
sudo ufw disable
|
||||
|
||||
# Consul
|
||||
|
||||
curl -L $CONSULDOWNLOAD > consul.zip
|
||||
|
||||
## Install
|
||||
sudo unzip consul.zip -d /usr/local/bin
|
||||
sudo chmod 0755 /usr/local/bin/consul
|
||||
sudo chown root:root /usr/local/bin/consul
|
||||
|
||||
## Configure
|
||||
sudo mkdir -p $CONSULCONFIGDIR
|
||||
sudo chmod 755 $CONSULCONFIGDIR
|
||||
sudo mkdir -p $CONSULDIR
|
||||
sudo chmod 755 $CONSULDIR
|
||||
|
||||
# Vault
|
||||
|
||||
curl -L $VAULTDOWNLOAD > vault.zip
|
||||
|
||||
## Install
|
||||
sudo unzip vault.zip -d /usr/local/bin
|
||||
sudo chmod 0755 /usr/local/bin/vault
|
||||
sudo chown root:root /usr/local/bin/vault
|
||||
|
||||
## Configure
|
||||
sudo mkdir -p $VAULTCONFIGDIR
|
||||
sudo chmod 755 $VAULTCONFIGDIR
|
||||
sudo mkdir -p $VAULTDIR
|
||||
sudo chmod 755 $VAULTDIR
|
||||
|
||||
# Nomad
|
||||
|
||||
curl -L $NOMADDOWNLOAD > nomad.zip
|
||||
|
||||
## Install
|
||||
sudo unzip nomad.zip -d /usr/local/bin
|
||||
sudo chmod 0755 /usr/local/bin/nomad
|
||||
sudo chown root:root /usr/local/bin/nomad
|
||||
|
||||
## Configure
|
||||
sudo mkdir -p $NOMADCONFIGDIR
|
||||
sudo chmod 755 $NOMADCONFIGDIR
|
||||
sudo mkdir -p $NOMADDIR
|
||||
sudo chmod 755 $NOMADDIR
|
||||
|
||||
# Docker
|
||||
echo deb https://apt.dockerproject.org/repo ubuntu-`lsb_release -c | awk '{print $2}'` main | sudo tee /etc/apt/sources.list.d/docker.list
|
||||
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y docker-engine
|
||||
|
||||
# Java
|
||||
sudo add-apt-repository -y ppa:openjdk-r/ppa
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y openjdk-8-jdk
|
||||
JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")
|
||||
|
||||
# Spark
|
||||
sudo wget -P /ops/examples/spark https://s3.amazonaws.com/nomad-spark/spark-2.1.0-bin-nomad.tgz
|
||||
sudo tar -xvf /ops/examples/spark/spark-2.1.0-bin-nomad.tgz --directory /ops/examples/spark
|
||||
sudo mv /ops/examples/spark/spark-2.1.0-bin-nomad /usr/local/bin/spark
|
||||
sudo chown -R root:root /usr/local/bin/spark
|
||||
|
||||
# Hadoop (to enable the HDFS CLI)
|
||||
wget -O - http://apache.mirror.iphh.net/hadoop/common/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz | sudo tar xz -C /usr/local/
|
||||
Loading…
Reference in New Issue