Skip to content

Latest commit

 

History

History
207 lines (161 loc) · 9.99 KB

install_run_coreos.md

File metadata and controls

207 lines (161 loc) · 9.99 KB
Raw

Installing and Running PX-Dev on CoreOS

This guide takes you through an install from prerequisites through the PX-Dev setup steps. For the sake of illustration, our example uses AWS EC2 for servers in the cluster, AWS Elastic Block Storage for storage devices, and Compose.IO for a hosted etcd service. As long as your configuration meets the Deployment Requirements, you can use physical servers, another favorite public cloud, or virtual machines.

Once this installation is complete, you can continue with walk-throughs for:

Prerequisites

PX-Dev requires a server with storage devices, Docker 1.10, and use of a key-value store for the cluster configuration. This guide uses CoreOS (1010.6.0 minimum) as the OS.

Step 1: Launch servers

First, we create three servers in AWS, using:

  • Image: CoreOS (HVM)
  • Instance Type: c3.xlarge
  • Number of instances: 3
  • Storage:
    • /dev/xvda: 8 GB boot device
    • /dev/xvdb: 64 GB for container storage
    • /dev/xvdc: 64 GB for container storage
  • (optional) Tag: add value 'px-cluster1' as the name

Volumes used for container data can be magnetic or SSD. PX-Dev will apply different policies based on storage devices capabilities.

Step 2: Configure Docker

  • SSH into your first server. Default user for CoreOS is "core". Docker should be installed by default
  • Verify Docker is running at 1.10 or later : docker -v
  • Configure Docker to use shared mounts.
  • Copy docker.service file for editing
    • sudo cp /usr/lib64/systemd/system/docker.service /etc/systemd/system
    • Edit the docker service file for systemd.
    • sudo vi /etc/systemd/system/docker.service
    • Remove the MountFlags line.
  • Reload the daemon : sudo systemctl daemon-reload
  • Restart Docker : sudo systemctl restart docker

The shared mounts configuration is required, as PX-Dev exports mount points.

Step 3: Provision etcd

You can use an existing etcd service or stand-up your own. In this example, we chose Compose.IO for its ease of use.

  • Create a new etcd deployment in Compose.IO.
  • Select 256MB RAM as the memory.
  • Save the connection string, including your username and password.
  • Example: https://[username]:[password]@[string].dblayer.com:[port]
  • If you are using Compose.IO, the connection string might end with [port]/v2/keys. Please omit the /v2/keys for now.

You only need to do this etcd step once. You can use the same etcd service for multiple PX-Dev clusters.

Specify Storage

Step 1: View disks on servers

PX-Dev pools the storage devices on your local server and creates a global capacity for containers. We will use the two non-root storage devices (/dev/xvdb, /dev/xvdc) from our first step in Prerequisites.

Important: Back up any data on storage devices that will be pooled by PX-Dev. Storage devices will be reformatted!

To view the storage devices on your server:

  • Command line: run # lsblk
  • Note the devices without the part(ition)

Example output:

  $ lsblk
  NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
  xvda                      202:0    0     8G  0 disk 
  └─xvda1                   202:1    0     8G  0 part /
  xvdb                      202:16   0    64G  0 disk 
  xvdc                      202:32   0    64G  0 disk

Step 2: Edit the JSON configuration

The PX-Dev config.json lets you select the storage devices and identifies the key-value store for the cluster.

  • Download the sample config.json file:
  • https://raw.githubusercontent.com/portworx/px-dev/master/conf/config.json
  • Create a directory for the configuration file.
  • # sudo mkdir -p /etc/pwx
  • Move the file to that directory. This directory later gets passed in on the Docker command line.
  • # sudo cp -p config.json /etc/pwx
  • Edit the config.json to include the following:
  • clusterid: This string identified your cluster and should be unique within your etcd k/v space.
  • kvdb: This is the etcd connection string from the third step in Prerequisites.
  • devices: These are the storage devices that will be pooled from the prior step.

Example config.json:

    {
      "clusterid": "make this unique in your k/v store",
      "kvdb": [
          "etcd:https://[username]:[password]@[string].dblayer.com:[port]"
        ],
      "storage": {
        "devices": [
          "/dev/xvdb",
          "/dev/xvdc"
        ]
      }
    }

The definition for all attributes in config.json is [here] (https://github.com/portworx/px-dev/tree/master/conf) The format for the kvdb section is as follows:

kvdb:[
  "etcd:[http/https]://[....]"
]

IMPORTANT: If you are using Compose.IO, the kvdb string might end with [port]/v2/keys. Please omit the /v2/keys for now. Before running the container, make sure you have saved off any data on the storage devices specified in the config.

  Warning!!!: Any storage device that PX-Dev uses will be reformatted.

Run PX-Dev

When you run Docker and PX-Dev, your storage capacity is aggregated and managed by PX-Dev. As you run PX-Dev on each server, new capacity is added to the cluster.

Once PX-Dev is running, you can create and delete storage volumes through the Docker volume commands or the pxctl command line tool. With pxctl, you can also inspect volumes, the volume relationships with containers, and nodes.

Step 1: Run PX-Dev

Start the PX-Dev container with the following run command:

# sudo docker run --restart=always --name px-dev -d --net=host --privileged=true \
                 -v /run/docker/plugins:/run/docker/plugins                       \
                 -v /var/lib/osd:/var/lib/osd:shared                              \
                 -v /dev:/dev                                                     \
                 -v /etc/pwx:/etc/pwx                                             \
                 -v /opt/pwx/bin:/export_bin:shared                               \
                 -v /var/run/docker.sock:/var/run/docker.sock                     \
                 -v /var/cores:/var/cores                                         \
                 -v /lib/modules:/lib/modules                                     \
                 --ipc=host                                                       \
                portworx/px-dev

Explanation of the runtime command options:

--privileged
    > Sets PX-Dev to be a privileged container. Required to export block  device and for other functions.
    
--net=host
    > Sets communication to be on the host IP address over ports 9001 -9003. Future versions will support separate IP addressing for PX-Dev.
    
--shm-size=384M
    > PX-Dev advertises support for asynchronous I/O. It uses shared memory to sync across process restarts
    
-v /run/docker/plugins
    > Specifies that the volume driver interface is enabled.
    
-v /dev
    > Specifies which host drives PX-Dev can see. Note that PX-Dev only uses drives specified in config.json. This volume flage is an alternate to --device=\[\].
    
-v /etc/pwx/config.json:/etc/pwx/config.json
    > the configuration file location.
    
-v /var/run/docker.sock
    > Used by Docker to export volume container mappings.
    
-v /var/lib/osd:/var/lib/osd:shared
    > Location of the exported container mounts. This must be a shared mount.
    
-v /opt/pwx/bin:/export_bin:shared
    > Exports the PX command line (pxctl) tool from the container to the host.

Step 2: View global capacity

At this point, PX-Dev should be running on your system. You can run Docker ps to verify.

The pxctl control tools are exported to /opt/pwx/bin/pxctl. These tools let you control storage. For more on using pxctl, see Using Portworx storage.

  • View the global storage capacity by running
  • # sudo /opt/pwx/bin/pxctl status
  • See the example output below
  • Use pxctl to manage volumes, for example to create, snapshot, and inspect.
  • View all pxctl options by running # /opt/pwx/bin/pxctl help

The following output of pxctl status shows that the global capacity for Docker containers is now 128 GB.

    # /opt/pwx/bin/pxctl status
    Status: PX is operational
    Node ID:  c553a764-9565-4f6b-b70c-10d963096b76
     	IP:  172.31.23.134 
     	Local Storage Pool:
     	Device		Caching Tier	Size	Used
     	/dev/xvdb	true            64 GB   2.0 GB
     	/dev/xvdc	true            64 GB   2.0 GB
     	total		-               128 GB  4.0 GB
    Cluster Summary
     	ID:  86428a61-a22a-4a7a-a79a-1cfbbb846f7e
     	IP: 172.31.23.134 - Capacity: 119 GiB/3.7 GiB OK (This node)
    Global Storage Pool
     	Total Capacity	:  119 GiB
     	Total Used    	:  3.7 GiB

You have now completed setup of PX-Dev on your first server. To increase capacity and enable high availability, repeat the same steps on each of the remaining two servers. Run pxctl status to view the cluster status. Afterwards, continue with Quick Start Guides for application scenarios that use PX-Dev.

Step 3: Add nodes

To add nodes to increase capacity and enable high availability, complete the same steps on each server.

  • JSON configuration:
  • If you have the same storage device configuration on every node, then copy the config.json you created the first time to all the nodes.
  • If you have different storage device configurations on the nodes, make sure to use the same clusterid and kvdb on all the nodes.

Afterwards, continue with Using PX-Dev storage.