A guide to using

BOSH

This tutorial gives a simple introduction to BOSH. If you never deployed with BOSH or never heard about it this step-by-step guide will bring you up to speed.

Learn more:

Prepare

We are going to use local BOSH environment provided by VirtualBox (known working version 5.1+). We will install BOSH server (Director) using BOSH CLI. Once VM with Director is running we are going to use BOSH CLI to send commands to the Director.

Learn more:

1

Setup BOSH environment

BOSH CLI is a command line interface to the Director. Follow these instructions to install it.

We are going to use bosh create-env command to install BOSH Director. bosh create-env command provides a way to install initial VM with BOSH Director on any IaaS (AWS, GCP, etc). In our case we will use VirtualBox as our infrastructure provider.

First get configuration files that specify BOSH environment in VirtualBox. And run bosh create-env as following:

$ git clone https://github.com/cloudfoundry/bosh-deployment bosh-deployment

$ mkdir vbox

$ bosh create-env bosh-deployment/bosh.yml \
--state vbox/state.json \
-o bosh-deployment/virtualbox/cpi.yml \
-o bosh-deployment/virtualbox/outbound-network.yml \
-o bosh-deployment/bosh-lite.yml \
-o bosh-deployment/bosh-lite-runc.yml \
-o bosh-deployment/jumpbox-user.yml \
--vars-store vbox/creds.yml \
-v director_name="Bosh Lite Director" \
-v internal_ip=192.168.50.6 \
-v internal_gw=192.168.50.1 \
-v internal_cidr=192.168.50.0/24 \
-v outbound_network_name=NatNetwork

In case of any issues see bosh-deployment VirtualBox docs.

Learn more:

2

Log in

Once VM with BOSH Director is running, point your CLI to it, saving the environment with the alias vbox:

$ bosh -e 192.168.50.6 alias-env vbox --ca-cert <(bosh int vbox/creds.yml --path /director_ssl/ca)

Using environment '192.168.50.6' as anonymous user Name Bosh Lite Director UUID ... Version 261.4.0 (00000000) CPI warden_cpi Features compiled_package_cache: disabled config_server: disabled dns: disabled snapshots: disabled User (not logged in) Succeeded

Obtain generated password to BOSH Director:

$ bosh int vbox/creds.yml --path /admin_password

Log in using admin username and generated password:

$ bosh -e vbox login

Now we are ready to deploy!

Deploy

Before we proceed we need to understand what BOSH needs to deploy software.

What to deploy

Software that is deployed with BOSH needs to be packaged in a special format called a release. For each service that will be deployed, a release needs to contain source files, configuration files, installation scripts, etc. For example, a redis release would contain the source code for redis, redis configuration defaults and redis init scripts.

How to deploy

Each BOSH deployment needs to provide a specially structured configuration file - deployment manifest. This file defines what resources are going to be deployed, what services are going to be running on each of resources and properties that will be passed to services configuration files. For example, for a redis deployment manifest, there are entries for how many and what size redis VMs there should be and how redis should be configured.

Learn more:

1

Create BOSH release

We are going to use a simple BOSH release that deploys an http server.

$ git clone https://github.com/mariash/learn-bosh-release learn-bosh-release

$ cd learn-bosh-release

$ bosh create-release

Upload generated release to Director:

$ bosh -e vbox upload-release

Check uploaded releases:

$ bosh -e vbox releases

Using environment '192.168.50.6' as client 'admin' Name Version Commit Hash learn-bosh 0+dev.1 fd90933 1 releases

Learn more:

2

Upload stemcell

A Stemcell is an operating system image that BOSH uses to create VMs. Official BOSH stemcells are maintained with security updates at bosh.io.

Upload stemcell to Director.

$ wget --content-disposition https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent

$ bosh -e vbox upload-stemcell bosh-stemcell-*-warden-boshlite-ubuntu-trusty-go_agent.tgz

See uploaded stemcells:

$ bosh -e vbox stemcells

Using environment '192.168.50.6' as client 'admin' Name Version bosh-warden-boshlite-ubuntu-trusty-go_agent [your stemcell version] 1 stemcells

Learn more:

3

And deploy

Run deploy by providing path to deployment manifest. Deployment manifest specifies what services to deploy, their properties and resources configuration.

$ bosh -e vbox -d learn-bosh deploy manifest.yml

See the list of deployed instances as it was specified in manifest:

$ bosh -e vbox instances

... Deployment 'learn-bosh' Instance Process State AZ IPs app/guid running - 10.244.0.2 1 instances ...

Add route to VirtualBox network:

$ sudo route add -net 10.244.0.0/16 192.168.50.6 # Mac OS X

$ sudo route add -net 10.244.0.0/16 gw 192.168.50.6 # Linux

$ route add 10.244.0.0/16 192.168.50.6 # Windows

See that our service is up and running.

$ curl 10.244.0.2:8080

Hello, Maria from ...

Learn more:

Modify Deployment

Now we will update our deployment with new version of software. We will modify some properties. And we are going to scale our deployment.

1

Modify release

BOSH makes it easy to modify and deploy new versions of software. Let's modify our release source files.

In release folder open src/simple_server/app.rb and change the name to yours.

Create new version of release (force option is used to ignore warning about local changes), upload new version of release to Director and deploy!

$ bosh create-release --force

$ bosh -e vbox upload-release

$ bosh -e vbox -d learn-bosh deploy manifest.yml

See that the updated version was deployed:

$ curl 10.244.0.2:8080

Hello, you from ...

2

Scale deployment

With BOSH it is easy to scale deployments. All you need to do is modify number of instances in manifest file.

Open manifest.yml and change number of instances under job from 1 to 2. Add another IP to list of job static IPs: 10.244.0.6.

Run deploy:

$ bosh -e vbox -d learn-bosh deploy manifest.yml

Check that 2 instances were deployed:

$ bosh -e vbox instances

... Deployment 'learn-bosh' Instance Process State AZ IPs app/guid-1 running - 10.244.0.6 app/guid-2 running - 10.244.0.2 2 instances ...

See that we have 2 instances of our service running:

$ curl 10.244.0.2:8080

Hello, you from <uuid-1>

$ curl 10.244.0.6:8080

Hello, you from <uuid-2>

3

Change properties

Every release can specify a set of properties that need to be set in deployment manifest and provided to service. For example, that can be database credentials, address of another service, etc.

Our release allows to change property port on which server is listening. You can see the list of properties that can be modified in learn-bosh-release/jobs/app/spec. Let's open manifest.yml and under the section properties set the value of port to 8888. Now we can just re-deploy our manifest changes. Note, we don't need to build new release version, configuration files will be regenerated with new properties.

$ bosh -e vbox -d learn-bosh deploy manifest.yml

Let's see that our property was changed:

$ curl 10.244.0.2:8888

Hello, you from ...

When something goes wrong

BOSH provides a set of recovery mechanisms. Let's break our deployment and find ways to fix it.

1

Failing service

BOSH is using monit to monitor running services. If the service goes down it will bring it up. Let's watch how this works. SSH to one of instances:

$ bosh -e vbox -d learn-bosh ssh app/0

$ sudo -i

# watch monit summary

The Monit daemon 5.2.5 uptime: 2m Process 'app' running System 'system_localhost' running

In a separate window (on host) let's kill our runnning server:

$ curl 10.244.0.2:8888/kill

Back in the instance window notice that monit will report process as 'Does not exist' and after some period service will be brought back up by monit again.

Learn more:

2

Failing VM

What if there is a problem with instance that is running our service? BOSH offers manual and automatic recovery when there are problems with infrastructure resources like VMs or disks. In this exercise we are going to kill one of our instances and use manual recovery option.

Lets destroy one of our instances. SSH to VirtualBox VM.

$ bosh int vbox/creds.yml --path /jumpbox_ssh/private_key > jumpbox.key

$ chmod 600 jumpbox.key

$ ssh jumpbox@192.168.50.6 -i jumpbox.key

Our VirtualBox environment is using Warden to manage resources. Warden is a Linux container management tool. Each container represents one instance. On VirtualBox VM kill one of Warden containers.

$ wget -qO - http://127.0.0.1:7777/containers

{"handles":["id-1","id-2"]}

$ wget -qO - --method=DELETE http://127.0.0.1:7777/containers/id-1

Let's see that one of the instances is in a bad state:

$ bosh -e vbox instances

... Instance Process State IPs app/guid-1 running 10.244.0.6 app/guid-2 unresponsive agent 10.244.0.2 ...

One of the components in BOSH is the Health Monitor. It independently watches system health and will bring missing instances back up by instructing infrastructure to recreate missing resources like VMs with the required persistent disk. Keep running bosh -e vbox instances and see that instance is brought up and service is running eventually.

Now let's turn off automatic repair and manually resolve the issue.

$ bosh -e vbox update-resurrection off

Kill one of the containers again as described above. Run cloud check and select option "Recreate VM and wait for processes to start".

$ bosh -e vbox -d learn-bosh cloud-check

Cloud check command allows to manually resolve issues when resources (VMs and persistent disks) are in a bad state. Run bosh -e vbox instances to see all instances running again.

Learn more:

3

Debugging failing deploy

When deploy command fails there are could be a number of reasons:

  • Invalid network configuration in deployment manifest (e.g. IP address is in use or out of subnet range)
  • Infrastructure provider failed to create VM or disk (e.g. quota exceeded, instance type is not available)
  • Properties required by release were not provided in manifest

    • Let's add another job to our manifest router (it will redirect all requests between servers). Note, that since uploaded release already contains this job, we don't need to update release.

    - name: router templates: - name: router instances: 1 resource_pool: default networks: - name: default static_ips: [10.244.0.10]

    Re-deploy with new job.

    $ bosh -e vbox -d learn-bosh deploy manifest.yml

    ...Failed: `router/0 (...)' is not running after update.

    Oh-oh, looks like deploy failed. Let's get our service logs, untar them and check stderr log.

    $ bosh -e vbox -d learn-bosh logs router/0

    We should find this error: "At least one server must be provided". Router fails to route because there is no servers specified.

    Let's add a property to router job to specify our servers pointing to their static IPs and ports ("http://10.244.0.2:8888", "http://10.244.0.6:8888"), re-deploy and see it succeeds.

    Now running curl -L http://10.244.0.10:8080 should give us responses from different servers.

Done!

In this tutorial we used BOSH to deploy services, updated our deployment with source changes, scaled the number of services and changed their properties. We recovered from failing service, failing VM and failing deploy.

We were using VirtualBox environment with Warden CPI (Cloud Provider Interface). Warden is a Linux container management tool and Warden CPI abstracts VMs as Linux containers. The Director can work with any CPI that implements a certain API to manage IaaS resources. There are several supported CPIs for different IaaS providers: AWS, GCP, Openstack, vSphere and vCloud. Read more about CPIs here: http://bosh.io/docs/cpi-api-v1.html.

Learn more: