A guide to using

BOSH

This tutorial gives a simple introduction to BOSH. If you never deployed with BOSH or never heard about it this step-by-step guide will bring you up to speed.

Learn more:

Prepare

We are going to use bosh-lite in this tutorial. bosh-lite is a vagrant VM that comes with pre-installed BOSH server (Director). Once VM with Director is running we are going to use BOSH CLI to send commands to the Director.

Learn more:

1

Install BOSH-lite

Bring bosh-lite VM up with vagrant:

$ vagrant init cloudfoundry/bosh-lite

$ vagrant up --provider=virtualbox

Bringing machine 'default' up with 'virtualbox' provider...

In case of any issues see bosh-lite's README

2

Install BOSH CLI

BOSH CLI is a command line interface to the Director. It is distributed as a Ruby gem.

$ gem install bosh_cli

In case of any issues see detailed installation instructions.

Let CLI know about bosh-lite Director (use admin/admin to log in):

$ bosh target 192.168.50.4

Target set to 192.168.50.4

Now we are ready to deploy!

Deploy

Before we proceed we need to understand what BOSH needs to deploy software.

What to deploy

Software that is deployed with BOSH needs to be packaged in a special format called a release. For each service that will be deployed, a release needs to contain source files, configuration files, installation scripts, etc. For example, a redis release would contain the source code for redis, redis configuration defaults and redis init scripts.

How to deploy

Each BOSH deployment needs to provide a specially structured configuration file - deployment manifest. This file defines what resources are going to be deployed, what services are going to be running on each of resources and properties that will be passed to services configuration files. For example, for a redis deployment manifest, there are entries for how many and what size redis VMs there should be and how redis should be configured.

Learn more:

1

Create BOSH release

We are going to use a simple BOSH release that deploys a sinatra server.

$ git clone https://github.com/mariash/learn-bosh-release learn-bosh-release

$ cd learn-bosh-release

$ bosh create release

Upload generated release to Director:

$ bosh upload release

Check uploaded releases:

$ bosh releases

Acting as user 'admin' on 'Bosh Lite Director' +------------+----------+-------------+ | Name | Versions | Commit Hash | +------------+----------+-------------+ | learn-bosh | 0+dev.1 | 1867f1d2 | +------------+----------+-------------+ Releases total: 1

Learn more:

2

Set Deployment Manifest

Deployment manifest specifies what services to deploy, their properties and resources configuration. The manifest contains a placeholder director_uuid that needs replacing.

Update the director_uuid value in the manifest.yml with the value from:

$ bosh status --uuid

Set the deployment manifest:

$ bosh deployment manifest.yml

Deployment set to '..learn-bosh-release/manifest.yml'

Learn more:

3

Upload stemcell

A Stemcell is an operating system image that BOSH uses to create VMs. Official BOSH stemcells are maintained with security updates at bosh.io.

Upload stemcell to Director.

$ wget --content-disposition https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent

$ bosh upload stemcell bosh-stemcell-3147-warden-boshlite-ubuntu-trusty-go_agent.tgz

See uploaded stemcells:

$ bosh stemcells

Acting as user 'admin' on 'Bosh Lite Director' +---------------------------------------------+---------+ | Name | Version | +---------------------------------------------+---------+ | bosh-warden-boshlite-ubuntu-trusty-go_agent | 3147 | +---------------------------------------------+---------+ (*) Currently in-use Stemcells total: 1

Learn more:

4

And deploy

$ bosh deploy

See the list of deployed VMs as it was specified in manifest:

$ bosh vms

... +-----------+---------+---------------+------------+ | Job/index | State | Resource Pool | IPs | +-----------+---------+---------------+------------+ | app/0 | running | default | 10.244.0.2 | +-----------+---------+---------------+------------+ ...

Add route to bosh-lite network:

$ sudo route add -net 10.244.0.0/19 192.168.50.4

If your workstation runs Linux, use this command instead:

$ sudo ip route add 10.244.0.0/19 via 192.168.50.4

See that our service is up and running.

$ curl 10.244.0.2:8080

Hello, Maria from ...

Learn more:

Modify Deployment

Now we will update our deployment with new version of software. We will modify some properties. And we are going to scale our deployment.

1

Modify release

BOSH makes it easy to modify and deploy new versions of software. Let's modify our release source files.

In release folder open src/simple_server/app.rb and change the name to yours.

Create new version of release (force option is used to ignore warning about local changes), upload new version of release to Director and deploy!

$ bosh create release --force

$ bosh upload release

$ bosh deploy

See that the updated version was deployed:

$ curl 10.244.0.2:8080

Hello, you from ...

2

Scale deployment

With BOSH it is easy to scale deployments. All you need to do is modify number of instances in manifest file.

Open manifest.yml and change number of instances under job from 1 to 2. Add another IP to list of job static IPs: 10.244.0.6.

Run deploy:

$ bosh deploy

Check that 2 VMs were deployed:

$ bosh vms

Acting as user 'admin' on 'Bosh Lite Director' Deployment `learn-bosh' Director task 22 Task 22 done +-----------+---------+---------------+------------+ | Job/index | State | Resource Pool | IPs | +-----------+---------+---------------+------------+ | app/0 | running | default | 10.244.0.2 | | app/1 | running | default | 10.244.0.6 | +-----------+---------+---------------+------------+ VMs total: 2

See that we have 2 instances of our service running:

$ curl 10.244.0.2:8080

Hello, you from <uuid-1>

$ curl 10.244.0.6:8080

Hello, you from <uuid-2>

3

Change properties

Every release can specify a set of properties that need to be set in deployment manifest and provided to service. For example, that can be database credentials, address of another service, etc.

Our release allows to change property port on which server is listening. You can see the list of properties that can be modified in learn-bosh-release/jobs/app/spec. Let's open manifest.yml and under the section properties set the value of port to 8888. Now we can just re-deploy our manifest changes. Note, we don't need to build new release version, configuration files will be regenerated with new properties.

$ bosh deploy

Let's see that our property was changed:

$ curl 10.244.0.2:8888

Hello, you from ...

When something goes wrong

BOSH provides a set of recovery mechanisms. Let's break our deployment and find ways to fix it.

1

Failing service

BOSH is using monit to monitor running services. If the service goes down it will bring it up. Let's watch how this works. SSH to one of VMs:

$ bosh ssh app/0

$ sudo -i

# watch monit summary

The Monit daemon 5.2.4 uptime: 10m Process 'app' running System 'system_ec1f1ae3-80cb-448b-a024-a24985271f76' running

In a separate window (on host) let's kill our runnning server:

$ curl 10.244.0.2:8888/kill

Back in VM window notice that monit will report process as 'Does not exist' and after some period service will be brought back up by monit again.

Learn more:

2

Failing VM

What if there is a problem with VM that is running our service? BOSH offers manual and automatic recovery when there are problems with infrastructure resources. In this exercise we are going to kill our VM and use manual recovery option.

Kill one of VMs. bosh-lite is using Warden to manage deployments. Warden is a Linux container management tool. Each container represents one VM. On vagrant VM kill one of Warden containers:

$ vagrant ssh

vagrant@agent-id-bosh-0$ wget -qO - http://127.0.0.1:7777/containers

{"handles":["id-1","id-2"]}

vagrant@agent-id-bosh-0$ wget -qO - --method=DELETE http://127.0.0.1:7777/containers/id-1

Let's see that one of the VMs is in a bad state:

$ bosh vms

... +-----------------+--------------------+---------------+------------+ | Job/index | State | Resource Pool | IPs | +-----------------+--------------------+---------------+------------+ | unknown/unknown | unresponsive agent | | | | app/0 | running | default | 10.244.0.2 | +-----------------+--------------------+---------------+------------+ ...

One of the components in BOSH is the Health Monitor. It independently watches system health and will bring missing VMs back up. Keep running bosh vms and see that VM is brought up and service is running eventually.

Now let's turn off automatic repair and manually resolve the issue.

$ bosh vm resurrection off

Kill one of the containers again as described above. Run cloud check and select option "Recreate VM".

$ bosh cck

Cloud check command allows to manually resolve issues when resources (vms and persistent disks) are in a bad state. Run bosh vms to see all VMs running again.

Learn more:

3

Debugging failing deploy

When deploy command fails there are could be a number of reasons:

  • Invalid network configuration in deployment manifest (e.g. IP address is in use or out of subnet range)
  • Infrastructure provider failed to create VM or disk (e.g. quota exceeded, instance type is not available)
  • Properties required by release were not provided in manifest
  • Let's add another job to our manifest router (it will redirect all requests between servers). Note, that since uploaded release already contains this job, we don't need to update release.

    - name: router templates: - name: router instances: 1 resource_pool: default networks: - name: default static_ips: [10.244.0.10]

    Re-deploy with new job.

    $ bosh deploy

    ...Failed: `router/0' is not running after update

    Oh-oh, looks like deploy failed. Let's get our service logs, untar them and check stderr log.

    $ bosh logs router 0

    We should find this error: "At least one server must be provided". Router fails to route because there is no servers specified.

    Let's add a property to router job to specify our servers pointing to their static IPs and ports ("http://10.244.0.2:8888", "http://10.244.0.6:8888"), re-deploy and see it succeeds.

    Now running curl -L http://10.244.0.10:8080 should give us responses from different servers.

Done!

In this tutorial we used BOSH to deploy services, updated our deployment with source changes, scaled the number of services and changed their properties. We recovered from failing service, failing VM and failing deploy.

We were using bosh-lite that comes with Warden CPI (Cloud Provider Interface). Warden is a Linux container management tool and Warden CPI abstracts VMs as Linux containers. The Director can work with any CPI that implements a certain API to manage IaaS resources. There are several supported CPIs for different IaaS providers: AWS, Openstack, vSphere and vCloud. Read more about CPIs here: http://bosh.io/docs/cpi-api-v1.html.

bosh-lite comes with pre-installed Director. bosh-init provides a way to install initial VM with BOSH Director on any IaaS. Read more about it here: http://bosh.io/docs/using-bosh-init.html.

Learn more: