We are going to use local BOSH environment provided by VirtualBox (known working version 5.1+). We will install BOSH server (Director) using BOSH CLI. Once VM with Director is running we are going to use BOSH CLI to send commands to the Director.
BOSH CLI is a command line interface to the Director. Follow these instructions to install it.
We are going to use bosh create-env command to install BOSH Director. bosh create-env command provides a way to install initial VM with BOSH Director on any IaaS (AWS, GCP, etc). In our case we will use VirtualBox as our infrastructure provider.
First get configuration files that specify BOSH environment in VirtualBox. And run bosh create-env as following:
In case of any issues see bosh-deployment VirtualBox docs.
Once VM with BOSH Director is running, point your CLI to it, saving the environment with the alias vbox:
Obtain generated password to BOSH Director:
Log in using admin username and generated password:
Now we are ready to deploy!
Before we proceed we need to understand what BOSH needs to deploy software.
Software that is deployed with BOSH needs to be packaged in a special format called a release. For each service that will be deployed, a release needs to contain source files, configuration files, installation scripts, etc. For example, a redis release would contain the source code for redis, redis configuration defaults and redis init scripts.
Each BOSH deployment needs to provide a specially structured configuration file - deployment manifest. This file defines what resources are going to be deployed, what services are going to be running on each of resources and properties that will be passed to services configuration files. For example, for a redis deployment manifest, there are entries for how many and what size redis VMs there should be and how redis should be configured.
We are going to use a simple BOSH release that deploys an http server.
Upload generated release to Director:
Check uploaded releases:
A Stemcell is an operating system image that BOSH uses to create VMs. Official BOSH stemcells are maintained with security updates at bosh.io.
Upload stemcell to Director.
See uploaded stemcells:
Run deploy by providing path to deployment manifest. Deployment manifest specifies what services to deploy, their properties and resources configuration.
See the list of deployed instances as it was specified in manifest:
Add route to VirtualBox network:
See that our service is up and running.
Now we will update our deployment with new version of software. We will modify some properties. And we are going to scale our deployment.
BOSH makes it easy to modify and deploy new versions of software. Let's modify our release source files.
In release folder open src/simple_server/app.rb and change the name to yours.
Create new version of release (force option is used to ignore warning about local changes), upload new version of release to Director and deploy!
See that the updated version was deployed:
With BOSH it is easy to scale deployments. All you need to do is modify number of instances in manifest file.
Open manifest.yml and change number of instances under job from 1 to 2. Add another IP to list of job static IPs: 10.244.0.6.
Check that 2 instances were deployed:
See that we have 2 instances of our service running:
Every release can specify a set of properties that need to be set in deployment manifest and provided to service. For example, that can be database credentials, address of another service, etc.
Our release allows to change property port on which server is listening. You can see the list of properties that can be modified in learn-bosh-release/jobs/app/spec. Let's open manifest.yml and under the section properties set the value of port to 8888. Now we can just re-deploy our manifest changes. Note, we don't need to build new release version, configuration files will be regenerated with new properties.
Let's see that our property was changed:
BOSH provides a set of recovery mechanisms. Let's break our deployment and find ways to fix it.
BOSH is using monit to monitor running services. If the service goes down it will bring it up. Let's watch how this works. SSH to one of instances:
In a separate window (on host) let's kill our runnning server:
Back in the instance window notice that monit will report process as 'Does not exist' and after some period service will be brought back up by monit again.
What if there is a problem with instance that is running our service? BOSH offers manual and automatic recovery when there are problems with infrastructure resources like VMs or disks. In this exercise we are going to kill one of our instances and use manual recovery option.
Lets destroy one of our instances. SSH to VirtualBox VM.
Our VirtualBox environment is using Warden to manage resources. Warden is a Linux container management tool. Each container represents one instance. On VirtualBox VM kill one of Warden containers.
Let's see that one of the instances is in a bad state:
One of the components in BOSH is the Health Monitor. It independently watches system health and will bring missing instances back up by instructing infrastructure to recreate missing resources like VMs with the required persistent disk. Keep running bosh -e vbox instances and see that instance is brought up and service is running eventually.
Now let's turn off automatic repair and manually resolve the issue.
Kill one of the containers again as described above. Run cloud check and select option "Recreate VM and wait for processes to start".
Cloud check command allows to manually resolve issues when resources (VMs and persistent disks) are in a bad state. Run bosh -e vbox instances to see all instances running again.
When deploy command fails there are could be a number of reasons:
Let's add another job to our manifest router (it will redirect all requests between servers). Note, that since uploaded release already contains this job, we don't need to update release.
Re-deploy with new job.
Oh-oh, looks like deploy failed. Let's get our service logs, untar them and check stderr log.
We should find this error: "At least one server must be provided". Router fails to route because there is no servers specified.
Let's add a property to router job to specify our servers pointing to their static IPs and ports ("http://10.244.0.2:8888", "http://10.244.0.6:8888"), re-deploy and see it succeeds.
Now running curl -L http://10.244.0.10:8080 should give us responses from different servers.
In this tutorial we used BOSH to deploy services, updated our deployment with source changes, scaled the number of services and changed their properties. We recovered from failing service, failing VM and failing deploy.
We were using VirtualBox environment with Warden CPI (Cloud Provider Interface). Warden is a Linux container management tool and Warden CPI abstracts VMs as Linux containers. The Director can work with any CPI that implements a certain API to manage IaaS resources. There are several supported CPIs for different IaaS providers: AWS, GCP, Openstack, vSphere and vCloud. Read more about CPIs here: http://bosh.io/docs/cpi-api-v1.html.