Continuous Deployment with Docker: Slides

Publish date: 2024-03-08

@sublimino

Delivery Engineer

News UK, Visa, British Gas

www.binarysludge.com

codes

Containment

data

efficacy

Principles

Partial or whole stack deployable on developers' machines
- Ease of full stack debugging
- Build server verifies tests rather than running them for the first time
- No staging environment, feature toggles instead
- Devs can run system integration tests, but they're slow to boostrap
  - System integration faster, incremental deployments

pick any two

more Principles

Containers identical from dev through test to production
Logic branched on environment variables
Deterministic pipeline
Constant cycle-time optimisation

Continuous delivery/deployment

Always ready to deploy vs always deployed
All code/containers must be testable
- Test teams should never be bigger than dev teams
- Humans are fallible, testers more so
- Machines are all that will be left one day

BRING PAIN FORWARD

deliver continuously

developer Workflow

Hack some code
Run local tests
Commit to build server
Profit

On green deploy

Push new code version to production alongside current
Balance traffic over to new version
Monitor cluster health, error rates and response times
Run post-deployment acceptance tests against the cluster
Revert to previous version if KPI tolerances exceeded

Implementation

Shared code repository
- npm module
- Present in every repository/image (even couchbase/redis)
- Contains linting rules, extensible task runner, read-only credentials
- Idempotent npm post-install task creates test directories and manipulates filesystem
- Any change triggers cascading builds in dependent jobs

Task runner

Stream processing and parallel task execution
Abstract invocation of language/library specificities
Tight dev feedback loop, reduced cycle-time

Unified task interface

Abstracts docker, fleet and test commands

$ gulp <tab> browserify fleet:restart test:functional build fleet:start test:functional:remote build-properties help test:functional:watch build:watch less test:integration default lint test:integration:remote docker:build livereload test:integration:watch ...

4 different test runners, correct one inferred and invoked
Defers test execution to cross platform BASH scripts
- Simple CLI options abstracted, i.e. --xml
- BATS acceptance tested
- OSX vs Linux - some GNU-ish complexities
Build server has no logic

dev to docker

Component developed with tests
Developer builds Dockerfile for component
- FROM common base image
- ADD source files
- RUN installation of dependencies

testing, testing

Test setup script

Global setup/teardown and test override hooks
External services brought up with fig
Each repository's fig.yml configuration file:
- Defines the component's invocation
- ...and that of any immediate dependencies

fig.yml

fig.yml of component under test is copied to fig-test.yml
Dependent components' fig.yml files merged into fig-test.yml

Brings in transitive dependencies

Environment variables are templated
- NODE_ENV
- Database names
- IPs and ports

fig on a build server

Used to complain at the absence of a TTY
Hacked around it...

export SCRIPT_COMMAND="$NOHUP_COMMAND script --quiet --command"

funky cross platform fixes required

export TARGET_FIG_FILE=${TARGET_FIG_FILE:-fig-test.yml} export PROJECT_NAME=$(basename "$(pwd)" | sed 's/[^a-z]//g'); export FIG_COMMAND="fig --file=${TARGET_FIG_FILE} --project-name ${PROJECT_NAME}"; ... figUp() { $SCRIPT_COMMAND $FIG_COMMAND up } ...

gnu gnu gnu

This is the price paid for cross platform compatibility

# linux defaults export BASE64_COMMAND="base64 -w 0" export NOHUP_COMMAND="nohup" export SCRIPT_COMMAND="$NOHUP_COMMAND script --quiet --command" # osx overrides [[ $(uname) = 'Darwin' ]] && { darwinScript() { $NOHUP_COMMAND script -q typescript $(echo $1 \ | sed -e 's/$^"//g' -e 's/"$//g') } SCRIPT_COMMAND="darwinScript" BASE64_COMMAND="base64" [[ $(command -v reattach-to-user-namespace) ]] && { NOHUP_COMMAND="reattach-to-user-namespace nohup" } }

back to test Setup script

HTTP health checks and /health endpoints are used to determine availability
Sleep is avoided where possible
Other necessary pre-test config

API oAuth authentication
Endpoints for exposing tunnels to Sauce labs
JS/CSS build chains

docker image tests

Tests are run against a container's public interface - i.e. exposed ports
Database fixtures in a separate image, invoked using links
On build server test completion, image is tagged
Tagged image pushed to private docker registry

Promotion

Once the component's tests are passed, system integration tests occur
These ensure the interfaces between components match
On completion, non-functional tests are run in an isolated environment
If all pass, the component is marked for deployment

Deployment

Orchestration still nascent
Current design informed by Kubernetes
- in lieu of 1.0 we've implemented a stripped-down feature set
  - Service discovery via etcd
  - HAProxy routing via confd
  - Rolling updates
- Added cluster immune system and rollback

In conclusion

Continuous deployment
- Start with tests
- Build a deterministic pipeline
- Deploy to production
- Monitor everything
- Use KPIs for a cluster immune system
- Roll back if KPIs exceed tolerances
- Halt the pipeline, analysis/5 whys?
- Write a failing test case before fixing it
- Re-enable the pipeline
- Sleep well at night

fin