Today we’re announcing Dusty, a new tool for running Docker-powered development environments on OS X. Here, I’ll talk about why we needed Docker for development and how Dusty helps address some shortcomings in the other tools available.
Building Services with Docker
We use Docker to deploy all of our services in production. The simplicity of the Docker build process allows us to scale our engineering organization by largely removing operational concerns from our application developers. If a team can ship a functional Docker image, we can run it in production.
The simple deployment model of Docker gave us the freedom to aggressively split out our monolith into many services. Typically, we’d deploy a new service on a dedicated autoscaling group and put it behind a dedicated load balancer. This one-to-one relationship of infrastructure to services was simple and worked well in production. However, this is the opposite of a typical development environment, where every service runs on a single machine. Our production tooling didn’t fit our local use case, so we were stuck with our Vagrant-based development environments after our initial move to Docker in production.
Managing our VM-based environments became more difficult as we added more services. The flexibility we enjoyed with Docker in production was hard to emulate locally on a single VM, as having multiple isolated instances of a language runtime or database installed in a single VM was not easy. With no standardized way to enable or disable services, our VMs ran every service we had and got bloated. Rolling out changes to the structure of an existing service would break everyone’s environments until they applied the necessary changes.
After dealing with this for a few months, we accepted that our VM-based system was untenable and that we needed to use Docker locally in order to accurately reflect what we were running in production. We were hopeful that, with the right tooling, we could come up with something that brought the benefits of Docker to our development use case.
Making Docker “Just Work” on OS X
We quickly hit a snag with the strategy of utilizing Docker for our development environments. At GameChanger, we’ve created the most popular amateur sports app in the App Store. Naturally, our developers and designers all use OS X, which can’t run Docker natively.
To address this, tools like boot2docker and the newer Docker Machine can create a lightweight Linux VM for running the Docker daemon. This gave us a way to use Docker on OS X, but the VM layer came with significant downsides. Previously simple tasks like accessing a service’s exposed ports or sharing files between your host filesystem and the containers were now complex. We needed to make working with Docker easy if we had any hope of selling our developers on moving from their current setups to a Docker-based solution.
This is when we first realized we needed to create Dusty. These usability problems are outside the scope of the existing tools, so we needed an application sitting on top of them to glue together a working solution and provide a user-friendly interface. We do a few things with Dusty to take away the pain of Docker on OS X.
- Automated Port Forwarding: Dusty integrates with nginx and your hosts file to traverse the network stack from OS X, through the Docker VM, and into your running container. All the user sees is a host and port accessible from their Mac which behaves the way they expect.
- Fast File Syncing: Dusty uses rsync for fast file syncing between your local filesystem and the VM filesystem accessible by the containers. This replaces VirtualBox shared folders, which are incredibly slow.
- File Management: Dusty provides a CLI command to copy files between your OS X filesystem and running containers.
These features go a long way towards making Docker pleasant to use on OS X. We can comfortably run applications inside of containers now, but there’s still more to do. We don’t just need to run applications, we need to actively develop them!
Developing with Dusty
Once we had Docker running well on OS X, we needed to move past just running our applications and start supporting a true development workflow. Our starting point for this was Docker Compose (née Fig), which bills itself as the preferred way to run development environments with Docker.
Compose gave us a good way to configure and run large, pre-defined sets of containers, but this only partially solved our problem. We still didn’t have a way to easily override code running inside these containers. If we hard-coded these overrides into the Compose specs, we forced every developer to check out the source for every application and keep it updated. With Compose, we couldn’t run isolated tests on our new code without writing separate Compose specs. We wanted a tool to save us from the additional cruft necessary to make Compose really work for active development.
We needed our tools to minimize this friction around changing code for running services. As our stack grew to include more services, we also realized we needed to make it easy for developers to run services they needed but were not actively developing. For example, our iOS app API consists of four services right now. If I’m only changing code in one of them, my tools should take care of running the other three without placing a cognitive load on me.
Dusty is built on top of Compose and adds functionality to enable an efficient development workflow.
- Repository Management: By default, Dusty manages all required repositories for you and ensures they stay up to date. You can override any individual repository if you want to modify its code on the fly. Dusty takes care of mounting the correct version into the running containers.
- Dependency Resolution: Dusty knows how to stitch together your containers to come up with a running environment. When you need to restart containers, Dusty restarts them in the correct order to pick up your changes and not break things.
- Library Development: Dusty lets you declare development library dependencies on running applications. You can make changes to your library and Dusty will handle restarting any dependent applications so they pick up your changes.
- Standardized Testing: Dusty lets you store test commands in the Dusty specification for an application or library, then runs those tests in a separate container. This lets you share tests easily across a development team and reduces friction when moving between different language ecosystems. Tests can also run against their own isolated database containers.
Dusty provides a powerful and flexible way of running development environments that meet the needs of a developer working on a large, service-oriented architecture.
Wrap-Up
With Dusty, we’re trying to make Docker a viable solution for development environments on OS X. We’ve made great progress with the general usability issues of having to run Docker through a VM, and the development workflow of Dusty manages away the complexity of changing code inside running containers. We’re excited to keep improving Dusty and make it hugely useful to ourselves and the community.
Dusty is available now for OS X. If you’d like to try it out, you can install Dusty and play around with our Getting Started docs, which include links to functioning example specs.
If you’d like to hack on Dusty yourself, the project is open source and made available under the MIT license.