Automating a Dusty Workflow with Watchdog

If you haven’t heard, some of my colleagues here at GameChanger released a pretty awesome project called Dusty for running Docker-based local development environments. I’ve been using Dusty for a few weeks now, feeling out the best ways to do the things I do most often. I work a lot with the front end of our website, so I change a lot of HTML, CSS and JS, as well as messing with the Python views that serve all of that content.

My Development Workflow

Let’s talk about what my workflow was when I ran our app without Docker and without any VMs. We run our main website on Django, so when I changed Python code, I’d have to restart the Django process. If I changed HTML, I didn’t have to do anything, Django would start serving out the new HTML immediately. We write CoffeeScript and LESS here at GameChanger, so if I changed those types of file, I’d need to re-compile those files to get them to take effect on my local copy of the site. In our old setup we ran a Grunt process that watched for changes to the source files and then did that compilation. So, all in all, the only thing that I needed to do on a regular basis was to bounce the Django process, and even that I could have automated away if I really needed to.

Dusty Workflow

Now on to Dusty. In my Dusty dev environment our Django app runs in a Docker container. That’s the same Docker container that we deploy to our production boxes, with a few hooks to do some dev specific things. First, we are able to update the container’s file system with the code we want it to run locally with dusty sync. Second, we’re able to run scripts against the container using dusty scripts. The particular script I’m talking about is our script to run the Grunt job to compile our LESS and CoffeeScript. Finally, I can bounce my Django process by running dusty restart. Now that I have the elements of my Dusty workflow, let’s talk about how it works out in practice.

My initial Dusty workflow required a lot of manual interaction. My Python changes workflow is just about the same because a dusty restart does a sync beforehand. The simplest thing from my original workflow, the change to an HTML template, now requires me to do a dusty sync to get the new HTML to the container, instead of taking effect immediately. My CoffeeScript/LESS workflow is also slower than it was before. We don’t want to run the “watcher” process inside the container, because we wouldn’t want to run the process in our production environment, so I have to run my dusty scripts command on every change I want to take effect. Also, just like with the rest of the code, I need to sync my changes before my dusty scripts call will do anything.

So, right off the bat, I’m seeing that its taking more of my time to do the same things I always did just because I’m using Dusty. That’s not good. So what do I do? I do what any engineer should, I automate those manual tasks. This is probably something you’ve heard over and over, but it’s absolutely true. Always optimize the things you do the most, and when possible, automate them. If you save 6 seconds on something you do 100 times a day, you just got 10 extra minutes. Additionally, if you can automate the things that you’re doing manually now, you don’t spend the time thinking “why isn’t this working” when you’ve forgotten to do the manual step.

Automation

Travis (one of the creators of Dusty) introduced me to this nice Python package and command line tool called watchdog. Watchdog watches for filesystem events and lets you define some jobs to run based on those events. I took a look at the documentation and examples and wrote the command to handle my CoffeeScript/LESS workflow needs. It looked like this:

watchmedo shell-command \
    --patterns="*.less;*.coffee" \
    --recursive \
    --drop \
    --command='dusty sync gcweb; dusty scripts gcweb grunt' \
    /gc/gcweb

Now, that’s not the ugliest thing I’ve ever seen, nor is it the prettiest. But there are two big problems. First, I want to be able to share this with my teammates here at GameChanger and telling them to paste a complex command in their terminal is not too helpful. Second, this only solves one part of my three part workflow. So to fully solve my problem, I’ll need to run at least two more of these commands.

Luckily, watchdog can run commands from files rather than command line args. They can read in what they call “tricks” files which are yaml files that provide similar information to what is in the command above. This solves my two problems with my first command. I can commit these files to the repository that they are watching, which distributes them to my team. Also you can put multiple commands into one file, which makes for far less overhead when running all of the commands. So let’s look at what that looks like:

# dusty-autosync.yaml
tricks:
- watchdog.tricks.ShellCommandTrick:
    patterns: ["*.less", "*.coffee"]
    shell_command: 'dusty sync gcweb; dusty scripts gcweb grunt'
    drop_during_process: true
- watchdog.tricks.ShellCommandTrick:
    patterns: ["*.html"]
    shell_command: 'dusty sync gcweb'
    drop_during_process: true
- watchdog.tricks.ShellCommandTrick:
    patterns: ["*.py"]
    shell_command: 'dusty restart gcweb'
    drop_during_process: true

Here you can see that when I change LESS or CoffeeScript files I sync and run my Grunt script, when I change HTML files I sync my code over, and when I change some Python code, I restart my app. Now all I have to do is run watchmedo tricks-from dusty-autosync.yaml when I start working and when I make changes to my code my Dusty container is updated appropriately to reflect them on my local copy of the site. I omitted part of the shell_command that is in the actual file above to be concise, but I also use the terminal-notifier command line tool to pop up a OS X notification when each of these commands finish so I never need to think “is it done yet?”.

Doing this small amount of automation has made Dusty a much more viable tool for me to use. If you’ve been using Dusty and have run into some similar annoyances, check out watchdog. If you use Docker and haven’t used Dusty yet, check it out.

Since I automated my workflow I’ve noticed myself waiting on the Grunt script to finish more often than I’d like, so now I’m going to try to shave a few seconds off of it.