We pride ourselves on getting things right. Our company values, posted on our wall, say, “Detest the unplanned, unpolished and unfinished.” Yet, several months back, we shipped an unstable version of our app. We didn’t realize that anything was wrong until the crash reports came in. Because of Apple’s approval process, we couldn’t get a fixed app released right away. We scrambled to notify our customers about workarounds.
The experience prodded us to adopt new principles for shipping apps. We’ve successfully applied those principles to our recent releases, which included a redesign of global navigation and an overhaul of basketball scoring.
Nothing about these principles is specific to apps. I actually could have titled this series, “How to ship anything.”
In this post, I’ll describe our first principle.
Principle 1. Define release criteria based on quality, not time.
We used to try to fit every release within a strict three week time box. There’s nothing inherently bad about time boxes. The problem was that the time box was our most salient release criterion. We’d feel good about ourselves when we released within our time box. We’d feel like we’d failed when our release stretched beyond our time box.
Last-minute bugs put us under a lot of pressure. Should we violate our time box, thereby admitting failure? Or should we rush and attempt to meet our deadline? We’d usually rush.
Rushing has a few predictable outcomes (see Seth Godin’s post), all of which happened:
- Heroism. We fix the bug in time to meet our planned release date.
- Delay. Our rushed fix causes more problems that require more fixes.
- Disaster. Our rushed fix causes more problems that we don’t detect. We ship a bad app.
Rushing is bad. In truth, deadlines don’t usually matter that much! We were psyching ourselves out to meet deadlines when taking an extra couple of days would have been better.
How did we stop rushing? We redefined our release criteria based on quality and not time. We still try to release every three weeks or so, but we’ll never release until we’ve met our quality criteria:
- All new functionality must be tested for functional and aesthetic bugs, and all critical bugs must be fixed before release.
- Our regression tests — automated and manual — must pass.
- If we make any fixes, we must retest the changed functionality and rerun the full regression test suite. When we realize that we need to retest, we allow ourselves the time to do it well. If we slip by a day or two, it isn’t a big deal.
Quality criteria apply to every tier
Public releases aren’t the only ones that have quality criteria. Our release process has always had three tiers: Alpha, Beta, and Release. We defined quality-based entrance criteria and exit criteria for each tier.
- Alpha: All code that passes unit and functional tests is promoted to Alpha.
- Beta: Eric Han, our release manager, must approve all changes that are promoted from Alpha to Beta. Only approved Alpha code that passes a small subset of our regression tests is promoted to Beta.
- Release: Only Beta code that passes our full regression test suite is released to Apple.
The success of our ‘gcios’ build causes us to automatically release an Alpha build, which is downloadable via HockeyApp.
The higher entrance criteria and the higher release criteria complement each other. We only run through our extensive Beta test suite on changes that we deem important enough for retesting. We’ll leave other changes out of our Beta tier and we’ll defer them to the Alpha tier for the next release.
It’s about what you do under pressure
Pressure situations make mistakes more likely. A good release process should handle pressure situations well. Under pressure, we make bad decisions and fall back on habits. Our company values don’t tell us to “meet deadlines at all costs.” Yet, under pressure, we found ourselves rushing a bad app to customers.
Since we’ve introduced quality-based criteria, we’ve reacted the opposite way when under pressure. When we’ve found late-breaking bugs in a recent release, we decided to slow down and re-test. We took an extra couple of days and launched when we were happy with our app.
What’s next?
Quality matters more than speed, but speed matters too. This first principle is about ensuring quality. The next two posts will discuss how to do so efficiently.