I’ve been working on a new project recently, and when I say new, I mean brand new. This isn’t just a lateral shift on the team to get my hands dirty in a new area of focus. Nor is it even adding a new product to our existing stack. This is building a brand new backend API for a brand new app. I’m really excited about being able to develop a new codebase, but the sheer number of unknowns and amount of work ahead is daunting. I wanted to share how my teammate Alex Etling and I embraced modular design to make our decision making process faster and less stressful.
Let’s start this story a few weeks ago, at the inception of the new team. The purpose of the app was clear, but I’m not sure I needed two hands to count the number of decisions that had been made. In that setting, Alex and I were given the mandate to figure out what the stack looks like and start building. We are both veterans at GameChanger, but neither of us had taken on something with this combination of future scale and current uncertainty. We’ve both architected solutions to a hard problems before, but when looking at a blank text editor and wondering what database should we use?, what language should we write this in?, and most terrifyingly what if we make the wrong choice?, it was hard to figure out where to start.
Alex and I were getting started a few weeks ahead of the team that was going to be building the client app that will use this API. That means that we didn’t have to spin something up immediately, but we knew that in a few weeks we had to have decisions made and a basic API ready for consumption. Everyone else who was working on the project at this point were focused on high level research, not implementation details. Alex and I had to start implementing crucial parts of the stack that could have long lasting consequences if we chose the wrong solution, without the clear definition of future product requirements and direction. It was a bit stressful.
So we started. Alex and I split up and did some research for a few days around things like which language, framework and database we would use to build the app. These questions took some time and research to get comfortable with a decision, but we eventually both came back together and presented our research and recommendations to each other. We settled on writing TypeScript, using a Node framework (Koa) and storing our data in Postgres. We felt good about making some early quick but well-researched decisions and we were ready to move on to the next thing.
Researching the Unknown
The next decision that I looked into was deciding whether an ORM should sit above our database, and if so, which ORM did we want to use? In our earlier decisions, Alex and I were able to pull from some past experience to narrow our search field and get a small set of options to really dig into. When looking into ORMs however, I didn’t have that past experience, especially not with any of the options available for our Node app. I wasn’t even able to make the first determination of do we need an ORM? quickly, because I didn’t know enough about the possible benefits and consequences of the choice.
Because of my lack of knowledge, I spent several days reading blogs, books, everything I could find about using ORMs, and never really finding a consensus. Especially because there was so much that was undefined about the future of our project, I couldn’t even search out the opinions of people who built systems with similar constraints. After a few days of this I realized that I wasn’t getting anywhere with research, but we had a schedule to stick to. We had to have something stood up for the client app team to start using sometime soon. I was just going to have to guess and hope I made the right decision.
Even when I finally made a decision, I wasn’t happy with the ORM I chose, Sequelize. While prototyping with it I realized that it didn’t quite fit what we needed. There was always a feeling in the back of my head that I was forcing something to work instead of feeling like I was gaining a lot by using Sequelize. It was the best tool out there, although the ideal solution may be building a solution that fits our specific needs. A custom tool wasn’t an option because we needed to start building a working API, not building out a TypeScript ORM.
What if we got it wrong?
I was getting pretty frustrated that I was not going to be able to make a well informed decision. I really didn’t want to set ourselves up for failure by locking in decisions that I had no idea were right. I’ve seen projects that made decisions that seemed perfectly rational given the constraints that have gone off the rails as they evolved. I was worried because I wasn’t even at rational yet, I felt like I was just guessing. Settling on Sequelize felt like I was selling out my future self, the person who was going to have to deal with the fallout of my bad decision.
We came to grips with this frustration by recognizing that we needed to focus not on are we choosing the right technology?, but instead on how can we enable ourselves to change this decision? We started talking about layering the code in our project with strong interfaces between the layers. Then, when it becomes apparent that a better choice is out there, we will be able to replace our earlier choice with the new hotness. Even the language choice is something that we can move away from if need be. Modules can be connected via an HTTP interface to allow us to move sections of responsibility to other microservices, written in other languages. That should also keep our main service small enough that a rewrite in a new language wouldn’t be untenable.
This idea of modularizing the component parts of a system is in no way new. Martin Fowler espouses the exact idea that I am talking about in Is Design Dead. It is a great practice for building maintainable systems, and it doesn’t just apply to software. Real-life objects like cameras use modularity to enable photographers to use expensive equipment in more ways. The epiphany for me here was how modular design makes the decision making process less stressful. Once we realized that our decision shouldn’t have permanent consequences, saying I’m comfortable going with Sequelize becomes much easier. I knew that a future switch to another ORM would be work, but that work would be contained.
How is it going?
With visions of modularity dancing in our heads, we dove headlong into our next set of decisions and building out the first iteration of the API. The code that we’re writing has to work, but also has to create barriers between modules that will help us replace a module if need be. Typed languages help a great deal here, we’ve been really happy with TypeScript. If you’re interested in understanding how we’ve implemented these modules in practice, check out the more technical exploration of modularization (coming soon). The project is still relatively small, so we haven’t explored the microservices tactic yet, but I think we’ve been relatively successful in writing our code in a way that makes it easy to replace.
Overall the project is going great, we’ve successfully provided a basic API to the client app team, and we’re ready to keep expanding the breadth of resources that we’re able to handle. We’re also looking forward to adding depth of responsibility to the API, adding features that turn it from an interface on top of a database to something that provides real customer value. There are lots of interesting projects in the pipeline, and we’re hiring people to come help us build them.