The Mini-Monorepo

“Can you revert your last commit and use that third-party Dynamo locking library instead?”

My team lead looked at me with his sharp, blue eyes. They were tucked under a strong brow, and just visible over a large auburn beard. It gave him a permanent expression of seriousness.

“Why’s that?” I asked earnestly. I was a little over a year into my first job, and I had just received an AWS certification. I was eager to put it to good use, starting by using Dynamo’s conditional writes in this library I was working on.

“I want us to get rid of as much of our own code as we can. With three teams touching this monorepo it’s too easy to add dependencies across projects. That’s what got us into this mess.”

I was a little surprised by his answer. I expected him to say “we don’t trust your code as much as we trust this third-party”, which was a concern I was ready to dispute. I had my counter points all lined up. I was not, however, prepared to object to dependency concerns. Truth be told, I hadn’t even realized there were dependency concerns.

Dependencies as clear as mud

The problems weren’t hard to see after he pointed them out though. We did have a mess, and it was caused primarily by poor dependency management. We had services depending on packages that should’ve been internal to other services, thereby breaking encapsulation. This meant that changes to the package could cause unintended consequences to the service that took on the dependency. Suddenly, the engineer working on that package had to be aware that there was another dependent, and had to make sure to update both services to accommodate changes to the package. Sometimes this also meant that both services needed to be deployed simultaneously, to prevent any breakages to the contracts between them. We had accidentally created a tightly-coupled mud ball where every change caused ripple effects and every deploy required deploying all the services in the monorepo at the same time. The more code we added, the more difficult it became to make changes or to deploy.

What had gone wrong? There wasn’t a lack of engineering leadership, since each team had very skilled, very experienced engineers. It also wasn’t an issue of knowledge, since the people in charge had taken great pains to stay up to date with industry best practices. We were moving from an on-prem set of services to cloud-native ones, so everyone was strongly encouraged to read about and understand cloud architecture, microservices, CI/CD, and so on.

“The [homomorphic] Force will be with you. Always.”

Looking back, the primary issue was our monorepo. Specifically, that the monorepo made boundaries less salient. It became easy to take a dependency on any code that was there, even if it should’ve remained private. A one-line change could take a once-private class and expose it to many other services for use. The change could be made right in the same pull request! Without regular maintenance on the CODEOWNERS file, there was little automated enforcement of the line between Team A’s code and Team B’s code. Ownership could become fuzzy, so it wasn’t always clear who had the final say on how a piece of code should be used. And to top it off, with multiple teams working in the same repo communication wasn’t perfect. Sometimes folks didn’t even realize taking a dependency would cause future problems, so they wouldn’t bother asking around. There were few structures in place to prevent ignorance or inexperience from damaging the overall system.

In short, the issues didn’t come from fundamental failures with the tech vision or the engineering team. Rather, the issues were inherent to the communication and collaboration structure we were using. The structure guiding how we wrote code affected the code itself.

This phenomenon is an example of what’s called the homomorphic force—an invisible pressure that causes one system to look like another. When that pressure is making your software system look like your organizational structure, we call that Conway’s Law. Conway discovered that software systems will always end up looking like the organization they’re built in, meaning that the organization is the “stronger” system in a sense, and the homomorphic force will always be pushing your software architecture to match it. So it didn’t matter how good our engineers were; the fact that we had multiple teams in one monorepo with no boundaries meant our code was doomed to start losing its boundaries too.

Before I go any further, let me make a caveat: monorepos are not inherently bad. There are companies that use monorepos effectively. Using a monorepo can be a great way for an organization to reuse application code, infrastructure code, and deployment configuration. Plus, it allows a lot of boilerplate to be written once, which saves engineering time and reduces copy/paste bugs. However, monorepos carry real organizational implications and therefore, due to Conway’s law, they bring real architectural implications too. Let me explain what that looks like in practice, as well as a way to use something I call the “mini-monorepo” to get the benefits of a monorepo while protecting the boundaries of your system.

Repositories are organizational markers

Let’s start by nailing something down: what makes an organization? Is it a manager? A reporting chain? Is an organization created by drawing circles on a piece of paper, calling those circles teams, and then convincing the people in the team to do standup together?

I’d argue that all of these things matter—managers, reporting chains, and regularly scheduled meetings with team members—insofar as they’re able to get a group of people to act like an independent, cohesive unit. These units are the true “teams” that make up your organization. If you do standup with the same group of people every day, but none of you work on the same code or the same projects, are you truly a team in this “independent and cohesive” sense? If no one talks or pairs on any work outside of standup or planning, is there really any cohesion? Or, let’s say the backend engineers on one team spend more time collaborating and designing with backend engineers on another team than they do with frontend engineers on their own team. What’s the real team unit here? Clearly it’s not enough to call a group of people a “team” on paper. There are many factors that go into creating a truly independent and cohesive team.

One of the most important factors is the way code ownership is understood and enforced. When a group collectively sees itself as the owner of a piece of code, you get cohesion. When it sees itself as the owner as opposed to any other team, then you get independence. However, this is just the understanding portion; enforcement is also necessary to make the understanding a tangible reality. The tangible reality then reinforces the understanding. Otherwise, as engineers join the team and leave the team, or as leadership changes, we run the risk of losing our understanding of code ownership.

This is where repositories become relevant: they are the fundamental unit of enforcement. With the right configuration, you can make a single team the owner of a repo, and you can require that team to approve any changes to the codebase. Further, thanks to automated CI/CD pipelines, this typically means owner approval is required for any code deployments. As such, the repository models and reinforces the team’s ownership of the code, which reinforces the team identity. And since the repo helps define the team, the repo helps define the org itself. It is an organizational marker.

The trouble with sharing

However counter-intuitive it may seem, we know a repository is an organizational marker capable of defining and reinforcing a team identity. It does this by making a shared understanding of code ownership a tangible reality. But, for those of us used to thinking in the other direction—that a team is the stable entity and a repo exists simply to store some code for the team—we might object that code ownership can be configured in a monorepo! Surely keeping code separate and owned by the right teams is just a matter of discipline. If we’re careful, then the teams will still be what they are on paper and boundaries will still exist between code components.

While I agree it’s theoretically possible, I’d contend that this approach denies the homomorphic force. It assumes that teams are able to create loosely-coupled architecture in a vacuum, free from the threat of organizational entropy. It sneers at our “sad devotion to that ancient religion”—but Conway’s Law tells us that’s simply not the case (and proceeds to Force choke the idea). Simply put, a monorepo makes boundaries less salient, which makes it easy for boundaries between domains and services to become less salient too. That leads to tight-coupling and generally bad architecture. If creating and enforcing boundaries relies on human beings continually exercising discipline in repo configuration, it will eventually fall apart. We are not creating a pit of success where it’s easy to fall into the right thing; rather, with a monorepo, we’re digging a pit of problems and dancing around the edge.

Here’s what I mean by organizational entropy, and how that erosion can cause us to fall into the pit (either of success or of problems, depending on which one you’ve dug). Let’s imagine for a second that you have a repo with a single service and multiple teams owning it. With all those owners, you’d think it’d be easier to get a review on a PR and ship things, or that on-call would be nicer with more people to share the load. However, the reality is often that having more people who can help means no one will help. This is called the Ringelmann effect, which notes that the more people there are in a group, the less productive each member becomes. Where a single, smaller team might’ve exhibited strong ownership over a service, having many teams “own” a service effectively means that none of them do.

This problem persists in a monorepo. If you take the lazy approach to repository permissions and pull request reviews, you just throw all of the teams who work in the monorepo on the CODEOWNERS file and let them sort it out. This basically guarantees no one will do that logistical work, which means the reviewer will become the closest person the author can find. This is exactly how ignorance and inexperience lead to taking on ill-advised dependencies, which then makes future changes really difficult.

If you take the disciplined approach and actually define which teams own what code in your CODEOWNERS file, you can force code authors to get reviews from team members who own the code before it gets merged in. This decreases the size of the group that owns a particular piece of code, which theoretically can make individuals more motivated to own it. However, you now need to make sure to keep that CODEOWNERS file up to date for the rest of eternity. As we know, any task that relies on a human remembering and doing it manually is bound to end up forgotten or done incorrectly. This is a large part of the problem statement that motivated infrastructure as code and automated CI/CD. In other words, that CODEOWNERS file will eventually fall into disrepair, and then it will become a bottleneck to the development process, and then it will be scrapped.

Here’s the point: don’t set yourself up for failure by creating boundaries by hand. They will erode over time, and eventually so will your team boundaries, and then so will your code. Instead, put a single team in charge of a repo. Then your team and code boundaries will stand firm. Without any effort at all, your structure will be promoting a loosely-coupled architecture.

The mini-monorepo: the benefits of reuse but with strict organizational boundaries

Finally, we’re in a place to discuss how to keep the benefits of a monorepo without accidentally digging a pit of problems. We know a monorepo can help us cut down on code duplication, and it can make it really easy to reuse code. At the same time, we know that reusing code across service or domain boundaries can be problematic. We also know that it’s best to have one team own a repo in its entirety. So, we need to push reuse as far as possible while staying within our constraints. The result is what I call a “mini-monorepo”.

The mini-monorepo is a repo owned by one team, that stays within one domain, yet contains multiple independently deployable components (what we might call “services”). If we’ve paid attention to our Clean Architecture, we know that we should have business logic (or what Uncle Bob calls “use cases”) depend on our core business entities, and then other interactions (like with a database, or an HTTP server, or a message queue) should depend on our use cases. Entities and use cases don’t need to know whether they’re being served in a REST API, or consumed from a Kafka topic. Those are implementation details that don’t affect our core business logic.

Because all dependencies point inward to our core business logic, a really nice way of organizing our repo emerges. We can write all of our business entities and logic once and tuck them into a package in our mini-monorepo. Then, all of our services (e.g. a REST API, a Kafka consumer, some background worker, etc.) can depend on that core package. Now, rather than services depending on other services in a circular mudball, we end up with multiple services all dependent on the same core set of logic. The services are separate, so changes to one service don’t affect any other services, and because they depend on the business logic (and not the other way around), changes don’t affect our business logic either. We have a dependency tree rather than dependency spaghetti, which minimizes the blast radius of any changes we make to our services. This makes them easy to work on in the future even as the codebase grows.

It goes without saying that it’s important to maintain this tree structure. If you start writing too much business logic and find that some of your components depend on part of the logic, but other components depend on a different part of the logic, that’s a hint that you’ve gone past your domain boundaries. Also, if services start depending on each other and not just on your core business logic, that’s another hint you’re causing problems for your future self.

If you stick to the mini-monorepo—one owning team, one domain, one set of core business entities and rules, and then multiple independently deployable components using that core set of rules—then you will be able to maximize your code and config reuse while maintaining important organizational boundaries. This will keep your architecture boundaries intact too. Your future self (and future coworkers) will thank you.