One of the most impressively sized books on my bookshelf is Introduction to Database Systems, an artefact of my university years. It had an equally “impressive” price tag, or should that be “punishing” given my student wage? I regarded this expensive and weighty tome to contain unquestionable wisdom from on high, and in the very early 2000s it practically was. I had taught myself SQL in the past, but concepts such as normalisation, ACID and referential integrity were new and became to me immutable aspects of a good database.
From where we sit now, basking in the light of dozens of NoSQL database technologies (a terrible name, but we’re stuck with it), this is obviously not true. The hegemony of RDMS is over, document databases, graph databases and wide column stores are all widely used and acceptable options. With this revolution came the discarding of many of these “immutable” aspects, the argument being that by making this trade off an advantage is conferred. People make these same trade offs in RDMS schemas all the time, de-normalisation for example might be considered a cowboy move, but its read performance advantage is indisputable.
So what does all this have to do with microservices? Well trade offs have to be made, and this becomes obvious fairly early on, often when a developer is first introduced to the concept. With a distributed finely grained architecture spread across different databases and technologies, transactions won’t always be an option, neither will foreign keys. With the speed and agility that a microservice architecture provides this is the cost.
This is a scary idea, especially for those of us weaned on SQL, what will become of our data? First of all transactions. Is there a reason your whole system needs to be able to run in a single transaction? More often a web call will generate several data calls, but many of them are read only, and most of them touching only a few tables or rows. The read only calls probably can run outside the transaction and the others likely centre around some domain concept. That domain concept in turn probably makes sense to be collected into a microservice, which can then run a transaction. This won’t always be the case and hard decisions are inevitable somewhere along the line.
Giving up referential integrity is an easier task as it comes with a big reward. Removing foreign keys and replacing them with API calls means the owner of the data is free to change their internals. As long as the contract with the consumer of the API is obeyed then the owner can change as fast as requirements change, without the consumer having to also be updated. Databases aren’t the only line of defence for referential integrity, most applications we write already deal with this, often checks happen in a few layers as data travels through our systems. Without the database enforcing referential integrity we’re relying on our services and applications behaving correctly manner, something we already do to prevent errors in any case.
Everything old is new again, we aren’t dealing with new concepts, distributed systems have always had to face these trade offs. A micro-service architecture makes these trade offs more visible and explicit creating a tension developers must address. Even if a team chooses a more course grained approach, they’ve evaluated what is going to work for their project best and this can only be a good thing.