Podcast Review: Software Engineering Radio

I’ve been catching up on some podcasts while catching up on tasks around the home this past week. One of them is Software Engineering Radio, which unsurprisingly is about software engineering. The podcast invites on a lot of well known guests to talk about their experinces and their opinions.

Follows are some recent ones I have particularly enjoyed.

James Lewis on Microservices

If you have been closely following Martin Fowler and James Lewis’ microservice commentary, this won’t have anything that you haven’t heard before. But if you’re just discovering microservices this is an excellent summary of the architural style.

Gang of Four – 20 Years Later

The surviving authors of the seminal Design Patterns discuss their book, and its relevance today. They share their favourite patterns and their least favourites. I’ll give you one guess which pattern they regret. Probably my favourite interview on se-radio to date.

James Turnbull on Docker

An excellent introduction to its history and why you should care about this paradigm shifting technology. Another interview that probably won’t provide you any insights if you have been following the subject closely.

Josiah Carlson on Redis

Being a long time memcached user I’d never bothered getting deep into Redis. But after listening to this interview I am resolved to get more familiar with it. The interview goes through Redis’ features and design, advantages and drawbacks.

Stefan Tilkov on Architecture and Micro Services

A comparitively less bullish interview on microservices than the James Lewis interview. That isn’t to say that Stefan doesn’t think that the pattern isn’t a good idea, but he does come across a little more cautious than James. Stefan is also somewhat less prescriptive with his definition than James and Martin.

Mage Knight: Board Game Play Report #1

I’ve owned Mage Knight a long time and previously only played a co-op scenario. It is a long game with a complex set of rules. As the Shut Up & Sit Down guys explain below in their review it is a “gamers game”, not something non board gamers will enjoy. So I have been waiting for a chance to play it with a friend who knows the game as well as the right 3rd and 4th players. Today we got together for a 4 player game and played a competitive scenario.

All the players were experienced board gamers so the lead time was “only” around 30 minutes for setup and basic rule explanation. Once we got started the play was reasonably smooth, having an experience player to answer questions instead of delving into the dense rule books helped. We were soon exploring the hexes with ease, womping on the orcs in our way.

mage-knight

As things progressed however the game slowed down. Our decks expanded along with the map, giving us more choices, combinations and decisions. This combined with more player interaction ruining pre-planned turns made for a fair bit of downtime. And we weren’t even playing PvP rules which I imagine would have slowed down things considerably.

I found the game frustrating, mostly because I made a series of silly mistakes in part as I was learning, but also because I got distracted. I moved to a village and ended my turn but on my following turn I forgot I had done so, my character covering the icon. I then moved off the village towards the castle I had in my sights, ending my turn next to it ready for an assault on my following turn. Once it was my turn again however I was informed I would need to be on the same space as the castle to attack it.

It was then I realised my mistakes, wasting a lot of movement without gaining anything. I was most annoyed with myself. I proceeded to throw away a lot of good cards to get myself to the castle, and more taking it with a less than formidable deck. By this stage I was falling behind in levels and far from where the action (and therefore points) was. Dumb and unlucky, but I didn’t allow to foul my mood apart from a couple of choice swear words.

I cannot relate much of my fellow players experiences in detail. To me it seemed they had a target rich environment and the opportunity to pull of some fun combos and generally enjoyed themselves. As with my last play everyone found movement painful, getting about the expanding map being slower and more costly than one would expect. There were also complaints about downtime as well, passively watching someone pull together a turn with a set of cards we were only vaguely familiar with wasn’t much fun.

The next games I play are going to be 2 player and maybe try I will try the 1 player variant. I found enough satisfaction in the game with the promise of plenty more to continue with it, albeit aware of its weak points.

Yet Another Reason I Cannot Use Docker

I would love to use Docker but keep finding edge cases that block its usage. Today the hurdle is with our private repositories and docker build.

The sites we build have external dependencies on private libraries we have written, both Python and Javascript. These dependencies are currently fetched via mercurial/git over ssh, but at build time a docker container doesn’t have access to private keys with permission to access them. The current solutions I’ve seen are to copy in a password-less private key with access to the repositories you need. But then you’re in the horrible situation of either passing around a password-less private key or everyone generating one of their own. Both options being a security risk.

Someone raised a ticket 6 months ago addressing this, Forward ssh key agent into container #6396 which would nicely solve this problem by giving access to SSH_AUTH_SOCK. But only a little discussion has been had on the ticket since and it doesn’t look like it is even in the docker road-map right now. So I am left either:

  • Ignoring the security risks.
  • Taking on the ticket myself.
  • Providing another way of adding dependencies.

The latter is the way I will likely go as I am not happy with ignoring security risks and I don’t know much Go in order to contribute usefully to Docker. However it isn’t clear at this stage what other implementation can be used. Python has devpi but you still need a username/password to access private indexes. Bower can only pull things from Git repositories, so that one might be a real blocker. Though I at least know Javascript so adding a new way for Bower to retrieve packages might be an option.

Accelerando by Charles Stross

If you’re into AI or futurism I highly recommend reading Accelerando by Charles Stross, it is a brilliant example of science-fiction’s power to explore the “real” future. The first two thirds are the best living through the singularity fiction I have ever read. The progression is so reasonable, written mostly from the perspective of an entrepreneur working in the AI field. It actually gave me future shock at a couple of points and I had to put it down to work through the implications of what Stross had lead me through. A bit post-human, a bit trans-human and a bit cyberpunk in flavour but not transcendence thank god (pun intended).

Microservices: But What About Foreign Keys & Transactions?

One of the most impressively sized books on my bookshelf is Introduction to Database Systems, an artefact of my university years. It had an equally “impressive” price tag, or should that be “punishing” given my student wage? I regarded this expensive and weighty tome to contain unquestionable wisdom from on high, and in the very early 2000s it practically was. I had taught myself SQL in the past, but concepts such as normalisation, ACID and referential integrity were new and became to me immutable aspects of a good database.

From where we sit now, basking in the light of dozens of NoSQL database technologies (a terrible name, but we’re stuck with it), this is obviously not true. The hegemony of RDMS is over, document databases, graph databases and wide column stores are all widely used and acceptable options. With this revolution came the discarding of many of these “immutable” aspects, the argument being that by making this trade off an advantage is conferred. People make these same trade offs in RDMS schemas all the time, de-normalisation for example might be considered a cowboy move, but its read performance advantage is indisputable.

So what does all this have to do with microservices? Well trade offs have to be made, and this becomes obvious fairly early on, often when a developer is first introduced to the concept. With a distributed finely grained architecture spread across different databases and technologies, transactions won’t always be an option, neither will foreign keys. With the speed and agility that a microservice architecture provides this is the cost.

This is a scary idea, especially for those of us weaned on SQL, what will become of our data? First of all transactions. Is there a reason your whole system needs to be able to run in a single transaction? More often a web call will generate several data calls, but many of them are read only, and most of them touching only a few tables or rows. The read only calls probably can run outside the transaction and the others likely centre around some domain concept. That domain concept in turn probably makes sense to be collected into a microservice, which can then run a transaction. This won’t always be the case and hard decisions are inevitable somewhere along the line.

Giving up referential integrity is an easier task as it comes with a big reward. Removing foreign keys and replacing them with API calls means the owner of the data is free to change their internals. As long as the contract with the consumer of the API is obeyed then the owner can change as fast as requirements change, without the consumer having to also be updated. Databases aren’t the only line of defence for referential integrity, most applications we write already deal with this, often checks happen in a few layers as data travels through our systems. Without the database enforcing referential integrity we’re relying on our services and applications behaving correctly manner, something we already do to prevent errors in any case.

Everything old is new again, we aren’t dealing with new concepts, distributed systems have always had to face these trade offs. A micro-service architecture makes these trade offs more visible and explicit creating a tension developers must address. Even if a team chooses a more course grained approach, they’ve evaluated what is going to work for their project best and this can only be a good thing.

Design and Implementation of Microservices Workshop

Today I attended a full day workshop presented by Sam Newman and Scott Shaw on micro-services. Most people seemed pretty up with the why of micro-services, if you’re not James Lewis and Martin Fowler have talked at length on the subject. The workshop covered a wide range of topics and gave an excellent overview of the how and what of micro-services, something which is still lacking in literature (but is coming, see below).

I have seen Sam talk before, in fact his talk at Yow! last year was my first introduction to micro-services and I have read an early release copy of his book Building Microservices. A lot of the concepts were very familiar to me, however being a workshop meant that there was significant discussion around those ideas. This gave me a great deal of insight into the thinking around micro-services and how people have dealt with the trade offs and choices they present.

It was especially nice to have confirmation that some of the ways I am heading with Biarri’s architecture are tried and tested paths. This goes doubly for the contract testing library Pact (and Pact.rb) which a good friend of mine has been encouraging me to port to Python for Biarri’s use.

A few other books, videos and tools popped up in the talk which I have noted for further investigation:

  • Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans
  • Implementing Domain-Driven Design by Vaughn Vernon
  • Apache Zookeeper, useful for managing services, something like consul.io which I discovered last week.
  • Postel’s law: Be conservative in what you do, be liberal in what you accept from others (often reworded as “Be conservative in what you send, be liberal in what you accept”). A useful principle for micro-service communication.
  • Hystrix, a “latency and fault tolerance library” by Netflix. Probably overkill for Biarri’s architecture but it could come in handy in the future.
  • 12factor.net, SaaS app principles which I also discovered last week but re-affirms it’s usefulness.
  • A video by Stuart Halloway (I think) dealing with real time data and versioning. I did a google and couldn’t find a link so I am going to chase up Sam about it.

In all well worth my time.

Infracoders June Meetup

I’ve set myself the goal of attending one software meetup per week. I enjoy attending conferences, and meetups are like micro-conferences. You meet people, you learn something and often there is beer and food at the end like this evening.

This evening I attended Infracoders (http://www.meetup.com/Infrastructure-Coders/), a devops group focused on tools that make devops easier and more fun. Given my current focus on Biarri’s overhaul of infrastructure and development workflows these sorts of tools are on my mind a fair bit.

The presentations were interesting. First up was Alexey Kotlyarov and Ross Williamson from Infoxchange talking about some Docker tools called Pallet and Forklift (https://github.com/infoxchange/docker-forklift). Their talk was rather information dense and I found it a little hard to follow. From what I understood with my small knowledge of Docker is they automate some common development and deployment tasks in a platform/language agnostic manner. They look like they have a similar stack and problems to Biarri in many ways with lots of projects and a fragmented environment, so I will be looking into it deeper.

They also linked to some interesting things in their talk which are worth looking at:

  • Zato (https://zato.io/), a python ESB and application server. I am not clear on what its value proposition is yet but it has something to do with managing SOA which is something I am spending a lot of time thinking about right now.
  • The twelve-factor app (http://12factor.net/), a set of guidelines or principles for building SaaS applications. I’d not seen this before and at first glance looks like good reading.
  • Serf (http://serfdom.io), cluster management.

The second presentation was by Colin Panisset from REA Group about Credulous (http://credulous.io/), a AWS credential management system. He was an excellent and amusing speaker but Credulous solves a problem we don’t have and unlikely to have in the medium term at Biarri. It does look like a good solution if you have a large team with access to your AWS infrastructure. His lack of usage of GPG as a tool to solve the key sharing and encryption part of Credulous’ bothered me a little, but his criticisms of the installation and setup of GPG weren’t without merit. He did highlight some aspects of AWS security that I had not considered and will discuss with my co-workers.

The evening concluded with free beer and dumplings which was nice. I will certainly consider attending again, there seems to be significant overlap with the Devops Melbourne meetup, though perhaps their focuses are different.

2013 Year in Review

A pretty landmark year was 2013.

April 6 first of all, as it was the day I got engaged to the spectacular Li atop the almost as spectacular Mt Sturgeon.

Taken atop the nearby Mt William

The other major change was I donned Lycra and took up the sport of cycling. This happened later in the year but I already attempted and completed a ride through the Strathbogie Ranges with Orica-GreenEDGE [GPS record of my ride]. I was majorly stuffed by the end of the ride I can tell you.

Some good friends got married, or engaged, or had babies, there is a lot of that going around at the moment so I don’t think I’m alone in thinking is was a pretty big year.

Some Stats/Favourites/Lists

  • Favourite Book: Leviathan Wakes by James S.A. Corey
  • Books read: 24
  • Favourite Movie: I don’t know! Need to use Goodfilms more.
  • Favourite TV: Game of Thrones, Modern Family, Almost Human.
  • Kilometres cycled: 1,100.2 km (4 months of owning a bicycle)
  • Online courses started: 2
  • Online courses finished: 0
  • Conferences attended: 2 (PyconAU and Yow! Melbourne)
  • Conferences talked at: 0
  • New programming languages learned: 0
  • Favourite programming library: Angularjs
  • Girls of my dreams got engaged to: 1

So 2014?

Well the obvious one that will be dominating the first half of the year is getting married in April and then honeymooning in North Africa and Europe. That will likely consume the first four months of the year.

Beyond that I have some aspirations:

  • Read more books: 26
  • Talk at a conference: PyconAU?
  • Properly learn a pure functional language: Scala? Closure? Haskell? I will leave the term properly vague.
  • Complete a couple of online courses.
  • Finish one of my hobby programming projects.
  • Blog once a week.

The Lost Fleet by Jack Campbell

I don’t think I have read a more cheesy science fiction book since I was a teenager reading early Heinlein novels. The Lost Fleet series is pretty simple, a legendary war hero returns from the dead (found in a escape pod after 100 years) to lead a fleet trapped behind enemy lines.

The setting is well put together, two large human space faring civilisations in a drawn out war where neither side can hold sway. Each book is a series of battles in various solar systems as the fleet fights to escape largely intact.

These fleet actions are some of the best I’ve read, realistic, using believable technology and science. They’re sweeping battles spread across light hours of space employing clever tactics and it makes for some good clean fun.

However these epic battle broken up by some weak intrigue with characters who for the most part a little more than cardboard cut outs. Apart from the main character you never get a feel for the other characters, the few there are and what characterization there is can be stilted and painful at times. It really holds back the series from greatness which it a pity as the kernel is solid and enjoyable.

3/5 stars

Attack on Titan

I haven’t watched much anime lately, Planetes and Death Note were the last two I watched through. However a few friends have been raving about Attack on Titan and one of them posted an AMV trailer which made it look really fun so I gave it a try.

I was both happy and disappointed, first the happiness. It is a beautiful production, the setting is gorgeous, the character design is solid and the art work is world class. The plotting is well paced, able to introduce a lot of characters without my head spinning. However the plot itself doesn’t leave the familiar rut anime has trodden in the whole 20 years I’ve been watching it. It doesn’t stifle my enjoyment but it does make the diamonds in the rough, Planetes I am looking at you, shine just that much brighter.