Avoiding R&D in Production
Since the days when computers first came into common use, there must have been tens of thousands of accounts receivable programs written. There are probably a dozen or more accounts receivable projects underway as you read these words. And somewhere today, one of them is failing. — Peopleware, DeMarco & Lister
We software developers are notoriously tribal. Whether it’s our editor, or language, or framework, or whatever else, we’re somewhat prone to… well… let’s just say… debates about the minutiae of our relative choices.
As a young freelancer, I could never quite grasp why these finer details wouldn’t come across to a prospective client. Frankly, they didn’t give a damn that we were going to use Technology A, that I was proposing, rather than Technology B, featured by a competitor.
— Is it somehow not possible with B? — Oh, no B can do it (just as well, I guess).
And, of course, it can. What does it really matter if you use Python, or Ruby, or Go, or whatever, for your web application backend? In the Python world, whether you use Django, or Flask, or FastAPI, the same. Nothing hangs on it, not from the client’s perspective.
What they’re asking is, first, is this software going to get built at all, since project failure is still all too commonplace. Then, assuming that, they want to know what the ongoing commitment looks like, what’s it going to cost to maintain?
The key to unlocking that wasn’t technology. It wasn’t language or framework features that mattered. It wasn’t even about the bottom line — price is a thing, sure, but it’s rarely the main thing. Rather, it was about risk. If you could show that your approach was the safer — across the whole product lifecycle — you’d often win the bid. It’s that I want to discuss today.
Choosing Boring Tech
There’s a maxim that circulates, from a now classic Dan McKinley talk and essay, to choose boring tech.
What counts as boring? That’s a little tricky. “Boring” should not be conflated with “bad.” There is technology out there that is both boring and bad. You should not use any of that. But there are many choices of technology that are boring and good, or at least good enough. MySQL is boring. Postgres is boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. Cron is boring.
The nice thing about boringness (so constrained) is that the capabilities of these things are well understood. But more importantly, their failure modes are well understood.
The idea is to select a small set of established technologies — such as Django and PostgreSQL — on which to build your business. It’s a proven way, what Craig Kerstiens has described as The Money Stack, and one that’s highly influential.
There are two aspects to choose boring tech that, I think, are important. The first is focusing your skills, and those of your team.
Each technology you pick up has a cost. Let’s say you’re using PostgreSQL for your application database, (1 point). Next you want to add full-text search, so you add Elasticsearch, (2 points). Then there’s a push to add vector-based storage, to power some new AI magic, so you search around and go for Pinecone, (3 points).
Here you’ve got three technologies, to learn, integrate, and maintain. Instead, you can leverage the tech you already have, and get more bang for your buck.
So you still use PostgreSQL for your application database, (1 point). But then you decide to keep using Postgres for your full-text search, (still 1 point). And when that AI requirement comes up, you decide that Postgres is still your best option, (and still 1 point).
Here, instead of three technologies, you’re using just one. The cost to learn, and integrate the new features is minimal in contrast to picking up a whole new tool. It’s something you’re already maintaining; the marginal increase is again likely small. Your team is gaining in expertise in a tool you’re already using. You’re levelling-up rather than diluting your skills.
Now, maybe there’s some reason why the specialist tool might outperform your existing one — why Elasticsearch might outperform PostgreSQL, say. But can you make that case? Does it apply already at the point you’re at? Is that benefit worth the cost of taking on the extra dependency? If you can ask these questions, your analysis — and your proposal — is already at a deeper level than if you just run for the specialist tool without considering the options already available to you.
§
The second aspect, then, of Choose boring tech is the relative lack of unknowns.
For a mature technology like nginx, or PostgreSQL, or Django, almost everything that ever could go wrong in production already has, to many people, many times. Those failure modes are well understood, and solutions are only ever a search away.
The deployment and scaling patterns are standard. You’re never inventing anything here. It’s boring in exactly the right sense in play here. That’s good precisely because it frees you to focus on the novel parts of your application, rather than on rabbit holes that people have been down a million times before.
It’s here that choose boring tech ties back into your proposal because it enables you to frame your discussion in terms of risk-management, for which it’s quite likely your competitors haven’t presented, or perhaps even considered.
Two kinds of risk
The standard risk is implementation. How likely is it that you can get this built, within a given timeframe, for a given budget?
In Shipping Software on time and on budget, I suggested that one powerful technique for task estimation was to provide a best case, a worst case, and then a most-likely estimate of how long a task will take. You can apply some math to those estimates, but, the very process of thinking through the three outcomes helps make your most-likely estimate much more realistic than it would otherwise be. (Tl;dr: You ask a programmer for a most-likely estimate, and you’ll get the best case automatically, unless you take measures to make sure that doesn’t happen.)
Programmers notoriously hate giving estimates. Part of that — again argued in the essay linked above — is often that you’re being tasked with estimating for delivery when you should be doing discovery, but let’s assume we’re past that.
The other reason programmers hate estimating is that they’re trying to give, particularly, worst case and most-likely estimates for a novel technology, that they don’t really know, for which they have no grounds on which to base such a number. (Best case is generally easier: New and Shiny invariably promises to deliver in half to two-thirds of the best case estimate of Old and Boring; exactly that amount to be captivating, without triggering the No Silver Bullet Alarm 🚨)
You ask me about a small feature addition in Django. I can say: best case 2 days; worst case 5 days; most-likely 3 days; and a weighted probable of 3.5 days. What’s more, though, I can put my money where my mouth is. Because I am confident in the worst case estimate, I can offer the client a bounded maximum cost quote, that de-risks the implementation for them. Because I am confident in the most-likely estimate, I know there’s a good chance I walk away from the project with my profit margin intact.
In contrast, even often for simple features, with a novel framework, that I don’t really know, I can perhaps offer the fairy-tale best case scenario1, but when it comes to worst case, and hence most-likely, it’s not clear that I can say anything at all. We’ve all been in situations with an unknown technology where some unforeseen complexity had arisen, and implementation has taken multiples of what we’d originally thought. (We’ve all been in situations where we’ve failed entirely.)
With the novel technology, I simply can’t offer the client a bounded maximum quote. I’m asking them to shoulder the risk of my novel technology choice.
The second risk then is operations. For a new service, a new web server, a new application protocol, a new database, for example, we have no idea how it’s going to scale.
For The Money Stack, nginx in front of gunicorn running Django, using PostgreSQL, the scaling patterns are entirely known. You can tell the client how much a single box costs, how much traffic that will serve, when they’ll need to scale, and how much that will cost. With New and Shiny, the only reference is back to well, this is what it costs with a more traditional approach.
Again here, the problems are important. With the established tech, they’re known; with the newer options, you cannot rule out running into some novel operational issue that can cost an unspecified amount to resolve.
In both the cases of implementation and operations, by choosing established technology, you’re able to present your proposal with realistic bounded maximums that you can commit to. In a competitive tendering situation, merely presenting that confidence, and the reasons for it, can be enough to put your competitors on the back-foot. Business owners aren’t interested in tech — not per se — but risk? Yes, risk they understand.
So we never innovate?
The original Choose boring technology essay noted that the maxim contained the seeds of its own reductio ad absurdum:
Taking this reasoning to its reductio ad absurdum would mean picking Java, and then trying to implement a website without using anything else at all. And that would be crazy. You need some means to add things to your toolbox.
Elsewhere, I’ve seen it described as reactionary and (even) bordering on anti-intellectualism. I don’t know quite what the latter piece there is arguing against — it presents a similar model to the original essay as far as I can understand — but likely it’s pragrammers. We are, as a bunch, so incredibly likely to take a nice sounding maxim, and then proceed to apply it.
Like all these things, choose boring tech needs to be used with sensitivity. Software development is a good business. If we have any hopes that it will stay that way, we need to keep advancing the state of the art.
As a pragmatic concern as well, you need to give your team room to experiment. If you don’t, they’ll do it anyway, but it’ll be on a client project, where (as we’ve seen) it matters that they don’t.
A quick search suggests that the big tech firms spend maybe between 5-10% or 15-20% of revenue on research and development (R&D). At the higher end there, that corresponds to a day a week. Roughly Google’s old 20% time.
Let’s say you put that aside for experimentation, for R&D in your team. Possibly, some weeks it doesn’t quite happen — perhaps there’s a project commitment, say. But most weeks, yes. So you sit somewhere between the higher and lower ends of the scale. Not a bad place to be.
Then set a norm that the core of that time be about keeping up to date.
Take Django. It’s archetypal boring tech: it’s mature, it’s stable, and so on. But it’s also pushing forward every single release. The sweet spot for Django is to be on the latest major release. Not only does that make for the easiest upgrades, but it gives your team access to the new features as they arrive, and those go straight into your development practices, keeping your offerings and your team’s interest fresh.
Same with Python, same with PostgreSQL, and so on. The new versions offer constant feature and performance improvements that you can leverage just by staying up to date, just by being in that habit.
That’s the sweet spot too for riding the wave as technology changes.
Vector DBs come out, to take an example from earlier. Within a short period — shorter than you can ever productise one of the specialist offerings — extensions become available for PostgreSQL, that you can experiment with, without taking on the cost of that whole other tech. In Django, the new (exciting) third-party packages always target the latest version first2. And so on.
Your boring tech choices aren’t standing still. There’s no reason you need to either.
Then, over the medium term, nice and slowly — at a sensible pace — your team can experiment with the other things that they’ve got on the radar. Some of those you’ll bring in, some of those — most of those, likely — you won’t. That’s totally normal. That’s R&D.
The one thing you’re not doing is R&D in production.