Slow open source.
There’s this kind of feeling that open-source projects need to move quickly.
As if they were Silicon Valley startups. Without a constant stream of releases with new headline features, how can they compete to gain share?
It’s commonplace on projects to see questions about when a next the release is due, and even whether the project is still alive, if there isn’t this rapid update cadence.
These doubts are independent of almost any other metric you might choose to look at. A project can have been maintained for years, be super stable, have exemplary test coverage, CI passing against all the latest language and dependency versions, and all the rest of it — and yet still, there’ll be questions.
The heading here was slow open source. Needless to say, I think this attitude is backwards. I like my projects to evolve slowly. It’s a sign of maturity and actual usage. The last thing I want is my dependencies evolving like they’re the first few versions of a weekend side-project that’s never actually seen battle. God no.
If they’re actually adding value, our dependencies capture deep patterns in the domain we’re working in. They’re hard won, and they emerge out of practice.
If something is evolving rapidly, it may well be interesting, and exciting, but at a very deep level it says that we don’t really know what we’re doing here.
That’s OK. Exploring is one thing. As I argued, though, before, it’s best to avoid doing R&D in production. When it’s time for work, we need our dependencies to guide us into proven patterns for success. It’s how they do that I want to talk about today.
§
In searching one does not know in advance how — Piet Mondrian
It’s common that when we sit down to write a new feature, or a whole application, we don’t know what we’re doing. Not that we’re incompetent. Far from it. Rather, when we sit down to work, there are two kinds of uncertainty that are, it seems, essential to the process.
We neither know exactly what it is we're going to build, nor exactly how it is that we will build it.
And that’s totally normal. Fred Brookes1, emphasis mine:
The most important function that software builders do for their clients is the iterative extraction and refinement of the product requirements. For the truth is, the clients do not know what they want. They usually do not know what questions must be answered, and they almost never have thought of the problem in the detail that must be specified.
To be clear, it’s not that just a bit more thinking is all that’s needed here. More planning is almost always to the good but, there is an in principle unknowability: the only way you can find out what your program really requires is to actually write it.
Get something working. Then respond to that. How’s this? Yes, like that. No, not like that. And in this way, we elicit what we’re aiming at — both the problem, and its solution.
§
In the same breath, it’s not often that the outline of our application requires real technical innovation. We’re treading well-worn paths. We’re writing web apps, right?
For this, we have established design patterns, where…
Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, is such a way that you can use this solution a million times over, without ever doing it the same way twice. — Christopher Alexander2
The notion of design patterns draws some criticism. Historically, folks forget the without ever doing it the same way twice bit, and slavishly follow the particular examples, given for a particular language, at a particular point in time. Obvious problems arise.
The deeper insight is surely right, though. There are repeating general structures that emerge from the very medium we’re working in. Codifying and labelling those allows us to lean on them, at least long enough for the details of the concrete problem we’re working on now to emerge.
If a framework is a structure upon which we build, then, the role of our software frameworks is to embody those general patterns for our domain. To give us the foundations, and free us to work the details, on what’s unique in our particular case.
§
This is what I mean when I talk about Django having a grain — a way it wants you to do things. Django is built for the web, and its structure mirrors the natural patterns that web development wants you to follow.
On the podcast, Will and I have often joked that 95% of all apps is CRUD — the basic create, retrieve, update, delete operations.
In the same way as calling the very best technology Boring, this is slightly dangerous because it creates a misconception. Those not doing web development start thinking that folks are somehow earning big salaries for doing work that is essentially trivial.
The reality is (of course) that you earn the salary not for the CRUD, but for everything else, which is all the interesting and difficult bits, that come up on every project.
Nonetheless, it’s probably true, or not far off it, that CRUD is at least the starting point of almost all applications. It’s in laying down the structures around those basic operations that you start to see the real shape of the actual application you’re building.
It’s here that the slow evolution of the framework really matters. Django has been going for twenty years. In that time, it’s proven, millions of times, that it got the fundamental pattern of the web right. We know we can rely on that.
Innovations are often appealing. The latest framework. The new hotness. Yeah! They’re typically built specifically to address a felt problem with the established way — too much boilerplate, normally.
But a pattern isn’t just about what it makes easy. To go deeper, it also needs to expose the right flex points too. Does it have the appropriate seams, when you need to do something different? All too often, the new hotness will paste over the cases it just didn’t have the exposure to realise that it had to account for. Essentially, an attractive nuisance gets marketed as a new best practice, but is in reality a regression from the status quo.
Our domains are complex. That something might be done in a weekend is a joke. You might have something that starts to look like the easy bit in a weekend, sure. But the rest of it? That takes time. We use the mature libraries and frameworks because they’ve proven their worth, that they got the abstractions right. We watch, in wonder, as folks (and whole ecosystems) burn countless cycles jumping from tool to tool, before any of them have had the chance to be properly tested. We experiment, absolutely, but we don’t do R&D in production.
§
Django gets a brief mention above there, but none of that is really Django specific. It generally applies to whatever tool you’re using.
But Django tends to draw a decent amount of ire. Folks who know better don’t like any number of ways in which it couples the wrong things together, in ways that cannot possibly scale. And yet lets you solve the web problem quickly, as it’s done successfully for all these years.
I think it’s interesting to ask, how can this be? It’s not that folks are wrong. I can listen to a talk on application architecture or software design, and I can nod along all the way. Yeah, absolutely, separating this from that is a wonderful idea.
The only trouble is, it’s theoretical.
I wrote last year about Growing Your Business Logic With Django. About how we can migrate our business logic through our Django application, to manage complexity, as our application grows. I’m not going to repeat the argument from there. Just go and read it again. The conclusion was twofold.
As our application evolves, we approach the more sophisticated solutions that are held up as superior. But we do so as needed. More abstract solutions are not cost free. They’re more complex to implement, and they’re more difficult to comprehend.
We’ll grow to them if required, but many times we’ll find they’re not. Because we grew to them, rather than adopting them from the start, we can often rest at an earlier, simpler point that’s better tuned to the needs of our actual application.
In the very first Stack Report, I wrote about Greg Wilson’s Software Design by Example. I still recommend this.
Wilson talks about the difference in comprehension at different levels of abstraction.
[This] means that for any given task, the code that is quickest for a novice to comprehend will almost certainly be different from the code that an expert can understand most quickly.
But novice and expert are not static concepts.
Relevant to this discussion is that when you’re writing a new application, you’re a novice in that domain. Remember, we don’t know what we’re building, and we don’t know how we’re going to build it.
We need to aid our understanding, not make things overly complex. By keeping the abstraction level down, until we’re sure we need it, not only do we avoid over engineering, but we bring the most important task, the iterative extraction and refinement of the product requirements, closer to home.
Delayed abstraction is a superpower. It’s one that Django pulls off because the abstractions it does provide — Django’s grain — map so well onto the web domain.
§
I’ll finish with an example.
I have these things come up all the time. I often say how I still have fun using Django after all these years. The reason is that I’ll be working on a problem, and I’ll need a particular tweak, and then Django will just have exactly the hook, or function, or outlet, that I need. And then I’ll smile. After all these years, Django still surprises me. I couldn’t ask for more.
This one isn’t a surprise. More bread and butter.
I use Neapolitan’s CRUDView as a scratchpad when I’m laying down new functionality. You can do the same with Django’s generic class-based views, but Neapolitan lets me treat sets of views as one.
It’s on my roadmap to pull out from the work project, handling parent objects declaratively as part of the CRUDView API. To take Neapolitan’s example Bookmark model, you’d want to associate that with the parent User, say.
As I showed in the Growing your business logic piece, I like to encapsulate associating the parent object in the form class for the create view. This both ensures that the association is made, and keeps the view logic as close to the default patterns as can be.
From the view’s perspective, the key logic here goes into the get_form_class() method, where we define that form needs to take the parent object, and get_form() where we actually pass it for the create action.
Now, depending on the circumstance, get_form_class() can get quite complex. Then maybe I need to share that logic with another CRUDView. Two child models with the same parent, say.
At that point, we have to abstract. With tests in place, we pull out a helper function, that takes (probably) some values from the view. We adjust how we validate those. We tighten the constraints a little. We have a function that’s more testable than the view bound first version was. We add those tests. We use the new helper in the second view. It’s remarkably smooth.
You’ve all done this. It’s not special.
At each point, though, the process remains addressable. CRUDView gives you a namespace to throw down and evolve your logic. The standard view flows are easy to test (integration tests at first pass, those may well be). And in so doing, the needs of your application emerge quite naturally. No wonder it’s fun.
It’s not the only process, of course. But it’s a good one. And it’s robust. I wouldn’t worry if it’s not to everyone’s taste. The question is whether it’s to yours.