This month I've been reading: Software Design by Example

Samuel Johnson said of Paradise Lost¹ that it…

… considered with respect to design, may claim the first place, and with respect to performance the second, among the productions of the human mind².

That's pretty high-praise.

But it doesn't help with the core problem, which is, How on Earth am I going to read the thing?

Good as it might be, C17th epic poetry is not easy going. The language is difficult, it's long, and the topic is (at least, now) more than a little obscure.

That — difficult, long, obscure — sounds a lot like most books on programming. They might be good, but really? (I glance at the shelf. Compilers, Principles, Techniques, and Tools looks down at me with a knowing smile. I'm pretty sure your shelf has similar.)

The trick with Paradise Lost is that it's divided into books that are specifically written to be the length that it's reasonable to read in a single sitting.

Half the poetry is in the rhythm, the metre. With that, and with the intricacy of the plot, if you get disturbed when you're reading it, you're likely better-off going back to the beginning of the bit you're on, and reading it again, rather than trying to pick up the thread of where you were at — exactly what Satan was trying to do, and to whom, at the point you left off.

Let's say a book of Paradise Lost takes you roughly an hour. You set aside that time, you get it read, and you get a natural pause, at just the right time, just when you need one.

Which brings me to my topic.

I've been reading Software Design by Example: A Tool-Based Introduction with Python, by Greg Wilson. I can't recommend it enough.

First-off it's open source, and freely available online.

• The site is here: <https://third-bit.com/sdxpy/>

• The GitHub repo has all the examples: https://github.com/gvwilson/sdxpy

I bought the print edition — which is well put together and pleasing to both the hand and eye. It's a handsome thing:

I sometimes get ebooks or use online versions, but mostly I'm reading at bedtime, and if I don't have a book, I won't read it. I don't want more screens at that hour.

Then — despite computer books being difficult, long, obscure — Software Design by Example (SDXPY) is a pleasure to read. It gets into some advanced topics quickly: you're building parsers, test runners, validators, template engines, package managers, databases, web servers, … — but it never overwhelms.

I'll read for perhaps half-an-hour to forty minutes. In that time I'll read a chapter, or early-on maybe two. And then I can stop, having covered the topic, and having reached a natural break.

You're never left mid-way, unable to finish the chapter. The quality of exposition is exemplary. SDXPY builds up in bite-sized chucks that you can read in a single sitting.

So, if you take nothing else, go read SDXPY. Highly recommend.

The Big Ideas

I wanted to discuss the "big ideas" behind SDXPY's approach, that are talked about in the introductory chapter.

Complexity

First, as the number of components in a system grows, the complexity of the system increases rapidly…

Meanwhile, our ability to hold things in our head doesn't (ever) change much. So:

If we want to build large programs that we can understand, we therefore need to construct them out of pieces that interact in a small number of ways. Figuring out what those pieces and interactions should be is the core of what we call “design”.

That's from page 2. You hit that, and you think, "Oh, yes! We're in for a good ride".

Determining the components that comprise our system is the core of what we do as programmers. Every function, every class, we're constantly thinking about, and refining the interfaces that we're building upon. There are two thoughts here:

• That such thought is the process of design accounts for so much that's both wonderful about being a software developer, and also that we fret about. Even in the best case you never have a design that you simply then go and implement. Rather, the active thought means we're creating that design as we go. We're designing whilst making. No wonder it's such fun³.

• Then, the prospect of a book showing that design process in action, with examples that are by any standard, non-trivial, in easily digestible chunks… – well, that's catnip. I'm all here for that, and I'm assuming you are too.

Comprehension

The second big idea in SDXPY is that experts and novices have different levels of comprehension for a given level of abstraction.

One example, that's short and clear (from Chapter 8 on Functions and Closures), gives this code:

env.append(dict(zip(params, values)))

The exercises say to rewrite that line using a loop to insert parameter names and values in the dictionary. "Do you find your rewritten code easer to read?", it asks?

I'll tell you plain, after all these years, I still find the for-loop easier to grok first time. (There's something about zip that I always have to think about.)

Regardless of your particular answer there, though, this difference in comprehension, Wilson says: 

… means that for any given task, the code that is quickest for a novice to comprehend will almost certainly be different from the code that an expert can understand most quickly.

This has obvious implications for teaching code (and for books like SDXPY itself). Novices need different examples to experts. But, also, expert is relative. You put someone in front of a new code base, they're a novice too.

In technical docs, we often seek for what we'd do in production. We want examples that are realistic. But is that what really serves?

We're used, from Diátaxis, to the idea that different parts of documentation need to play different roles, are we learning or applying, doing or thinking?

Folks coming to the docs to learn need to see good patterns, sure. But can we craft the introductory examples, so they show the idea, without requiring that you already have lots of other context. Can they spot what's important right now, leaving other things for a bit later?

In giving a conference talk, you know your audience (whilst technically savvy) aren't familiar with your code: if you don't simplify — and radically — you're not going to get your point across.

And then there's future you. Coming back to this code, you're the novice here again. Are you really able to understand what was going on (in reasonable time) if you don't keep it relatively simple? You don't have to answer: I'll put my hand up for you "No".

Metaprogramming

The third (and final) big idea of SDXPY is that "programs are just another kind of data".

How do you build a test runner? Well, you introspect to build a list of test functions, and then you run them. How do you build a linter? You walk a tree code objects and examine each one. And so on.

In just shy of 30 – short, totally readable, remember – chapters, SDXPY shows you the basic design of all the tools you might think of. Git? Yep. A database? Yep. A package manager? Yep.

All the while, these powerful techniques show that source code is just text, that a program is just a data structure. It's impressive. It's liberating.⁴

This kind of metaprogramming – treat programs as data — isn't new to us. Of Django, its Models, its forms and their factories, the admin, and so on all leverage introspection in order to work. Indeed Django includes such as part of its Design Philosophy:

Django apps should use as little code as possible; they should lack boilerplate. Django should take full advantage of Python’s dynamic capabilities, such as introspection.

But such doesn't come without cost. You open up Django's ModelBase.__new__() — it's hairy. That same Django page has a warning against too much magic: "Magic shouldn’t happen unless there’s a really good reason for it".

Wilson:

Treating code like data enables us to solve hard problems in elegant ways, but at the cost of increasing levels of abstraction in our programs.

Increased abstraction means less comprehension: that was big idea two.

Once again, finding the balance is what we mean by "design".

He's giving us these powerful metaprogramming techniques, but with concrete examples, he's asking us to think about the costs of using them too.

Wilson is giving us the tools to think about that in general. To think about how to design software.

§

I look back at my shelf of books on computers and programming. It's my profession; I get a lot of them; You always need to be learning. But there's only so much shelf-space. I've got rid of a lot of books over the years too.

Of those books that remain, there are a few that stand out, as having changed the way I think about development. It feels like a lot of those are pretty old though⁵. I feel like I've been waiting a while for a book might join them.

It's maybe too early to say but, in case you couldn't tell, I'm really enjoying Software Design by Example. I think it might be that book. I think it might earn its spot on the shelf.