A Single System Perspective
In an early issue of the Nomadness Report, I found myself discussing the mental model necessary to take on something so insanely complex as a homebrew starship of sorts. Companies handle this sort of thing by creating a hierarchy of departments and design groups, but I’m working more or less alone on this tour de force of geekery… and that is frankly overwhelming. I have already lived through some of the risks of this: watching completed systems become obsolete while spending years building the physical substrate that should have come first (Microship), getting seduced by gizmology and letting it overwhelm issues of usability (BEHEMOTH), diving into design without a clear internal “elevator pitch” to keep things focused (Shacktopus), buying gadgets long before they are really needed and then watching them go stale on the shelf (Nomadness). If it weren’t for the existence proof of well-executed successful projects (Winnebiko, Winnebiko II, Bubba, & Polaris), I’d start to worry…
So now that all the logistical impossibilities have been resolved and my lab is right at the head of the dock, I’m in the unfamiliar position of being able to work full-time on the Nomadness project. It’s almost like starting from scratch, though the design goals are already clear; I’m treating this fresh perspective as an intellectual luxury and making a deliberate effort to avoid the panoply of deadly phenomena cataloged above.
I thus opened the latest issue of the Nomadness Report with a philosophical piece on designing a complex system. Reader feedback has been most gratifying, so I decided to reproduce it here in its entirety… and some of this was later integrated into my Gonzo Engineering piece that wraps all this up much more tidily.
The Perils of Segmentation
As the conceptual components of this daunting project have been shoved to and fro over the years, they have by nature become segmented… and here there be dragons.
Imagine all the arbitrary boundaries and overlaps. Power is a good example: it’s one big chunk that has always been on my radar, further broken down into Battery Management and Distribution. But if you really think about it, both of those are woven so deeply into the rest of the ship that any attempt to draw a box around them immediately carves out (or calves off, depending on your perspective) sub-categories more logically associated with other systems.
This may not sound like an important issue, and indeed there are clear enough “edges” when it comes to physical panels and schematics that I don’t spend much time fretting over ephemeral classification ambiguities. But it still leads to a sort of sloppiness in the project-management domain: is this piece of the puzzle a packaging task, a power distribution job, or just the upstream DC connection to that subsystem over there?
It may seem that the more neatly I can parse this huge system into a tidy hierarchy, the easier it will be to visualize… and thus, to build. But not so fast.
Fundamentally, what we are talking about when designing any huge interconnected system is the need to reduce complexity. If a project has the opposite effect, then it ventures into the domain of wankage – cool in a geek sense, but not very helpful when it comes to effective day-to-day operation.
It does not follow that a segmented design process will necessarily yield a choppy user experience, but things do have a tendency to go that way despite our best intentions (I’ve done it). The components take on lives of their own, with their various interfaces and affordances defining our view of them… as well as the way they interact with each other. Indeed, we must think of things with this kind of object model, lest the entirety turn into a giant mass of spaghetti.
The problem is that a complex system presents design challenges at every level of magnification. This easily translates into a “sliding scale,” with one extreme a well-conceived object-oriented chunk of code (where development tools and geek-cultural norms enforce discipline). But when you step back from this utopia of bits, you find a box with hard edges and arbitrary behavior, nestled into a complex application environment that incorporates not only other subsystems developed in parallel, but multiple incarnations of unrelated concepts from third-party vendors, orthogonal cultures, previous technological generations, cobbled gizmos of necessity, and random glue.
Suddenly the elegance of the Big System is a function of the level of magnification with which you approach it, and, well, that’s just confusing.
On Nomadness, this phenomenon might manifest itself as an elegant yacht with traditional user-interface affordances (lines, pedestal, winches), augmented by a complex multi-generational aggregation of marine electronics: chart plotter, autopilot, nav/weather instruments, radar, power systems, engine controls… interfaced, if at all, via a melange of protocols new and old. In the background is a panel of circuit breakers, and scattered here and there are little control panels, valves, through-hulls, and stand-alone electronic modules… none particularly well documented.
Added to this, thanks to years of obsessive effort reflecting designs and purchases scattered along the timeline of a rapidly evolving industry, we might find a second technological overlay: a console system in which are embedded a variety of radios, computers, networking tools, audio production gear, lab equipment, and homebrew gizmology… along with a database-backed server with a dozen or so nodes around the boat contributing data. Atop all this (assuming I get around to it while still alive) is a rich layer of software intended to bring it all into a single conceptual space.
And it’s all intended to reflect the dream of a cohesive integrated system, reminiscent of the starship Enterprise. “Computer… report!”
OK, what’s wrong with this picture, other than sounding like something that will never get finished?
An Experiment in Conceptual Cohesion
A key component of my work on previous technomadic substrates (most notably, BEHEMOTH) has been the avoidance of wheel-reinvention through the expedient of taking one product after another and incorporating them into a general toolset. This typically involved serial and audio crossbar networks, hacking power supplies and original manual controls to allow load shedding and software-driven operation, and otherwise shoehorning a diverse range of gadgets into a single environment.
That’s pretty much what I’m doing here, and in some ways it’s easier now… most of the gadgets that I need have better tools for remote control than their counterparts of a quarter century past. With a signal-agnostic crossbar network and a bigger power budget to accommodate things that need an IP stack, it should not be too difficult to make anything talk to anything.
But here’s the problem: all that is limited to my own geek overlay onto the existing ship. Naturally, that will incorporate the NMEA2000 network and I/O from independent subsystems, but one of the biggest chunks of this whole project boils down to plain old interfacing: using Arduino nodes and trivial sensors to bring everything under the aegis of the central server (see the “Shacktopus Overview” in Issue #3 for more context). I’m not fundamentally changing the design of all the systems made by other people, just slurping them into my microworld.
This is a considerable oversimplification, though it appears to be the solution to the problem: interface the hell out of everything, wrap it in code, and cast off the docklines with a tablet-based GUI added to the list of traditional yacht affordances. Tighten that winch. Head up a few degrees. Pinch to zoom that video image from the masthead, and verbally inquire about the ETA at our next waypoint.
But let’s look at how all this impacts the development process, including the ongoing maintenance issues of replacing ailing or obsolete subsystems. If we get to control every aspect of the design, then this would not be a problem… but I am annoyingly finite, and would not reinvent all these arcane wheels even if I had the time and skills to do so.
Back in the Microship epoch, I developed my network of controller nodes in FORTH… a lovely vocabulary-based language that embodies a “bottom-up” programming methodology that makes testing trivial and enforces clear chunking of functionality. I found myself adapting the same philosophies to hardware design, trending toward clearly defined circuits that do one thing well and have precise interface specifications.
We don’t have that luxury when trying to assimilate huge complex systems made by other people… so the broad character of this project is starting to emerge: All we have to do is become intimately familiar with every system (both now and in the future), write the tools to bring it into our current paradigm, create new hardware for every un-augmented device or data point of interest, create a database standard and a metaphor for dealing with this “internal model of the ship,” and cobble interface code that works from every connected device ranging from tablets and phones to voice and packet radio. All that, and packaging too.
This convoluted philosophical sketch brings us back to the point of this article: if we’re going to take on something this absurdly ambitious, it is essential that we think of it correctly. If there is an attempt to accomplish all this by outlining a collection of isolated to-do list items, each defined by the flavor of the target device and the software tools du jour, then it is going to be nigh impossible to avoid a gradual emergence of the scenario I imagined above… so many different concepts in the same system that I can neither keep it all in my head nor successfully document it.
The solution, I believe, comes down to the way every task is contextualized. This almost sounds like HMB (Hip Metaphysical Bullshit), but it’s true: when we build machines, they become the crystallization of our dreams. Get the dream right, avoid letting the tools shape our work, and keep the big picture in context… and we at least have a shot at creating something that reflects our design objectives.
What does this mean in pragmatic day-to-day terms as I resume focus on the boat? Just this: Nomadness (or any huge convoluted system) is not a collection of subsystems as much as it is a single entity. The temptation to treat the project as a massive hierarchy of stand-alone sub-projects is dangerous, for the inevitable conceptual drift will change the flavor from one to the next. The solution boils down to two things:
- Keep a clear dream uppermost in mind, even if that requires reducing it to an “elevator pitch” so it is not vulnerable to creeping featuritis.
- Make sure every task is colored and shaped by that dream, and is not allowed to wander off into the seductive domain of its own intermediate objective.
Let’s put a practical spin on this, then get back to specifics!
Engineering in a Nutshell
This little snippet about how engineering really works is from my Gonzo Engineering essay, a sort of geek manifesto. Here’s how to manage a huge, complex project:
- Accept going in that your first tentative decomposition of the fundamental concept will yield an over-simplified TO-DO list, distorted by misunderstanding of key issues.
- Avoiding all the items labeled TBDWL (To Be Dealt With Later) or ATAMO (And Then A Miracle Occurs), dive headlong into the well-defined parts, finishing some of the electronic design so early in the game that it is guaranteed to be obsolete before the physical substrate is built.
- Blunder ahead on the non-obvious parts, getting pleasantly distracted by learning curves and occasional moments of certainty, only to discover basic flaws in your reasoning.
- Now that you are forced to re-think the initial concept, map it onto newly recognized reality to yield a fresh TO-DO list (with new lab notebooks and computational tools to keep things lively) and another cycle of enthusiastic activity
- Repeat steps 3-4 countless times at varying levels of abstraction ranging from the entire system down to individual components.
- Meanwhile, since technology evolves with frightening rapidity, acknowledge the fact that any computer-based system is such a moving target that if it’s not completed quickly, it will be irrelevant by the time it ships.
- Respond by simplifying the design, further refining your objectives and abandoning dead-end ideas while doggedly pursuing others that have come to represent too large an economic or emotional investment to allow a graceful retreat.
- Compromise here and there, bang out a few things that weren’t on the list, then add them and cross them off to make yourself feel good.
- Get totally sidetracked a few times, and periodically dive into major development marathons to meet public deadlines like trade shows, pulling all-nighters in PFD mode (Procrastination Followed by Despair).
- Announce new completion dates whenever a previously predicted one has passed, and keep driving your PR engine to maintain interest during a process that is a textbook illustration of Hofstadter’s Law (“Everything takes longer than you expect, even when you take into account Hofstadter’s Law.”)
Part of this development heuristic is just sloppy management, but it also reflects the way we think. This is why engineering is, at its heart, an art form (and why the average completion time of a homebuilt boat is 137 years).
Perhaps the most interesting thing about this seemingly ugly process is that it’s iterative and self-correcting. Grandiose or stupid ideas may not be obvious during first-pass blue-sky analysis (when the project is glued together by wishful thinking), but it’s another story entirely when it all has to be converted into Clearly-Defined Tasks (CDTs) and drawings that make sense to machinists. Without some kind of closed-loop intellectual process to fine-tune your thinking, it would be impossible to get to the point where you can start using engineering tools to convert fantasies into contraptions.
Trying to shortcut this by starting on Day One with formal design methodologies can have the catastrophic effect of committing you to an ill-defined goal state, whereupon the end result is shaped more by your toolkit than by the supposed objective. That’s why so many products seem malformed, patched, and otherwise inelegant: management loves formal methods and looks askance upon such frivolous notions as approaching product design as a delicate blend of art and engineering. The exceptions, when they occur, are a joy to use. The rest miss the point, no matter how stylish their exterior or sophisticated their underlying technology.
So it appears that designing a system isn’t nearly as rigid a process as typical engineering textbooks would have you believe. Your component choices affect the shape of the thing you’re building; said shape in turn creates constraints that affect your choice of components. Such psychological race conditions can only be resolved by tweaking the granularity knob while adding inputs to your evolving mental model, until the correct solution congeals in a flash.
It’s easy, and here’s how to do it: Prop your feet up on your desk, relax, and form a fantasy of the desired results. Now turn it slowly in your head while calmly examining it from all sides, allowing input variables to float until an unanticipated combination satisfies your psychic fantasy-comparator and generates a flash of recognition. Since all your noodling is naturally saved in a big circular buffer called short-term memory, let this recognition event pre-trigger a snapshot of the conditions that immediately preceded it (before accumulated pondering-propagation delays introduce conceptual drift). There’s your design specification. Take that and run with it.
This is probably not an engineering methodology that makes managers comfortable, though it’s a good summary of life in the trenches. There is a pervasive myth that structured methods and sequential procedures, used in isolation, will get you there… but I’ve never seen it work that way. The tools don’t actually start to become useful until you’re quite thoroughly immersed, and that can take weeks of appearing, to outside observers, as if you are loafing.
You must log in to post a comment.