Systems programming

Posted on November 1, 2011 by Tommy McGuire

Labels: software development, c, programming language

People frequently come up to me and ask, "Bob, what is systems programming?" or, "Hey, Bob, what is a systems programming language?"

Those are more difficult questions than they appear. For part of the confusion, I blame John Ousterhout, the creator of the TCL programming language. TCL does not seem to get much use these days, but was one of the early, popular dynamic languages along with Perl, Python, and PHP. In "Scripting: Higher Level Programming for the 21st Century", Ousterhout wrote,

System programming languages were designed for building data structures and algorithms from scratch, starting from the most primitive computer elements such as words of memory. In contrast, scripting languages are designed for gluing: they assume the existence of a set of powerful components and are intended primarily for connecting components together.

I believe he was wrong on both counts. On the one hand, all programming languages are, or should be, designed for building data structures and algorithms; to do otherwise severely limits what can be done with the language, even more so than not being Turing complete[1]. On the other hand, designing a language for connecting components primarily written in some other language is a losing game since most users will not create separate components.[2][3]

What have Ousterhout's comments got to do with systems programming? His definition has, I believe, either led to (or been a result of the same cause of) many other languages such as Java being called "systems programming languages" when they rightly are not. The right question is not "What is the difference between systems programming languages and scripting languages", but "What is the difference between systems programming and applications programming?" To my mind, there is less real difference between TCL, Python, Perl, Ruby, and so on, and Java, Go, C#, etc., than between C and anything on that list.

So, what exactly is systems programming? It's hard to say. What are some of the characteristics of systems programming? That's an easier question. Systems programming usually involves the creation of software for use by other software, not by humans. Systems programming is almost always about managing resources for use by that other software. Systems programs may, and likely will, be built around complex data structures and algorithm and will also likely be built from the "most primitive computer elements" such as words of memory, if only because either

there is nothing below them to create "less primitive" abstractions, or
it is absolutely vital that their performance, or memory, or some other footprint not directly related to whatever the ultimate task is, be optimized.

Operating system kernels are definitely systems programming. As are device drivers. Compilers? Not systems programs. Interpreters and virtual machines? Maybe. Embedded programming is a bit of a special case: many of the tasks involved in embedded software may be similar to the tasks involved in applications, but because the resources available to embedded software are so relatively expensive, their management becomes essential to every part of the device. On the other hand, technological advances have moved some of what was embedded development solidly into the applications arena; witness smart phones in the past few years.

Here's a rule of thumb I came up with back when it seemed necessary to choose between C and one of those other, applications languages for some task: Does the problem statement explicitly mention having to manage your own memory? If not, then you shouldn't be using C, and you probably aren't doing systems programming.

James Iry made a very insightful observation about the nature of the C programming language:

Alan Perlis famously said "a programming language is low level when its programs require attention to the irrelevant." What's relevant depends on the problem and the domain, so a corollary would be that a language that's at too high a level prevents paying attention to some of what is relevant for your needs.

I believe that statement is insightful (and it is the casus belli for this post) because it touches on exactly the key of systems programming. What would it mean to write a memory manager or garbage collector in a language that has garbage collection? How could you use a language for systems programming if you cannot manipulate a device whose control system is mapped into memory at a specific location? Alternatively, why the stinking hell would you write an application in C if you didn't have to do some of those things?

I recently discovered a 2006 paper by Jonathan Shapiro, "Programming Language Challenges in Systems Codes: Why Systems Programmers Still Use C, and What to Do About It", describing the motives for BitC, a systems programming language that integrates advances of the last thirty years of programming language technology. Shapiro provides a very idiosyncratic, but also very accurate description of systems programs:

Systems programs operate in constrained memory.
Systems programs are strongly driven by bulk I/O performance.
Performance matters. (And as a corollary, data representation matters.)
Systems programs retain state. (And as a corollary, user-managed storage is a requirement.)

Shapiro's description of the first ("On a 32-bit [...] machine with [...] 64 gigabyte physical memory, it is extremely difficult to get the per-page book-keeping data structures to fit within a 0.5 gigabyte kernel virtual region."), second ("...the goal is to [perform] zero copies of that data [..., requiring] the creation of pointers that do not point to the beginning of an object [...or objects] multiply aliased in both the virtual and physical maps (e.g. by DMA) [...and sub-objects that] reference the storage of the original."), and third points ("...in the critical paths [...], the addition of only a few cache references — or worse, cache misses — in an inner loop can mean the difference between winning and losing customers") are largely what I mean by "resource management", specifically memory. Shapiro's fourth point is a little less obvious:

The most common pattern in systems programs is event loops. While there is short-lived data within these processing loops, the pattern overall is that there are (relatively) large amounts of state that live for the duration of the event processing cycle. This tends to penalize the performance of automatic storage reclamation strategies. To make matters more interesting, there are caches. These update frequently enough that “old space” techniques may not perform well, but are large enough that scanning them can be quite expensive.

Consider that the event loops he is speaking about may continue for the up-time of the device. My original thought was that implementing memory managers would be problematic in a language that provided automatic memory management; Shapiro is making the point that the presence of automatic memory management may be detrimental.

Oh, and when someone comes up to me and asks one of those "Bob,...?" questions, I always answer the same way: "My name's not Bob."

[Edit: Added the link to and paragraphs about Jonathan Shapiro's paper. Thanks to Eli Gottlieb and "The design and implementation of a modern systems programming language" for pointing it out (and Deca). And my name's still not Bob.]

[1] For example, TCL itself was designed with sole concern for glue capability, which leads to most of the reasons it has lost favor.

[2] TCL's glue capability created Tk, Expect, and AOLserver, but many more, complex, programs were simply written in TCL (likely using Tk) than were built by gluing other components together.

[3] Modulo command line shells such as sh, bash, etc., which are examples of programming languages as user interfaces. Also, the term "scripting" is much more appropriate for these languages. Complex applications in these languages are some of the most disturbing things I have ever seen.