Code That Built the Internet: The Impact of BSD, Part 2

The first article in this series introduced BSD and its tight relationship with the internet. We continue looking at particular features in this final part.

The Most Important Digital Communication Medium

Nowadays, we have many protocols and media for sharing information with our friends, our colleagues, and the world. There’s traditional phone text messaging, chat programs, and numerous collaboration platforms competing to guide us through our workday and apply AI to relieve us of tasks.

But for most of the history of digital computing, people depended almost completely on email, which was invented before the internet in 1965. Even today, many services require an email address to sign up and log in, with the assumption that email is a universal, unique identifier.

Let’s see what BSD did for email.

An enormous number of protocols preceded the Simple Mail Transfer Protocol (SMTP) that internet designers ultimately settled on. SMTP is indeed simple, following the same style as the FTP and Telnet protocols that preceded it, and the famous HTTP protocol that followed. Everything is plain text, making it easier to code applications, transmit data, and troubleshoot problems. Every message starts with header lines that give information such as the sender, and continues with a body containing the payload (the user’s message).

But designing a mail program was not so simple, because there were so many alternative mail systems that had to interoperate.

Eric Allman’s sendmail program was first released (under the name delivermail) in 1979. This predated the official existence of SMTP, which was defined in an RFC in August 1982. Merely parsing email formats was a nightmare task. Most of the formats existed before the IP protocol stack, let alone the Domain Name System, and had their own idiosyncratic routing strategies. For instance, the popular uucp protocol sent messages from one host to another using a „bang path,“ which separated the hostnames with exclamation marks.

sendmail handled everything. The complexity of the parsing was exacerbated by a configuration format that emphasized terseness, responding to the cramped space in 1980s memory and disks. It’s no surprise that a typical sendmail configuration looks like an encrypted espionage report.

And the brevity of the sendmail configuration format was complemented by the loquaciousness of the documentation. O’Reilly’s classic sendmail book (originally penned by Allman himself) kept growing and eventually topped out at more than 1,300 pages. But for a long time it was a must-have for system administrators.

The only statistic I can find for the popularity of sendmail was gathered in 1996, when sendmail ran on 80% of the reachable SMTP servers, far outstripping any other mail server program.

We shouldn’t leave sendmail without a tip of the hat to another key role it played in computer history: facilitating the first major distributed, internet-wide cyberattack.

The role of sendmail is significant because it had a known security flaw for years. The feature containing the flaw could be disabled, but many administrators didn’t bother to do so. It was an obscure flaw, to be sure, but it had been identified and left unfixed. Although sendmail was nominally open source, it was not developed on the basis of community contributions, like most modern free software projects. Instead, like most BSD components, it was the bailiwick of a small team.

In the late 1980s, I worked at a Unix vendor named MASSCOMP. It was an informally organized company where employees routinely sent email on any topic of interest to them company-wide. In one such mailing, a programmer got a number of peeves off his chest. After complaining about particular tools, about how projects were managed, and about corporate decisions, he ended with a comically flippant remark, „Why is the bug still in sendmail?“

It was this bug, I assume, that was exploited by a graduate student in 1988 to launch a program that „went viral,“ traveling from computer to computer until it brought down about 10% of hosts on the internet.

Like many first-time security breaches, this „worm“ abruptly shifted threats that were abstract and theoretical into the realm of the immediate and the practical. And while developers try hard to deal with security vulnerabilities, many users lag at updating their software, so known flaws still lead to successful attacks.

Fertile Ground for Computing Innovation

BSD has had an indelible impact on modern operating systems and programmer tools, separate from the networking contributions described in this series. Besides what I’ve covered in this series, a few of the notable contributions of BSD include:

• vi: the powerful text editor used by most advanced users on BSD, Linux, and other Unix-like systems
• Jails: the predecessor to the container concept that underlies Docker and Kubernetes
• Berkeley Packet Filter (BPF): a framework for running user-space programs inside the kernel, now ported to Linux and used for many advanced features
• termcap and curses: C libraries that create a graphical interface on a plain text screen (no eye-rolling, please; this was a big deal in the 1980s, and undergirded vi in particular)
• BSD Make: a complete rewrite of the old AT&T Unix build system, providing a comprehensive set of tools for managing program builds that’s very different from the Autoconf/Automake suite that uses the GNU version of Make to build the Linux kernel

BSD was the key product licensed by perhaps the most successful and historically important Unix-based corporation in history: Sun Microsystems. Founded in 1982 by Bill Joy and other key BSD contributors, Sun was the vendor of choice for researchers and other professional Unix users. The company invented two fixtures of computing in its time: the NFS distributed file system and the Java programming language.

So why didn’t BSD become the universal free operating system? For that story, see my four-part series From Unix to Linux: Key Trends in the Evolution of Operating Systems. In particular, part 3 of the series surveys the hurdles that blocked BSD’s dissemination.

Still, BSD thrives in some contexts. It plays mission-critical roles in many large companies. There are administrators who praise its stability and performance, and who wouldn’t exchange it for Linux for all the penguins in Patagonia. Companies that use BSD at the heart of their products include Apple (which released the essential, BSD-based part of its macOS operating system as the open source Darwin project), Network Appliance, Netflix, Juniper Networks, New York Internet, Sheridan Computers, Beckhoff, Metify, rgNets, Cloudium, Antithesis, ecard, and the Sony Playstation. Many companies don’t publicize their use of the BSD operating system.

And people who have chosen other operating systems still rest on the legacy of BSD, because it was on BSD that the internet grew to maturity.

<< Read the first part of this series