API Rules for designers: use one encoding, not two

 

API Rules for designers: use one encoding, not two

 

"If you haven't tested your code, it's probably wrong"

 

Today's bad API example is from the Bluetooth Heart Rate system. Most of the protocol is fine: you get a notification every so often with updated heart rate information (e.g.,  heart rate in beats per minutes (BPM) plus optional stuff for overall energy and the "RR Interval" for fine-grained heart data).

 

Testing this protocol is extra expensive because the heart rate (BPM) can be sent in two ways. It might be a single byte, and it might be two bytes. There's a single bit in a flags structure to say which way it is. If the person's heart is 255 BPM or less, it's one byte. If it's 256 or more, it's two bytes. The app code (and device code) has to handle this.

 

For an app, the code is hard to test for two reasons. Most importantly, a human doesn't ever have a heart rate that high (according to Copilot, "255+ BPM is possible, but it’s always pathological"). The second is that even if I could find a person with a heart rate that high, I have no confidence that any specific consumer-grade device will ever produce this data. As just a person coding in their spare time, I can buy a couple of heart monitors. But imagine I still worked at Microsoft: what are the changes that a VP (that's the clearance it takes for approving purchases) would sign off on an unknown number of devices on the chance that one of them would produce this data?

 

In the end, I've got code in my new Bluetooth app (not on the Microsoft app store yet) to handle the special flag. But my confidence that it works isn't great.

 

The right API choice was to always send 2 bytes of data and not try to make something in the name of spurious efficiency.

 

For the nay-sayers: the awkward protocol doesn't support any interesting new scenarios, and doesn't have any appreciable amount of energy.

 

If they had kept the protocol as-is, developers would have the same abilities. The RR Interval data size would shrink from 9 entries to 8 -- but in typical use, there's only 3 or 4 entries. And if there's a case for needing more than 8 entries, the Bluetooth device could simply transmit more often.

 

Takeaway: Protocol designers should always include "how will a developer test their code" when considering complicated APIs.

 

 Everything wrong with the FINGER protocol 

For those of you who have never heard of it, Finger is one of the old "litle"¹ TCP services. As a user of a big multi-user machine, you can edit the ".plan" file in your directory; people can then run a command like finger person@example.com and it will retrieve your .plan file along with other information like where and when you last logged in. It was a super useful way to coordinate with teammates back in the days before cell phones had been created. 

 The protocol itself is pretty simple: the finger command sends a single line of data with the user name, and the server replies with a bunch of text and then closes the connection. So what could go wrong? In this minor screed, I list both things that should have been known at the time, and also things that we know about protocols today that weren’t known then. 

TL/DR: the spec is wrong, confusing, incorrectly implemented and potentially dangerous. But other than that, it works pretty well :-) 

The protocol spec is incorrect (/W). 

 Firstly, the finger spec, RFC  1288, is wrong. The "BNF" query notation, section 2.3, with query type #1, attempts to allow an optional /W before the user. The /W is the verbose switch (W stands for "whois") and servers can reply with more information when it's provided. (This is accessed by the finger -l person@example.com switch; -l stands for long). But that's not what the BNF actually says. What the BNF says is that the /W switch is required whenever a username is provided. What should be an optional switch into a mandatory one. 

Good news! Every actual finger client implements what the spec tried to say and not what it failed to say. Which is good, because a number of existing (as of February 2021) Finger servers implement the earlier RFC 742, which doesn’t allow the /W switch. 

The protocol BNF is clumsy. 

The protocol “BNF” in general is more formalistic than useful. There’s an old saying that every level of indirection makes code harder to follow; the corresponding saying for BNF is that simple and common definitions like CRLF should be spelled out each time they are used, not hidden behind a layer of naming indirection. The BNF also loves using short name; {C} is the name of the rule that eventually expands to CRLF, and {U} the rule for user names. 

Additionally, the BNF is split into two rules: one for direct user lookup, and one for an indirect network lookup (these are Q1 and Q2 in the BNF). But this makes the Q1 clumsy, as it has to handle both user lookups with no user, and user lookups with a user. A better split would be three query types: a NULL query (with or without a /W), a user query (also with or without a /W) and a network query. 

On-behalf-of is not good networking 

We can totes forgive the original spec from adding in the slightly weird “Q2” format. This format is used when we're asking server “A” to ask server “B” for information. It’s like the user can’t get the information they want directly; they have to go through a gatekeeper server. The other servers are called Remote User Information Program (RUIP). Back in the 1970s when the RFC was created, the internet was often provided to a single computer at a site; the site then used other protocols and network to connect to other computers at the site (hence the internet used to be described as a “network of networks” which were expected to use non-Internet protocols). 

But in modern times, the Q2 “on behalf of” experience isn’t needed. Indeed, none of the servers I found would handle it. 

Massive security issues 

 Finger servers often return the time and location of user logins. For example, FINGER might say that a particular user is currently logged in at a particular terminal in a particular room. This is handy when dealing with friendly teammates, but is totes wrong when dealing with stalkers and worse. Lots of people really don’t want other people to know where they are. 

Giant compat issues with modern servers 

You might be confused by this one – what could I possibly mean about modern Finger servers? Have there even been any modern Finger servers at all? Why would anyone build a new Finger server given that the Finger protocol is often blocked by firewalls and provides very few features needed by people. 

It turns out that just looking on GitHub shows a bunch of different Finger servers. These servers are mostly derived from the original RFC 742 Finger protocol. It’s almost the same as the RFC 1288 Finger, but doesn’t allow for the /W switch. Other servers attempt to handle the /W switch, but don’t do it correctly (finger.farm, for example, failed until recently).  

One more thing about the /W switch spec: case-insensitive

[Later edit]: the RFC set of specs has long declared that just strings in the BNF descriptions should always be assumed to be case-insensitive: "monday" is the same as "Monday" and "MONDAY" and "MoNDAy". The FINGER spec takes the opposite approach: the /W switch, AFAICT, is actually case-sensitive and should always be upper-case.

As a fun aside: the RFC editors are, in the instance, wrong. While I understand why they decided that BNF should be case-insensitive (it's part of our text-based heritage), it's also the case that the workaround they use (specify case-sensitive strings as hex characters) is demonstrably error-prone. I've personally filed about a half-dozen different bugs against Internet protocols for getting the HEX representation of strings wrong.

The best solution is to require each BNF description to say if they are case-sensitive or not.

Use these learning for your own protocols! 

Finger is part of the old tradition of text-based services that are almost designed for direct command-line manipulation. As such, it’s now mostly out of favor (when was the last time you read your email by directly talking to a POP server?). That said, there are still lessons from FINGER for today. 

• Simple, direct protocol descriptions are easier to debug than complex ones. 
• Be aware of bad actors. Don't let your APIs enable stalkers and thieves. 
• Make sure that the easy path for handling your protocol also allows servers an upgrade path. 

 

Note¹: Finger is one of the litte TCP services noted in RFC 848 along with echo, discard, systat, netstat, quotd chargen, finger and a couple of time-related services. 

 

A Network Reading List

I get to be  mentor to an incoming program manager where I work. Most newcomers aren't super familiar with networking, and many people aren't familiar with the actual history of our part of computer technology.

Here's a hopefully useful curated list of useful links to source material on the creation of the internet.

The first internet was created for ARPA by BBN

The American Deparment of Defense "Advanced Research Projects Agency" (ARPA) asked for the internet in this original request. The winning proposal was from Bolt, Beranek and Newman (BBN). About a decade later, BBN wrote the Completion Report.

Ton of BBN history is online.

RFCs and Protocols everyone should know

The internet is governed (as much as it is governed) by a set of "Request for Comments" (RFCs). There's also this handy list for all 8649 current RFCs. These are all numbered; there's a society that curates them. However, anyone can be part of the RFC process. In addition to RFCs, there are also "IEN" paper with their own numbering

One of the most accessible papers (and a great way to start) is the Endian RFC, published by Danny Cohen of the Internet Engineering Task Force (IETF) as IEN 137. A good introduction to network humor is RFC 1149.

As you might guess, creating an RFC can be a long, slow slog, starting with basic proposals in a proposed state, getting consensus from experts, and converging on a draft standard and finally being voted in as an official internet standard. You might also guess that that many important protocols have gone all the way through the entire process, but here you would be wrong. The IMAP Email standrd 3501 (from 2003) is still "proposed".

You might also think that conforming to the official standards is critical. Actually, not so much. 

Internet Mail

EMail was one of the earliest super-popular uses of the Internet. There are two main ways to read email: the Post Office Protocol (POP) formats POP (RFC 918) , POP2 (RFC 937) and the current and most popular POP3 (RFC 1939) and the much more powerful / complex IMAP (currently IMAP 4.1 RFC 3501 from 2003). Sending email is via SMTP, which is a different protocol, and calendars use CALDAV (or other)

The Internet before HTTP

Telnet (RFC 318) lets you type into a computer here and have it be accepted over there. Now replaced with SSH

Gopher (RFC 1436) is kind of like HTTP/HTML, but is more of a "menu" oriented system. There still exist Gopher servers, and there's even a yearly conference. Amazingly, there's only one Gopher program for Windows in the Windows Store (and I wrote it)

Usenet News (RFC 977) is like a distributed Reddit but without reputation points.

FTP (RFC 2065) is how we used to download files. Unlike many other protocols, two sockets are used: one is for sending commands, and the other is for getting a response. This actually works really badly with firewalls. FTP is also a great example of RFCs being updated: the original RFC is 114 and then runs up to 172 265 354 542 765 959. After that there are merely a string of updates. Also notice that the orignal email protocols were handled by FTP.

Finger (RFC 742) is what was facebook. People would write information in their .plan file and it would be retrieved.

Internet Relay Chat (IRC) (RFC 1459) is like Teams (or text messages). Unlike some of these other protocols, plenty of people still use IRC.

The internet and HTTP + HTML

HTML is the markup language; HTTP is the protocol for sending HTML over the internet. It's HTML plus a bunch of headers. HTTP and HTML was first created in 1991 by Tim Berners-Lee. The NeXT computer he used when creating it is on display at the Science Museum in London. 

HTTP is defined originally in RFC 1945 which also defines Uniform Resource Locators (URLs)

HTML is defined in RFC 1866 and heavily updated since then.

Security and the internet

A common thread through all of the original RFCs is a complete lack of care about security. For example, RFC 475 says:

It has been suggested that FTP specification should require that mail
   function (for receiving mail) should be "free", i.e., FTP servers
   should not require the user to "login" (send the USER, PASS, and ACCT
   commands).

Furthermore, note that all communications are send in the clear with no encryption whatsoever. Amazingly, a recent Senate proposal for all states to email a number of sensitive documents to a senate committe proposed using a completely in-clear mail system!

Reading the Transport Layer Security (TLS) RFCs is not recommended. They are very complex, and won't help you understand the protocol. You should know that the current common TLS version is 1.2 and the new one is 1.3 (but it's not in common use yet). TLS 1.0 is barely acceptable for security and is expected to be completely broken before long. The old Secure Sockets Layer (SSL) is completely broken and must never be used if you want security.

VPN and WiFi access points often used RADIUS (RFC 2059) for authentication and authorization. One neat thing about RADIUS is that it's designed to be secure, but thanks to advances in breaking encryption, it's not at all secure on the internet.

You'll see tons of stuff about X.509 certificates. The most important thing to know is that all certificates are X.509. Other than that:

1. A certificate is just a thin wrapper for a security key with some extra goo
2. People handled their certificates incorrectly so often (e.g., using the code signing cert for TLS) that they are now marked with the intended purpose.
3. The Public Key Infrastructure (PKI) is a way to 'chain' from one certificate to the next. This is always done so that there are just a few super-trusted certs and everything else flows down from there.
4. There are custom mechanisms for trusting a certificate. Azure uses a custom mechanism inside the datacenter; for their purposes it's much faster, more robust and more secure than using the standard PKI
5. Combining #3 and #4: the bar for inventing a new scheme to validate certificates is spending three month with the security experts. 
6. "Browser level" certificate checking is pretty secure. The entire chain is examined; each cert along the chain is validate for expiration and to make sure that each cert correctly signed its part of the chain.
7. Adding to browser-level certificate check is safe (security-wise). 
8. Bypassing the browser-level certificate check is not safe. People screw this up constantly.

Read The Most Dangerous Code in the World for how common security mess-ups are.

Multicast -- it's not really hard at all

So, I've been trying to learn some multicast programming. What I really want, of course, is to run a program on one computer in my local network (otherwise known as "home"), and then discover it on another computer.

So I figure multicast is the answer. And, in truth, it is! It's great! It's even simple! But every single silly blog post about it -- especially from the vendors -- makes it sound hard.

Here's the real deal on Multicast. It's just UDP. You send it to an IP address like 224.0.50.5. The exact values are pretty unimportant. And the port is just a port, like 9883. The server (listener) is just a hair more complex: bind to the same port, and set the "multicast join" socket option to the IP address.

Tada! Packets sent by the sender go to the listener. And you can reply from the server back to the sender by looking at the remote IP address.