Directory entry says what? Current Gopher type field types

Ready to dig into the details of the modern Gopher ecosystem?  

The Gopher network protocol is like the precursor to modern HTML web browsers. Like HTML, a Gopher client could display links and text, download files, and display images. Unlike HTML, a Gopher client is using very spare, without the opportunity to display colorful, interactive displays. 

A typical Gopher screen is a directory, a menu-like list of lines, each of which does one thing. A line might display some text, or be a link to a file to view or download, or an image, or might be  a link to another Gopher directory, possibly on another server.  

The Amadeus Gopher Server, served up by my own Simple Gopher Client

I started looking more into modern Gopher sites and how they actually work when creating my own Simple Gopher Client for the Window Store.

 The Gopher RFC 1436 lists 14 different directory entry type fields, each of which is given a single letter identifier. 0, for example, means that the entry refers to a text file that can be displayed; 1 stands for a link to another gopher server. The 'g' type field is for a GIF file and 'I' is for an image type (but the type isn't explicitly given). Uniquely, directory entries can point to other protocols: '8' for a Telnet server, '2' for a particular phone-lookup protocol, '7' for a search engine, and 'T' for an IBM 3270-style terminal connection. 

Over the years other type fields have been informally added to the list. I recently did a crawl of the Gopher space as it exists in February, 2019 to see what kinds of directory entry type fields are in current usage across the current Gopher space. 

Many Gopher files are served up using generic descriptions and not the more precise descriptions. Type 9, "Binary file" is the most common, followed by type 'd', document (a modern addition that's not part of the official Gopher spec). These two account for 87% of the non-image files served up by Gopher. The third most common type is type 5, DOS Binary, followed by BinHex, PDF and UUEncoded. 

Something similar happens with picture image formats. The 'I' generic image field type is used about 30x more often than the more specific 'g' GIF field type. 

Looking at the most popular field types, the 0=file, i=information and 1=directory are the top three field types by far, accounting for about 90% of the field types. 

Some of the original field types are hardly present. The "T" type field that indicated a IBM TN3270 style interaction is entirely missing. The type field '2' CSO Phone book lookup is present on just a 4 pages total, but most of them seem to be samples of what a CSO phone book would be like, not a real phone book. There are actual field type '3' error pages, and no surprise, they seem to result from correctly handling errors from the scripts that generate some Gopher pages. There are also no Duplicate Server '+' type field entries. 

 (Note: I removed from the numbers pages that are test pages whose purpose is to validate Gopher clients) 

Type Fields (Alphabetical order) 

I'll finish this blog post with a handy table of existing Gopher tag types. Every tag that was found at least 10 times, or is part of the official RFC, is listed here 

Field  Count  Type  Status  ;  11  Video  +  0  Duplicated Server  RFC  0  60976  File  RFC  1  29335  Directory  RFC  2  5  CSO Phone  RFC  3  36  Error  RFC  4  223  File (BinHex)  RFC  5  631  File (DOS Binary)  RFC  6  3  File (UUEncoded)  RFC  7  257  Index-search server (Veronica)  RFC  8  479  Telnet  RFC  9  4799  File (binary)  RFC  D  12  Some kind of binary file?  d  1590  File (document)  g  102  Image (gif)  RFC  H  4  h  3914  HTML Link  I  3300  Image  RFC  i  13216  Information  M  115  Mail file?  P  26  PDF File  p  15  Image (PNG)  s  278  Sound  T  0  IBM TN3270  RFC  w  9  Wiki edit link

Field  Count  Type  Status  ;  11  Video  +  0  Duplicated Server  RFC  0  60976  File  RFC  1  29335  Directory  RFC  2  5  CSO Phone  RFC  3  36  Error  RFC  4  223  File (BinHex)  RFC  5  631  File (DOS Binary)  RFC  6  3  File (UUEncoded)  RFC  7  257  Index-search server (Veronica)  RFC  8  479  Telnet  RFC  9  4799  File (binary)  RFC  D  12  Some kind of binary file?  d  1590  File (document)  g  102  Image (gif)  RFC  H  4  h  3914  HTML Link  I  3300  Image  RFC  i  13216  Information  M  115  Mail file?  P  26  PDF File  p  15  Image (PNG)  s  278  Sound  T  0  IBM TN3270  RFC  w  9  Wiki edit link

Field  Count  Type  Status  ;  11  Video  +  0  Duplicated Server  RFC  0  60976  File  RFC  1  29335  Directory  RFC  2  5  CSO Phone  RFC  3  36  Error  RFC  4  223  File (BinHex)  RFC  5  631  File (DOS Binary)  RFC  6  3  File (UUEncoded)  RFC  7  257  Index-search server (Veronica)  RFC  8  479  Telnet  RFC  9  4799  File (binary)  RFC  D  12  Some kind of binary file?  d  1590  File (document)  g  102  Image (gif)  RFC  H  4  h  3914  HTML Link  I  3300  Image  RFC  i  13216  Information  M  115  Mail file?  P  26  PDF File  p  15  Image (PNG)  s  278  Sound  T  0  IBM TN3270  RFC  w  9  Wiki edit link

Field  Count  Type  Status  ;  11  Video  +  0  Duplicated Server  RFC  0  60976  File  RFC  1  29335  Directory  RFC  2  5  CSO Phone  RFC  3  36  Error  RFC  4  223  File (BinHex)  RFC  5  631  File (DOS Binary)  RFC  6  3  File (UUEncoded)  RFC  7  257  Index-search server (Veronica)  RFC  8  479  Telnet  RFC  9  4799  File (binary)  RFC  D  12  Some kind of binary file?  d  1590  File (document)  g  102  Image (gif)  RFC  H  4  h  3914  HTML Link  I  3300  Image  RFC  i  13216  Information  M  115  Mail file?  P  26  PDF File  p  15  Image (PNG)  s  278  Sound  T  0  IBM TN3270  RFC  w  9  Wiki edit link

Field	Count	Type	Status
;	11	Video
+	0	Duplicated Server	RFC
0	60976	File	RFC
1	29335	Directory	RFC
2	5	CSO Phone	RFC
3	36	Error	RFC
4	223	File (BinHex)	RFC
5	631	File (DOS Binary)	RFC
6	3	File (UUEncoded)	RFC
7	257	Index-search server (Veronica)	RFC
8	479	Telnet	RFC
9	4799	File (binary)	RFC
D	12	Some kind of binary file?
d	1590	File (document)
g	102	Image (gif)	RFC
H	4
h	3914	HTML Link
I	3300	Image	RFC
i	13216	Information
M	115	Mail file?
P	26	PDF File
p	15	Image (PNG)
s	278	Sound
T	0	IBM TN3270	RFC
w	9	Wiki edit link

First lessons learned: Magnetometers!

Magnetometers -- how do they work?
I'm having fun with an NXP Rapid IOT kit as part of a Hackster.io contest. This is the first time I've really tried to deal with a magnetometer and how to convert the seemingly random output into something that can be used for a compass. Here's what I've learned!


You have to calibrate your code to match the magnetometer values. I simply looked at the range of X and Y values as I rotated the IOT kit (range in this case meaning to grab the highest and lowest values). This gives you a midpoint for the X and Y values.

Calculate an adjusted X and Y: Xadjust = (Xmidpoint - Xraw) where the values have the obvious meaning. I suppose a better adjustment would also divide by the overall range so that the X and Y values are also normalized, but so far I haven't needed to do that.

Calculate an angle with the always handy atan2(x, y) function. You might remember from trigonometry class that the tangent of an angle is equal to the opposite side of the triangle divided by the adjacent side. If you know opposite/adjacent, then you automatically know the tangent of the angle, and by using the arc-tangent, can get the original angle back. In computers, we use atan2 and provide the X and Y (aka, adjacent and opposite) values directly; otherwise, there would be a divide-by-zero problem when the adjacent is zero (aka, when pointing magnetic north/south)

In computers we always get the angle in radians; multiply by 180/3.1416 to get the angle in degrees. No, that value for PI is a little off, but take a look at the results you get; they jitter by several degree no matter what.

But then there was a big discovery: my laptop has magnets that really, really interfere with the readings! This isn't helped by the short USB cable provided!

TL/DR: compass heading is atan2 (xadjust, yadjust) where the adjusted values are relative to the actual midpoint that you measure. You have to do the measure locally; it's not a globally-valid amount all over the world. The phrase "all over the world" includes "especially near a modern laptop with magnets"

Fun historical fact: iron ships included "deviascopes" to adjust the magnetic compass and to offset the residual magnetism left over from building the ship. A ship's magnetic field will potentially change every time the ship goes in for servicing (and on battleships, every time the big guns are moved!)

Another grumpy post, this time about Application Insights

What is Application Insights, and why is it awful?

Hey, maybe for some people it's not a terrible time suck. Maybe for some people, it's providing value. How many ways has it sucked up my time so far? Let me count the ways:

1. When I made my Windows App, Microsoft not very helpfully added a reference to it in the project. I spent time to understand what it was (it is supposed to send up "telemetry" to some dashboard somewhere). I decided to keep it because I didn't want to offend the Awful Scary Microsoft Monsters.
2. When I got scary email saying that I'm about to spend hundreds of dollars a month to get some alleged insights because Microsoft decided that hosting a ton of data was expensive, and they didn't want to foot the bill forever. Luckily my actual usage is almost nothing (I think).
3. When my updated projects simply refused to compile. Except when they did compile. But they never compile when I'm making an app store
4. And just now, when I finally ripped off the bandage, and remove the Application Insights packages from my app.
5. Oh, and while I'm at it, I figured that I might as well look and see what awesome insights the Application Insights is giving me. I turns out that whatever data has been sent up is gone; when I go to my Azure Portal, there's a grand total of no data at all, or even a hint that any data has ever been sent up

Total wastage: too much.

Ethereum uses what base encoding?

Base 58 is about the dumbest thing ever

I've been learning Ethereum (because, you know, bitcoin). Being a networking kind of person, I'm looking at the networking protocols. Let's leave aside questionable choices like using Keccak-256 which can be argued are "forward looking" and not "completely unsupported by major languages".

No, lets look at encoding. Each Ethereum address is, of course, a big binary number. It's written out as hex (arguably silly, but whatever). It's then translated using, not base-64, but using base-58. As far as I can tell, this is something they just made up.

I'll ignore the lack of support in major languages.

Base-64 has the nice property that it's exactly 2^^6. This means that one byte transforms into one complete Base-64 value with 2 bits left over. Three bytes transforms neatly into four output bytes. A reader or writer can deal with small, finite-sized, easily handled values.

Base-58, on the other hand, is 5.858 bits. That's nothing useful. It means that any hand-crafted library is more likely to be wrong than to be right. The supposed benefit? So that a few characters that might be misinterpreted can be dropped.

That “google” screed

My email-protocol team learned that some servers mishandle one particular part of the email protocol relating to calendar appointments.  Did we give up, and just ignore customers trying to use those servers?  No!  We dug in, and worked around the problem.  That’s what computer programmers do: we find problems whose solution is hard, and we solve them anyway.

There’s been a recent document written by a person at Google about how women are genetically different from men and make worse programmers.  The author would rather sit and complain about the world instead of fixing the real problems in it.

Is their statement (“women are genetically different”) even true?  There used to be a wave of studies all about how different men and women are.  But there’s a more recent wave of studies that looks at the question sideways: “how easy is it to reverse the gender effect?”  It turns out that the answer is that it’s pretty darn easy; that seemingly trivial and cheap changes are all that is needed to reduce or eliminate differences as seen by common tests.

Even there a genetic difference, then what?  I have genetically bad eyes.  And with a hundred dollars of glasses, I get the same 20/20 vision as anyone else.  Am I to be condemned to a miserable, non-computer programming existence for the sake of some glasses?  My height is genetic, too, but it’s also helped by a good diet when I was a kid.  Just because part of an effect is beyond our control doesn’t mean that we give up all control; it means we have to apply the levers that we have.

Worse, it’s clear that genetics can only play a limited role in the effect.  Taking the author’s paper and assuming everything presented as a fact is true, he still acknowledges an large and unjust set of societal influences that hinder many women from succeeding in a field that I love and that I know many women would have loved, too.  It’s immoral when a person’s best possible career is thwarted for no reason other than some outmoded thinking.  The author, however, feels that keeping people from a great career is perfectly OK.  That wrong.

What particularly ticks me off is the number of people, apparently in my profession, who agree with the author – saying, “he’s misunderstood” instead of recognizing how wrong the analysis is.  We solve problems; we don’t sit wringing our hands and complaining about people who are making a change.

[Disclaimer: I work at Microsoft, which of course competes with Google in many areas.  Despite that, I recognize both that Google is full of fine people, and that (sigh) my own company presumably also has some of the left-behinds]

Infineon Sensor Hub–figuring it out

The Infineon Sensor Hub is a little Bluetooth device (but really, aren’t they all?) which the nice folks at Infineon use to demonstrate their DPS310 digital barometric pressure sensor.  The sensor, BTW, also produces Altitude data, although they don’t quite explain how they calculate that little bit.

The device on the right is the sensor hub nano
The problem, of course, is that Infineon is a hardware company.  Need to know some hardware related detail?  They have reams of charts and graphs.  But the Bluetooth spec for the device so we can actually poke at it and get data outside of their Android app?  That’s a much harder task.
Well, here’s a little dump of what I’ve learned.  The device acts like a serial port (and therefore commits one of the sins from my ‘your Bluetooth is bad’ post). 
Commands going in start with a $ and end with just a newline.  Data coming back is either JSON format (for the meta-data you can get) or not.  But either way, the return data also starts with a $.  Except when it starts with a >, like it echoes (some) bad commands.
Initial Messages:
Send these three strings
    $hello
    $info
To the $hello message the device replies with
${"pt":"urn:{557C199C-D246-43D5-8079-A68986BBAEB1}","uuid":"","username":""}
${"pt":"urn:{557C199C-D246-43D5-8079-A68986BBAEB1}","uuid":"","username":""}
To the $info message the device replies with

---

${"name":"IFX_NanoHub","manufacturer":"Infineon","protocol":"1.1","fw":"BSL111_FW312-b16ca0f","baud":115200,"sensors":["1"]}
${"name":"IFX_NanoHub","manufacturer":"Infineon","protocol":"1.1","fw":"BSL111_FW312-b16ca0f","baud":115200,"sensors":["1"]}

---

Yes, it seems to send the same data out twice.  And note the silliness with the baud rate – as you all should know, you don’t ever set the internal Bluetooth baud rate.
The you get get more info about the sensors with
    $sinfo id=1
The device responds with

---

${"id":"1",
     "manufacturer":"Infineon",
    "type":"DPS310",
    "chip_id":0,
    "port":0,
    "dds":[
        {
        "name":"Pressure",
        "id":"p",
        "type":"d",
        "unit":"mBar"
        },
        {
        "name":"Altitude",
        "id":"a",
        "type":"d",
        "unit":"m"
        },
        {
        "name":"Temperature",
        "id":"t",
        "type":"d",
        "unit":"degC"
        }
        ]
    }
${"id":"1","manufacturer":"Infineon","type":"DPS310","chip_id":0,"port":0,"dds":[{"name":"Pressure", "id":"p", "type":"d","unit":"mBar"},{"name":"Altitude", "id":"a", "type":"d","unit":"m"},{"name":"Temperature", "id":"t", "type":"d","unit":"degC"}]}

---

BTW, I took the results and split them onto multiple lines.  Yes, you once again get the duplicated answer. But see how we can tell what data we’re going to get, and how we can tell what kinds of sensors are available.  In theory, Infineon engineers can use the “same” protocol for multiple boards.
Then you can set the sensor modes.  It seems like you can get by with just sending the $start_sensor command

    $set_mode sid=1;md=mode;val=bg
    $set_mode sid=1;md=prs_osr;val=16
    $set_mode sid=1;md=prs_mr;val=32
    $start_sensor id=1
A set_mode command with correct parameters results in $ack and a bad one results in $nack
Stop the flood of sensor data with the $stop command
    $stop
The flood of sensor data looks like this.  The first number is the sensor id, the second is what's being produced (a=altitude, t=temp, p=pressure) based on the sinfo results.  The last value is a timestamp.

---

$1,a,27.8411,282787
$1,t,35.0797,284036
$1,p,1009.9077,284036
$1,a,27.8638,284036
$1,t,35.0619,285286
$1,p,1009.9113,285286

---

Good luck hacking your infineon!

Your Bluetooth is bad (and you should feel bad)

Alternative title: your Bluetooth is great, and you should feel great!

Little Bluetooth devices are awesome!  What’s not to like about teeny little things with switches and lights, magnetic sensors and motors?  And over the years, I’ve learned what makes a device easier or harder to control.
But first, a quick TL/DR about Bluetooth for people who aren’t as familiar.  Some devices are like a serial port: you send bytes, and you get bytes, and the device maker has to invent a little protocol for what the bytes mean.
Picture break: some of the Bluetooth devices mentioned

Left to right: MetaWear, SensorTag, BERO robot, TI Lamp kit
Other devices are BLE devices: each device has a set of “services”; each service has a set of “characteristics” with a name and value which can often be read, written, or can send a notification on change.  There are a bunch of standard services and characteristics.  For example, service 180f is the battery service; it includes characteristic 0x2a19, battery level which is a  single byte with a value 0 to 100.

Extra-wordy documentation!

I’m looking at you, otherwise awesome BBC micro:bit.  Here’s a short snippet from one characteristic:

Read                    Mandatory
Write                   Excluded
Write Without Response  Excluded
Signed Write            Excluded
Reliable Write          Excluded
Indicate                Excluded
Broadcast               Excluded
Writeable Auxiliaries   Excluded
Extended Properties     Excluded
 

That’s a big table with lots of rows, and it could all have been replaced with a single entry: “Supported operations: Read”.

Sloppy details!

Service 0x180a, device information, includes characteristic 0x2a29, Manufacturer name string.  It should be something like a company name, right?  Like “Texas Instruments” should be the manufacturer name on their delightful SensorTags.  In reality, the most common manufacturer names I see is a blank string, and literally the words, “Manufacturer name”. 
Similarly, each characteristic can be named.  TI is really good about adding clear names to their device characteristics; you can almost use just the names to guess how to control the device.  The micro : bit, less so. I had to update my Network Explorer program to convert the BBC GUIDs into more readable names.
More bluetooth devices

Left to right: BBC micro: bit, DOTTI, Hexiwear, Magic Lamp

Slow commands!

The Dotti is an 8x8 pixel light where each pixel can be set to any color using a Bluetooth command.  Too bad the write is a “write with response” which is extra-slow, so that actually filling up the screen is slow and painful instead of fast.  The BBC micro : bit, in contrast, has a single command that can fill its entire 5x5 pixel display in a single faster command.

Make the app do all the math!

The original 2541 SensorTag is a small device with a bunch of sensors – temperature, pressure, and more.  TI decided that “raw” access to the data was more important than “simple” access.  This is the temperature and pressure code that each app needs to write:
t_a = ((c0 * t_r / Math.Pow(2, 8) + c1 * Math.Pow(2, 6))) / Math.Pow(2, 16);
S = c2 + c3 * t_r / Math.Pow(2, 17) + ((c4 * t_r / Math.Pow(2, 15)) * t_r) / Math.Pow(2, 19);
O = c5 * Math.Pow(2, 14) + c6 * t_r / Math.Pow(2, 3) + ((c7 * t_r / Math.Pow(2, 15)) * t_r) / Math.Pow(2, 4);
p_a = (S * p_r + O) / Math.Pow(2, 14);
p_a = p_a / 100.0;
The newest SensorTag, the 1350, has values that can just be read directly. Except that they are the only 3-byte results I’ve ever seen.  Fun fact: there aren’t any helper libraries for reading in 3 byte values.

Ignore the math details!

Looking at you, Google Eddystone!  One of the values that an Eddystone beacon can produce is the temperature of the beacon; it’s documented to be a floating number in “8.8” format.
Specifically, here’s what the Eddystone spec says:

• Beacon temperature is the temperature in degrees Celsius sensed by the beacon and expressed in a signed 8.8 fixed-point notation. If not supported the value should be set to 0x8000, -128 °C.

Now, that actually doesn’t look bad; there’s a (mediocre) example and a link to the exact details.  The link, however, goes to the main page for the current Cornell ECE 4760 course in Microcontrollers.  It’s not even to a specific lecture.  I eventually found lecture #11 to be useful.
(Weirdly, there’s a python library for the Ruuvi tag that says that the temperature data is in “8.8” format.  But they decode it using the RuuviTag way, where the second byte just has a value 0..99, and where the first byte actually isn’t in two’s complement.)
And some more little devices

Left to right: another MetaWear device, NOTTI, the 1350 SensorTag, and the AutoBike with a Bluetooth shifter

Pretend it’s serial!

The Magic Light BLE is a pretty standard Bluetooth-enabled, color-changing bulb.  It’s weirdness: that instead of having a simple characteristic for setting the color (like the TI beLight and pretty much all of the others!), they have a “serial protocol”.  There is a single characteristic to write data to, and a single one to read from, and you have to send in a set of bytes using some other protocol that they just made up.
For extra weirdness: they don’t think Bluetooth is reliable, so there’s a bizarre checksum on the command.
This is different from the puck.js device, where they do the same kind of thing, but it’s OK because the data you send is literally a stream of JavaScript commands (how cool is that?)
I also give a pass to my Autobike that has a computer-controlled continuously variable transmission with a stream of Bluetooth data.  It’s an older device from before all the BLE stuff was more standardized.  And I really like the bike.

And a word about my apps…

Some of my apps directly control devices: BERO Robots, Quirky Nimbus, Autobike, TI SensorTag 2541, Magic Light, TI BLE Lamp, MetaWear.  And my do-it-all Network Inspector gives a more raw approach to investigating Bluetooth devices.
And then there’s the Best Calculator, IOT edition.  It is programmable in BASIC (!), and the BASIC has a bunch of extensions to make all kinds of IOT and Bluetooth programming easy.  It’s got a free trial, and will save you time when you’re automating your life.  Give it a try!