From the past: Win32s was both awesome and awful

 WIN32S: a programmer's dream from the Windows 3.1 era    

Gather 'round, young 'uns, and let me give you some wisdom from the days of the Win32s compatibility library. When Windows was being made, it was a 16-bit system, meaning that pointers were short and it was painful to address much memory. I'll also share a key feature of technical decisions: how to make an early decision that you don't know the ramifications of.

Quick aside: the people who made the 8086 and compilers as well as the PC and the OS knew this, so pointers were "actually" 20 bits (1 meg) thanks to the selectors and corresponding weird compiler switches, plus there was weird back-compat with the "A20" shim, and of course there was additional weirdness for some bank-selection.

Enter the RS/1 statistical program, which I helped port from its VAX/VMS and Unix heritage to the PC running Windows 3.1 and NT. It was a "big" program that had been in active development by a team of capable developers for ten years or more, with a ton of features. It had been designed originally to work on constrained computers like the PDP-11 (also a 16 bit machine with its own pointer weirdness).

A key early decision was to use the Win32s library on Windows 3.1. This library let us use 32-bit pointers in our code, which was a big plus. A downside, though, is that we didn't have any way to know about any potential downsides. Marketing material then, as now, talks big about how great new technology is, and doesn't much mention possible issues.

But there was an incredibly tiny flaw in the Win32s implementation of "unlink" that was to have enormous implications. To explain the bug, you have to know two things:

Firstly, the unlink call is used to "delete" a file. It's called unlink because technically the file isn't "deleted": it's merely removed from a directory. If the file is only present in one directory and no program is using it, the file is also deleted. 

Secondly, RS/1 kept track of every "table" of data as a separate file, and it used tables for pretty much everything. This was actually awesome, and I think more programming environments would benefit from a table-first approach. Some tables were permanent, but others were "temporary" and might be just in memory or might have a file backing depending on the available memory. Remember that RS/1 was designed for low-memory environments, so this automatically shuffling of data between disk and memory was a critical part of the program.

The bug

The bug in unlink was that if you did an unlink on a file that didn't exist, it returned the wrong value. Specifically, it returned success. The Posix standard was to return failure. This was critical to the underpinnings of RS/1: it's how the program knew if a temporary file was just in memory or was actually backed by a file on disk. By returning the wrong values, some internal bookkeeping got confused, and would eventually cause a crash.

This was caught, BTW, by the incredibly good $systemtest() function that RS/1 shipped with. Any user, at any time, could run $systemtest() and the program would do a pretty solid job of verifying that it was all running correctly.

On a Windows 3.1 machine, the program crashing also meant that the whole computer crashed, which was not ideal for debugging. I finally tracked it down by running RS/1 in a debugger on two machines, one running Windows 3.1 (where it used the Win32s library and crashed) and one running Windows NT (which used the native implementation which worked perfectly). I then started doing a sort of binary search to trace what was different about the two systems as it did the $systemtest().

Key takeaways for making early technical decisions

Bet on the future, not the past. We could have just made a 16-bit app. But the future was clearly 32-bit for Windows

Everything that isn't mainstream has bugs. Your schedule should include time for them. A constant in my technical career is that libraries that don't get much use have bugs, and there isn't much management resolve to fix them. In our case, the mainstream was either 16-bit code on Windows or 32-bit code on VMS or Unix. Win32s was a weird little library (as was the "PharLap" DOS extender that an earlier PC version used). 

Workarounds are better than hoping for a bugfix. If your library has an issue, it's best to figure out a workaround. Waiting for a fix that might never come will delay shipping, possibly forever. 

Modbus: deciphering the CRC protocol

 The Modbus over Serial protocol doc has a clear, simple set of instructions on how to decode. You just have to know that two of the steps are in a reverse order, but it actually makes sense.

The Modbus protocol is used by the Daybetter LED light Bluetooth protocol. It's arguably a terrible fit for this: the protocol includes a bunch of stuff that isn't even slightly relevant with Bluetooth (like the CRC), but doesn't leverage any of the Bluetooth strengths (like the ability to split "color" from "on/off")

Modbus CRC Calculations explained

Here's the official explanation of the CRC calculations from page 14+15 of , modified into a numbered list. Absolutely no changes were made except to make it a list -- that's why there's still weird commas and periods. It's substantially the same as the steps in page 39 to 41 in section 6.2.2 AKA Appendix B.

During generation of the CRC,

1. each 8–bit character is exclusive ORed with the register contents. 
2. Then the result is shifted in the direction of the least significant bit (LSB), with a zero filled into the most significant bit (MSB) position.
3.  The LSB is extracted and examined. 
4.  If the LSB was a 1, the register is then exclusive ORed with a preset, fixed value. If the LSB was a 0, no exclusive OR takes place.  

Note that clear ordering: first you do a shift, then you look at the LSB and do the XOR. BTW, the "preset, fixed value" is  0xA001 (decimal 40961 or binary 1010 0000 0000 0001)

But when you look at the commonly-available Modbus CRC calculations from random Github repositories, the code always switches steps 2 and 3! The LSB is grabbed first! What the heck! why are the clear and unambiguous steps in the official docs not what everyone implements?

Everyone is right because of Microcontrollers!

When you look at high-level languages, a right-shift is just a right shift, possibly with the ability to decide what gets shifted into the MSB (either duplicating the old bit, so a negative number stays negative, or filling it with zeros). But that's not what microcontrollers programmed in assembly do!

A typical microcontroller will do the shift (often with lots more control over the bits) and will also set the carry flag. The next instruction you do can then be a conditional jump based on the carry bit and therefore based on the original LSB.

 And indeed, the flowchart diagram from the Modbus protocol doc, appendix B, page 40, labels the item 4 "if" statement as "Carry over" yes/no. They are expecting implementors to use the carry flag or the overflow flag depending on the microcontroller being programmed!

The Modbus flowchart is reproduced below.

Microcontroller assembly

Motorola (NXP) 6805 chip has arithmetic shift right and logical shift right as two different operations. In both cases the operand is shifted one bit to the right, with the LSB moving into the carry bit.

The 6502 is similar, as is the ARM chip where setting the flags is optional (LSRS in assembler as opposed to LSR). The Intel 8051 does this with a RRC A opcode () but you have to set the carry flag to zero before doing the instruction (if it's 1, then a 1 gets put into the MSB)

Helpful Links:

Modbus over Serial Line specification and implementation guide

History: writing fancy code on a plain compiler (Irix version)

 Writing fancy code on a plain compiler

This is a story of porting C++ code using all the latest features to a machine whose compiler was (ahem) definitely not supportive of advanced features :-/

Back in about 1997 and 1998 I was a software consultant and got hired by an old coworker at Avid to help them port their shiny new high-performance file-copying software to the Irix. IIRC, they had written it in “portable” C++ which I discovered was anything but ☹

Writing code with no strings attached

 Amongst the other delights of 1997 era SGI workstations: SGI was a leader in creating the C++ STL. But the SGI people really, really didn’t like the proposed STL string types, so they just … didn’t. One of the many, many steps on my journey to porting this code was to create enough of a string class to compile.

 Other issues were that vector<> wasn’t compatible either, so I had some lovely #ifdef’s in the code.

What's in a namespace? Nothing. 

 The compiler also didn’t handle namespaces; they were read in an ignored. For most code this was a minor inconvenience, but the people writing this “portable” C++ code were a different breed. They had many, many classes with the same name and similar but different functionality. My solution was to create enough of a C++ “parser” to re-write the code. Turns out that if you ignore enough of the rules, you can make a C++ parser with just Lex 😊

Exceptionally fine threading

 But the absolute worst part was that the “portable” C++ code used both multiple threads (typical for networking code) and exceptions (still a new thing). The SGI compiler (which was the only compiler – I did a thorough look to find anybody else with a compiler) could handle threads, and could handle exceptions, but created incorrect binaries when dealing with both. And it didn’t matter if you threw any exceptions; the generated code was wrong regardless.

 My simple solution was to note that every single exception was uniformly caught exactly one level higher, and that none of their code ever returned a value. So I just made the exception-throwing methods return a value, instead. Simple, quick to implement and IMHO made the code a little nicer looking. This solution was rejected.

 The alternative solution was to use processes instead of threads. A “thread spawn” became a “process spawn”. And not just a process spawn: a process spawn all of the processes sharing their C++ memory so the data structures should be shared (and mutually updated, meaning using cross-process mutexes).

 This was an unglodly heavyweight project. But the pay was very nice.

TL/DR

 Moral of the story: never outrun your compiler?

Review: God and Golem, Inc (Norbert Weiner) -- 1964, MIT

TL/DR: I'm glad to have read the book but can't recommend it. The interesting ideas are now widely accepted (computers can learn, and we can't rely on computers to make decisions).

Best Quotes

"A goal-seeking mechanism will not necessarily seek our goals" (page 63)

"This is only one of the many places where human impotence has hitherto shielded us from the full destructive impact of human folly" (page 64)

"A digital computer can accomplish in a day a body of work that would have the full efforts of a team of [human] computers for a year, ..." (page 71). A modern 2022 computer can do the work of 40,000 people for a year in about a second (a Core I5 can do 34969 million FLOPS).

"Written for the intellectually alert public"

The book cover flaps are, unusually, one long continuous text that summarizes the text. The final paragraph: "... written for the intellectually alert public, does not require of the reader that [they] have a highly technical background." I suspect that this is the editor code for "all the glamor of a calculus textbook, but without the equations."

I originally picked up this book second hand as part of my overall interest in everything in the history of my computing profession. This is the first time I've managed to get all the way through while also grasping what the heck Norbert is trying to say. It helped that I put in lots of annotations and had access to the internet.

Theme: Computers will be like humans

If you accept that the Star Trek character "Data" is a "sentient being", then you already agree with Weiner. The entire book is trying to get us people to understand that eventually computers will have all of the parameters of sentient life.

The book was written in the early 60's (the publication date of 1964 is misleading; the book is a rewritten amalgam of earlier lectures), which is before "Star Trek" and sentient robots for the general public, but it's written long after Isaac Asimov's Robot series (including the books with Daneel Olivaw).

Weiner's basic thesis is that "computers" need to be considered in three ways: can a computer learn, can a computer reproduce, and what functions should be handled by humans and which by computer?

Can computers learn (spoiler: yes)

The "can computers learn" is now well understood: yes, they can. Weiner has a highly intelligence-is-everything point of view: in his opinion, as soon as a game is theoretically understood, it ceases to be of any interest at all to anyone. The obvious counter-example -- that people still play tic-tac-toc -- is entirely unconsidered.

This section, BTW, is what propelled me to write notes in the book. Weiner will bring up a person's name on one page, mess about for 15 pages, and then bring back that name assuming that you remember it.

Can computers make a new computer? (spoiler: eventually, yes)

The section on whether computers can duplicate themselves can only be understood by people who understand the complex dead-end mechanism used in WW2 artillery fire control systems. This is something Weiner excelled at, and he has great enthusiasm for it. But a better example is the numerically controlled machine tools that were already available -- a computer can guide the tools needed to build more computers.

The section is also somewhat weird. Biologists love to use "can reproduce themselves" as part of the important distinction between living and non-living. But from a legal or religious perspective, it's bunk: people don't have more or fewer rights because of their ability to reproduce.

What's the right place of computers? (helper, not decider)

Weiner correctly foreshadows the problems of having computers be the ultimate decider of critical actions, while also missing most of the problems that we're bedeviled with currently.

He's got a lot to say about nuclear war (fifty years later, we thankfully have never had another nuclear war, although arguably several wars have been highly influenced by the nuclear capabilities of the sides). He's rightfully skeptical of automated launch systems -- the reality of most alerts is that they are false alarms.

So, he says that computers will be like humans? (answer: no)

On the one hand, he's got a lot to say about how computers can theoretically learn, mutate, and reproduce. But he doesn't carry this to the logical conclusion: that computers will eventually be sentient (which he doesn't bring up at all). Instead, he argues that we humans must block any attempt to have computer make decisions that affect us humans. He's firmly in the camp that computers are good helpers for the human intellect but are ill-suited to being in control.

And right now, I'd say he's right. We see computers making "unbiased" decisions on health care that turn out to be racist (*), or "unbiased" justice decisions that put one set of people into jail. And we see clearly during these days of the Ukraine war that computerized messaging can be a tool to amplify one position or another.

If it's not physics, it's crap

Holy cow, there's an entire chapter devoting to bashing the mathematical formulations of anything that isn't physics. He's got a lot to say about how (for example) mathematical economics can't possibly ever be useful because getting good data is hard. What he misses is that we can deal with the data being wonky. During the pandemic times, we all saw the strange way that death rates would fluctuate, only to be explained that this state or that state was behind in their processing, and would periodically catch up by providing one giant batch of data. Similarly, the reason that some states (like Florida) have a low death rate is that all visitor deaths are reported by the home state.

One problem some academics have is that they can see how their own field is impacted by whatever the new thing is, but they can't imagine how this will impact other fields. Famously, after WW2, the British government commissioned an academic to decide if these new "computers" would be useful. The academic could easily see how their own particular field would benefit (x-ray crystallography), but couldn't imagine that computers would be useful in any other field.

Wait -- what's all this religious stuff?

Weiner love to talk religion. He's not very good about being particularly coherent. FYI: the sin of simony isn't related to Black Masses.

Where's the golem?

The golem is the Golem of Warsaw. It's mentioned in passing on page 49. Considering that it's the overarching theme, you'd think it would be mentioned a bit more. It's also mentioned on page 95, the conclusion, where it's mentioned once in an attempt to explain why the book is called God and Golem, Inc.

What's with the ", Inc"?

The title is best parsed as "(God) and (Golem, Inc)". For years I've been assuming it was best read as "(God and Golem), Inc". He's comparing the for-profit creators of computing machinery ("Golem, Inc") with God. 

(*) I can hear the "well, actually" crowd now. "Well, actually, the computers are racists, they merely use racist data to implement racist policies that have disproportionate impact on different races in a way that dehumanizes people and creates additional stumbling blocks, but the computers themselves aren't racist". Well, actually, that attitude is bogus.