Showing posts with label Statistics. Show all posts
Showing posts with label Statistics. Show all posts

Sunday, August 24, 2025

From the past: Win32s was both awesome and awful

 WIN32S: a programmer's dream from the Windows 3.1 era    

Gather 'round, young 'uns, and let me give you some wisdom from the days of the Win32s compatibility library. When Windows was being made, it was a 16-bit system, meaning that pointers were short and it was painful to address much memory. I'll also share a key feature of technical decisions: how to make an early decision that you don't know the ramifications of.

Quick aside: the people who made the 8086 and compilers as well as the PC and the OS knew this, so pointers were "actually" 20 bits (1 meg) thanks to the selectors and corresponding weird compiler switches, plus there was weird back-compat with the "A20" shim, and of course there was additional weirdness for some bank-selection.

Enter the RS/1 statistical program, which I helped port from its VAX/VMS and Unix heritage to the PC running Windows 3.1 and NT. It was a "big" program that had been in active development by a team of capable developers for ten years or more, with a ton of features. It had been designed originally to work on constrained computers like the PDP-11 (also a 16 bit machine with its own pointer weirdness).

A key early decision was to use the Win32s library on Windows 3.1. This library let us use 32-bit pointers in our code, which was a big plus. A downside, though, is that we didn't have any way to know about any potential downsides. Marketing material then, as now, talks big about how great new technology is, and doesn't much mention possible issues.

But there was an incredibly tiny flaw in the Win32s implementation of "unlink" that was to have enormous implications. To explain the bug, you have to know two things:

Firstly, the unlink call is used to "delete" a file. It's called unlink because technically the file isn't "deleted": it's merely removed from a directory. If the file is only present in one directory and no program is using it, the file is also deleted. 

Secondly, RS/1 kept track of every "table" of data as a separate file, and it used tables for pretty much everything. This was actually awesome, and I think more programming environments would benefit from a table-first approach. Some tables were permanent, but others were "temporary" and might be just in memory or might have a file backing depending on the available memory. Remember that RS/1 was designed for low-memory environments, so this automatically shuffling of data between disk and memory was a critical part of the program.

The bug

The bug in unlink was that if you did an unlink on a file that didn't exist, it returned the wrong value. Specifically, it returned success. The Posix standard was to return failure. This was critical to the underpinnings of RS/1: it's how the program knew if a temporary file was just in memory or was actually backed by a file on disk. By returning the wrong values, some internal bookkeeping got confused, and would eventually cause a crash.

This was caught, BTW, by the incredibly good $systemtest() function that RS/1 shipped with. Any user, at any time, could run $systemtest() and the program would do a pretty solid job of verifying that it was all running correctly.

On a Windows 3.1 machine, the program crashing also meant that the whole computer crashed, which was not ideal for debugging. I finally tracked it down by running RS/1 in a debugger on two machines, one running Windows 3.1 (where it used the Win32s library and crashed) and one running Windows NT (which used the native implementation which worked perfectly). I then started doing a sort of binary search to trace what was different about the two systems as it did the $systemtest().

Key takeaways for making early technical decisions

Bet on the future, not the past. We could have just made a 16-bit app. But the future was clearly 32-bit for Windows

Everything that isn't mainstream has bugs. Your schedule should include time for them. A constant in my technical career is that libraries that don't get much use have bugs, and there isn't much management resolve to fix them. In our case, the mainstream was either 16-bit code on Windows or 32-bit code on VMS or Unix. Win32s was a weird little library (as was the "PharLap" DOS extender that an earlier PC version used). 

Workarounds are better than hoping for a bugfix. If your library has an issue, it's best to figure out a workaround. Waiting for a fix that might never come will delay shipping, possibly forever. 


Sunday, February 5, 2023

Wi-Fi Performance samples

 Wi-Fi Performance, broadly

Number 2 in a series on Wi-Fi performance. This time, Wi-Fi (and networking in general) is compared in difference places is compared. Sneak preview: Bluetooth PAN is the slowest :-)


I ran a simple speed test and gathered latency information (round-trip UDP times) in a variety of places and network. No surprise, Ethernet is the fastest and Bluetooth is the slowest. Airport Wi-Fi had generally high throughput (it's higher throughput than my house Wi-Fi network, although note that I have fiber with a low bandwidth cap) but with medium latency.

PAN? Bluetooth Personal Area Network? What's that?

Bluetooth PAN (Personal Area Network) is a technology that lets you share an internet connection from one laptop to another using Bluetooth instead of Wi-Fi. I'm not quite sure why people would do this, but it's still supported in Windows. It's got a non-winning combination of low speed and high latency. 

To actually share a network connection using PAN:

1. On the "host" laptop, open the Mobile Hotspot settings, and share your Wi-Fi (or Ethernet) connection with Bluetooth. 

2. On the "using" laptop, you need to get to the PAN settings page. AFAICT, there's only one way to do this: 

  • right-click on the Bluetooth icon and click "Join a Personal Area Network"
  • in the resulting Vista-era "Devices and Printers", select the "host" computer whose network you want to join. You might need to first pair with the "host" computer.
  • click connect using and select direct connection.


Wi-Fi: Is it faster on AC? Is it slower on Battery?

 Wi-Fi Performance: AC versus DC

One of a series of comparisons to learn what makes a difference in Wi-Fi performance.

TL/DR: they aren't difference, at least on my setup.

In the screenshot, I've done several latency, download, and upload tests against one specific server.

Test methodology: With the laptop plugged in, run all the speed tests twice with "AC" as the note. Then remove the docking station (the part that is plugged in) and run the same tests against the same server and mark them "DC".

Data Analysis: 
  • Download mean throughput increased from 16.2 Mbps to 18.4 Mbps going from DC to AC. This isn't a large difference.
  • Upload mean throughput decreased from 31.3 Mbps to 31.28 Mbps going from DC to AC. This is almost certainly not a real difference.
  • Mean Latency decreased from 12 msec to 10.3 msec. My stats program tells me this is probably a real difference. The DC had a little bit less jitter, but not by a real amount.
Conclusion: Wi-Fi is about the same speed on DC as it is on AC.

Saturday, January 7, 2023

The BEST way to calculate Standard Deviation

 The BEST way to calculating Standard Deviation 

The "classic" formulas aren't always the "easiest to program" formulas. Let's look at the classical formula to calculate a sample standard deviation:

σ = √ (Σ(xᵢ-x̅)²)/(n-1)

where σ is the standard deviation, x̅ is the mean and n is the sample count. But note that we have to walk through the numbers twice: once to calculate the mean, and then again for the rest of the calculations. And yet, old fashioned scientific calculators with just a handful of registers (memory locations) could do this calculation by just entering in a column of numbers. What gives?

The answer is that although conceptually you are summing the square of the difference between each sample and the mean, you can actually do the calculation differently as long as you can keep track of n, the sum of x, and the sum of x². The resulting calculation is:

σ = √(Σ(xᵢ²) - ((Σxᵢ)² / n))/(n-1)

or, slightly more computery: for each new x:

1. n++

2. sumX += x;

3. sumXSquared += x*x;

4. varianceEstimate = (sumXSquared - ((sumX*sumX) / n) / (n-1)

5. stdDevEstimate = SQRT(varianceEstimate)


And you get a running estimate for the standard deviation. You also get the variance, but that's not often really needed.

Statisticians call these the "estimates" because the statistical theory is that there's a magic, universal reality for the actual population variance and standard deviation for which these samples provide an estimate. They aren't wrong :-)