Tuesday, January 10, 2023

Adding Pivot Item causes massive crash in UWP XAML

 Surprise exception when adding PivotItem

Quick background: my Simple Wi-Fi Analyzer program is getting a hidden feature which is unlocked by manipulating the UX in a secret way. I won't say what the secret is, but you can always look for "Unlock" in the source code :-)

The Analyzer app uses the nice Pivot control: I define a bunch of tabs for different features, and the user picks the tool they want to use. The new feature is a hidden tab, and my unlock code simply unhides it.

Except it doesn't. You can't actually hide a PivotItem in a Pivot control; it's not supported. Instead you have to create the PivotItem in code and then add it to the pivot.Items collection, it is shows right up!

Except you actually get this Vista-era dialog:

The problem (not even on Stack Overflow) is simple: you can't update a pivot.Items collection while you're in a PivotSelectionChanged callback! The solution is to 

Instead add in code like this:

                    var task2 = this.Dispatcher.RunIdleAsync((arg) => {

You don't have to wait for the task; it will just run when it's appropriate.

Saturday, January 7, 2023

Calculating Standard Deviation

 Calculating Standard Deviation 

The "classic" formulas aren't always the "easiest to program" formulas. Let's look at the classical formula to calculate a sample standard deviation:

σ = √ (Σ(xᵢ-x̅)²)/(n-1)

where σ is the standard deviation,  is the mean and n is the sample count. But note that we have to walk through the numbers twice: once to calculate the mean, and then again for the rest of the calculations. And yet, old fashioned scientific calculators with just a handful of registers (memory locations) could do this calculation by just entering in a column of numbers. What gives?

The answer is that although conceptually you are summing the square of the difference between each sample and the mean, you can actually do the calculation differently as long as you can keep track of n, the sum of x, and the sum of x². The resulting calculation is:

σ = √(Σ(xᵢ²) - ((Σxᵢ)² / n))/(n-1)

or, slightly more computery: for each new x:

1. n++

2. sumX += x;

3. sumXSquared += x*x;

4. varianceEstimate = (sumXSquared - ((sumX*sumX) / n) / (n-1)

5. stdDevEstimate = SQRT(varianceEstimate)

And you get a running estimate for the standard deviation. You also get the variance, but that's not often really needed.

Statisticians call these the "estimates" because the statistical theory is that there's a magic, universal reality for the actual population variance and standard deviation for which these samples provide an estimate. They aren't wrong :-)

Thursday, January 5, 2023

Handling the TITLE option

The information (i) element isn't any part of the Gopher RFC (most of Gopher is part of RFC 1436, and the Gopher URL is documented in RFC 4266), and therefore neither is the common practice of putting displaying an information element differently when a TITLE is put into the directory entry selector.

Thanks to a GopherSpace crawl, I know that out of about 2100 menus crawled, there were 305 information elements with a TITLE on them -- that's actually a pretty impressive percentage, and presumably it's driven by automated Gopher menu file creation. I'm wondering, though: as the programmer for a Gopher client, how many different Gopher client will render the TITLE elements specially? For that matter, how many Gopher pages put something into column 5, where the "+" in Gopher+ puts the Gopher+ indicator?

(I know that the first version of my own doesn't because I didn't know about the TITLE concept).

Let's start by analyzing pages that have too many columns (e.g., more than 5 columns, which corresponds to having more than 4 embedded tabs). There are five such directory entries in my GopherSpace crawl. Three are from a single page which, when I looked at is, is really an HTML page that's being served up as if it was Gopher. Of the remaining two entries, one has a user string consisting of three tabs instead of real data, and the other (from a different menu) has three tabs too many at the end of the line but and doesn't have any actual data in the columns.

We can also analyze the number of columns in directory entries

Almost all of the Gopher directory entries are the standard four columns. You can hardly tell the actual numbers, but 1907 entries were a single column, 9 were 2 columns, 895 were three, and 6590 were 5. There were 151840 entries with 4 columns.

History: writing fancy code on a plain compiler (Irix version)

 Writing fancy code on a plain compiler

This is a story of porting C++ code using all the latest features to a machine whose compiler was (ahem) definitely not supportive of advanced features :-/

Back in about 1997 and 1998 I was a software consultant and got hired by an old coworker at Avid to help them port their shiny new high-performance file-copying software to the Irix. IIRC, they had written it in “portable” C++ which I discovered was anything but

Writing code with no strings attached

 Amongst the other delights of 1997 era SGI workstations: SGI was a leader in creating the C++ STL. But the SGI people really, really didn’t like the proposed STL string types, so they just … didn’t. One of the many, many steps on my journey to porting this code was to create enough of a string class to compile.

 Other issues were that vector<> wasn’t compatible either, so I had some lovely #ifdef’s in the code.

What's in a namespace? Nothing. 

 The compiler also didn’t handle namespaces; they were read in an ignored. For most code this was a minor inconvenience, but the people writing this “portable” C++ code were a different breed. They had many, many classes with the same name and similar but different functionality. My solution was to create enough of a C++ “parser” to re-write the code. Turns out that if you ignore enough of the rules, you can make a C++ parser with just Lex 😊

Exceptionally fine threading

 But the absolute worst part was that the “portable” C++ code used both multiple threads (typical for networking code) and exceptions (still a new thing). The SGI compiler (which was the only compiler – I did a thorough look to find anybody else with a compiler) could handle threads, and could handle exceptions, but created incorrect binaries when dealing with both. And it didn’t matter if you threw any exceptions; the generated code was wrong regardless.

 My simple solution was to note that every single exception was uniformly caught exactly one level higher, and that none of their code ever returned a value. So I just made the exception-throwing methods return a value, instead. Simple, quick to implement and IMHO made the code a little nicer looking. This solution was rejected.

 The alternative solution was to use processes instead of threads. A “thread spawn” became a “process spawn”. And not just a process spawn: a process spawn all of the processes sharing their C++ memory so the data structures should be shared (and mutually updated, meaning using cross-process mutexes).

 This was an unglodly heavyweight project. But the pay was very nice.


 Moral of the story: never outrun your compiler?

Monday, January 2, 2023

Your Bluetooth is bad, January 2023 edition

 Your Bluetooth protocol is bad, January 2023

I've finally gotten over a big hump in my Bluetooth Device Controller program -- I've been poking around with adding and fiddling with devices, and that means that the code has been getting much more "experimental". That's not a good thing for an app that I ship, and which has over 35 thousand downloads! I've been working hard to convert the experimental code into an app that people can use without too much frustration.

With that, it's on to the next installment of this series on crappy Bluetooth protocols, focused on the Govee line of air sensors. The version 1.10 app supports the Govee 5074; the next version (presumably 1.11) will support the 5075 and 5106. All of them suffer from the same three flaws, and the 5106 has a unique and fun new flaw.

Don't shut off communications too early. All of the Govee devices like to shut down their Bluetooth connections really fast -- after about 4 seconds, they shut down the connection even if you've been talking on it. Other devices will wait until the connection has no traffic before shutting down.

Just transmit your freaking data. If you provide data, just provide it: make a characteristic, and make it readable and notifiable. 

Don't fold multiple values into decimal values. This is harder to explain. The Govee Air Sensor, as an example, sends out temperature, humidity, and air quality data in a single advertisement. But instead of just filling in 3 two-byte integer values, they instead take the temperature and multiple by 1_000_000. Then then take the humidity and multiple by 1_000. Then they add the air quality. This is all written as a single 4-byte integer.

To decode this monstrosity, you have to read in the 4-byte unsigned integer (in big-endian mode, even though Bluetooth is mostly little-endian). Then do a weird combination of MOD and integer divide operations to split out the three numbers.

Use the right Manufacturer code. The Govee devices mostly use a made-up EC88 manufacturer code; this is an unassigned value that nobody should be using. But the 5106 Air Quality monitor, for no apparent reason, uses the Nokia Phone code (they are manufacturer #1).

FYI: Common Timeout connection parameters

Each Bluetooth LE device can provide a set of connection parameters. These are decoded (now) by the Bluetooth Device Controller; they are part of the "Connection Parameters" (2A04) characteristic of the "Common Configuration" service (1800). The timeout is the last two bytes in little-endian format. For example, if the last two bytes are "90 01" in hex, that's 0190(hex) which is 400 (decimal). The value is in 10s of milliseconds, so the 400 (decimal) means 4 seconds for a timeout.

Looking at my device library, common settings here are:

  • 100 ms used by the SensorBug
  • 175 ms used by the Sphero
  • 4 sec used by the microbit, the govee, the kano coding wand, the viatom, the vion, and skoobot, smartibot and espruino
  • 5 sec used by the gems activity tracker
  • 6 sec used by the Mipow and the sense peanut
  • 10 sec used by the inkbird, lionel, the pyle, the powerup, the various sensor tags, and the dotti

Tuesday, September 6, 2022

Clipboard data for Excel

 I recently wrote a new app for Windows, the Simple Wi-Fi Analyzer. It will scan for nearby Wi-Fi hotspots and present you will lots of information about each one. But that's not what this post is about. This post is about how to put text on the clipboard that Excel can read.

You might think that CSV is the answer. It's not; Excel will treat it badly. An Excel expert will know to do a Paste Special and then laboriously tell Excel all about the delimiters, but that's  terrible solution.

The right solution is to use HTML. You can put text onto the clipboard as text, but encoded as HTML. In my new app, the HTML on the clipboard looks like this:








And shazam! it pastes perfectly! Except that Excel seems to think that setting the column width is beneath its dignity, but whatever.

BTW: the <body> isn't needed. And you'll need to encode the strings with System.Net.Webutility.HtmlEncode?view=net-7.0. That method will escape all of the HTML characters (like "<") into their HTML-safe version (&LT;)

Sunday, March 20, 2022

Review: God and Golem, Inc (Norbert Weiner) -- 1964, MIT

TL/DR: I'm glad to have read the book but can't recommend it. The interesting ideas are now widely accepted (computers can learn, and we can't rely on computers to make decisions).

Best Quotes

"A goal-seeking mechanism will not necessarily seek our goals" (page 63)

"This is only one of the many places where human impotence has hitherto shielded us from the full destructive impact of human folly" (page 64)

"A digital computer can accomplish in a day a body of work that would have the full efforts of a team of [human] computers for a year, ..." (page 71). A modern 2022 computer can do the work of 40,000 people for a year in about a second (a Core I5 can do 34969 million FLOPS).

"Written for the intellectually alert public"

The book cover flaps are, unusually, one long continuous text that summarizes the text. The final paragraph: "... written for the intellectually alert public, does not require of the reader that [they] have a highly technical background." I suspect that this is the editor code for "all the glamor of a calculus textbook, but without the equations."

I originally picked up this book second hand as part of my overall interest in everything in the history of my computing profession. This is the first time I've managed to get all the way through while also grasping what the heck Norbert is trying to say. It helped that I put in lots of annotations and had access to the internet.

Theme: Computers will be like humans

If you accept that the Star Trek character "Data" is a "sentient being", then you already agree with Weiner. The entire book is trying to get us people to understand that eventually computers will have all of the parameters of sentient life.

The book was written in the early 60's (the publication date of 1964 is misleading; the book is a rewritten amalgam of earlier lectures), which is before "Star Trek" and sentient robots for the general public, but it's written long after Isaac Asimov's Robot series (including the books with Daneel Olivaw).

Weiner's basic thesis is that "computers" need to be considered in three ways: can a computer learn, can a computer reproduce, and what functions should be handled by humans and which by computer?

Can computers learn (spoiler: yes)

The "can computers learn" is now well understood: yes, they can. Weiner has a highly intelligence-is-everything point of view: in his opinion, as soon as a game is theoretically understood, it ceases to be of any interest at all to anyone. The obvious counter-example -- that people still play tic-tac-toc -- is entirely unconsidered.

This section, BTW, is what propelled me to write notes in the book. Weiner will bring up a person's name on one page, mess about for 15 pages, and then bring back that name assuming that you remember it.

Can computers make a new computer? (spoiler: eventually, yes)

The section on whether computers can duplicate themselves can only be understood by people who understand the complex dead-end mechanism used in WW2 artillery fire control systems. This is something Weiner excelled at, and he has great enthusiasm for it. But a better example is the numerically controlled machine tools that were already available -- a computer can guide the tools needed to build more computers.

The section is also somewhat weird. Biologists love to use "can reproduce themselves" as part of the important distinction between living and non-living. But from a legal or religious perspective, it's bunk: people don't have more or fewer rights because of their ability to reproduce.

What's the right place of computers? (helper, not decider)

Weiner correctly foreshadows the problems of having computers be the ultimate decider of critical actions, while also missing most of the problems that we're bedeviled with currently.

He's got a lot to say about nuclear war (fifty years later, we thankfully have never had another nuclear war, although arguably several wars have been highly influenced by the nuclear capabilities of the sides). He's rightfully skeptical of automated launch systems -- the reality of most alerts is that they are false alarms.

So, he says that computers will be like humans? (answer: no)

On the one hand, he's got a lot to say about how computers can theoretically learn, mutate, and reproduce. But he doesn't carry this to the logical conclusion: that computers will eventually be sentient (which he doesn't bring up at all). Instead, he argues that we humans must block any attempt to have computer make decisions that affect us humans. He's firmly in the camp that computers are good helpers for the human intellect but are ill-suited to being in control.

And right now, I'd say he's right. We see computers making "unbiased" decisions on health care that turn out to be racist (*), or "unbiased" justice decisions that put one set of people into jail. And we see clearly during these days of the Ukraine war that computerized messaging can be a tool to amplify one position or another.

If it's not physics, it's crap

Holy cow, there's an entire chapter devoting to bashing the mathematical formulations of anything that isn't physics. He's got a lot to say about how (for example) mathematical economics can't possibly ever be useful because getting good data is hard. What he misses is that we can deal with the data being wonky. During the pandemic times, we all saw the strange way that death rates would fluctuate, only to be explained that this state or that state was behind in their processing, and would periodically catch up by providing one giant batch of data. Similarly, the reason that some states (like Florida) have a low death rate is that all visitor deaths are reported by the home state.

One problem some academics have is that they can see how their own field is impacted by whatever the new thing is, but they can't imagine how this will impact other fields. Famously, after WW2, the British government commissioned an academic to decide if these new "computers" would be useful. The academic could easily see how their own particular field would benefit (x-ray crystallography), but couldn't imagine that computers would be useful in any other field.

Wait -- what's all this religious stuff?

Weiner love to talk religion. He's not very good about being particularly coherent. FYI: the sin of simony isn't related to Black Masses.

Where's the golem?

The golem is the Golem of Warsaw. It's mentioned in passing on page 49. Considering that it's the overarching theme, you'd think it would be mentioned a bit more. It's also mentioned on page 95, the conclusion, where it's mentioned once in an attempt to explain why the book is called God and Golem, Inc.

What's with the ", Inc"?

The title is best parsed as "(God) and (Golem, Inc)". For years I've been assuming it was best read as "(God and Golem), Inc". He's comparing the for-profit creators of computing machinery ("Golem, Inc") with God. 

(*) I can hear the "well, actually" crowd now. "Well, actually, the computers are racists, they merely use racist data to implement racist policies that have disproportionate impact on different races in a way that dehumanizes people and creates additional stumbling blocks, but the computers themselves aren't racist". Well, actually, that attitude is bogus.