Thursday, December 28, 2023

Overview: Installers for .NET Core

 Installer types for .NET Core applications

I've got a new application, and I want a nice installer for it. Visual Studio has a metric ton of different installer projects, variously called "publisher" and "setup" and "install" and "deploy" -- so many, many different adjectives.

TL/DR: Make a setup project

My goal is an installer that is a single file, probably .EXE but I'll do a .MSI or .MSIX or whatever. Additional points are awarded for easy instructions, instructions that are less than 5 years old, a simple build process, output that can be placed somewhere other than the "bin" directory, and, of course, an installer that actually works.

My project, FWIW, is a .NET "Core" (now it's just called ".NET") -- that's the new thing that replaces the "Framework" version of .NET. It's using some of the WinRT APIs, so the output type is "WinExe" and the target framework is "net7.0-windows10.0.22000.0".

Publish (Folder) from main project (23 MB)

Creates a folder with your real .EXE plus a .DLL plus some more DLLs plus a .JSON file plus more random files. They are pretty much all needed, and there's no actual "installer"; it's just random files that you can write an installer for.
Points deducted: not a single file, no installer

Publish (Click Once) from main project (24 MB)

Creates a folder with an MSI for installing, plus also a .EXE for installing, plus also a random set of "Application files" which is full of random files that have all been renamed. The output folder must be in the bin folder.
Points deducted: terrible multi-install experience, wizard is broken, very old docs

The multi-install experience happens when you do an install to test the installer, and then install from literally any other place. Instead of upgrading, or removing the old app, or installing side-by-side, the installer just fails with no suggested course of action.

The publish "wizard" is broken: you can select stuff to change in the installer. But when you select, for example, the "Sign Manifest" "tab", you don't get brought to that experience; the dialog box is unchanged. Instead you have to click the "next" button to get from one "tab" to the next. 

The documentation for "What's new" is from 2012. 

Setup Project from new project (7 MB)

Creates an .MSI for installing, plus a pointless SETUP.EXE that only works with the .MSI. This means I had to waste time figuring out which file I really needed. 

Points deducted for: bizarre and undocumented "editor". To use it, right-click the "Application Folder" and select "Add" and then "Project Output" and give the original project. 

The created installer is a clunky, old-fashioned installer, so the user has to click "next", then look at stuff they won't want to change, then click next again, click next on the UAC prompt, and then click close. And it gives a little warning to be sure to check with Windows Update for updates to .NET Framework, which this app, AFAICT, doesn't use at all (it's .NET Core). 

I really want to deduct points for the "Transitive" property, the description of which is "Determines whether the installer will reevaluate the Condition property for the selected item when reinstalling on a target computer". The words condition, property, select, and target computer really just raise new questions. 

Sunday, September 24, 2023

Project: Govee E-Ink display

 The Govee H5074 E-Ink display project!

All complete! This project uses an Adafruit nrf52840 board running CircuitPython and their 2.13 inch e-ink display (the tri-color one) to pull in data via Bluetooth LE from a Govee H5074 temperate and humidity sensor.

Watch it on YouTube. Also, the code is up on Github; take a look!

Some of the challenging / fun parts: on reset, the device will look around for a Bluetooth "Current Time Service" so that the clock is set automatically (no need to every manually set it!). Reading in Bluetooth advertisements for the Govee H5074 was not as obvious as it should have been (and I've got a blog post about it), and the e-ink display was much more challenging than I though it would be.



Using E-Ink displays with CircuitPython

 Using E-Ink displays with CircuitPython


Adafruit has some very nifty e-ink displays. It's something I've always wanted to try out: I've got an e-book reader that uses e-ink, and of course I've seen retail proce tags that use them. What I found is that although the demo code worked great, there were some major gotcha moments when using them in a project.

Firstly, the update process for the display is both very slow and very obvious. The display will go into a reverse-video mode and clear to white and to black several times before settling in and showing the new display contents. I've got a YouTube video that shows the refresh cycle.

Secondly, the display can only be updated every 3 minutes. This is locked by the code: if you try to refresh the screen more often, the CircuitPython library will just throw an exception.

Lastly, the sample code, as written, doesn't actually let you refresh the display. This may seem weird: the library code (displayio) certainly includes the concept of a display refresh, and when used with OLED displays you can do a refresh. But as it turns out, there's some kind of bug in the CircuitPython library (as of 2023-09-24), and when I do a refresh, my labels simply turn into black rectangles.

All my project code is up on Github. Take a look at the code.py file for the complete code; also look at the GoveeDisplay.py file; that's the file that makes all of the text labels for the device and handles text updates (e.g., it knows how to take the GoveeData object and update the right labels using either degrees Fahrenheit or Celcius).

The ideal way to make a display is to 

  • Set up the display
  • Create  all the text labels, background images, and layout groups
  • Call display.show() (and maybe display.refresh()) to update the display

After that, we can update the label.text fields, and the display will refresh (or, for e-ink displays, we have to call the display.refresh() method to force a refresh). This is simple and clean: updating the display is done just with the labels.

But for the e-ink display, you have to do something else. You can reuse the text labels and whatnot, but you have to rebuild the display all over again. In code.py, this is done in the while True: loop at about line 229. Looking at the code, on every refresh (which happens every 5 minutes), I have to redo pretty much everything display related: release the old display, make a new display_bus, make a new display, call display.show() (with the existing groups, etc.), and then finally I can do a refresh() and have the display update.


    displayio.release_displays()
    display_bus = displayio.FourWire(
        spi, command=epd_dc, chip_select=epd_cs, reset=epd_reset, baudrate=1000000
    )
    time.sleep(1)

    # For issues with display not updating top/bottom rows correctly set colstart to 8
    display = adafruit_ssd1680.SSD1680(
        display_bus,
        colstart=8,
        width=DISPLAY_WIDTH,
        height=DISPLAY_HEIGHT,
        busy_pin=epd_busy,
        highlight_color=0xFF0000,
        rotation=270,
    )
    # We can use the existing group with the new display.
    display.show(mainDisplayGroup)

    #
    # Wait for a time that's evenly divisible by 5. That means that sometimes
    # the scan results will be a little late.
    #
    wait = TimeToWaitForEvenClock(clock.datetime)
    if wait < 4 * 60:  # If it's e.g., 3:44:59
        time.sleep(wait)

    voltage = get_battery_voltage()
    gd.ShowGovee(userprefs, scanResult, clock.datetime, voltage)
    display.refresh()


One more thing -- it's not super reliable

Regretfully, my experience with the e-ink display is that it's also not super reliable. Every now and then, I'd refresh my code, and then the display just wouldn't update. Or maybe it didn't initialize, or it wasn't connected somehow. There's no exception thrown, or useful trace, or debug message.

In at least one case, I was trying to use a different physical microcontroller board of the same type (an NRF52840). And for whatever reason, the display just didn't want to work that the other micro, but it wouldn't say why. Is there a hardware fault? Did I mess up some connection? I have no idea, and no useful path forward.

In the end, I just used the original microcontroller, and eventually everything worked, mostly. And when it doesn't, pressing "reset" will eventually make everything work again.

Links to devices:






Reading Bluetooth LE (BLE) sensor data advertisements in CircuitPython

 Bluetooth Sensors -- reading the Govee H5074 advertisement data




Reading Bluetooth LE (BLE) sensor data from CircuitPython isn't always obvious. In this example, I show how I pull data from the Govee H5074 Bluetooth sensor. The sensor is a small, battery-powered sensor that reads the temperate and humidity and transmits it inside of Bluetooth advertisements.

The code is all up on Github. You'll want to look in the code directory at the GoveeScanner.py file. There's also a Govee5074.py file which can parse an advertisement once it's been seen, and a GoveeDisplay.py file that can display the data to an E-Ink display.


The key to the code is a pair of methods: the Scan() method and the ScanOnce() method. ScanOnce will start a scan ble.start_scan(timeout=...) . In practice, I use very short timeouts (.1 second) but then do a bunch of them in a loop in the Scan method. The start_scan returns a  generator, which is like a list but the contents will flow in over time. The contents will be both advertisements and advertisement scan responses. A scan response is like additional data in the advertisement, but it only comes in when it's requested. The request will be automatic; we don't have to do anything to request a scan_response.

The Govee sensor data is send in the scan_response, in what's called the "Manufacturer Data" section.

The critical thing to know is that the two values -- the advertisements and scan_responses -- aren't linked together by CircuitPython. We have to do that ourselves, and we do it by saving the advertisements in a dictionary that's indexed by the Bluetooth address.
  

The advertisements will have the name of the device. We're looking just for Govee H5074. In the code at about line 88 we look for the complete_name :

            name = advert.complete_name
            if (name is not None) and ("Govee_H5074" in name):
                # print("TRACE: 30: Found main govee advert", advert.address, name)
                goveeAdverts[advert.address] = advert


If the device is one we want, then we save away the advert in the goveeeAdverts dictionary, indexed by the Bluetooth address. This is critical because when we see an advertisement scan_response, we actually don't get the name but we do get the address.

The other part of the ScanOnce method is reading the scan responses. We filter first based on whether it's a response we want (we'll get lots of responses from lots of devices, but we only care about the Govee H5074 devices here).

            if advert.scan_response:
                # Check to make sure that this response address is in the list
                # of adverts that have the right name ("Govee_H5074...")
                # You can't tell from just the response advert; you need the
                # original advert, too.
                if advert.address not in goveeAdverts:
                    # very frequent -- set is only govee adverts, but the
                    # scan_response can be for anything
                    # print("TRACE: 15: not in set", advert.address)
                    continue

Once we know it's a scan_response we want, we pull out the manufacturer data and parse it. If it parses OK, then we prepare to return it.

                MANUFACTURER_SECTION = 0xFF  # Per Bluetooth SIG
                if (MANUFACTURER_SECTION in advert.data_dict):
                    annunciator.Found()
                    buffer = advert.data_dict[MANUFACTURER_SECTION]

                    g5074 = Govee5074(buffer)
                    if (g5074.IsOk):
                        annunciator.Read()
                        if retval is None:  # only print the first one
                            print("TRACE: 28: GOT DATA", g5074)
                        retval = g5074


The annunciator is the code that lets the user know what's going on. "Read" is an indication that we've read the data OK.


TL/DR: when reading Bluetooth advertisements, be sure to check the scan_responses too, because that's where the data might be.

Friday, June 30, 2023

Clocks that set themselves!

The Adafruit Clue-Clock


I'm always frustrated when I have to reset a bunch of clocks after a power outage. Did you know that setting the time on an IOT device can be easy when you add support for the Bluetooth Current Time Service? (On the SIG site you will want the first doc, "Current Time Service 1.1"). In this blog post I show some of the code I wrote for the Adafruit Clue using CircuitPython and the adafruit_ble library. I've even got a Youtube video! to show the device and step through the code. 

There's also a handy Windows app that is the server side of the time setting; it will broadcast out the current time. Download "Simple Bluetooth Time Service" on the Windows store; that's how I set the time. The complete source code for it is on GithubUpdate: see also my [Govee Ink Display](ElectronicsProjects/2023-Adafruit-Python-InkGoveeListener at main · pedasmith/ElectronicsProjects (github.com) project; it has a newer and easier-to-use version of the clock code.

In the video I step through three interesting features of the clock.

Feature 1: Display stuff to the screen

We'll want to print text to the screen; this is done with the clue.simple_text_display() object. The clock uses the simple_text_display, so we can display several lines of text, but nothing fancier. 


A key point (that took me far to long to figure out!) is that after you update one or more a lines of text, you must call the .show() method -- otherwise, nothing gets displayed!


Sample Code

from adafruit_clue import clue

colors = ((0xff, 0xff, 0xff),)
display = clue.simple_text_display(title="Clock", 
                                   title_scale=2, text_scale=4,
                                   title_color=(0xa0, 0xa0, 0xa0),
                                   colors=colors
                                   )
str = "{:02d}:{:02d}:{:02d}".format(currHour, currMinute, currSecond)
clue_display[0].text = str
clue_display.show()


The simple_text_display is documented on the circuitpython site.

The text_scale value of 4 fits a time display with a format of HH:MM:SS (8 characters long) with room for two more characters (eg, enough room for an AM/PM indicator, if desired)

The title_scale is relative to the text_scale.

The colors set the colors of each line. I set it so tht the time and date are white, and the day (and the bluetooth scan results) are blue.

Feature 2: Use the Real Time Clock

The chip used to track time accurately is the "real time clock" -- without it, the code would slowly drift. Fun fact: the first IBM PC did not include a battery-backed real-time clock chip. Every time you turned on the computer, you had to enter the date and time.

The real-time clock uses struct_time for many operations; it's just a tuple where you can grab values by index. the current hour, for example, is index 3.

The real-time clock needs power to work; if it loses power, it will stop stracking an accurate date and time (it says "real time clock", but it does dates, too). How we set it is the topic of the next section.

The CircuitPython rtc module is a delight to use: there's just a simple way to set the initial value and a simple way to pull out the current time.

Feature 3: Connect to Bluetooth Current Time Service

Now we get the hard stuff: reading data from an external Bluetooth "Current Time Service" source. The idea is that a nearby PC will broadcast out a "current time" (there's a standard for this); the clock will pick it up and use it to set its time.

To make it work, you should have already added the adafruit_ble to your lib directory. It's not on the list of libraries in the Adafruit Clue documentation.

The bulk of the Bluetooth code is in BtCurrentTimeServiceClient.py. There's two critical classes in that file: the BtCurrentTimeServiceClient class which matches the Bluetooth Special Interest Group (SIG) standard and which is compatible with the Adafruit CircuitPython Bluetooth setup, plus a helpful wrapper class BtCurrentTimeServiceClientRunner class which listens for Bluetooth advertisements and connects to the time service.

You will want to look at the code while reading this description :-)

 BtCurrentTimeServiceClient

The BtCurrentTimeServiceClient class is less than 20 lines of code. The Adafruit CircuitPython Bluetooth system isn't too hard to use, but there isn't a very good tutorial on it. Hopefully this explanation will help!

The BtCurrentTimeServiceClient class exists for only one reason: it's the "glue" between the Bluetooth system and your code. When you get an advertisement for a Bluetooth device you want to connect to, you'll provide this class (the class and not an object) and will get back an object that's mostly this class (it will have been updated)

The object you get back will only be valid until the connection is broken. In the code, the connection is broken almost as soon as the data is read.


class BtCurrentTimeServiceClient(Service):
    uuid = StandardUUID(0x1805)
    data = StructCharacteristic(
        uuid=StandardUUID(0x2A2B),
        # Don't need to provide these; they should be discovered
        # by the Bluetooth system.
        # properties=Characteristic.READ | Characteristic.NOTIFY,
        struct_format="<HBBBBBBBB"
    )
    def GetTimeString(self):
        (y, m, d, hh, mm, ss, j1, j2, j3) = self.data
        retval = "{0}-{1}-{2} {3}:{4}:{5}".format(y, m, d, hh, mm, ss)
        return retval


The data value is set to be a StructCharacteristic. But when you examine the data later on (like after it's been updated by the remote side!), it will instead be a tuple of the data, parsed by the struct_format string. You just have to know from other sources what the data values actually mean.


BtCurrentTimeServiceClientRunner

The BtCurrentTimeServiceClientRunner is the class you'll actually call to get the Bluetooth current time data. Just call Scan, passing in a bluetooth "ble object; it's the Bluetooth from the clue device (```ble = adafruit_ble.BLERadio()```). There aren't any other methods in the class that should be called.

The runner Scan method will scan for Bluetooth advertisements for a set amount of time (15 seconds in this case); the scans can complete in less time, so I loop around as needed. The inner loop of Scan calls ScanOnce to do a single advertisement scan, returning a connected Bluetooth device. Once this method has a connected Bluetooth device, we hook up the service connection with the ConnectToCurrentTimeService method (yeah, I'm using the word "connected" here in kind of two different ways). Once we have a service connection, we can pull out the time data directly.

The ScanOnce returns a connected connection to the remote device (or None, of course). It does a single advertisement scan, up to a maximum amount of time, looking for an advertisement that says it supports the current time service. When one of those is found, we connect to that device. In my case, the device will just be my laptop when it's running the Simple Bluetooth Current Time Service app. 

The ConnectToCurrentTimeService creates a 'live' (connected) service object given a connection to a device. 

To convert a connection to a bluetooth device into a useable per-service object, you need to provide a class with a uuid that matches the service you need to use, plus a data object which needs to be one of the Characteristic types (for example, StructCharacteristic). When you "get" an object from the connection, the smart connection "array" will create a brand-new object for you, of the class you specify, that's hooked to (connected to) the live Bluetooth object. As part of this, the "data" value in the class, which had been, e.g., a StructCharacteristic, will now just be a tuple of data. Reading that tuple will get you the latest data.

To recap: the Scan method will scan advertisements for an appropriate BT device, will connect to it, will make a service connection, read the characteristic data, put that data into a tuple, and return the tuple. In case of errors, it will just return None.

Once the tuple of date is read, we just set up the real-time clock at about line 74 of the code.py file. Once this happens, the clock will be updated!

You can make this work for your device, too -- just pop in the BtCurrentTimeServiceClient.py, and call the Scan() method with a BT radio. Just don't forget to include the adafruit_ble library on your device!


Good luck!


Sunday, April 23, 2023

Hints on injecting input into Windows (use 'ScanCode' for Unicode!)

 Use ScanCode to inject Unicode characters!

Got a Unicode string and need to inject it using the InputInjector class? Wondering how to inject it since the InjectedInputKeyboardInfo class has a ScanCode and a Virtual Key but no Unicode char? It turns out to be easy!

In the InjectedInputKeyboardInfo class, set the KeyOptions to Unicode and then fill in the ScanCode with the unicode char. Since you've got a string, you'll make a list of InjectedInputKeyboardInfo items and then send them all with the InjectKeyboardInput method.

I know this works because my new keyboard will happily inject  Unicode strings like this: 😋👩‍👩‍👧‍👧!

Handy Links

MSDN learn.microsoft.com ScanCode
Old [UWP]InjectedInputKeyboardInfo with Unicode forum questions that says they have an answer (but don't say what it is): here



Friday, April 14, 2023

Hints on using CircuitPython's adafruit_ble Bluetooth

 Hints on using CircuitPython's adafruit_ble Bluetooth

Some APIs and libraries for Bluetooth are a joy to use: they fit right into our basic concepts of how the protocol work, and match the kinds of tasks we want to do.

And then there's the adafruit_ble library. 

I've been working on a little project using the very nifty AdaFruit  Feature nRF52840, and there's a lot to like about it. They got a ton of the details just *french kiss*, starting with the font on the main device (the 840 is in extra-large letters so you can quickly tell one from the other) and running to their library of compatible "feather" devices and integration into CircuitPython.

But the ble library? They have clearly spent a ton of time and effort on it (which I thank them for). But everything in it is just that little bit backwards from everything I know about Bluetooth.

Example: why doesn't this code work?

Here's a simple example: I've found a device and I know that it has supports the Current Time Service that I want to read. I know it does that because I check to make sure that the service's GUID (0x1805) is part of the device advertisement. This is done with the CircuitPython statement: 

        if BtCurrentTimeServiceClient.uuid in connection

which means I should be able to get that service, right? So this code should work?

    service = connection[BtCurrentTimeServiceClient.uuid]

Because that's what a collection is for: you can check to make sure a key is in the collection, and then pull the value out of the collection. But that's not how CircuitPython works. You can check the services via a guid, but to get a value, you can only do that by providing the type of the object you want out (and it had better derive from Service)

Why this is horrific, and how much time I wasted

Given code that doesn't work, the next step is to figure out how to fix it. Potential fixes I tried included:
  • using a different kind of UUID (UUID versus StandardUUID)
  • getting the UUID from a different place (afafruit_ble versus bleio)
  • testing to make sure that the 'in' wasn't just always returning true by trying a fake UUID
  • connecting at different times
  • stopping the scan before getting data
  • doing a time.sleep(5) before connecting or getting data
  • connecting to the address versus the advertisement
  • getting the service twice
  • pairing
  • iterate through the services (this was really weird)
  • disconnecting when I was done
  • putting it in a try/except block to investigate the exception

That's a lot of investigation just to learn that what's going on is "magic": the CircuitPython library, instead of just boringly reporting on a connected devices's services and characteristics, demands that you carefully create a complete-enough CircuitPython replica of the device you're connecting to.

Doc shortcomings examples

Little snippets of code would have been super handy. All of the documentation assumes that I'm already a Python expert and will instantly understand the Pythonesque ways to getting information

For example, there's a handy "StructCharacteristic" that will put chunk of data out of a Bluetooth characteristic. This is handy when you need to read something like the Current Time Service where the time data is split into 9 different fields, mostly unsigned byte, but one is a 2-byte unsigned short.

But how does one actually read the data? It's not mentioned in the docs or in the source code. Turns out this will read the data:

    char = service.data
    print("    value=", char)
    print("    year=", char[0])
    print("    mon=", char[1])
    print("    day=", char[2])

This works because the data is the characteristic, and seemingly when you get it you get the unpacked version of the raw data.

What should they have done?

Take a look at the Windows Bluetooth (UWP) API. Once you have a device, you can list all the services, and for each service, list all of the characteristics.

Or, if you know more about the device you're connecting to (often the case), you can get just the services and characteristics you're interested in, which is almost certainly going to be faster (no point in poking the Bluetooth device for a bunch of service and characteristic data that you don't care about).

Handy Links

Link to Adafruit Feather nRF52840 Express
Link to the Characteristic class on GitHub which includes Struct































Friday, March 31, 2023

WTF is “Exact time 256” : diving into Bluetooth SIG documents

 

WTF is “Exact time 256”    

The Bluetooth Special Interest Group (SIG) has a metric ton of Bluetooth LE device specs in a massively confusing pile. In this walkthrough I’ll show how I figured out the details of a fairly simple LE protocol. In particular, I’ll be creating something that needs to match the “Current Time Service”, services # 0x1805.

TL/DR: the good document is the “Gatt Specification Supplement”.

On the Bluetooth.com site, you want “Specifications” and under specification you’ll need a tab open for both the “Assigned numbers” and the “Specifications” directory.

In “Assigned numbers”, there’s two important documents. The first is “Assigned Numbers Document”. It’s a giant list of all of the GATT services by name and all of the characteristics. The second is “Gatt Specification Supplement”.

In the “Specifications” directory, find the listing for the "Current Time Service” and click the link to get the service page. It's got a bunch of the lest useful programming documents ever. The only document that’s interesting for most developers is the Current Time Service 1.1 PDF file. Click the link to read the long, complicated document.

Page 9 starts to be interesting: the Current Time Service supports three characteristics: the “Current Time” (page 10), “Local Time”, and “reference Time”. On page 10 it’s mentioned that we’ll be reading the Exact Time 256 field. This is the first bit of useful (and critical) information in the spec.

BTW, when they say “Unknown”, it’s not clear to me which “Unknown” value they mean. But in the Assigned Numbers document there are 8 time that “Unknown” is mentioned; all of the numeric values (6 of them) are zero. The others are weird strings for something having to do with telegrams. Or not, it’s a Bluetooth SIG spec, so nothing is clear.

At this point you might wonder what the actual bytes are. The Bluetooth SIG doesn’t care that you’re wondering. You might even try typing “Exact Time 256” into the main page search box on the main page, but you won’t get any hits. However, you can find it in the Assigned numbers where it has number 0x2A0C in the “Characteristics by name” and “Characteristics by UUID” section.

Note that in the “Specifications” it lists “Current Time” as the characteristic; that’s characteristic 0x2A2B.

And then finally, take a look at the GATT Specification Supplement, page 78, section 3.62, “Current Time”. It lists the bytes: there’s an Exact Time 256 and then a U8 “Adjust Reason” which says why the time changed. Even better, there's a decent overview of the Exact Time 256 fields!

And here the path through the forest of all Bluetooth knowledge ends. If only there was some way to know that the "supplement" is in fact the useful document, and not the pile of other docs.

Wednesday, March 15, 2023

It takes a lot of people to ship a complex product

 I'm a Program Manager (PM) for one of Those Big Companies. I don't actually program except as a hobby, and I don't manage. Where I add value is making sure that what we're building is what our customers need.

"I could build a replacement with just ___ people"

No, you couldn't, if it's a complex project. I was recently reminded of the true history of the automobile: a long, hard slog by an enormous number of people to get each element of a car to actually work. In particular, take a look at Motor-Car Principles on Project Gutenberg. Among the many, many things that had to be painfully figured out


How to stop a car. The book lists a whole series of brake types (none of them disk brakes like in modern cars) 



How an axle works. Car axles are not wagon axels, and you can't use one instead of the other.

Building a transmission. This is something steam engines, for example, simply don't have. A steam engine on a train is directly connected to the wheels with no intervening transmission.



Making a piston. Steam engines, in contrast to car engines, are forgiving: as long as the timing is roughly correct, the steam engine will work. A gas engine has much higher tolerances.

The book goes on with a long, long list of technologies that are unique to cars.

Moral of the story: some projects are pretty simple and can be done with a small team (or even just one person). But once you get to new technology, it's a quagmire that requires an extraordinary amount of work just to know what needs to be done, and from there to co-ordinate the work so that all the parts show up on time.

Thursday, March 2, 2023

Fixing "The type or namespace 'Windows' could not be found" error CS0246

"The type or namespace 'Windows' could not be found" solution!

Every time I install a new version of Visual Studio, I run into just a metric hand-full of errors from previously working projects. The projects haven't changed, but suddenly Visual Studio is seemingly incapable of finding the "Windows" namespace.

My Solution: run the Visual Studio Installer". Pick your installed version and click Modify. The modification you probably want is to under Universal Windows Platform development in the Optional section. Look at the list of available Windows SDK and just install them all (assuming you have enough disk space).

There's no harm (AFAICT) in getting them all, and it will save you heartache and frustration down the line.

This entire process is just a cluster. What's more common than building apps for Windows using the compiler tools from Microsoft that are specifically designed to build Windows apps? Well, the only thing more common then developing windows apps is seeing yet another place where Visual Studio once again makes development more painful.

For me, the #1 reason I get the error is that I don't have enough of the Windows SDKs installed. By default, the Visual Studio installer only installs the "most recent" SDK. You might think that's enough, but it's not.

Each project will have a reference path that looks like this:

    <Reference Include="Windows.Foundation.UniversalApiContract">
      <HintPath>..\..\..\..\..\..\Program Files (x86)\Windows Kits\10\References\10.0.22010.0\Windows.Foundation.UniversalApiContract\14.0.0.0\Windows.Foundation.UniversalApiContract.winmd</HintPath>
    </Reference>

Note how the HintPath specifies exactly one particular SDK. If you have a more recent version, it doesn't count even though the contracts are specifically designed to be upwards compatible.

What's worse, Visual Studio loves to kind of silently update your project files, so you'll silently be updated, potentially breaking the project on a different computer.

Error CS0246 The type or namespace name 'Windows' could not be found (are you missing a using directive or an assembly reference?)

Sunday, February 5, 2023

Wi-Fi Performance samples

 Wi-Fi Performance, broadly

Number 2 in a series on Wi-Fi performance. This time, Wi-Fi (and networking in general) is compared in difference places is compared. Sneak preview: Bluetooth PAN is the slowest :-)


I ran a simple speed test and gathered latency information (round-trip UDP times) in a variety of places and network. No surprise, Ethernet is the fastest and Bluetooth is the slowest. Airport Wi-Fi had generally high throughput (it's higher throughput than my house Wi-Fi network, although note that I have fiber with a low bandwidth cap) but with medium latency.

PAN? Bluetooth Personal Area Network? What's that?

Bluetooth PAN (Personal Area Network) is a technology that lets you share an internet connection from one laptop to another using Bluetooth instead of Wi-Fi. I'm not quite sure why people would do this, but it's still supported in Windows. It's got a non-winning combination of low speed and high latency. 

To actually share a network connection using PAN:

1. On the "host" laptop, open the Mobile Hotspot settings, and share your Wi-Fi (or Ethernet) connection with Bluetooth. 

2. On the "using" laptop, you need to get to the PAN settings page. AFAICT, there's only one way to do this: 

  • right-click on the Bluetooth icon and click "Join a Personal Area Network"
  • in the resulting Vista-era "Devices and Printers", select the "host" computer whose network you want to join. You might need to first pair with the "host" computer.
  • click connect using and select direct connection.


Wi-Fi: Is it faster on AC? Is it slower on Battery?

 Wi-Fi Performance: AC versus DC

One of a series of comparisons to learn what makes a difference in Wi-Fi performance.

TL/DR: they aren't difference, at least on my setup.

In the screenshot, I've done several latency, download, and upload tests against one specific server.

Test methodology: With the laptop plugged in, run all the speed tests twice with "AC" as the note. Then remove the docking station (the part that is plugged in) and run the same tests against the same server and mark them "DC".

Data Analysis: 
  • Download mean throughput increased from 16.2 Mbps to 18.4 Mbps going from DC to AC. This isn't a large difference.
  • Upload mean throughput decreased from 31.3 Mbps to 31.28 Mbps going from DC to AC. This is almost certainly not a real difference.
  • Mean Latency decreased from 12 msec to 10.3 msec. My stats program tells me this is probably a real difference. The DC had a little bit less jitter, but not by a real amount.
Conclusion: Wi-Fi is about the same speed on DC as it is on AC.

Tuesday, January 10, 2023

Adding Pivot Item causes massive crash in UWP XAML

 Surprise exception when adding PivotItem

Quick background: my Simple Wi-Fi Analyzer program is getting a hidden feature which is unlocked by manipulating the UX in a secret way. I won't say what the secret is, but you can always look for "Unlock" in the source code :-)

The Analyzer app uses the nice Pivot control: I define a bunch of tabs for different features, and the user picks the tool they want to use. The new feature is a hidden tab, and my unlock code simply unhides it.

Except it doesn't. You can't actually hide a PivotItem in a Pivot control; it's not supported. Instead you have to create the PivotItem in code and then add it to the pivot.Items collection, it is shows right up!

Except you actually get this Vista-era dialog:


The problem (not even on Stack Overflow) is simple: you can't update a pivot.Items collection while you're in a PivotSelectionChanged callback! The solution is to 

Instead add in code like this:

                    var task2 = this.Dispatcher.RunIdleAsync((arg) => {
                        MethodThatAddsPivotItem();
                    });


You don't have to wait for the task; it will just run when it's appropriate.


Saturday, January 7, 2023

The BEST way to calculate Standard Deviation

 The BEST way to calculating Standard Deviation 

The "classic" formulas aren't always the "easiest to program" formulas. Let's look at the classical formula to calculate a sample standard deviation:

σ = √ (Σ(xᵢ-x̅)²)/(n-1)

where σ is the standard deviation,  is the mean and n is the sample count. But note that we have to walk through the numbers twice: once to calculate the mean, and then again for the rest of the calculations. And yet, old fashioned scientific calculators with just a handful of registers (memory locations) could do this calculation by just entering in a column of numbers. What gives?

The answer is that although conceptually you are summing the square of the difference between each sample and the mean, you can actually do the calculation differently as long as you can keep track of n, the sum of x, and the sum of x². The resulting calculation is:

σ = √(Σ(xᵢ²) - ((Σxᵢ)² / n))/(n-1)

or, slightly more computery: for each new x:

1. n++

2. sumX += x;

3. sumXSquared += x*x;

4. varianceEstimate = (sumXSquared - ((sumX*sumX) / n) / (n-1)

5. stdDevEstimate = SQRT(varianceEstimate)


And you get a running estimate for the standard deviation. You also get the variance, but that's not often really needed.

Statisticians call these the "estimates" because the statistical theory is that there's a magic, universal reality for the actual population variance and standard deviation for which these samples provide an estimate. They aren't wrong :-)




Thursday, January 5, 2023

Handling the TITLE option

The information (i) element isn't any part of the Gopher RFC (most of Gopher is part of RFC 1436, and the Gopher URL is documented in RFC 4266), and therefore neither is the common practice of putting displaying an information element differently when a TITLE is put into the directory entry selector.

Thanks to a GopherSpace crawl, I know that out of about 2100 menus crawled, there were 305 information elements with a TITLE on them -- that's actually a pretty impressive percentage, and presumably it's driven by automated Gopher menu file creation. I'm wondering, though: as the programmer for a Gopher client, how many different Gopher client will render the TITLE elements specially? For that matter, how many Gopher pages put something into column 5, where the "+" in Gopher+ puts the Gopher+ indicator?

(I know that the first version of my own doesn't because I didn't know about the TITLE concept).

Let's start by analyzing pages that have too many columns (e.g., more than 5 columns, which corresponds to having more than 4 embedded tabs). There are five such directory entries in my GopherSpace crawl. Three are from a single page which, when I looked at is, is really an HTML page that's being served up as if it was Gopher. Of the remaining two entries, one has a user string consisting of three tabs instead of real data, and the other (from a different menu) has three tabs too many at the end of the line but and doesn't have any actual data in the columns.

We can also analyze the number of columns in directory entries

Almost all of the Gopher directory entries are the standard four columns. You can hardly tell the actual numbers, but 1907 entries were a single column, 9 were 2 columns, 895 were three, and 6590 were 5. There were 151840 entries with 4 columns.







History: writing fancy code on a plain compiler (Irix version)

 Writing fancy code on a plain compiler

This is a story of porting C++ code using all the latest features to a machine whose compiler was (ahem) definitely not supportive of advanced features :-/

Back in about 1997 and 1998 I was a software consultant and got hired by an old coworker at Avid to help them port their shiny new high-performance file-copying software to the Irix. IIRC, they had written it in “portable” C++ which I discovered was anything but

Writing code with no strings attached

 Amongst the other delights of 1997 era SGI workstations: SGI was a leader in creating the C++ STL. But the SGI people really, really didn’t like the proposed STL string types, so they just … didn’t. One of the many, many steps on my journey to porting this code was to create enough of a string class to compile.

 Other issues were that vector<> wasn’t compatible either, so I had some lovely #ifdef’s in the code.

What's in a namespace? Nothing. 

 The compiler also didn’t handle namespaces; they were read in an ignored. For most code this was a minor inconvenience, but the people writing this “portable” C++ code were a different breed. They had many, many classes with the same name and similar but different functionality. My solution was to create enough of a C++ “parser” to re-write the code. Turns out that if you ignore enough of the rules, you can make a C++ parser with just Lex 😊

Exceptionally fine threading

 But the absolute worst part was that the “portable” C++ code used both multiple threads (typical for networking code) and exceptions (still a new thing). The SGI compiler (which was the only compiler – I did a thorough look to find anybody else with a compiler) could handle threads, and could handle exceptions, but created incorrect binaries when dealing with both. And it didn’t matter if you threw any exceptions; the generated code was wrong regardless.

 My simple solution was to note that every single exception was uniformly caught exactly one level higher, and that none of their code ever returned a value. So I just made the exception-throwing methods return a value, instead. Simple, quick to implement and IMHO made the code a little nicer looking. This solution was rejected.

 The alternative solution was to use processes instead of threads. A “thread spawn” became a “process spawn”. And not just a process spawn: a process spawn all of the processes sharing their C++ memory so the data structures should be shared (and mutually updated, meaning using cross-process mutexes).

 This was an unglodly heavyweight project. But the pay was very nice.

TL/DR

 Moral of the story: never outrun your compiler?

Monday, January 2, 2023

Your Bluetooth is bad, January 2023 edition

 Your Bluetooth protocol is bad, January 2023

I've finally gotten over a big hump in my Bluetooth Device Controller program -- I've been poking around with adding and fiddling with devices, and that means that the code has been getting much more "experimental". That's not a good thing for an app that I ship, and which has over 35 thousand downloads! I've been working hard to convert the experimental code into an app that people can use without too much frustration.

With that, it's on to the next installment of this series on crappy Bluetooth protocols, focused on the Govee line of air sensors. The version 1.10 app supports the Govee 5074; the next version (presumably 1.11) will support the 5075 and 5106. All of them suffer from the same three flaws, and the 5106 has a unique and fun new flaw.

Don't shut off communications too early. All of the Govee devices like to shut down their Bluetooth connections really fast -- after about 4 seconds, they shut down the connection even if you've been talking on it. Other devices will wait until the connection has no traffic before shutting down.

Just transmit your freaking data. If you provide data, just provide it: make a characteristic, and make it readable and notifiable. 

Don't fold multiple values into decimal values. This is harder to explain. The Govee Air Sensor, as an example, sends out temperature, humidity, and air quality data in a single advertisement. But instead of just filling in 3 two-byte integer values, they instead take the temperature and multiple by 1_000_000. Then then take the humidity and multiple by 1_000. Then they add the air quality. This is all written as a single 4-byte integer.

To decode this monstrosity, you have to read in the 4-byte unsigned integer (in big-endian mode, even though Bluetooth is mostly little-endian). Then do a weird combination of MOD and integer divide operations to split out the three numbers.

Use the right Manufacturer code. The Govee devices mostly use a made-up EC88 manufacturer code; this is an unassigned value that nobody should be using. But the 5106 Air Quality monitor, for no apparent reason, uses the Nokia Phone code (they are manufacturer #1).

FYI: Common Timeout connection parameters

Each Bluetooth LE device can provide a set of connection parameters. These are decoded (now) by the Bluetooth Device Controller; they are part of the "Connection Parameters" (2A04) characteristic of the "Common Configuration" service (1800). The timeout is the last two bytes in little-endian format. For example, if the last two bytes are "90 01" in hex, that's 0190(hex) which is 400 (decimal). The value is in 10s of milliseconds, so the 400 (decimal) means 4 seconds for a timeout.

Looking at my device library, common settings here are:

  • 100 ms used by the SensorBug
  • 175 ms used by the Sphero
  • 4 sec used by the microbit, the govee, the kano coding wand, the viatom, the vion, and skoobot, smartibot and espruino
  • 5 sec used by the gems activity tracker
  • 6 sec used by the Mipow and the sense peanut
  • 10 sec used by the inkbird, lionel, the pyle, the powerup, the various sensor tags, and the dotti