Issues with LIB files (a riff on Raymond Chen)

Today's The Old New Thing blog talks about a subject near and dear to my heart: why the Microsoft Linker needs some TLC when it comes to making lib files.

Let's back up.  What's a lib file, and why do they exist?  Well, a long time ago, computer programmers decided that the mechanics of converting what a programmer writes into what a computer understands should be done in two steps: a compiler turns the "source file" (what the programmer wrote) into a "object file" (a temporary file).  Then a linker turns the object file into a program.  It turns out that in general compiling is much slower than linking, and that most programs are composed of many source files, and that you don't have to re-compile a source file that hasn't changed.  So if your program consists of a hundred source files, and you change one, you only have to compile one, and then link them all together, and you get a program.

Now the twist come in: with bigger projects, it's handy to lump all of the object files together into a "lib" (library) file.  That way you don't have to tell the linker about all of your files; you just point it to the one lib file.

That's where the Microsoft Linker falls down, badly.  A common pattern for using lib files is to split the people using the lib file from the people who are making it.  As such, I want to pack my lib files with "everything" the user might want to use.  

But you can't.

As Raymond Chen points out in his (much better) blog , The Old New Thing, today (link: 

http://blogs.msdn.com/oldnewthing/archive/2008/12/29/9255240.aspx), it's eay to compile a file one way, put it into a library, and have a user compiling and linking another way to almost but not quite use your symbol.  It's very frustrating and requires a bit more fixing ability then we should expect end-programmers to have.

There are three things I'd like to see with the linker:

1. On failure, the linker should try a fuzzy match ("I didn't find 'xyz' but I did see 'xyxz@4')
2. The linker should keep track of how all of the files are being compiled ('the library was linked with /FOO but you are compiling with /NOTFOO)
3. A library should be "stuffable" with all of the different versions of a file -- I should be able to pop in the unicode and ansi, debug and release, statically linked and dynamically linked versions of all of my files. 

The goal, of course, is to let me as a library-creator make a library that other people can "just use" -- not "just use with a lot of swearing".

About Me and Where I Work

Thanks to the local economy, I recently was part of a layoff at a company which I'll simply describe at the "Purple Company".  People who work there will know what I mean.  Luckily I picked up a new job very quickly as (cue ominous music) one of the networking people for Microsoft.

I wouldn't bother mentioning it, but at the NEO (New Employee Orientation -- everything at MSFT has an acronym) it was mentioned -- and repeated -- and then emphasized:

We can write a blog.  MSFT likes us to write blogs.  But the legal staff is worried about looking sneaky, and they'd like all of us MSFT bloggers to mention that we are, indeed, MSFT employees.

So there.  I work at Microsoft.  I won't put in a disclaimer that I don't speak for Microsoft because IMHO that should be the default legal status on the internet.  And you'll see from previous entries that I've been definite about MSFT's poor documentation (and sometimes bizarre code).

PIPING in CMD.EXE

So, I'm trying something simple: redirect the output of a command to a file.  For no obvious reason, the utility decided that some of the data was 'stdout' and the rest was 'stderr'.  So all I need to do is the 2>&1 redirect trick, right?

Right.  But it's much stranger than that.

cvs annotate >data.dat 2>&1

is the correct command.  The wrong command is 

cvs annotate 2>&1 >data.dat

See the very subtle difference?  I sure didn't.  Because it didn't work right away, I had to start debugging it -- was it really putting output into those two pipes? (yes).  Can I pipe them each, one at a time? (yes)  Can I google for the correct syntax (yes).  Nowhere did I see the warning: you must do the 2>&1 pipe AFTER the >data.dat redirect.

Interestingly, if you first redirect output 1 (stdout), then pipe output 2 (stderr) into 1, then redirect output 1 to a diferent file, output 2 goes to the console.  I'd expect it to go into one of the two files that 1 was redirected to.  And output 1 eventually goes to the last file it was directed to.

ObLink: Microsoft badly documents the pipe operators at:

http://technet.microsoft.com/en-us/library/bb490982.aspx

Dreaded LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs

Perhaps the most irritating misfeature in the Microsoft compiler/linker family is the complete inability to make proper library files. What's a library file, you ask? Simple: it's where one person writes a bunch of code in a nice high-level language and then converts it into a single, neat "lib" file that a second person can then use in their program.

Conceptually, it's a no-brainer. Getting a good implementation of this, though, is beyond Microsoft's ability. It's easy to make a library -- it's just not possible to make a library that an arbitrary other person can use. (I'm currently interfacing a small program to Erlang; they have to ship three different libraries on Window just to get around the issues).

Microsoft keeps on changing their run time, so you have to match versions. They have multi thread and non multithreaded, and using MFC and not using MFC, and link against DLLs and link to a static library, debug and non debug -- there are too many options when what we really want to do is make a single (gigantic) "lib" that other people can use seamlessly.

Case in point: the dreaded LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library message that pops out when you go to use someone else's library. What it means is that some compiler setting that most people don't care about at all is set one way in the library and another way in your own code. Microsoft's advice ('use /NODEFAULTLIB') is useless.

What often works for me is the /VERBOSE:LIB switch (which you have to type in yourself). It lets you see exactly what libraries are being pulled in; from that list you can (generally) tell what has to be updated.

Shell Blog -- interesting blog ruined

So I've been poking at lots of interesting bits of Microsoft APIs, and was casting about for a new world to conquer. In the Shell blog I saw the 'Shell Namespace Extension: Created and Using the System Folder View Object'.

It's very interesting, it might solve some of my integration questions, and it's another couple hours out of my life that I would like back. What the entire (deleted) article didn't seem fit to mention is that when they say Microsoft Windows provides a default implementation of IShellView that what they really mean is most existing copies of Microsoft Windows do not provided any of these facilities

Because their code is (deleted) Vista only. Now, I don't object to Vista only code. But the Windows Explorer has only been around since what? Windows 95? Any code that talks about Shell Integration had better have a pretty good reason for not supporting their existing code base. And any code that does talk about integrating should mention any huge honking holes in the usability.

Microsoft: I want those hours back!

(Link to the Shell Revealed blog post: http://shellrevealed.com/blogs/shellblog/archive/2007/09/05/Shell-Namespace-Extension_3A00_-Adding-Custom-Command-Module-Items.aspx)

%PROGRAMFILES% - WTF?

Thesis: C:\Program Files was OK, but now it's horrible, and Microsoft is the reason.

So, in the old days there was C:\Program Files. That's totally OK -- it's good to have some centralized place to install programs to. But then Microsoft screwed it up. Someone noticed that Program Files isn't a localized name -- if you're German, or Mongolian, or anything but an English-speaker, it might as well be named C:\Giberish Stuff.

So there are two solutions. One is to change the name of the directory. The other is to change not what the actual place is but just how it's viewed in the Explorer. In typical Microsoft fashion, they did both. So I don't get the simplicity of a constant location, and I don't know what the user will see in Explorer.

Game Installers -- picking a directory

Over in Raymond Chen's blog I wrote a short comment about game installers and letting the user pick their installation directory. You can read it now at The Old New Thing and look for 'Peter'.

Raymond is right: your computer is already making hundreds of decisions for you, and nobody says word one. Then it makes one other decision that has just as much impact on you, and you get upset. That's silly. He's also right that exposing these decisions to the user is expensive: it wastes the time of people who don't care, it sucks up QA resources, and when people set it wrong (and they will) they call and complain. Or they don't call and uninstall my company's product, and I don't like it when that happens.

To put the problem in perspective: do you argue with the computer about the order in which USB hubs are enumerated? Do you ever find yourselve wishing you could load your program into one of your memory SIMs and not the other? Why can't it go in the other direction? Who picked "HKEY_CLASSES_ROOT\CAPICOM.Store.3" as a registry entry anyway? Why can't you pick that, too? And the answer is: because someone picked a way that's as good as any other way, and you just have to live with it.

Let me describe the actual program I work on. It's a sort of program manager that lets you to pick some programs, and the manager application downloads and installs them. One of our big advantages over the "classic" way of doing things is that we try to have a very smooth download and install experience with no pointless pop-up messages. I recently had the mispleasure of installing the same three games (that my company doesn't control) on a half a dozen laptops. In each case there was just dialog after dialog -- please pick a directory to install to, HEY! THAT DIRECTORY DOESN'T EXIST! OMFG, THE WORLD WILL EXPLODE! DID YOU REALLY MEAN THAT? -- would you like the program to actually work? -- would you like us to spam you? -- would you please read a this long text and mindlessly click OK? -- you have to scroll to the bottom first!. Yuck, yuck, yuck.

So what about the comments people left? Well, my company is picky about what programs are offered, and only allow programs that work when installed to %PROGRAMFILES% and in standard user mode, so Alexey's comments don't apply. We also handle the namespace issues that Will raised. I'm dubious about the namespace collision issue anyway -- the registry would have the same issue, as might the class names, along with the 'universally writable data' directory in c:\ProgramData. As far as disk space issues go (like SuperKoko and Igor Levicki raised) -- well, the programs aren't very big. cjm thought I had a bad attitude, but I don't. Indeed, my bosses often appreciate my general willingness to work on what needs to be worked on. cjm also denies that it causes more work, but that's not true and I have the bug reports to prove it.