PIPING in CMD.EXE

So, I'm trying something simple: redirect the output of a command to a file.  For no obvious reason, the utility decided that some of the data was 'stdout' and the rest was 'stderr'.  So all I need to do is the 2>&1 redirect trick, right?

Right.  But it's much stranger than that.

cvs annotate >data.dat 2>&1

is the correct command.  The wrong command is 

cvs annotate 2>&1 >data.dat

See the very subtle difference?  I sure didn't.  Because it didn't work right away, I had to start debugging it -- was it really putting output into those two pipes? (yes).  Can I pipe them each, one at a time? (yes)  Can I google for the correct syntax (yes).  Nowhere did I see the warning: you must do the 2>&1 pipe AFTER the >data.dat redirect.

Interestingly, if you first redirect output 1 (stdout), then pipe output 2 (stderr) into 1, then redirect output 1 to a diferent file, output 2 goes to the console.  I'd expect it to go into one of the two files that 1 was redirected to.  And output 1 eventually goes to the last file it was directed to.

ObLink: Microsoft badly documents the pipe operators at:

http://technet.microsoft.com/en-us/library/bb490982.aspx

Dreaded LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs

Perhaps the most irritating misfeature in the Microsoft compiler/linker family is the complete inability to make proper library files. What's a library file, you ask? Simple: it's where one person writes a bunch of code in a nice high-level language and then converts it into a single, neat "lib" file that a second person can then use in their program.

Conceptually, it's a no-brainer. Getting a good implementation of this, though, is beyond Microsoft's ability. It's easy to make a library -- it's just not possible to make a library that an arbitrary other person can use. (I'm currently interfacing a small program to Erlang; they have to ship three different libraries on Window just to get around the issues).

Microsoft keeps on changing their run time, so you have to match versions. They have multi thread and non multithreaded, and using MFC and not using MFC, and link against DLLs and link to a static library, debug and non debug -- there are too many options when what we really want to do is make a single (gigantic) "lib" that other people can use seamlessly.

Case in point: the dreaded LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library message that pops out when you go to use someone else's library. What it means is that some compiler setting that most people don't care about at all is set one way in the library and another way in your own code. Microsoft's advice ('use /NODEFAULTLIB') is useless.

What often works for me is the /VERBOSE:LIB switch (which you have to type in yourself). It lets you see exactly what libraries are being pulled in; from that list you can (generally) tell what has to be updated.

Shell Blog -- interesting blog ruined

So I've been poking at lots of interesting bits of Microsoft APIs, and was casting about for a new world to conquer. In the Shell blog I saw the 'Shell Namespace Extension: Created and Using the System Folder View Object'.

It's very interesting, it might solve some of my integration questions, and it's another couple hours out of my life that I would like back. What the entire (deleted) article didn't seem fit to mention is that when they say Microsoft Windows provides a default implementation of IShellView that what they really mean is most existing copies of Microsoft Windows do not provided any of these facilities

Because their code is (deleted) Vista only. Now, I don't object to Vista only code. But the Windows Explorer has only been around since what? Windows 95? Any code that talks about Shell Integration had better have a pretty good reason for not supporting their existing code base. And any code that does talk about integrating should mention any huge honking holes in the usability.

Microsoft: I want those hours back!

(Link to the Shell Revealed blog post: http://shellrevealed.com/blogs/shellblog/archive/2007/09/05/Shell-Namespace-Extension_3A00_-Adding-Custom-Command-Module-Items.aspx)

%PROGRAMFILES% - WTF?

Thesis: C:\Program Files was OK, but now it's horrible, and Microsoft is the reason.

So, in the old days there was C:\Program Files. That's totally OK -- it's good to have some centralized place to install programs to. But then Microsoft screwed it up. Someone noticed that Program Files isn't a localized name -- if you're German, or Mongolian, or anything but an English-speaker, it might as well be named C:\Giberish Stuff.

So there are two solutions. One is to change the name of the directory. The other is to change not what the actual place is but just how it's viewed in the Explorer. In typical Microsoft fashion, they did both. So I don't get the simplicity of a constant location, and I don't know what the user will see in Explorer.

Game Installers -- picking a directory

Over in Raymond Chen's blog I wrote a short comment about game installers and letting the user pick their installation directory. You can read it now at The Old New Thing and look for 'Peter'.

Raymond is right: your computer is already making hundreds of decisions for you, and nobody says word one. Then it makes one other decision that has just as much impact on you, and you get upset. That's silly. He's also right that exposing these decisions to the user is expensive: it wastes the time of people who don't care, it sucks up QA resources, and when people set it wrong (and they will) they call and complain. Or they don't call and uninstall my company's product, and I don't like it when that happens.

To put the problem in perspective: do you argue with the computer about the order in which USB hubs are enumerated? Do you ever find yourselve wishing you could load your program into one of your memory SIMs and not the other? Why can't it go in the other direction? Who picked "HKEY_CLASSES_ROOT\CAPICOM.Store.3" as a registry entry anyway? Why can't you pick that, too? And the answer is: because someone picked a way that's as good as any other way, and you just have to live with it.

Let me describe the actual program I work on. It's a sort of program manager that lets you to pick some programs, and the manager application downloads and installs them. One of our big advantages over the "classic" way of doing things is that we try to have a very smooth download and install experience with no pointless pop-up messages. I recently had the mispleasure of installing the same three games (that my company doesn't control) on a half a dozen laptops. In each case there was just dialog after dialog -- please pick a directory to install to, HEY! THAT DIRECTORY DOESN'T EXIST! OMFG, THE WORLD WILL EXPLODE! DID YOU REALLY MEAN THAT? -- would you like the program to actually work? -- would you like us to spam you? -- would you please read a this long text and mindlessly click OK? -- you have to scroll to the bottom first!. Yuck, yuck, yuck.

So what about the comments people left? Well, my company is picky about what programs are offered, and only allow programs that work when installed to %PROGRAMFILES% and in standard user mode, so Alexey's comments don't apply. We also handle the namespace issues that Will raised. I'm dubious about the namespace collision issue anyway -- the registry would have the same issue, as might the class names, along with the 'universally writable data' directory in c:\ProgramData. As far as disk space issues go (like SuperKoko and Igor Levicki raised) -- well, the programs aren't very big. cjm thought I had a bad attitude, but I don't. Indeed, my bosses often appreciate my general willingness to work on what needs to be worked on. cjm also denies that it causes more work, but that's not true and I have the bug reports to prove it.

Command-line parsing: more details

In the June 26, 2007 post "CMD.EXE Parsing - Splitting into Arguments" I listed the rules that Microsoft's C RTL uses when console apps split their command line into arguments. Rule number two was "2. The first argument (the program name) is parsed specially".

The actual rule for the first argument isn't terribly complex. Like regular arguments you're either in 'inquote' mode or not; when in 'inquote' mode the only end for the argument is the NUL character (but you can switch out of 'inquote' mode). Unlike the regular rules, there are escape characters -- a backslash is just a backslash, and there's no way to embed a double-quote. The double-quotes that switch in and out of 'inquote' mode are, of course, not considered part of the argument.

When not in 'inquote' mode a space or tab exits the parsing.

By the way -- Microsoft actually publishes the underlying source code for all of this. When you compile in DEBUG mode, you can even set breakpoints on the C RTL startup part of your code. In the "Visual Studio 8" directory (which is what they call 'Visual Studio 20005'), look for VC\crt\src\stdargv.c

Escaping Strings for MSFT RTL Parsing

Today's topic is how to convert any string into one that can be parsed an Microsoft RTL-using program. Note, however, that not all strings can be so converted -- in particular, any string with an embedded NUL character is not convertable.

Why do this? Because sometimes you have to generate output that other programs will read in through their command line. When you do, you have to escape your output so that the other program will handle it correctly. For instance, suppose I want to generate a string with a space in it (like "C:\Program Files\myprog\foo.txt") -- how can I write it out so that the other program gets exactly what I wrote?

This is the inverse problem of the last post. In the last post, the question answered was: what are the rules that the Microsoft Run-Time-Library has for parsing command line arguments. Note that today's solution is not complete: the various shells will ALSO do their own parsing (to expand environment variables) before handing off the command line.

The hard case, and therefore the one to tackle first, is to convert a string so that it can be put into Double-Quotes. The rules from the last entry say that the only character that has to be escaped is the duoble-quote, but it that it has very funny rules about multiple double quotes, and whether the number of preceeding backslashes is odd or even. It also depends on whether we're already in 'inquote' mode or not.

To make life easy, let's decide to always start off with a Double-Quote and always stay in 'inquote' mode. Further, we have to figure out how to deal with the following-Double-Quote rule (number 3.1.1 in 'inquote' mode). We could write fancy code to figure out if there was going to be a following Double-Quote, but instead I'll take the easy way out and always add in the following Double-Quote myself. That way the next Double-Quote doesn't have to be handled in an extra-special way.

The parsing rules say that backslashes aren't handled specially unless they are preceeding a Double-Quote. I'll have to keep track of how many there are in a row.

And finally we get the following code:

Buffer BORString::Convert::AnyToMSFTCArg (const Pointer& Src_In)
{
Pointer Src (Src_In);
Buffer Retval;
Retval.Resize (Src.StrLen() + 2 + 4);
  // Pre-size the return buffer once.  It will 
  // automatically resize as needed

 // p is the temporary pointer into Retval
Pointer p = Retval.p();

 // Keep track of how many backslashes we've 
 // seen that preceed a Double-Quote.
int NSlash = 0;

 // Temp. character.  Everything is wide-char,
 // but should trivially convert to narrow.
wchar_t c;

 // Always slip straight into inquote mode.
p.Append ('"');
while ((c=Src.GetW()) != (wchar_t)-1)
{
 if (c=='"')
 {
   // Double the backslashes
  for (int i=0; i<NSlash; i++)
  {
   p.Append ('\\');
  }
   // Always write out the Double-Quote 
   // twice just in case the next char
   // is also a quote.
  p.Append ('"');
  p.Append ('"');
 }
 else
 {
  p.Append (c);
 }
 NSlash = (c=='\\') ? NSlash+1 : 0;
}

 // The last slash, if any, also has to be doubled.
for (int i=0; i<NSlash; i++)
{
 p.Append ('\\');
}
p.Append ('"');

return Retval;
}

The 'Buffer' and 'Pointer' is part of a string library I use -- they are used so little here that I think you can figure out what they do.

The next topic: how CMD.EXE parses the command line before your program ever sees it.