Monday, December 29, 2008

Issues with LIB files (a riff on Raymond Chen)

Today's The Old New Thing blog talks about a subject near and dear to my heart: why the Microsoft Linker needs some TLC when it comes to making lib files.

Let's back up.  What's a lib file, and why do they exist?  Well, a long time ago, computer programmers decided that the mechanics of converting what a programmer writes into what a computer understands should be done in two steps: a compiler turns the "source file" (what the programmer wrote) into a "object file" (a temporary file).  Then a linker turns the object file into a program.  It turns out that in general compiling is much slower than linking, and that most programs are composed of many source files, and that you don't have to re-compile a source file that hasn't changed.  So if your program consists of a hundred source files, and you change one, you only have to compile one, and then link them all together, and you get a program.

Now the twist come in: with bigger projects, it's handy to lump all of the object files together into a "lib" (library) file.  That way you don't have to tell the linker about all of your files; you just point it to the one lib file.

That's where the Microsoft Linker falls down, badly.  A common pattern for using lib files is to split the people using the lib file from the people who are making it.  As such, I want to pack my lib files with "everything" the user might want to use.  

But you can't.

As Raymond Chen points out in his (much better) blog , The Old New Thing, today (link: 
http://blogs.msdn.com/oldnewthing/archive/2008/12/29/9255240.aspx), it's eay to compile a file one way, put it into a library, and have a user compiling and linking another way to almost but not quite use your symbol.  It's very frustrating and requires a bit more fixing ability then we should expect end-programmers to have.

There are three things I'd like to see with the linker:
  1. On failure, the linker should try a fuzzy match ("I didn't find 'xyz' but I did see 'xyxz@4')
  2. The linker should keep track of how all of the files are being compiled ('the library was linked with /FOO but you are compiling with /NOTFOO)
  3. A library should be "stuffable" with all of the different versions of a file -- I should be able to pop in the unicode and ansi, debug and release, statically linked and dynamically linked versions of all of my files. 
The goal, of course, is to let me as a library-creator make a library that other people can "just use" -- not "just use with a lot of swearing".

2 comments:

Raymond said...

1. Fuzzy matching would be even worse! Your program links and then crashes (because the calling convention doesn't match or because you passed the wrong number of parameters).

2. The linker does keep that information --- in the decoration!

3. You can do that today, just throw all the obj's into one giant library. As long as the different versions are differently decorated, you're good.

Sunrise Programmer said...

Re: Raymond:
Regretfully, not everything is in the name mangling -- for example, I can't make a /debug and /nodebug version of an OBJ file, put them both into a LIB file and expect the linker to sort everything out.

There are a lot of very irritating ways of compiling two files that aren't actually compatible with each other (for example: if one file uses the C RTL statically and the other as as DLL). The linker actually tracks these, but instead of producing a useful message merely says, "sorry: some of your files aren't compatible".

And for fuzzy matching -- that would, of course, only be for the diagnostics. Obviously if the signatures don't match then the functions aren't actually compatible (except in rare cases)