The Moth

The problem
When I was trying to call the WerRegisterFile API as described previously, I run into an interesting situation: Calling the API without specifying any CharSet should have attempted first the WerRegisterFile. If I specify CharSet.Auto, it should first attempt WerRegisterFileW and if it cannot find that then go back to try WerRegisterFile. Either way, you would have thought that this would succeed or blow. As it turns out, leaving the CharSet off, silently fails (no exception, return value is 0, Marshal.GetLastWin32Error is 0 etc) but specifying CharSet.Auto works as expected! For the record, CharSet.Unicode behaves like CharSet.Auto and CharSet.Ansi fails like when we omit altogether CharSet from the DllImport declaration.

Download a repro VS2005 project here (obviously requires Vista).


Solution
Looking at kernel32.dll (which exports this function) shows that there is only one entry for the API (screenshot of dumpbin). This seems to be a pattern with Vista, according to my colleague MikeT.

Well it seems that after a series of chats with Mike, he updated his post with useful thoughts. Read his 4th thought in the updated section and come back... I’ll wait...

[...time passes...]

So to prove the thought, I looked at another Vista API that takes strings: RegisterApplicationRestart. It too has no trailing A or W (screenshot). So calling this API without CharSet also gives no errors, but the string you get passed back to the application at the command line is definitely not the one I passed to it (something Chinese-looking on my system when I gave it something English).

That rests the case. I should have realised earlier that CharSet is not just about finding the correct entry point but also about how the string will be marshalled.

As an excuse I should state that, having mostly coded for WinCE, I think I have been spoilt since that is all Unicode and I can’t remember using CharSet with any of NETCF pinvokes and they all worked fine :)

Final thoughts
1. Whoever thought the default of CharSet should be Ansi has a lot to answer for. Note that it is a C# decision as the CLR defaults to Auto!
2. I wonder if the Vista change (that threw me off track) is going to bite others or if it just muppets like me that fell for it :S
3. Why is WerRegiserFile in kernel32.dll when all the other WER APIs are in wer.dll?!
4. WerRegisterFile should be giving some indication to the developer that it failed. At the end of the day, you give it a path (string), it thinks it is a different string (path), so it cannot find the file and then it silently fails! WTF?
5. I detest silent failures. Make an explosion next time. A big loud one or a quiet one, but not a silent one. Silent ones are the worst!