This is the comment I couldn’t post to his website:
“It's error upon error for this one. Code 2869 means that the dialog designated as an error dialog doesn't work how Windows Installer needs an error dialog to work - see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/windows_installer_error_messages.asp. So the real error is being lost. Visual Studio is generating you a broken Error dialog.
I'm going to guess that the real error is that your custom action is failing, because it isn't privileged. On Windows Vista, only custom actions marked NoImpersonate get to run with administrative permissions (actually, they run as LocalSystem). Visual Studio cannot be told to mark a custom action as NoImpersonate (as far as I know). If you want to fix it after generating the MSI, you can use Orca (the MSI table editor, part of the Platform SDK, search for Orca.MSI) to add 2048 to the Type column of the three rows which use the InstallUtil DLL (which is the native code that calls into your managed DLL). I've also heard of tools which can be used to execute SQL against an MSI - it should be possible to do this with VBScript using the MSI object model.
The Windows Installer team does not recommend the use of managed code custom actions. This message does not seem to have got through to the Visual Studio deployment team. The recommendation is to use as few dependencies as possible, which generally translates to statically-linked C++ code.
Digging around in Reflector shows that you're using the custom actions to add the SDF to the ActiveSync Add/Remove Programs box. I'm not really a fan of this idea - and I note that Microsoft doesn't do this with the Compact Framework itself. It would be simplest to scrap this custom action completely. I also note that you're not handling rollback or uninstall. You should also use the /register flag to CEAppMgr.exe so that it doesn't install immediately on the connected device (or install when the next device is connected).
Windows Installer does support finding and executing an EXE that's already on the system as a custom action, but I don't think you can do this in Visual Studio.
You might want to consider a better installation solution, such as Windows Installer XML (WiX, http://wix.sourceforge.net/)”
I’ve been getting into WiX recently. I was going to do a presentation at DDD4, but not enough people voted for it. If you fancy attending any of the proposed sessions and can spare a Saturday, sign up now. (I’m waiting for the final agenda to be posted, but the places may all go before that happens.)