Mike Dimmick's Bleurgh

MFC message handling

2010-03-05T16:32:00.003Z

Goran, a commenter at Raymond's blog, asks:

As one of those people who only have a hammer :-( ... It looks like MFC just told us off with this? WM_GETDLGCODE has meaningfull wParam and lParam, but CWnd::OnGetDlgCode has none of that. Anyone used OnGetDlgCode? Help?

The specific context is handling the return key - Windows tells you what key caused WM_GETDLGCODE to be fired in the wParam parameter, but MFC doesn't pass that information on to the OnGetDlgCode handler.

Something that often catches people out with MFC is the thought that you have to use the specific handler that MFC declares for a certain message. In fact, it's simply a shortcut - ON_WM_GETDLGCODE in the message map causes the message map handling to go down a certain branch that happens to call the handler function with no parameters.

Instead, you can simply use ON_MESSAGE in the message map, using WM_GETDLGCODE and declaring the function as taking WPARAM, LPARAM and returning LRESULT. Do make sure you don't also include ON_WM_GETDLGCODE.

The message map macros themselves simply produce a table of message number ranges and handler function pointers. The MFC WindowProc scans the table to find a handler for a given message number, then - based on an enumeration value set by the message map macro which indicates the argument types - decodes wParam and lParam, then calls the handler with the appropriate arguments.

Many of the predefined helper message map macros, which usually don't take any arguments, define the function name that you're expected to implement. ON_MESSAGE allows you to specify the function name (as you'd expect, as a generic macro).

The Visual Studio Wizards have always simplified the full power of message maps, not supporting custom message numbers (which you have to use ON_MESSAGE for) and not supporting ranges of command IDs (ON_COMMAND_RANGE, ON_NOTIFY_RANGE, etc). Few people even seem to realise that you can use the same handler function for more than one message map macro, for example if you have discontinuous ranges of control IDs.

Four-and-a-half months

2009-10-13T22:45:00.001+01:00

…that’s how long I was running Windows 7 RC1 without switching back to my hard disk with Windows Vista. The System event log shows I last booted it on 31 May 2009.

This is practically the last action I’m going to take before wiping this disk – the Vista disk – and installing Windows 7 RTM. Just a little check here and there to find anything that I might not have backed up when transferring over to 7 RC1, then it’s formatting time.

The only thing I’ve missed? The extra 200GB of disk space!

In retrospect Windows Vista has not been a bad operating system, but it had some pretty rough edges, particularly at the start with poor hardware support. It was a rush 18–month project, after the initial three-year disaster of Longhorn, and in some areas it shows. Windows 7 has had a further two-and-a-half years to bake, and a further two-and-a-half years of device driver and hardware development; it’s not fundamentally different but feels more complete.

Technical details on Humax problem

2009-07-11T12:47:00.001+01:00

I thought I’d break this into another post so that the fix is easier to understand for non-technical people.

Comparing the old and new driver source shows that the definition of the libusb_request structure had changed. Visual inspection suggested that this was probably the source of the compatibility issue, when using the old DLL with the new driver.

When using the new DLL, the following error occurs: “The ordinal 76 could not be located in the dynamic link library libusb0.dll.” Inspecting the program and DLL shows that while the DLL’s entry points are exported by name, the program links to the DLL by ordinal. This is peculiar; while it’s a little faster than string-matching, and it saves a little space, care has to be taken to keep the ordinals the same between versions of a DLL. Reading the .def file shows that the ordinals weren’t specified, and that numerous functions were added and removed between the two versions.

My fix is to add specific ordinal declarations to the .def file for functions that humax.exe is using, so that the correct functions are used. I did this, and recompiled the DLL with the Windows Vista WDK (Windows Driver Kit). Success! Or, at least, as much success as I’ve had with Windows XP 32–bit; the box still has a tendency to hang during file transfer.

File Attachment: libusb0.def (1 KB)

Downloading from your Humax PVR9200T on Windows x64

2009-07-11T12:24:00.001+01:00

The Humax PVR9200T is a pretty good Freeview/digital terrestrial personal video recorder. I’ve had mine for a few years now. Over time, the disk fills up with some programmes you want to keep. It’s possible to copy recorded programmes from the box to your computer, over USB.

Unfortunately the supplied eLinker software is not highly-regarded. As a result some replacements have been written, such as Humax Media Controller. However, the driver supplied only works on 32–bit Windows.

The USB library and driver used are a Win32 port of libusb, a driver which makes it possible to write your USB transfers fully in user-mode. A later version (0.1.12.2) has both 32– and 64–bit versions of the driver. However, the updated driver doesn’t work with the libusb0.dll supplied with Media Controller 1.05, and the program won’t run if you overwrite it with the new DLL.

I’ve rebuilt the new DLL so it will work with the program. The attached ZIP file contains the new version and the new drivers. To use, extract the contents of the folder. Copy the libusb0.dll in the top-level folder to the Humax Media Controller folder, overwriting the current version. Then, use Update Driver in Device Manager to install the driver from the Driver folder.

Windows Vista introduced a driver signing requirement for x64, and this driver isn’t signed. To get it to load on Windows Vista and Windows 7, reboot, and press F8 until the boot menu appears. From this menu, select Disable Driver Signature Enforcement, then continue to boot normally.

Hopefully this will help other people with the x64 versions of Windows!

File Attachment: HumaxPVR.zip (78 KB)

ADSL Router suggestions?

2008-07-30T22:29:00.001+01:00

OK, I think my router has become too hackable. The reliability has hit the toilet, and I’ve had two messages from my ISP warning me that there has been odd traffic. The first time I got a warning message I think I was using BitTorrent at the time, but the second was yesterday at around lunchtime – when my computers were all off and I was at work.

So, list of ~~demands~~ requirements:

Wired router with a few Ethernet ports
802.11g WiFi with at least WPA encryption (can’t do WPA2 on the handheld)
Xbox Live compatible
DHCP implementation that works
DNS server address passthrough from DHCP server (i.e. I want to use my ISP’s DNS servers, not have the box do DNS for me)
ADSL2+ compatible, for whenever that gets rolled out by Demon
PPPoA since this is of course a UK requirement
A firewall that can be completely switched off or is actually configurable intelligibly
Not excessively burdened with blinkenlights
Will be updated regularly with patches for whatever its OS is

That last point is critical. There are way too many routers out there that run Linux and aren’t patched. D-Link haven’t produced an update since last November and even that’s not officially official, you have to find it on the FTP site. The last official update is September 2006.

I don’t need 802.11n as a) it’s still a draft, b) none of my other equipment needs it, and c) 54Mbps is already sufficiently fast to access the Internet. I don’t do much copying between devices and I wouldn’t do it when they were running on battery anyway.

I’m half tempted to try building a Windows CE kernel for it myself.

How do I see what the JIT compiled, anyway?

2008-03-10T01:49:00.001Z

You use the SOS debugger extension’s !U command.

Yeah, that’s not very helpful. Here’s how to get from that terse description to something useful.

As an example, let’s use the following trivial program:

using System;

public class Simple
{
   public static int Main()
   {
       Console.WriteLine( "Hello World!" );
       Console.ReadLine();
       
       return 0;
   }
}

Dump this snippet into Notepad, save as simple.cs file and compile it. We want to see what it’ll be like when the user runs it, so we’ll do a release build. From the VS2005 Command Prompt, run csc /debug:pdbonly /o+ simple.cs. (I find it’s quicker to do command-line compiles for simple test programs than firing up Visual Studio. It’s not as if we’re designing a Form.) If you’re doing this on a 64–bit computer, like me, add /platform:x86 so we get consistent output across platforms. (Marking as x86 means that the executable headers are set to indicate a 32–bit program. The IL is the same whatever you set here, but if you leave it as anycpu, the default, the .NET Framework will create a 64–bit process which obviously gives different output.)

Why call Console.ReadLine? Well, the JIT compiler is helpful (or at least it thinks it is). If you start a program under the debugger, it will generate less optimized code because it thinks you want to debug it, but that changes the code from what the user will see. So I want to add a stop point in the program so I can attach the debugger after the process has started.

The next thing you’ll need is the Debugging Tools for Windows kit. Grab the 32–bit kit – the 64–bit kit can debug 32–bit processes, but SOS (the debugger extension DLL which implements the commands we’re going to use) doesn’t appear to work. We’re going to use WinDBG, which is slightly friendlier than the other debuggers although not a lot!

Open WinDBG. Run simple.exe and, when it’s waiting for input, go to File/Attach to a Process in WinDBG. You’ll see a list of other processes (and if running Windows Vista with UAC enabled, or on XP as a standard user, a load of access denied errors). Select simple.exe from the list and hit OK. WinDBG automatically stops once you’ve attached to a process so you can start manipulating the program straight away. This is the output I got:

Microsoft (R) Windows Debugger Version 6.8.0004.0 X86
Copyright (c) Microsoft Corporation. All rights reserved.

*** wait with pending attach
Symbol search path is: C:\Windows;C:\Windows\System32;C:\Windows\SysWOW64;SRV*C:\WebSymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
ModLoad: 008c0000 008c8000   C:\Users\Mike\Documents\Programming\simple.exe
ModLoad: 776a0000 77800000   C:\Windows\SysWOW64\ntdll.dll
ModLoad: 79000000 79046000   C:\Windows\system32\mscoree.dll
ModLoad: 75b70000 75c80000   C:\Windows\syswow64\KERNEL32.dll
ModLoad: 76ce0000 76da6000   C:\Windows\syswow64\ADVAPI32.dll
ModLoad: 76e10000 76f00000   C:\Windows\syswow64\RPCRT4.dll
ModLoad: 75850000 758b0000   C:\Windows\syswow64\Secur32.dll
ModLoad: 75a80000 75ad8000   C:\Windows\syswow64\SHLWAPI.dll
ModLoad: 77110000 771a0000   C:\Windows\syswow64\GDI32.dll
ModLoad: 77040000 77110000   C:\Windows\syswow64\USER32.dll
ModLoad: 76c30000 76cda000   C:\Windows\syswow64\msvcrt.dll
ModLoad: 75d00000 75d60000   C:\Windows\system32\IMM32.DLL
ModLoad: 76940000 76a08000   C:\Windows\syswow64\MSCTF.dll
ModLoad: 75a70000 75a79000   C:\Windows\syswow64\LPK.DLL
ModLoad: 759c0000 75a3d000   C:\Windows\syswow64\USP10.dll
ModLoad: 746c0000 7485e000   C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6001.18000_none_5cdbaa5a083979cc\comctl32.dll
ModLoad: 79e70000 7a3ff000   C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
ModLoad: 75390000 7542b000   C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.1434_none_d08b6002442c891f\MSVCR80.dll
ModLoad: 75e30000 7693f000   C:\Windows\syswow64\shell32.dll
ModLoad: 771b0000 772f4000   C:\Windows\syswow64\ole32.dll
ModLoad: 790c0000 79bf6000   C:\Windows\assembly\NativeImages_v2.0.50727_32\mscorlib\5b3e3b0551bcaa722c27dbb089c431e4\mscorlib.ni.dll
ModLoad: 79060000 790b6000   C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorjit.dll
(1dc.12b4): Break instruction exception - code 80000003 (first chance)
eax=7efaf000 ebx=00000000 ecx=00000000 edx=7770d2d4 esi=00000000 edi=00000000
eip=776b0004 esp=04c6fa00 ebp=04c6fa2c iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!DbgBreakPoint:
776b0004 cc              int     3

You enter your debugging commands in the edit box at the bottom, next to the prompt 0:003>. This means we’re debugging the 0th process we’re attached to, and our commands by default apply to thread number 3. Four threads? Well, thread 0 is our main thread, thread 1 was created by the CLR’s debugging support in case we attached a debugger, thread 2 is the finalizer thread, and thread 3 was just created by WinDBG so it could stop the process safely. (You can see this by running ~* k, although you’ll need to be set up to get debugging symbols from the symbol server to get good stack traces.) The Debugging Tools debuggers automatically stop the process when you attach, so that you can start manipulating the process straight away.

The first thing we need to do is ask WinDBG to load the SOS extension. This is installed with the CLR, so we ask it to load from the same folder that mscorwks.dll (the DLL which implements the virtual machine, effectively the guts of the CLR) lives in:

0:003> .loadby sos.dll mscorwks

Now we can see what state of the managed threads are in by running !threads. Any command beginning ! comes from a debugger extension DLL, and they can be disambiguated if necessary, but they’re searched in reverse order loaded (last loaded = first searched) so anything we want will be found in SOS anyway.

Right. To see what we can do, run !help.

0:003> !help
-------------------------------------------------------------------------------
SOS is a debugger extension DLL designed to aid in the debugging of managed
programs. Functions are listed by category, then roughly in order of
importance. Shortcut names for popular functions are listed in parenthesis.
Type "!help " for detailed info on that function. 

Object Inspection                  Examining code and stacks
-----------------------------      -----------------------------
DumpObj (do)                       Threads
DumpArray (da)                     CLRStack
DumpStackObjects (dso)             IP2MD
DumpHeap                           U
DumpVC                             DumpStack
GCRoot                             EEStack
ObjSize                            GCInfo
FinalizeQueue                      EHInfo
PrintException (pe)                COMState
TraverseHeap                       BPMD 

Examining CLR data structures      Diagnostic Utilities
-----------------------------      -----------------------------
DumpDomain                         VerifyHeap
EEHeap                             DumpLog
Name2EE                            FindAppDomain
SyncBlk                            SaveModule
DumpMT                             GCHandles
DumpClass                          GCHandleLeaks
DumpMD                             VMMap
Token2EE                           VMStat
EEVersion                          ProcInfo 
DumpModule                         StopOnException (soe)
ThreadPool                         MinidumpMode 
DumpAssembly                       
DumpMethodSig                      Other
DumpRuntimeTypes                   -----------------------------
DumpSig                            FAQ
RCWCleanupList
DumpIL

To find out a little more about !U, let’s run !help U:

0:003> !help U
-------------------------------------------------------------------------------
!U [-gcinfo] [-ehinfo] <MethodDesc address> | <Code address> 

Presents an annotated disassembly of a managed method when given a MethodDesc
pointer for the method, or a code address within the method body. Unlike the
debugger "U" function, the entire method from start to finish is printed,
with annotations that convert metadata tokens to names.


...
03ef015d b901000000       mov     ecx,0x1
03ef0162 ff156477a25b     call   dword ptr [mscorlib_dll+0x3c7764 (5ba27764)] (System.Console.InitializeStdOutError(Boolean), mdToken: 06000713)
03ef0168 a17c20a701       mov     eax,[01a7207c] (Object: SyncTextWriter)
03ef016d 89442414         mov     [esp+0x14],eax

If you pass the -gcinfo flag, you'll get inline display of the GCInfo for
the method. You can also obtain this information with the !GCInfo command.

If you pass the -ehinfo flag, you'll get inline display of exception info
for the method. (Beginning and end of try/finally/catch handlers, etc.).
You can also obtain this information with the !EHInfo command.

Right, so we need the address of a MethodDesc structure, or the address of the code itself. How can we get one of these? There’s a command called DumpMD…

0:003> !help dumpmd
-------------------------------------------------------------------------------
!DumpMD 

This command lists information about a MethodDesc. You can use !IP2MD to turn 
a code address in a managed function into a MethodDesc:

0:000> !dumpmd 902f40
Method Name: Mainy.Main()
Class: 03ee1424
MethodTable: 009032d8
mdToken: 0600000d
Module: 001caa78
IsJitted: yes
m_CodeOrIL: 03ef00b8

If IsJitted is "yes," you can run !U on the m_CodeOrIL pointer to see a 
disassembly of the JITTED code. You can also call !DumpClass, !DumpMT, 
!DumpModule on the Class, MethodTable and Module fields above.

We’re getting a bit closer. Let’s try DumpModule…

0:003> !help DumpModule
-------------------------------------------------------------------------------
!DumpModule [-mt] 

You can get a Module address from !DumpDomain, !DumpAssembly and other 
functions. Here is sample output:

0:000> !dumpmodule 1caa50
Name: C:\pub\unittest.exe
Attributes: PEFile
Assembly: 001ca248
LoaderHeap: 001cab3c
TypeDefToMethodTableMap: 03ec0010
TypeRefToMethodTableMap: 03ec0024
MethodDefToDescMap: 03ec0064
FieldDefToDescMap: 03ec00a4
MemberRefToDescMap: 03ec00e8
FileReferencesMap: 03ec0128
AssemblyReferencesMap: 03ec012c
MetaData start address: 00402230 (1888 bytes)

The Maps listed map metadata tokens to CLR data structures. Without going into 
too much detail, you can examine memory at those addresses to find the 
appropriate structures. For example, the TypeDefToMethodTableMap above can be 
examined:

0:000> dd 3ec0010
03ec0010  00000000 00000000 0090320c 0090375c
03ec0020  009038ec ...

This means TypeDef token 2 maps to a MethodTable with the value 0090320c. You 
can run !DumpMT to verify that. The MethodDefToDescMap takes a MethodDef token 
and maps it to a MethodDesc, which can be passed to !DumpMD.

There is a new option "-mt", which will display the types defined in a module,
and the types referenced by the module. For example:

0:000> !dumpmodule -mt 1aa580
Name: C:\pub\unittest.exe
......
MetaData start address: 0040220c (1696 bytes)

Types defined in this module

      MT    TypeDef Name
------------------------------------------------------------------------------
030d115c 0x02000002 Funny
030d1228 0x02000003 Mainy

Types referenced in this module

      MT    TypeRef Name
------------------------------------------------------------------------------
030b6420 0x01000001 System.ValueType
030b5cb0 0x01000002 System.Object
030fceb4 0x01000003 System.Exception
0334e374 0x0100000c System.Console
03167a50 0x0100000e System.Runtime.InteropServices.GCHandle
0336a048 0x0100000f System.GC

We still need a Module to pass to this command, but we’re getting there. Maybe DumpDomain can help us? I’ll skip the help text this time.

0:003> !DumpDomain
--------------------------------------
System Domain: 7a3bc8b8
LowFrequencyHeap: 7a3bc8dc
HighFrequencyHeap: 7a3bc934
StubHeap: 7a3bc98c
Stage: OPEN
Name: None
--------------------------------------
Shared Domain: 7a3bc560
LowFrequencyHeap: 7a3bc584
HighFrequencyHeap: 7a3bc5dc
StubHeap: 7a3bc634
Stage: OPEN
Name: None
Assembly: 0052f848
--------------------------------------
Domain 1: 00516800
LowFrequencyHeap: 00516824
HighFrequencyHeap: 0051687c
StubHeap: 005168d4
Stage: OPEN
SecurityDescriptor: 00517d30
Name: simple.exe
Assembly: 0052f848 [C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]
ClassLoader: 0050e470
SecurityDescriptor: 0051f5d0
  Module Name
790c2000 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll

Assembly: 00535ac0 [C:\Users\Mike\Documents\Programming\simple.exe]
ClassLoader: 0050e630
SecurityDescriptor: 00535a38
  Module Name
000f2c3c C:\Users\Mike\Documents\Programming\simple.exe

At last, we have the address of something we can use. Let’s get the module information for simple.exe. For that we need the address next to it in the module list for the simple.exe assembly.

0:003> !dumpmodule -mt f2c3c
Name: C:\Users\Mike\Documents\Programming\simple.exe
Attributes: PEFile 
Assembly: 00535ac0
LoaderHeap: 00000000
TypeDefToMethodTableMap: 000f0038
TypeRefToMethodTableMap: 000f0040
MethodDefToDescMap: 000f005c
FieldDefToDescMap: 000f0068
MemberRefToDescMap: 000f006c
FileReferencesMap: 000f0088
AssemblyReferencesMap: 000f008c
MetaData start address: 008c206c (740 bytes)

Types defined in this module

      MT    TypeDef Name
------------------------------------------------------------------------------
000f3030 0x02000002 Simple

Types referenced in this module

      MT    TypeRef Name
------------------------------------------------------------------------------
790fd0f0 0x01000001 System.Object
79101118 0x01000006 System.Console

Now we can get the MethodTable for the Simple class:

0:003> !dumpmt -md f3030
EEClass: 000f1174
Module: 000f2c3c
Name: Simple
mdToken: 02000002  (C:\Users\Mike\Documents\Programming\simple.exe)
BaseSize: 0xc
ComponentSize: 0x0
Number of IFaces in IFaceMap: 0
Slots in VTable: 6
--------------------------------------
MethodDesc Table
   Entry MethodDesc      JIT Name
79371278   7914b928   PreJIT System.Object.ToString()
7936b3b0   7914b930   PreJIT System.Object.Equals(System.Object)
7936b3d0   7914b948   PreJIT System.Object.GetHashCode()
793624d0   7914b950   PreJIT System.Object.Finalize()
00310070   000f3020      JIT Simple.Main()
000fc01c   000f3028     NONE Simple..ctor()

Finally we have something we can pass to !U. Note the JIT column. Here, PreJIT means that the code has come from a native image generated by ngen, JIT means that the JIT compiler in this process has generated the code, and NONE means that the method hasn’t been compiled yet. We didn’t override any of System.Object’s virtual methods, so they occupy the first four places in our MethodTable.

I placed the call to Console.ReadLine at the end of the program so everything has already been compiled – remember, the JIT compiles code on-demand, one method at a time, when that method is called (barring very simple methods which might be inlined into their callers). Let’s just get on and show the code for Main.

0:003> !U f3020
Normal JIT generated code
Simple.Main()
Begin 00310070, size 36
00310070 833d8c10920300  cmp     dword ptr ds:[392108Ch],0
00310077 750a            jne     00310083
00310079 b901000000      mov     ecx,1
0031007e e8e1580579      call    mscorlib_ni+0x2a5964 (79365964) (System.Console.InitializeStdOutError(Boolean), mdToken: 06000770)
00310083 8b0d8c109203    mov     ecx,dword ptr ds:[392108Ch] (Object: System.IO.TextWriter+SyncTextWriter)
00310089 8b153c309203    mov     edx,dword ptr ds:[392303Ch] ("Hello World!")
0031008f 8b01            mov     eax,dword ptr [ecx]
00310091 ff90d8000000    call    dword ptr [eax+0D8h]
00310097 e84c750d79      call    mscorlib_ni+0x3275e8 (793e75e8) (System.Console.get_In(), mdToken: 0600076e)
0031009c 8bc8            mov     ecx,eax
0031009e 8b01            mov     eax,dword ptr [ecx]
003100a0 ff5064          call    dword ptr [eax+64h]
003100a3 33c0            xor     eax,eax
003100a5 c3              ret

Interesting. The JIT clearly sees Console.WriteLine(string) and Console.ReadLine as trivial and has inlined the calls. In turn it’s inlined Console.get_Out. The reference to SyncTextWriter isn’t the JIT being clever – it’s SOS telling us that the address 0x0392108C is the base of SyncTextWriter’s vtable, as this call to TextWriter.WriteLine(string) is virtual.

I realise, and hope, that this isn’t everyday usage. More often than not you’ll have crashed somewhere and want to find out where (for which see !IP2MD). But it can be useful to see just how your C# code turns into machine instructions.

What the heck does 'rep ret' mean?

2008-03-10T00:13:00.001Z

I was looking at what the .NET x64 JIT compiler generates for some code, and saw something very odd at the end of the routine: the last instruction of the function was rep ret. Looking a bit further, this is the same at the end of every JIT-compiled routine.

The thing is, the rep prefix to an instruction is supposed to tell it to be repeated. Repeat the return? How do I do that?

The Intel architecture software developer’s manual set says it’s only defined for the ‘string’ instructions like movs (which moves a byte/word/dword from the address pointed to by ESI to the address pointed to by EDI). The rep prefix repeats the string instruction ECX times. Yes, this means that you can implement memcpy in a single instruction. (You can do memset with a single rep stos instruction, once AL is loaded with the value to be stored.) It’s explicitly undefined for anything else.

So where the heck has this illegal usage come from? I followed a couple of clues and found this patch notification for glibc on x64. And indeed the current version of AMD’s optimization guide [PDF] for Athlon 64 processors says that you should do this. The reason? The branch predictor gets it wrong if the ret instruction is jumped to directly by a branch instruction, or if the ret directly follows a branch instruction.

I’m not sure doing it throughout, even when you’ve got epilog code in there which prevents the bug, is necessary though.

AMD have now published a new optimization guide for their Family 10h processors [PDF] and guess what, the advice has changed. Instead of using a two-byte illegal instruction, they now recommend the three-byte instruction ret 0. The difference between a plain ret and a ret imm16 (where imm16 is an immediate 16–bit value) is that ret imm16 pops the return address, then specified number of bytes from the stack before jumping to the return address. It’s common to see this in 32–bit Windows WINAPI (__stdcall) code as this calling convention requires the called function to clean up the parameters from the stack. (64–bit Windows has only one calling convention and it mainly passes parameters in registers, so stack cleanup is not required.

Still, it’s a shame to see the JIT generating this on my Core 2 Duo laptop, which doesn’t have the bug (as far as I know, but there’s no mention in Intel’s optimization guide). And it’s an even bigger shame on AMD that they a) didn’t fix the damn bug in the processor and b) recommended an illegal instruction as a way round it.

BT and the case of the missing DSL service

2008-01-13T23:32:00.001Z

Recently, at the end of September, I got a new neighbour in the flat downstairs. The property is an end terrace, split into two flats in 1999, but the landlady never did a great job of separating the services out (electricity and gas usage is included in the meter counts downstairs, there isn’t a separate supply). When I moved in, the landlady lived downstairs; she allowed a previous tenant of this flat to put in extension wiring to use her old BT landline, as she’d got cable service with NTL (now Virgin Cable, of course), so I was using the BT line. After leaving, other tenants of the downstairs flat had simply taken over the cable service.

When the new bloke moved in downstairs, he wanted a BT service. He claims that he asked them for a new line – they interpreted this to mean taking over the old one, and started the process of transferring my line to him. First I knew of this was a letter dated 16 October saying “we understand you no longer live at the above address and so we’ll disconnect the phone on 30 October.” Obviously I phoned up straight away and said no, I’m not leaving, please cancel this transfer. They prevaricated a bit, then said, “oh, I can’t cancel this on your say-so, the person ordering has to cancel.” Fine, I say – can I give him a direct number to contact you? “Nope.”

Anyway, I ask the new guy if that’s OK, he says he got fed up of BT trying to transfer the line and went with cable anyway, and he’d been trying to get through to cancel and would try again. A day or so later he tells me he’s cancelled and I don’t think any more about it.

In early November my parents tell me they couldn’t get through. (They didn’t actually tell me they were getting a message saying ‘number not in service.’) A few days into November, suddenly, no connection. A horror begins to dawn on me, and my neighbour – BT have gone ahead and done it anyway.

Now comes the reason for the tale. Most of the time I can’t get through to BT Customer Service, even if I wait for half an hour on the phone. The first time I get off hold, there’s silence for a few seconds and the line goes dead. Next time I’ve apparently called the wrong department and they try to transfer me, but again I end up on hold for half an hour (when I give up). After several goes of this I go for an end-around: I call sales and ask for the line to be transferred back to me.

No problem, says the salesman, but there’s a 12–month minimum contract and you can’t have the line until the 27th of November, unless the current customer cancels before that, oh, and you can’t have your old number back. I did try explaining but it seemed futile. Actually I got pretty angry which I did apologise for.

I thought about it for a couple of days and decided to do what should have been done a very long time ago – get a separate line terminating in this flat. You’d expect that to take a long time, right? Longer than transferring the billing from one customer to another?

Nope. I could have it put in on the 20th. So I paid my £125 (£125!) and got it done.

Unfortunately you can’t order DSL until the phone line is in your name and it appears, under the new number, in BT’s database. So the earliest I could order ADSL was the afternoon of the 20th, then it takes another 14 days or so, taking us to an official activation date of 7 December. I got straight through to Demon sales on practically the first ring. They said I couldn’t have my old hostname/website/email address back initially but once activated I could ask for it back.

The 27th came and went and I suddenly realised I hadn’t cancelled the transfer! Never fear, either the second BT salesman cancelled it for me, or (more likely) they cocked up again. My neighbour still hasn’t managed to cancel and I think he’s refusing to pay.

Finally I got DSL back on the 6th, called up Demon support (got through on first ring again) and got my old hostname back, but the contents of my website were lost. So there are a lot of broken images around here, and if you emailed me in November, I didn’t get it.

I’d always heard that NTL, Telewest, and the unbundled broadband suppliers had worse customer service than BT. If it’s worse than BT they must send round people with pitchforks to stab you with if you even dream of calling them.

Return of Best Error Page

2008-01-13T22:58:00.001Z

CodeProject’s one of my favourite sites – indeed, one of the few real regular sites I visit – but it’s always had a bit of a problem with errors and with the load on the servers. I commented on this before, but sadly the image is gone – lost when my DSL account was closed (BT cock-up, nothing to do with Demon).

Anyway, they recently moved to an all-new ASP.NET implementation of their forums, but unfortunately the errors are still a common occurrence. And the error in the error handler is back:

(Yes, running Vista now.)

One of the things I love most about CP is the more… relaxed attitude to reporting problems.

Calling out EMBASSY Trust Suite

2007-12-07T04:12:00.001Z

Dear Wave Systems,

Fuck right off.

When I bought my Dell Latitude D820 laptop, I went through and deleted a certain amount of the shovelware that was preinstalled, but some of it, to be honest, I didn’t know what it did. I tried to work out what EMBASSY Trust Suite did, and it seems to be something to do with the Trusted Platform Module in the system. Encrypting files using the TPM key, or something like that.

Charles Petzold wrote an article yesterday commenting on random number generators, and I added a comment earlier on mentioning in passing the reported flaw in CryptGenRandom. I decided to see if this seemed to be the case in XP SP2. My answer was inconclusive, as the assembler was hard to follow. By chance I was debugging Internet Explorer with WinDBG (easiest way to force the RSA Enhanced Cryptographic Provider, rsaenh.dll, to load so I could disassemble it) and noticed an odd number of access violation exceptions occurring when I accidentally did a search in the instance of IE I was debugging. That’s weird, I thought – where am I on the stack?

In wxvault.dll. The stack trace was:

0012c888 100065ee wxvault+0x7967
0012cac4 42f8c769 wxvault+0x65ee
0012cadc 42f8cdc9 IEFRAME!PathFileExistsW+0x24
0012cb14 42f8ccf7 IEFRAME!HelperForReadIconInfoFromPropStore+0x97
0012cb98 42f78e53 IEFRAME!CInternetShortcut::_GetIconLocationWithURLHelper+0x156

Looks like we’re trying to get the favourite icon for the address bar. But how is IEFRAME calling into wxvault? Microsoft can’t know that this library exists. Is there something on the stack that somehow isn’t being included (can happen if a function was compiled with the Frame Pointer omitted and no symbols are available to get FPO data [which tells the debugger how to fix up]). Let’s disassemble around PathFileExistsW:

42f8c75e ff7508          push    dword ptr [ebp+8]
42f8c761 8bf8            mov     edi,eax
42f8c763 ff152c13ef42    call    dword ptr [IEFRAME!_imp__GetFileAttributesW (42ef132c)]
42f8c769 83f8ff          cmp     eax,0FFFFFFFFh

That’s weird, we called into GetFileAttributesW. How did we end up in wxvault?

0:000> u kernel32!GetFileAttributesW
kernel32!GetFileAttributesW:
7c80b74c e965ae7f93      jmp     wxvault+0x65b6 (100065b6)

Evil! They patched the running instance of kernel32! What else have they patched?

kernel32!CreateFileW:
7c810760 e9d0587f93      jmp     wxvault+0x6035 (10006035)

Note how they’ve failed to rebase the DLL, using the default 0x10000000 base address, making it collide with everything ever which also uses that default address.

Needless to say this is going to get uninstalled as soon as I take a full backup of the laptop! In my book, this is a user-mode rootkit. I don’t use the feature, so it’s going.

How should they have implemented this? Well, I’d start by seeing if it’s possible to change the algorithm for the Encrypting File System. It should be, it’s implemented using the Cryptographic API and CSPs (involving callbacks into LSASS in usermode!), so I would have thought that simply providing your own CSP would be sufficient.

If that’s not possible, my next port of call would be a file system filter driver. That would have the downside (like this) that every file system call would go through it, rather than the tiny amount of calls which actually target a file encrypted in this way.

The access violation looks like it might ultimately be caused by a bug in IE – it looks like IE tried to pass the URL to the favicon to GetFileAttributesW, which I would hope would fail (or would it try to invoke WebDAV?)

VS2005 Smart Device projects don't Dispose components

2007-11-04T20:09:00.001Z

I’ve posted about this on Connect and on Codeproject’s Lounge, and informed my colleagues, but not yet on here. I realise I’m repeating myself for some of my audience, but this is in a more readily searchable location.

I was trying to work out why Compact Framework applications seemed to be consuming a fair amount of memory, often causing the division between Program and Storage memory on Pocket PC 2003 to shift greatly toward Program, often causing inability to create files. I created a simple Windows Mobile 5.0 application with two forms, both with menu bars with actions on them, and had the first form continually create and destroy the second. I then loaded up .NET Compact Framework Remote Performance Monitor to monitor the application. I was seeing 400KB+ of GC heap still being used after a collection. Looking at the GC heap (in .NET CF 2.0 SP2’s RPM) showed that a large number of instances of the second form were still referenced.

Drilling down showed that the MainMenu and MenuItem objects that had been owned by the forms were still rooted, by their finalizers (shown as [root: Finalizer]). The forms were still referenced because I had event handlers connected to the menu items. That meant that all the other controls on the form were also still referenced, by the members of the form class.

To understand why they’re still rooted, even after GC, you have to know how finalization works. When a GC occurs, the collector marks every object that’s referenced by following roots, then sweeps away all those objects that are no longer referenced. When doing the sweep phase, if the object requires finalization (i.e. implements a Finalize method and GC.SuppressFinalize has not been called for that object), instead of actually being deleted and the memory reclaimed, it is placed on a finalization queue. This in itself is a source of roots. The finalizer is not run during the GC itself to reduce the time spent in GC. Instead, a separate finalizer thread processes the queue of objects to be finalized. The object will continue to be reported to GC until its finalizer is run, so may survive multiple GCs if memory demand is high and the finalizers are slow.

The reason I recommend that as far as is possible, you avoid the finalizer is that unless you’re very careful, you can end up with thread affinity problems (trying to destroy something from the wrong thread, or having to marshal back to the creating thread) and you can end up blocking the finalizer thread indefinitely. The issue of freeing memory later than would otherwise be possible isn’t as much of an issue on the desktop, but it also affects GC’s tuning of how often to run. It’s definitely an issue on the limited memory available on devices. For more on how GC works in the Compact Framework, see Steven Pratschner’s “Overview of the .NET Compact Framework Garbage Collector”. I’ve recommended before that you always call Dispose, on any object that implements it (although I’ve discovered it isn’t safe to Dispose the Graphics object passed to you in a PaintEventArgs structure).

So where does the finalizer come from? I studied the assemblies in Reflector. A side note here: the assemblies in Program Files\Microsoft Visual Studio 8\SmartDevices\SDK\CompactFramework\2.0\v2.0\WindowsCE do not actually contain the IL code, only the metadata. For the actual implementation see under SmartDevices\SDK\CompactFramework\2.0\v2.0\Debugger\BCL. Recent versions of Reflector will load assemblies from this location when you create a new assembly list in the File, Open List dialog (or when starting a newly downloaded copy, for the default list).

In Compact Framework 2.0, the System.ComponentModel.Component class implements a finalizer, so anything which derives from this class automatically gets one too. (It calls the virtual Dispose(bool) method, passing false as the parameter). The Dispose method in this class calls GC.SuppressFinalize, so if you dispose of a component properly, it won’t end up on the finalization queue. This matches what the desktop Framework has done since .NET 1.1 at least (I don’t have 1.0 installed to check).

For both desktop and device projects, a newly-created form declares a components member of type System.ComponentModel.Container, to contain the components dragged onto the designer surface. A Dispose(bool) override is generated for you (in Form.Designer.cs/.vb, for .NET 2.0 projects) which calls Dispose on the container, if components is not null. (Presumably the intent was that the container wouldn’t be created until needed, but in fact the initial code in InitializeComponent for a new form does create a Container and assigns it to components.) Container’s Dispose method disposes anything that was added to the container.

For a desktop project, when you drop something that derives from IComponent (I think) onto the form, and which doesn’t derive from Control, the designer generates code to add that component to the container, so it will be disposed of when the form is disposed. Simple. However – and here’s the bug – the device designer doesn’t. In fact it even deletes the creation of the Container object. Result, all your components end up running their finalizer, and thereby consuming memory past the first GC after they died, potentially increasing the overall memory use of the process.

To avoid the problem, you have to write the code to dispose the components yourself. The most straightforward way is to add the code that Visual Studio should have generated to your form’s constructor, after the call to InitializeComponent. That will typically look like:

this.components = new System.ComponentModel.Container();
this.components.Add( mainMenu1 );
// etc for other components

The difficulty is knowing exactly what to add and remembering to update it when you add or remove components. Reflector can help a bit – use the Derived Types view under Component to see the classes that are affected. The other one that affected us was the HardwareButton class (in Microsoft.WindowsCE.Forms). It doesn’t actually override Dispose(bool) so I think you ought to write code to clear AssociatedControl.

This bug still exists in VS2008 Beta 2. To help ensure it doesn’t continue in future versions, please vote for the bug on Connect.

In .NET Compact Framework 1.0, there isn’t a systematic finalization like this – Component doesn’t have a Finalizer or a Dispose method, although weirdly it does have a virtual Dispose(bool) method and a Disposed event. It also doesn’t implement IComponent, so you can’t add a Component-derived class to a Container! And it doesn’t implement IDisposable either.

The MainMenu class unfortunately does have a Finalizer, but it does not have a Dispose method. After a bit of digging around, you can see that you can at least clean up the native resources by setting the form’s Menu property to null. It still causes all the managed objects to remain referenced (if it has menu items that have event handlers). You’ll have to call GC.SuppressFinalize yourself once you’ve forced it to clean up.

The more I think about it, the more I think that garbage collection overpromises and under-delivers. At least the C++ resource-acquisition-is-initialization and automatic variable destruction model ensures that resources are freed in a timely fashion. That’s why the C++/CLI designers hooked that model up to IDisposable – there is no using block because local GC handle destruction does the job.

Tired of whacking rats

2007-10-12T23:59:00.001+01:00

I’m fed up with Windows Mobile. Fed up with it from an enterprise device perspective, anyway. See, most customers want a limited set of applications to run on their device. If it’s only one, you can write a full-screen app and hide the Start bar at the top, and maybe the menu bar at the bottom if you don’t need the on-screen keyboard. Two, and you have to find a way to switch between them (and recently it seems that the second one is TomTom Navigator, which doesn’t co-operate anyway) and stop anything else being run.

And so you do this, and bing! up pops a notification. So you work out how to kill that one and bing! here comes another one. It’s like playing Whack-A-Rat.

And I’m tired of it. I’m tired of trying to make my programs survive a cold boot, when your OEM subtly changes how it works from device to device (Symbol MC3090 – breaks the convention of ‘Application folder is for developers’ by lumping the wireless configuration utilities in there, and they have to match the wireless driver version, so you can no longer write to a hex file). But Windows Mobile 5.0 has persistent storage, right? Yes, but OS updates include a wipe of persistent storage, because the registry settings in persistent storage are a diff from the ROM version, so you change the ROM registry, you break the diff, and now you have to work out how to persist your program across clean boots. Or the OEM has inexplicably made the clean boot a simple keypress, with a confirmation prompt, but one so confusing where you press one half of a rocker for YES, I’d like to destroy my applications, and one for NO, actually I’d like to keep them please, and not marked them clearly or the instructions are confusing so the user that meant to say NO says YES by mistake, or they’re labelled the wrong damn way round in the first place and you still have to make the damn program survive!

…and breathe…

…and now you have to come up with a way to do program and firmware upgrades over the air, only it wasn’t designed in, because we were going to use the OEM’s system, but we can’t use that because a) it costs a bomb and b) it sucks and c) it blows as well and you find that you can’t do a complete device reload because the damn ROM image is too big to fit in RAM or persistent storage alongside the updates to your program because they couldn’t find room in ROM for .NET ‘Compact’ Framework and SQL Server ‘Compact’ Edition so you have to do it a bit at a time and now you’re trying to fudge it all at the last minute and then which bit has to chain which other bit and how do we do this on a schedule so updates don’t interfere with the user’s work and I DIDN’T SIGN UP FOR DEVICE MANAGEMENT DAMMIT!

Everyone working on databases should read this

2007-09-03T20:07:00.001+01:00

Rico Mariani (now Chief Architect of Visual Studio, formerly just a performance architect on the .NET Framework) has written a great article on database performance, but also covers correctness issues. A good read for any developer working on databases (and isn’t that most of us now?)

Database Performance, Correctness, Compostion, Compromise, and Linq too

Another reason not to overload the .NET Framework name

2007-07-10T21:00:00.001+01:00

This month’s security bulletin becomes a lot more confusing. It was pretty confusing already, but the extra detail of .NET Framework 3.0 is/is not vulnerable just adds an extra layer.

(Suggestion: update and let your customers know. Since .NET Framework patches are cumulative I expect Barry’s validators are also included.)

Wow, long time no post

2007-06-05T23:12:00.001+01:00

I have intended to on occasions, never got round to it. It’s been so long that Blogger have changed completely over to Google logins and my old configuration in BlogJet no longer worked because I’d had to switch to the Google login at one point (think I wanted to use my own identity on posting a comment on someone else’s blog).

Indeed the old Google API didn’t work anymore either and I had to grab BlogJet 2.0.x.

Google’s increasing privacy-invasion is making me want to get off this ship (and this one too).

When developers fight...

2007-06-05T23:08:00.001+01:00

Microsoft take their ball away.

The problem with biometrics

2007-01-25T20:58:00.000Z

If they change, they can’t identify you

Geek Dinner this Friday (19 Jan)

2007-01-16T23:46:00.000Z

My friend Colin Mackay, who I know from CodeProject, sent me a message last week to tell me that he was attending this weekend’s Vista and Office Developer Launch in Reading, and to ask if I’d like to meet up while he was here.

I was a bit slow responding and discovered tonight that he’d signed up for Zi Makki’s Geek Dinner on Friday night. If you’re in the area – whether or not you’re going to the event itself (I’m not, and nor is Ian) – why not come along?

How to misuse the Office 2007 Ribbon

2006-12-20T02:27:00.000Z

Dare Obasanjo has noticed a comment of mine on Jensen Harris’s post announcing Microsoft’s licensing of the concept of the Office 2007 ‘Ribbon’ UI. In that comment, I criticised (in a single sentence) Dare’s concept for a future version of RSS Bandit. I should say up-front that I’m a regular user of RSS Bandit; it’s my main RSS reader at home, in which I’m subscribed to over 100 feeds. I want this to remain usable, and my fear is that it won’t be.

Funnily, he doesn’t acknowledge that I made the first comment on that post, in which I go into detail. I said:

It doesn't belong. There's no need to go to an Office-style menu system in RSS Bandit because you barely ever use the menus anyway. It's not like there are loads of features hidden in the depths of the menus and dialogs, and the gallery is particularly over-the-top. How often do you think people will change the style of the newspaper view? Virtually never, in my opinion - they'll pick one that works, and stick with it. These options don't need to be 'in your face' the whole time. RSS Bandit is not document authoring software, it's a browser.

If anything you could follow IE7's lead and drop the menu bar entirely. There aren't that many menu options, and most of them are replicated with some other widget, on one of the toolbars, or in the case of View/Feed Subscriptions and View/Search, the two tabs in the pane.

Most of the other options that aren't duplicated could end up on an extended Tools menu.

Dare links to Mike Torres who comments on the menu-less UI of various Microsoft applications, suggesting that this is something recent. At least two of these have been menu-less for a while, in one case for five years: Windows Media Player. The original version of WMP in Windows XP was without menus:

(screenshot from http://www.winsupersite.com/reviews/wm9series.asp).

The highly-unconventional window shape was toned down in version 9.0 and became virtually conventional in 10.0, although all four corners are rounded whereas the normal XP themes have rounded top corners and square bottom corners.

It appears that the menus first disappeared from MSN Messenger in version 7.0, which was released in April 2005:

(screenshot from http://www.winsupersite.com/reviews/msn_messenger7.asp)

Which Office application is RSS Bandit most like? Word? Excel? No. It’s most like Outlook. Which major Office 2007 application does not get a Ribbon (in its main UI)? Outlook.

I’ve been following Jensen Harris’s blog more-or-less since the beginning. In it, he explains the motivations behind creating the Ribbon, and the data that was used to feed the process of developing it. The Ribbon is mainly about creating better access to creating and formatting documents, by showing the user a gallery of choices and allowing them to refine it. Which part of Outlook gets a Ribbon? The message editor (OK, this is actually part of Word).

RSS Bandit is about viewing other people’s content, for which the best analogy is probably IE7.

I haven’t done any UI studies. I’ve not taken part in any. But Microsoft have analysed their UIs. They’ve gathered data on how those interfaces are used – automatically, in some cases (the Customer Experience Improvement Programs). The Ribbon is an improvement for Office. It’s not going to be right for all applications. Many applications actually suffer in the classic File/Edit/View/Tools/Help system: the menus tend to either be padded with commands that are duplicated elsewhere, or are ridiculously short (e.g. RSS Bandit’s ‘Edit’ menu which only has a ‘Select All’ option, which if you’re currently looking at a browser tab appears to do nothing – it’s only when you switch back to the Feed tab that you notice it’s selected all the items in the current feed or feed group). They’ll suffer equally in the Ribbon, particularly if there are too few features to make a Ribbon worthwhile.

When designing a UI for your application, don’t be too slavish to a particular model. If you find yourself padding out the menus to conform to the File/Edit/View model, or if all your commands are on the Tools menu, a classic menu probably doesn’t fit. If you’re not offering a feature for the user to customise the formatting of something, which the user will use regularly, a Ribbon is probably also wrong. The standard toolbar is probably enough.

Another knock-on effect of the stupid WinFX->.NET 3.0 naming decision

2006-12-19T00:06:00.000Z

The next version of the Compact Framework will be called:

.NET Compact Framework 3.5.

Yeah.

Great way to confuse people.

Spotted: bad Google ads

2006-12-04T22:22:00.001Z

Saw this strip of ads on a site:

Test advert got onto a real site, perhaps? I know the site owner has only just added AdWords to their site, but it should show something useful!

(example.com is reserved by IANA for use in example URLs)

Spotted: bad Google ads

2006-12-04T22:22:00.000Z

Saw this strip of ads on a site:

Test advert got onto a real site, perhaps? I know the site owner has only just added AdWords to their site, but it should show something useful!

(example.com is reserved by IANA for use in example URLs)

How to fix the Smart Device Framework 2.0 installer

2006-11-03T21:13:00.000Z

Neil Cowburn noted that the Smart Device Framework 2.0 installer doesn’t work properly on Windows Vista.

This is the comment I couldn’t post to his website:

“It's error upon error for this one. Code 2869 means that the dialog designated as an error dialog doesn't work how Windows Installer needs an error dialog to work - see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/windows_installer_error_messages.asp. So the real error is being lost. Visual Studio is generating you a broken Error dialog.

I'm going to guess that the real error is that your custom action is failing, because it isn't privileged. On Windows Vista, only custom actions marked NoImpersonate get to run with administrative permissions (actually, they run as LocalSystem). Visual Studio cannot be told to mark a custom action as NoImpersonate (as far as I know). If you want to fix it after generating the MSI, you can use Orca (the MSI table editor, part of the Platform SDK, search for Orca.MSI) to add 2048 to the Type column of the three rows which use the InstallUtil DLL (which is the native code that calls into your managed DLL). I've also heard of tools which can be used to execute SQL against an MSI - it should be possible to do this with VBScript using the MSI object model.

The Windows Installer team does not recommend the use of managed code custom actions. This message does not seem to have got through to the Visual Studio deployment team. The recommendation is to use as few dependencies as possible, which generally translates to statically-linked C++ code.

Digging around in Reflector shows that you're using the custom actions to add the SDF to the ActiveSync Add/Remove Programs box. I'm not really a fan of this idea - and I note that Microsoft doesn't do this with the Compact Framework itself. It would be simplest to scrap this custom action completely. I also note that you're not handling rollback or uninstall. You should also use the /register flag to CEAppMgr.exe so that it doesn't install immediately on the connected device (or install when the next device is connected).

Windows Installer does support finding and executing an EXE that's already on the system as a custom action, but I don't think you can do this in Visual Studio.

You might want to consider a better installation solution, such as Windows Installer XML (WiX, http://wix.sourceforge.net/)”

I’ve been getting into WiX recently. I was going to do a presentation at DDD4, but not enough people voted for it. If you fancy attending any of the proposed sessions and can spare a Saturday, sign up now. (I’m waiting for the final agenda to be posted, but the places may all go before that happens.)

Petition to rename .NET Framework 3.0

2006-09-20T02:09:00.000+01:00

As soon as I heard that Microsoft were changing the name WinFX, an umbrella name for Avalon, Indigo – oh, excuse me, Windows Presentation Foundation and Windows Communication Foundation – and Windows Workflow, to .NET Framework 3.0, I thought it was an incredibly bad idea.

The trouble is that it confuses everybody. I’ve seen people commenting that they’ll delay moving to .NET 2.0 ‘because .NET 3.0 is just around the corner.’ They then get horribly confused – and normally angry – when you tell them that the CLR, BCL, Windows Forms, ASP.NET and the language compilers are completely unchanged in ‘.NET 3.0’ from .NET 2.0.

Someone’s started a petition to name it back to WinFX. I don’t care what name it has – does it even need an umbrella name? Can we not call the three subsystems by their own names? Even better, their codenames which despite not being descriptive were at least easy to say! Do I really need to even install WCF and WF just to get a WPF application to work?

What I suspect it does mean is that versions of .NET after 3.0 simply won’t install or work on Windows before XP SP2, Server 2003 SP1, or Vista. That’s a huge compatibility loss – .NET 2.0 works right back to Windows 98 and NT 4.0. Or, if new versions of the CLR and BCL will install and work on older operating systems, they’ll have another stupid naming decision to make.

It also means that even for downlevel systems, the new installers will be even more humungous than ever for the One That Is To Come After. People still complain about the size of the Framework installer; most end users will never have a web server installed on their machine – security considerations would suggest that they shouldn’t – so why in hell does .NET Framework include and install ASP.NET on every single box? This leads to people asking about and trying to invent jerry-rigged systems to either try to link the framework into their binaries or ship only bits of the Framework. It’s a recipe for disaster come servicing time.

Please, if you value everyone’s sanity, sign this petition. It probably won’t do any good but you can at least say you spoke up against the insanity.

Biometric scanners not particularly reliable

2006-09-19T21:03:00.000+01:00

Dana Epp posted a movie from Mythbusters cracking a fingerprint ‘lock’.

Not exactly secure.

Watch now. (YouTube, may get taken down when someone spots the copyright violation. What the hell, it’s Talk Like A Pirate Day. Arrr!)

Missing font on Vista RC1

2006-09-10T19:05:00.000+01:00

Anyone reading this blog from Windows Vista (Pre-RC1 build 5536 or RC1 build 5600) might notice a slight difference in appearance between Vista and XP. What is it?

They forgot to include Trebuchet MS Italic!

Windows Vista instead installs two copies of Trebuchet MS Bold Italic. When called upon to produce Trebuchet MS Italic, the Windows TrueType/OpenType renderer instead simply slants a copy of Trebuchet MS. This doesn't look very good - there's a reason that Vincent Connare drew a true italic.

I remember reporting a bug on Pre-RC1, but since I can't access the feedback site (being part of the Customer Preview Program rather than a 'beta tester', a differentiation that seems a little bizarre – do Microsoft not want bug reports from CPP members?) I don't know if anything's being done.

Perhaps Michael Kaplan could see what's happening here (although he does use Tahoma 'Italic' on his blog ;-) – there is no true italic for Tahoma).

VS6 family completely broken on Vista Pre-RC1 (or so it looked)

2006-08-30T21:08:00.000+01:00

Looks like my compatibility problems are solved: in the radical way, by completely breaking Setup on all three applications. Setup now crashes on clicking 'Next' at the welcome screen. No Program Compatibility settings work.

These tools are essential for my work. If they don't work I cannot upgrade.

UPDATE 2006-08-31: It appears that whatever was causing this may have been a temporary glitch; Visual Studio 6.0 is now installing. Still, given my experiences with eVC before, I can perhaps be forgiven for jumping the gun?

I can't go in and add a note to the bugs I filed, because as a Customer Public Preview user, while I can submit bug reports using the Beta Feedback tool, I can't log on to Connect to make changes or additional comments. This means I'm wasting someone's time to triage bugs that I now cannot repro.

People exhibit surprise that Windows Media DRM is 'cracked'

2006-08-29T23:16:00.000+01:00

Example surprise.

I’m not at all surprised this is possible. In order to decrypt data, you need two things: the encrypted data, and the decryption key. In order for media playback of DRM-protected files to be possible while disconnected from the Internet, both of those things need to be on your PC. If the key is already on the attacker’s PC, it’s only a matter of time before they find out where it is.

There are of course things that can be done – such as encrypting the decryption key with a master encryption key, so that it isn’t on disk in a usable form, then decrypting it only while it’s actually needed for playback – but ultimately, the key will be visible in the system’s memory somewhere for long enough to be copied.

TV Licensing has serious issues

2006-08-26T12:16:00.000+01:00

(Note for non-Brits reading this [if any]: in the UK we are required, if we want to watch broadcast television, to pay a licence fee of £131.50, which goes to support the BBC – BBC TV has no commercial advertising, except for other BBC programmes. You have to show that your TV is physically incapable of receiving broadcast television to avoid it.)

When I moved into this flat, I bought a TV licence using the TV Licensing website. I did not notice at the time that it had ‘auto-corrected’ the address I entered. This house, an early-20th-century end terrace, was split into two flats by the landlady in 1999 (based on the details from the Council Tax website). The landlady, and all the rental documents, refer to my upstairs flat as ‘17A’. However, the council, for council tax, refers to it as ‘First Floor Flat’. (Actually looking at it right now, the Council Tax website I linked above shows ‘1st Flr Flat’ – they clearly also have a stupidly short field length.)

When I moved my credit cards, I wasn’t aware of the council’s designation, so I used ‘17A’, and that’s what I entered on the TV Licensing website as well. Most UK websites have a gazetteer – a lookup of house number and postcode to pick the correct full address, and this one is no exception. It expanded the street name and town correctly, but dropped the ‘A’, so my licence is actually for number 17, which according to the council, no longer exists. The landlady calls the downstairs flat number 17.

Earlier this year, I decided to get a PVR (Humax PVR9200T, very good thanks). I ordered it with my credit card, and as usual when buying from a new supplier, they insisted it was sent to the card address (i.e. 17A). Whenever you buy TV equipment, this is reported to TV Licensing. Ever since then, I’ve been getting demands to buy a licence for 17A – which I can’t, because the website won’t accept it!

I’ve tried to change the address. You can still edit the address after it’s been expanded out on the change of address page, and I’ve tried that, but the ‘A’ is still dropped.

I’ve sent them letters. They’ve ignored them.

I’ve tried to phone them. They have an automated change of address system. It’s unusable. I’ve tried to leave a phone message. You get about 30 seconds, which is far too little to actually explain the problem. I’ve asked to be phoned back – they haven’t. I’ve tried to be put through to an agent – I just get disconnected.

I’ve sent emails through their website. They’ve been ignored or lost.

What’s actually almost more annoying is the lackadaisical attitude they’ve taken. I bought the PVR in February. I guess when I haven’t had a reply to one of the many attempts to contact them and get this corrected, I’ve been overoptimistic and assumed that they’d sorted it, whereas silence actually means I’m being ignored.

Today I’ve sent two more emails, one using the contact form and the other directly to the email address shown. Hopefully one of them will be processed this time, before the bailiffs come round.

I can’t even try to buy another licence (I’d lose about £40 because this licence still has four months to run, but that’s worth less to me than all this hassle), because I still can’t enter the correct address!

Generic components can only get you so far

2006-08-24T00:48:00.000+01:00

We had a strange issue with Meteor Server about two years back. Under stress, the Mem Size column (working set size, in fact) in Task Manager would be up and down like a yo-yo. I initially wondered whether the OS was trimming the working set too aggressively, and tried using the SetProcessWorkingSetSize function to increase the quota. Result: no improvement, it was still happening. The time spent in the memory allocator was causing the server to slow down significantly, and as it started to slow down, the problem would get worse, and worse, eventually virtually grinding to a halt.

To prevent overhead of context switching between multiple runnable worker processes, we moved a long time ago (before I started working on it) from a model where each client would have a dedicated worker process, to a much smaller pool of worker threads (the old mode can still be enabled for compatibility with older applications that don’t store their shared state in our session object or otherwise manage their state, but it is highly discouraged for new code). This does mean that there will be times where a client request cannot be handled because there is no worker process to handle it.

After some thought and experimentation, it became clear that what was happening was that when the server started to slow down, the incoming packets were building up in, of all things, the windows message queue. I should say at this point that we were using the Winsock ActiveX control supplied with Visual Basic 6 for all network communications. We already had a heuristic that would enable a shortcut path if the average time to handle a request exceeded a certain threshold. This shortcut path simply wasn’t fast enough.

To work around the problem, I added code that would actually close the socket when either of these conditions held. This was pretty tricky to get right as we had to reopen the socket in order to send a response out of it, and we would then need to close again if the average time still exceeded the threshold. There was at least one server release where the socket would not be reopened under certain conditions (if I recall, when both the time threshold was exceeded and a worker process became available). The memory allocation issue still occurred, but it was contained. I added an extra condition that would also close the socket if no worker process was available (this would prevent some retries from lost responses and some requests for additional blocks, both handled in the server process without using a worker, from being handled).

Then, recently, we discovered a problem with the code used to send subsequent packets of large responses, too large to fit into a single packet (the application server protocol is UDP-based). We weren’t setting the destination (RemoteHost and RemotePort properties) for these packets, assuming that this wouldn’t change. Wrong! If another packet from another client arrives (or is already queued) between calling GetData and SendData, the properties change to the source of the new packet. This sometimes meant that a client would receive half of its own response and half of a different one, which when reassembled would be gibberish (typically this would cause the client to try to allocate some enormous amount of memory, which would fail). I corrected that, but found that the log in which we (optionally) record all incoming and outgoing packets still had some blanks in it where the destination IP and port were supposed to be – these values retrieved from the RemoteHostIP and RemotePort properties. Where were these packets going? Who knows! Perhaps they were (eek!) being broadcast?

The WinSock control really isn’t designed to be a server component. Frankly it was amazing we were getting around 2,400 transactions per minute (peak) out of it. It was time to go back to the drawing board. Clearly I was going to need an asynchronous way of receiving packets, and the Windows Sockets API really isn’t conducive to use from VB6, so it was going to be a C++ component. Since string manipulation and callbacks were involved, I went with a COM object written with ATL.

I surmise that the WinSock control uses the WSAAsyncSelect API to receive notifications of new packets, and that’s why we were seeing the message queue grow with each packet received. The new component uses WSAEventSelect and has a worker thread which waits on the event for a new packet to arrive. When a packet arrives it synchronously fires an event, which has the effect of waiting until the server finishes processing the packet – either discarding it (as a duplicate, otherwise malformed, or due to excessive load), sending the next block in a multi-block response, or handing the request off to a worker process.

This does mean that there could be long delays between checking for packets. Doesn’t that cause a problem? Not really. The TCP/IP stack buffers incoming packets on a UDP socket in a small First-In-First-Out buffer. If the buffer doesn’t have enough space for an incoming packet, the oldest one in the buffer is discarded. That behaviour is perfect for our situation. You can vary the buffer size (warning, it’s in kernel mode and taken from non-paged pool, IIRC) by calling setsockopt with the SO_RCVBUF parameter.

For added performance the socket is in non-blocking mode, so on sending a packet, it simply gets buffered and the OS sends the data asynchronously.

Net result? No more problems with misdirected packets (my new API requires you to pass the destination in at the same time as the data), a step on the road to IPv6 support (the WinSock control will not be updated for that) – and a substantial performance improvement. My work computer now does 7,000 transactions per minute (peak) on the same application – and the bottleneck has moved somewhere else, because that figure was achieved with only three worker processes while the earlier one was with eight. (Hyperthreaded P4 3.0GHz). We saw much less difference in a VM (on the same machine) I’d been using to get performance baselines, but what we did see was that the performance was much more consistent with the new socket component.

The sizing for this application was previously around 1,500 transactions per minute per server, so this really gives a substantial amount of headroom.

My component would be terrible for general use – but it’s just right for this one.

Sometimes where code runs is more important than what it is

2006-08-23T22:58:00.000+01:00

Sometimes, to get the best performance from some code, you have to change the architecture.

Our application server product, Meteor Server is a complex beast. To be able to handle requests from clients concurrently, the main MeteorServer.exe process farms out those requests to a pool of worker processes. (Yes, we could switch to a single multi-threaded worker process even with VB6, but it’s a lot of effort and many existing applications may not be threadsafe, so we’d have to offer both schemes, and that’s even more effort.)

We can’t multi-thread the main MeteorServer.exe process because it’s written in VB6, and while you can make an out-of-process COM server process (a local server in COM parlance, an ‘ActiveX EXE’ in VB6 terminology) multi-threaded, you can’t make a ‘Standard EXE’ multithreaded. Oh, there are hacks, but I’m of the firm opinion that you shouldn’t subvert a technology to make it do something it wasn’t designed to do – when it goes wrong you will get no support.

A Meteor application is a COM object which exposes two methods through a dispatch (Automation) interface – well, one property and one method. The property, called VersionString, is simply to allow Meteor to pick up and display version information for the application. Every other piece of interaction is done through the TerminalEvent method, which receives a couple of interface pointers to allow it to call back into Meteor, a flag indicating whether this is a new client, a numeric event type indicating what the user’s last action was, and a string representing any event data. The application then calls methods on the interface to accumulate a batch of commands to be sent to the client – things like clearing the screen, setting the text colour, displaying text at a given location, sending a menu of options, defining an entry field. When one of these methods is called, it’s turned into an on-the-wire format, with an operation code and a wire-representation of the parameters. When the application returns from TerminalEvent, the server sends the complete batch to the client.

When I first started working on Meteor Server, when the application called a command-generating method, the stub of code in the interface made a call back into the MeteorServer.exe process to perform the wire-format translation. This meant it had to wait for the server process to finish whatever it was doing and go back to waiting for a window message. This made the server process a serious bottleneck – it was a ‘chatty’ interface, which is really not advisable across process boundaries. About three years ago, I looked at the code and realised it actually had no dependencies on any data in the server process, and had the idea to move this formatting code into the worker process that the application object was running in, to improve both performance and scalability. The commands would be batched up in the worker process then only sent across to the server when the batch was complete.

About a year later I actually made the change – we were seeking a significant performance improvement at the time. I don’t have a record of the performance change but I think it was some decent multiple – 3x or so the transaction rate.

About this time last year, or a little before, I was asked to add a new feature. Meteor provides a session state storage object which can store arbitrary strings that the application sets. The new feature was to allow an application to copy the session state data from another session to its own – this allows a user to resume their work on a different client, for example if the hardware is damaged or otherwise fails. I initially put the extra code directly in the batch-retire method that the worker process calls on completing a request, adding a new parameter, but when testing for performance, discovered that the simple test to see if the session should be transferred caused a regression of about 10%, and that would mean the difference between exceeding and failing to meet the performance requirement for a different customer.

The solution was to make this a separate method that the worker process would call if required, and to take it out of the mainline, returning the batch-retire method to its previous implementation. On doing this, the regression had gone – we were back up to almost exactly the same performance level we’d had before.

Know the environment in which your code has to run.

You must call Dispose

2006-08-23T21:21:00.000+01:00

You must call Dispose.

If an object implements the IDisposable pattern, or otherwise offers a Dispose or Close method, you must call it. No exceptions. If an instance member of your class implements IDisposable, your class should too.

OK, if you don’t, a finalizer might clean up after you. But you don’t want it to do that. The finalizer will only run once a GC has occurred. And a GC only runs based on heuristics of how much memory has been allocated (in general; there are a few cases in Compact Framework where the GC can be spurred to run, for example on receiving a WM_HIBERNATE message from the shell to say that physical memory is low). In .NET Framework 1.x and both versions of .NET Compact Framework, the GC has no idea how much unmanaged memory is being used by any object that manages an unmanaged resource. .NET Framework 2.0 does have the GC.AddMemoryPressure method, which can guide the GC to collect earlier than it might otherwise have done.

Finalizers don’t run on a regular application thread. They run on a special finalizer thread. This means you have to be careful around possible synchronisation issues. Objects to be finalized wait in a queue, and only one thread services that queue, so if a finalizer blocks, all undisposed objects will end up hanging around.

Once the finalizer has run, the managed memory isn’t automatically released. You have to wait for GC to run again. On the desktop or server, with the full Framework, you have to wait for it to collect the generation of the heap which the object is now in, which means an even longer wait for the managed memory to be released, which can keep the GC heap larger than it could have been.

The GC propoganda basically tells us we can be lazy. We can’t. We must clean up after ourselves. Treat the finalizer as a safety net.

Best practice is to manage one unmanaged resource with one managed object, and keep that managed object as simple as possible – ideally, just to manage that object. Make your resource manager class implement IDisposable and give it a finalizer as a backstop.

Anyone managed to get eVC working on Windows Vista Beta 2?

2006-08-20T16:12:00.000+01:00

My job involves a fair amount of C++ development for Pocket PCs and other Windows CE-based devices. I’ve been doing this for five years as of next week, so in that time I’ve used – and deployed projects developed with – eMbedded Visual C++ 3.0 with the Pocket PC 2000 SDK, the Pocket PC 2002 SDK, eMbedded Visual C++ 4.0 with Pocket PC 2003 SDK and now even a couple of things debugged with VS2005 (note: not compiled with VS2005 because they needed to be backwards compatible with PPC2003 and not redistribute the MFC 8.0 runtime – you can’t use the MFC 6ish supplied with the PPC2003 SDK with VS2005 because the headers won’t compile). To be able to upgrade to Windows Vista, compatibility with eVC is an absolute requirement – if it doesn’t work, I can’t upgrade.

(What about running under Virtual PC? Virtual PC doesn’t virtualise USB ports, new versions of ActiveSync don’t offer network sync, I don’t really fancy manual connection setup, and I can’t always rely on having devices with network support.)

Unfortunately it seems at the moment that eVC just doesn’t work under Vista. eVC 3.0 works with the Pocket PC 2000 SDK (only) installed, but as soon as you install the 2002 SDK (which shipped Platform Manager 4.0) it breaks, taking an access violation exception on opening a project. That’s installing with User Account Control enabled. With UAC disabled before installation, it doesn’t even get as far as an empty environment after installing the 2002 SDK.

eVC 4.0 barely even installs. If you launch Setup from the Program Compatibility Wizard, after setting Windows 2000 compatibility, it does install; using the default compatibility options (i.e. none) it crashes before even asking for the product key. I did find that a scripted build (using EVC /MAKE from the command line) would build for some platforms but not others, but again would crash on opening a project, making it impossible to debug. I’m guessing that UAC gets in the way of installing the SDKs to the right place.

This is a shame. I’d hoped that UAC would help with the myriad problems in trying to get eVC (either version) to work under a limited user account.

On Windows XP, to get them working as a limited user, both versions require users to have write-access to their installation directories. This stems from the Access/Jet database used to hold the processor type definitions, VCCEDB.MDB. Write access to this file is needed when running, and if you start more than one copy, Jet needs to create a ‘lock’ (.ldb) file in the same directory to manage concurrency. There are also registry keys which contain platform definitions under HKEY_LOCAL_MACHINE\Software\Microsoft\Windows CE Tools\Platform Manager, which contain things like the paths under Tools, Options, Directories. If eVC cannot write to this key, it creates an equivalent tree under HKEY_CURRENT_USER, but does not copy the original data, leaving you with a non-functional SDK. You have to copy the settings (maybe I should write a program to do this). I recall that there are other settings that you have to change the permissions on, but can’t recall what they are right now – it’s a while since I last had to do it.

eVC 4.0 also complains that it is unable to update its help system after you install any new SDKs. Running from a command prompt launched by makemeadmin doesn’t help, nor does running eVC as an administrator with Run As.

Somewhat related, VS.NET 2003 has a problem with device debugging as a limited user where it will fail to connect to a device that hasn’t been used for debugging before. You have to run as an admin (via makemeadmin) the first time you connect to a device (or the first time after cold booting). I surmise that some part or parts of the encryption keys are stored in a location that isn’t writable by a limited user.

Avoid interactive services

2006-07-22T00:32:00.000+01:00

It’s common, when thinking about providing configuration UI for a Windows service, for the developer to see the ‘Interactive’ checkbox in the service properties dialog. Or, maybe you have a server program which already has UI, which you’d like to turn into a service, and you need that UI to be available to the user.

Don’t be tempted.

Interactive services must run under the SYSTEM account (sometimes shown as LocalSystem). Well, OK, it’s not really a user account (you won’t see it in the user accounts database, it doesn’t have a password, it’s just assigned by the system to the special processes it creates at boot time like CSRSS and WINLOGON) but it behaves a lot like one. Anyway, the point is that it’s very highly privileged, and you should strive to have your services run with only the minimum privileges you need, rather than the maximum available. You cannot create a lower-privileged interactive service.

Interactive services also only work for session 0. On Windows 2000, XP and Server 2003, session 0 is the one which the user logs onto on the physical console – the actual keyboard, mouse and graphics hardware physically plugged into the machine. In Fast User Switching on XP, the first user to log on gets session 0, subsequent users get their own sessions. Windows XP Professional will remote session 0 if the user currently logged on locally to session 0 logs on remotely, or if no user is currently logged on. Windows Server 2003 will only remote session 0 if you pass the /console switch to mstsc.exe (Remote Desktop Connection). Windows 2000 never remotes session 0.

Users with other sessions never see the interactive service UI. If you’re trying to notify an administrator of a problem, they may not see the problem until physically logging on at a console, but the trend is towards remote administration. I have to walk all of probably 20 metres to administer servers at work, but it’s more convenient to do it at my desk, so I use Remote Desktop, when there isn’t a tool I can install on my workstation (for example, I’ve installed the Exchange Server 2003 management tools on my computer, and use RunAs to give myself appropriate privileges when using them).

On the current generation of operating systems, interactive services can be vulnerable to so-called “shatter” attacks, where an attacker sends window messages to the service to cause the service to execute code on his behalf. To mitigate this, Microsoft are making a change for Windows Vista and its successors including the next Windows Server version. Now, no users will log on to session 0 – it will be reserved for privileged processes only. Instead, the first user will log on to session 1. Since processes cannot send window messages across sessions, the shatter attack can no longer work. But this means that now no-one will see your UI. For more information, see the whitepaper “Impact of Session 0 Isolation on Services and Drivers in Windows Vista”.

So what to do? You will have to come up with some method of inter-process communication to allow a separate UI process to send commands to the service and display results. You might as well make it a network-capable IPC mechanism – then you can make your UI process capable of running on another machine.

Warning: Installing VS 6.0 on Vista Beta 2 leaves vulnerable Java VM

2006-07-16T16:30:00.000+01:00

I still need to use Visual Studio 6.0 at work – we have a lot of legacy software, and even some actively enhanced software, which uses VB 6.0 and Visual C++ 6.0. The installer for Enterprise Edition requires Microsoft’s Java VM to be installed and won’t continue without it. The version installed dates from 1998 (as does VS 6.0, of course) and is Build 2572.

Windows Update does not offer any updates for this VM on Vista Beta 2. There is no other source of updates. There were many security patches for this software.

When you have finished installing Visual Studio 6.0 you should immediately uninstall the VM.

You should also of course install Visual Studio 6.0 Service Pack 6. Some users have reported problems installing VS6 SP5 but I had no problems with SP6. Well, except that Windows Vista popped up a compatibility box claiming that the software hadn’t installed correctly, but it seemed to be OK.

If you need Java you should install Sun’s runtime environment (at time of writing version 5.0 Update 7). Note that when you have the JRE loaded, the Aero environment will shut down and you’ll revert to the Basic theme – no ‘cool’ 3D or glass effects – until the process that launched Java shuts down. This is because the JRE uses DirectX in an exclusive mode to do its drawing. This is supposed to be fixed by Vista RTM (source).

[Update: I have discovered that you can fake it by simply placing an empty file called msjava.dll in your System32 directory (presumably SysWOW64 on an x64 box). This also works for machines installed from clean with Windows XP Service Pack 2, which doesn’t include the Microsoft VM, although Windows Update does offer updates for the Microsoft VM after installing from the VS 6.0 CD, on XP.]

Don't bother using Google to search for Windows APIs anymore

2006-05-23T23:05:00.000+01:00

I’ve tried several times over the last week or so to look up details of an API, given its name, using Google (using a search engine in the web browser is still faster than waiting for the documentation browser to load) and have found that the official documentation (that is, msdn.microsoft.com) has just plummeted off the list.

MSN Search still brings up the API documentation usually as the first or second result (often, if it’s a routine that Windows CE also implements, it’s the first and second result). In case you think this is a quirk or that somehow Microsoft are blocking Google’s spider, it’s interesting to note that MSDN is usually the top hit on Yahoo search as well. Of course that doesn’t prove that Microsoft aren’t blocking Google’s spider.

If, like me, you’re using IE 7 Beta 2, you can use the MSDN Lab search (which uses MSN Search under the covers) directly from the browser search box by going to http://lab.msdn.microsoft.com/search/ and clicking the “Got IE7? – get our search” link in the bottom right-hand corner.

A question for the mobile networks: when are you going to deploy EDGE?

2006-04-02T16:53:00.000+01:00

Mobile data communications is starting to become a major business tool, for many logistics and distribution companies, and businesses who have a logistics or distribution component. Instead of downloading a whole batch of work to a handheld or other mobile computer, then uploading the results of the whole batch at the end of the working day, the business can get more timely information by having the mobile send its results as work is done. The business can also expose some of this information to its end customers, for applications like live package tracking. A system can even send live data updates of additional work required to the mobile computer, reducing requirements for the user to manually input job information.

For timely updates and responsive applications, it helps to have fast transfer speeds. The last generation of enterprise mobile handhelds, at least here in the UK, supported integrated GPRS. The new generation just coming on stream support EDGE – Enhanced Data rates for GSM Evolution (hmm, smells like an invented name to me). EDGE offers more bandwidth than GPRS, but is entirely compatible with GSM using the same basic radio format, unlike UMTS (3G). This enables existing GSM networks to be upgraded to support the higher data rates.

Unfortunately, on the whole, the UK networks haven’t. They’ve concentrated on 3G. Orange have recently announced some EDGE support, while (as far as I know) the others have made no such announcements.

The networks spent a huge amount of money on 3G licences, and the customers basically haven’t turned up (ok, Ian, I’m excepting you and your N90!) It shouldn’t matter where they recoup this investment – hell, it’s now a sunk cost. Recouping from the mass market of existing 2G customers, to me, makes more sense than trying to charge huge premiums on UMTS.

Adding EDGE would also improve bandwidth for 3G users outside the 3G coverage area, probably at a lower cost than expanding the 3G network, assuming that the user’s equipment also supports EDGE.

Now, I wonder if someone could ask this question for me?

Race condition in eVC linker?

2006-04-01T11:23:00.000+01:00

I’m not sure exactly why this happened. Yesterday I was trying to do a batch build for one of our most complex components – our Barcode Scanner Hardware Abstraction Layer (ScanHAL). This is a set of libraries which implement the same interface, to make our applications hardware independent with respect to the barcode scanner, making it possible to run more-or-less seamlessly on HHP Dolphin, Intermec and Symbol hardware, as well as on handhelds with no built-in barcode scanner (there’s a stub implementation). The clever bit (heh) is that all of the different libraries are included in the same CAB package, with a setup DLL which probes to decide which library is correct for the handheld it’s running on.

Anyway, the release process involves a batch build of everything, which rebuilds everything from source. For historical reasons the projects still support Pocket PC 2000, which ran on a bunch of different processor types (ARM, MIPS, SH3, x86 emulation). The build process currently builds both debug and release builds. For one particular customer, we made available a release build with debugging information, stripped of private symbol information using a tool by John Robbins called PrivateStrip – this means that they can tell us which function the program crashed on, if it does, but can’t easily disassemble the library. So in all there are about 20 different configurations of different libraries that need to be built.

Unfortunately since Wintellect reorganised their website, PrivateStrip is no longer available. I hope they’ll reinstate it.

Yesterday, the linker was repeatedly hanging in the last stage of building DLLs: it had just output the message about building the .LIB and .EXP files, but wasn’t completing the build. On a couple of occasions it did this on the last DLL – but it wasn’t consistently on any given DLL. My work machine is hyperthreading-capable, and this was happening with HT enabled. Turning HT off enabled me to complete the build.

I don’t know whether this would also happen on a dual-core or other multiprocessor machine.

You might need to be careful if you’re using older tools (eVC 3.0 and 4.0 both use modified versions of Visual C++ 6.0’s LINK.EXE) on a computer with more than one logical or physical processor.

Best line from Hustle tonight

2006-03-24T22:02:00.000Z

Tailor: “Do you dress to the right, sir?”

Danny: “No, I always vote New Labour.”

Tailor: “Ah, swinging to the right it is…”

Subtle interaction between WM_DESTROY handler and OnFinalMessage in ATL for CE

2006-03-18T18:20:00.000Z

There’s a subtle issue around handling WM_DESTROY in an ATL window and also overriding OnFinalMessage, if using the ATL windowing classes on Windows CE.

Typically, you would override OnFinalMessage if you wanted to do some final cleanup when the window has been destroyed and no further messages will be received. The canonical example – given in the documentation – is automatically deleting the object that manages the window using delete this.

On the desktop, OnFinalMessage is called by the CWindowImplBaseT::WindowProc function after the WM_NCDESTROY message is received. Since Windows CE top-level windows don’t have a non-client area, the non-client messages were removed from the platform. The Windows CE version of ATL therefore uses WM_DESTROY instead.

The code inside WindowProc only cleans up the window proc thunks (dynamic objects containing code used to patch between Windows’ expectation of a flat API, and a call to a C++ object member function) and calls OnFinalMessage if the message (WM_DESTROY or WM_NCDESTROY as appropriate) is not handled. This means it will work fine if you don’t override WM_DESTROY, but what if you need to do some cleanup here?

The last parameter of a message handling function in ATL is a BOOL reference typically named bHandled. This is set to TRUE by the message map macros (ATL message map macros generate a big if statement, rather than being table-driven like MFC) on calling the message handling function. If the value is still TRUE when the function returns, the message map considers it handled and stops processing; if it’s FALSE the macro code keeps looking for a handler.

If you handle WM_DESTROY on CE using ATL, you must set bHandled to FALSE, otherwise the window will never be cleaned up, and OnFinalMessage will not be called. The same is true if you handle WM_NCDESTROY on the desktop, but since this message is rarely handled, it’s less of an issue.

This issue was responsible for me leaking 96 bytes in a program every time an activity was completed, which doesn’t sound like a lot, but any long-running application needs to be free of leaks as far as possible. This was actually a Compact Framework program for the most part – this window implemented a signature capture control, which is actually pretty hard to do in CF 1.0, and we already had working C++ code (excepting leaks…). For some strange reason, though, a leak of 96 bytes here was causing an overall leak of over 300KB of virtual memory! Removing this leak has got it down to about 160KB but I’m stumped on where the rest of it’s going – the amount of VM allocated increases by 160KB whenever the form containing the control is created, but that isn’t returned when the form is disposed. A C++ program containing the control doesn’t exhibit the same behaviour.

Compact Framework seems to do wacky things to the process default heap, which usually means that you can’t actually view it with Remote Heap Walker – it aborts after several seconds of trying to walk the heap. To find this leak I created a non-default heap, overrode operator new and operator delete to allocate and free from this alternative heap, then viewed that.

Time to move on?

2006-03-13T13:35:00.000Z

I'm thinking about moving away from Blogger/BlogSpot. It's not a great fit for my content really. I'm considering either geekswithblogs.net or wdevs.com.

If you're subscribed and want to follow me if/when I do move, I've added a feed through FeedBurner which should allow me to automatically switch to whichever new service I go for. In the spirit of full disclosure I should add that it also allows me to track who's reading my feed, which I don't get at the moment from Blogger. I believe it also does content negotiation, so if your aggregator supports RSS you should get RSS while if it supports Atom you should get Atom.

Tired of cancelling ActiveSync Partnership Wizard?

2006-03-13T11:51:00.000Z

If you're a developer using Windows Mobile or Windows CE devices, you may be a bit fed up with the Partnership Wizard that appears when you connect a device. If you decide to create a partnership every time, your list of partnerships grows huge, you keep having to think of a unique name, and you still have to do it every time after hard resetting the device. Alternatively, if you cancel out or opt to connect as Guest, you have to do this every time you connect. After nearly five years of mobile device development I'm sick at the sight of this wizard.

There is another option: to use an undocumented registry key to stop the wizard appearing. Open Registry Editor (as an administrator) and navigate to HKEY_LOCAL_MACHINE\Software\Microsoft\Windows CE Services. Create a new DWORD value called GuestOnly and set its value to 1. The wizard will no longer appear on connecting a device. If you ever do need to create a partnership, you'll need to set this value back to 0 before connecting.

If you already have a partnership with a device, it will still connect with its partnership, rather than as Guest, even if this option is set.

What's the difference between x64 and IA-64?

2006-03-12T19:20:00.000Z

There still seems to be a bit of confusion over the identity of 64-bit processors for Windows operating systems. Windows runs on two types of 64-bit processor, identified as x64 and IA-64.

Going with the oldest first, IA-64 stands for Intel Architecture, 64-bit and was Intel’s attempt to move to a 64-bit architecture. It remains compatible with 32-bit user-mode code through hardware emulation, but this hardware emulation performs so poorly relative to a genuine x86 that all operating systems shipping for IA-64 now include a software x86 emulation called IA-32 EL. IA-64 is the instruction set; the family of processors which implement it are named Itanium (which you’ll see critics refer to as Itanic, suggesting that it’s sinking) and so you’ll sometimes see code written for them referenced as Itanium Processor Family or IPF. You need a new operating system to run any code on an Itanium – it cannot boot a 32-bit operating system.

x64 is actually a Microsoft term and stands for Extended 64. It is implemented by both AMD and Intel who respectively call it AMD64 and EM-64T (Extended Memory 64 Technology). AMD invented it as extensions to the existing x86 architecture. An x64-capable processor natively implements x86 in hardware – it is an x86 processor, with additional modes. It boots up in the same 16-bit ‘real mode’ that an x86 does. It can run existing 32-bit operating systems. You may well have an x64-capable processor without realising it. All AMD Athlon 64 and Opteron processors, and Intel Pentium 4, Pentium D and Xeon processors built within about the last year, implement x64. To check whether your Intel processor supports x64, use Intel’s Processor Identification Utility.

Itanium had the early lead and for a while held the general performance crown, but the relentless increase in x86 clock speeds eventually had AMD’s Opteron overtake it for integer calculation performance. Itanium still leads for floating point performance but has been stuck at 1.6GHz for about two years, if I recall correctly. It’s liable to be beaten by Intel’s own ‘Woodcrest’ Xeon-replacement later this year, in my opinion, if it remains stuck at this (now relatively low) speed.

Itanium is now pretty-much reserved to big-iron datacentre servers. It’s good for highly computationally-intensive applications. If you just need lots of memory, go with x64. Itanium used to have an advantage in number of supported processors too, but IBM recently started selling the xSeries 460 which supports up to 32 dual-core Xeons. This hits Microsoft’s limit of 64 logical processors which applies to both architectures.

Development Tools for Windows CE and Windows Mobile

2006-03-11T05:26:00.000Z

This is kind of a brain-dump. I get asked this a fair bit – which tools do you need to develop for Windows CE and/or Windows Mobile? (See here for the difference.)

Native code:

For Windows CE 3.0 custom platforms, Pocket PC 2000 and Pocket PC 2002: eMbedded Visual C++ 3.0. You cannot debug on these older devices using eVC 4.0; you can’t debug CE 4.x or later devices using eVC 3.0.

For Windows CE 4.x custom platforms: eMbedded Visual C++ 4.0. At least SP1 is required for CE 4.1, SP2 for CE 4.2, latest is SP4. SP1 and SP2 were mutually exclusive – if you installed SP2 you couldn’t develop for CE 4.1; this was rectified in SP3.

For Pocket PC 2003 (alternatively Windows Mobile 2003 for Pocket PC), Smartphone 2003 (Windows Mobile 2003 for Smartphone) and respective Second Editions: eVC 4.0 SP2 or later, or VS 2005.

For Windows CE 5.x custom platforms: eVC 4.0 SP4, or VS 2005. You will get link errors complaining about corrupt debug information if you use eVC 4.0 because the platforms are actually built using version 13.1 (VS2003–compatible) compilers while eVC 4.0 SP4 can only handle debug information from version 12.0 (VC 6.0–compatible) compilers, hence SP4 only includes 12.0 compilers.

For Windows Mobile 5.0: VS 2005 only. The SDKs do not install into eVC 4.0.

For whatever device you’re building for, you need the correct SDK. However, you will find that programs are binary-compatible across different CE platforms, if the APIs required by the program are present on the platform. Using the correct SDK ensures that you build for the correct processor type and don’t accidentally reference APIs that won’t be available at runtime.

Managed code:

.NET Compact Framework 1.0 is supported for Pocket PC 2002, Windows Mobile 2003 for Pocket PC, Windows Mobile 2003 for Smartphone and Windows Mobile 5.0, and custom CE 4.x platforms. For Pocket PC 2002 you must use VS.NET 2003; for Windows Mobile 5.0 you need VS 2005 (I think). For Windows Mobile 2003 you can use either, and I would strongly recommend using VS 2005 as soon as you can stand to convert your project. This will completely rewrite your resx files. Converting the project does not mean immediately upgrading to .NET Compact Framework 2.0, that’s a separate step.

.NET Compact Framework 2.0 requires VS 2005 and only runs on Windows Mobile 2003 for Pocket PC (not WM2003 Smartphone), and CE 5.0 and Windows Mobile 5.0 devices.

VS2005 requires ActiveSync 4.1, minimum, for deployment and debugging. I was originally annoyed at the loss of network synchronisation capability, but found that, if a wireless connection is present, you can begin a debugging session over USB and continue over wireless if you disconnect from the cradle or cable.

To complete the compatibility matrix, as far as I can see CF1.0’s SqlCeClient only works with SQL Server CE 2.0 while CF2.0’s only works with SQL Mobile 2005 (SQL Server 2005 Mobile Edition). If anyone knows different let me know.

I tried so hard

2006-02-27T21:00:00.000Z

I tried so hard, but I was defeated.

I was looking to buy Joe Satriani’s new album, “Super Colossal”, from a genuine UK retailer – ideally actually a UK copy. First of all, it doesn’t look like this is getting a UK release, or at least if it is, there’s no official information on it.

OK, so it has to be an import. I look on Amazon UK, it’s there, I pre-order it. About a day later I get an email – they’re cancelling the order. So now I’m buying from CD-WOW instead. Since this is Sony USA, I hope it’s not too badly DRM infested (although I am of course running as a limited user).

If you’re interested, Joe’s podcasting a little about each track, plus a one minute (or so) preview of the track. Schedule: Monday/Wednesday/Friday. If you want the videos, you’ll need QuickTime 7. I’m using Media Player Classic plus ffdshow codecs.

Back in time

2006-02-14T22:14:00.000Z

I’ve been meaning to switch around the samples of the band I was in, back in sixth form. This is an appropriate day to do it, because the song I’ve put up was written the day after Valentine’s in 1995, and it was written because of what happened.

Song: Torn In Two (MP3, 128kbps, 7.2MB, 7:53).

The song, basically, is about Dave having a crush on a girl in his music class, and her not being interested. He sent her a card, she sent him a note, and he was pretty cut up. He wrote some lyrics that night (first verse and chorus, if I remember right), the following morning he and Roger put together a chord sequence and basic vocal melody, then that afternoon I joined them to practice. After Roger had to leave, I added the second verse, and David and I put together the third. I took some of my inspiration from my own feelings at the time (yup, more unrequited crushing) and some from the note Dave received. There are harder things to take when you’re 17 than ‘I just want to be friends,’ but not all that many.

We practiced the song for about a week, then Dave called his guitar teacher and asked if, instead of a lesson, we could record it. Our then-drummer, James, couldn’t make it, so we had Nigel, the teacher, program a sequencer with a simple drum beat. Roger and Dave played together, with me singing a guide track, to get the keyboard track into the sequencer as well. Then Dave let it all out (and boy, did he let it out) on his lead guitar track – that’s all one take I think, or it might be two. I added the vocals, then we asked Nige to add a bass track for us. There’s a little ‘fill’ bit in the bassline where the fridge motor cut in and knocked the sequencer out for half a bar! A quick mix later and we had something. I can’t recall if Dave sent the girl in question a tape or not – he may well have!

I’m not sure if we were asked, or Dave asked, to perform Torn In Two at a school concert. The girl asked us not to, but by that point Dave had got over it a bit, so we did it anyway.

Six months later we had a new drummer, Chris, and returned to Nige’s studio to record four more tracks (among them, Survivor). We asked if we could add a new, live, drum track to Torn In Two. We found the tape, which had miraculously not been recorded over, but the sequencer program was gone: Roger had to re-record the keyboards. Nige programmed in a ‘click’ track for Chris to follow, since the sequencer timing data was still on the tape. I’m still amazed at just how well Chris was able to add drum fills building to some key parts in Dave’s solos.

I love this song. It’s my favourite of the ones we recorded. Now that I have my own guitar, it’s one of the songs I practice, although I’m only playing the chords in a semi-acoustic setup.

I’ve left Survivor up for the moment; I’ve re-encoded to 128kbps to save some space and download time.

WeWare?

2006-02-02T21:35:00.000Z

Eric Sink has a great article “Yours, Mine and Ours” in which he discusses different types of software:

“I claim here that there are three categories of software:
MeWare: The developer creates software. The developer uses it. Nobody else does.
ThemWare: The developer creates software. Other people use it. The developer does not.
UsWare: The developer creates software. Other people use it. The developer uses it too.”

Can I add WeWare to that list? I define it as MeWare but for your own development team. This gives it a slightly larger audience – requiring a touch more thought than MeWare in user interface and usability, but not really requiring the robustness or even completeness of true UsWare.

I spend a fair chunk of my time on WeWare – libraries for helping to complete a project rather than actually writing the code that solves the customer’s problem. Of course I do a lot of that too.

We’re still having trouble pushing Meteor Server over the chasm from WeWare to UsWare (from an application-development point of view, at least – there are plenty of installations where we wrote the application). We might have a couple of customers now, but it remains to be seen whether they’re able to run with it themselves.

Why you should install and enable a firewall on your PC

2006-02-02T21:13:00.000Z

…even if you have a hardware firewall/NAT/whatever.

Larry Osterman has a great post “Firewalls, a history lesson,” in which he makes an analogy to the first world war. An interesting read.

I should take up this fight with my colleagues again. They all think I’m crazy for running as a low-privileged user and having the XP SP2 software firewall on, but when one of the salesmen brings their horribly-infected notebooks into the office for me to disentangle, I’m glad of it.

I remain unconvinced of the merits of a two-way firewall: the trick is not to get the malware onto your PC in the first place. Two-way firewalls are pretty annoying whenever there’s a change to the client software you use; you only have to configure an incoming-only firewall when there’s a change to the services you provide. There’s a common problem in computer security – ensuring that you don’t train the user to just click ‘Yes’ all the time. That’s why the ‘enter root password for elevation’ prompts in Mac OS X worry me, especially since there doesn’t seem to be a way for the user to validate that the prompt came from a secure subsystem rather than J. Random Malware. I’m actually happier that the initial plan for Windows Vista is that “Consent Admins” will default to being presented simply with a dialog explaining the elevation, to which you click Permit to elevate or Deny to refuse.

MLP

2006-01-31T22:32:00.000Z

One reason you should clean up warnings

What's the difference between Windows Mobile 5.0 and Windows CE 5.0?

2006-01-12T21:39:00.000Z

I got the following question in email today:

Can you please explain me the difference in windows CE 5.0 and windows mobile 5.0? Isn't windows 5.0 is based on windows CE 5.0? If yes then why is the development tools for these are different, i mean windows CE based applications can be developed using evc++ 4.0 but for WM 5.0 based application we need Visual Studio 5.0?

Well, I’ve kind of answered this question already. The only new thing to add is that Windows Mobile 5.0 for Pocket PCs is based on Windows CE 5.1 bits according to the About screen (Microsoft using pre-release versions of Windows CE in Windows Mobile again? That caused a boatload of trouble for Pocket PC 2000 and 2002 and I thought they’d finally got over it.)

As for the development tools question, don’t ask me – ask Microsoft. The simple answer is that they didn’t generate an eVC-compatible SDK therefore it doesn’t register with eVC 4.0 therefore you can’t select Windows Mobile 5.0 as a target. As to why they didn’t do this, who knows? Perhaps something to do with the old Platform Manager, which was not the most reliable of software (understating the case severely). Also, it appears that MFC 6.0/CE and ATL 3.0/CE are not supported for new development; they don’t ship with the SDK any more, although the DLLs do ship on the device I think.

Two more alleged WMF 'vulnerabilities' - but there's a problem with the 'exploit'...

2006-01-10T20:28:00.000Z

A number of news sites are pointing to a post on the Bugtraq mailing list alleging more problems with Windows’ handling of the Windows Metafile format.

Just a quick recap on the original issue: I originally thought that this was simply a buffer overflow issue, but in fact it appears that it’s something different – that an intended feature can be used in an unintended way. As I said last time, a WMF file contains a sequence of GDI commands. One of the supported commands is the GDI Escape function, which allows the application programmer to pass additional commands to the graphics driver, which – because GDI is a unified screen and printing API – can be a printer driver. The exploit apparently uses the SETABORTPROC escape. This escape was intended to permit GDI to call the application back, periodically during printing, to determine whether the user had tried to abort the print job. The attacker can use the SETABORTPROC escape to point to another part of the WMF file which contains code, which will be executed by GDI. It’s a case of an overlooked feature with insufficient security protection, not a failure to correctly validate the input parameters – the parameters are valid.

To the new ‘vulnerability’. Here we are dealing with a malformed file. The attacker supplies sizes for some of the parameters which are larger than the amount of data supplied. There is no vulnerability here – all that happens is that Windows tries to copy more data than is supplied. When the source pointer goes off the end of the input buffer, it may encounter an unallocated page. When this occurs, an access violation exception occurs, which, unless the application has been written to guard against it, causes the application to crash.

Note that this cannot crash Windows itself. It can only crash the process performing the file parsing. Now, in many cases this will be Windows Explorer (explorer.exe) – but Explorer should restart after a crash (it always used to – I haven’t actually had a problem in a while so I don’t recall if it still does). If an attacker put a WMF malformed in this way on a website, and the user browsed to it, the browser would simply crash. So yes, it is a denial of service, of a sort, but it’s not a serious issue.

With this information in hand, Microsoft’s response seems pretty reasonable.

Don’t believe everything you read on Bugtraq.

WMF vulnerability patch to be released early

2006-01-05T20:35:00.000Z

2pm Pacific Time today. That’s 10pm GMT according to my handy World Clock app (recently updated to not crash if you’ve disabled automatic daylight saving adjustment).

Thoughts on the WMF vulnerability

2006-01-03T23:16:00.000Z

OK, so we know there’s a zero-day vulnerability (i.e. one which was not reported to any security organisation or vendor before being exploited) out there which utilises a malformed WMF file to execute code on a victim’s computer. This is being termed a ‘remote code execution’ vulnerability – this term is now being used to cover any situation where an attacker could cause code to be executed, but doesn’t differentiate between a situation where this can be done by the attacker actively sending data over a network to the victim, and one (such as this) where the victim must request data from the attacker. However, the attacker can cause software executing on the victim computer to automatically request the bad data – in this case, for example, by sending an email message to the victim containing a suitably malformed image file, which will cause some email packages to automatically render (draw) the image when the message is displayed.

Firstly, what is a WMF file? It stands for Windows Metafile. That’s a pretty meaningless name. What it actually contains is just a sequence of commands – that map one-to-one to GDI API calls – for producing a drawing. The easiest way to construct a WMF is to use the CreateMetaFile API which produces a GDI drawing surface, a device context, and returns a handle to it, an HDC. Once you’ve finished drawing – using the regular GDI API calls – you then call CloseMetaFile which gives you a HMETAFILE. You can then draw the metafile again using PlayMetaFile. It appears that this API is the one which contains the vulnerability – that some part of the format is insufficiently checked and the attacker can therefore cause the processor’s instruction pointer to end up pointing at a part of the supplied file.

This does suggest that any application that renders WMF data using the PlayMetaFile API could be an attack vector. Because it is such a venerable format, many applications will support it. You can include WMF drawings in your Word documents. You can process WMF files in Paint Shop Pro.

The current advisory from Microsoft suggests unregistering the shimgvw.dll component. This component is responsible for much more than WMF rendering. It performs all thumbnail rendering in Windows Explorer for all the best-known filetypes. It provides the size and other information for the Task pane and status bar. It also implements the ‘Windows Picture and Fax Viewer’ frame that appears if you click Preview on the context menu for an image. Unregistering this DLL kills all this functionality – but it does not protect against the vulnerability in other applications which call PlayMetaFile (except those which use shimgvw.dll as a proxy, such as Internet Explorer). This is my supposition, anyway – I would be astonished if shimgvw.dll did not render WMF simply by calling PlayMetaFile, and likewise Enhanced Metafiles by calling PlayEnhMetaFile.

While WMF files are most often used in the filesystem for storing vector-based clip-art (one among many other formats), you can also find them used within other formats, because of the native OS support. For example, when copying a diagram from Visio to Word, you will find that the prerendered version of the diagram (used for a linked or embedded diagram when the diagram is not active) is a metafile – although in this case it is most likely an Enhanced Metafile. Whether the Enhanced Metafile format can also be exploited is an unknown.

How to stop TlbImp requiring admin privileges

2006-01-03T20:32:00.000Z

It seems that the Type Library Importer tool, TlbImp.exe, sometimes needs administrative privileges to do its job, if you’re trying to create a Primary Interop Assembly (or otherwise a strong-named reference). At work today, even admin privileges weren’t enough – I’m not sure if something’s been broken after installing Visual Studio 2005 since it used to work fine. The error given is ‘Invalid strong name parameters specified.’

What seems to be required is that the strong-name-key needs to be added to a key container in order to be used, which TlbImp is doing for you behind the scenes. The container used by strong-name functions can be either a user or a machine container. The default seems to be a machine container (at least that’s been the case both here at home and at work).

You can switch to a user container using the following command from a Visual Studio command prompt (either from the Start Menu’s Visual Studio group, or by running vsvars32.bat from the Common7\Tools directory, or sdkvars.bat from the .NET Framework SDK Bin directory):

sn -m n

Having done this, you should no longer need administrative privileges to strongly-name an assembly.

This much laughing hurts

2005-12-23T12:01:00.000Z

I don’t often end up laughing hysterically at musical instruments!

In today’s feeds, an article from the Annals of Improbable Research blog linked to a Mr. Jay C. Easton, who plays the saxophone. Well, not just the saxophone, since there are many different types. Many more than the traditional four.

Including this monster.

There are some very smart saxophone makers out there, and the sound is actually pretty good. Well, except for the very lowest notes. I’m sure you can guess what they sound like, but there are plenty of clips on Easton’s site and Mr Eppelsheim’s. Another player, Fred Bayer, has had his Tubax modified to play much higher notes too (click on Tubax under Music in the navigation frame – a direct link takes you to the German language site).

Revisiting .NET CF Whinges

2005-12-22T21:05:00.000Z

18 months ago, I wrote an article called “.NET CF whinging”. Now that Compact Framework 2.0 has been released, what’s been fixed?

I complained that only DrawLine was available, not DrawLines (which maps better to the underlying Polyline call). In .NET CF 2.0, you can now use DrawLines.
Previously, you could only create a one-pixel-wide pen. Now you can create pens of any width.
The System.Diagnostics.Process class has been added.
WaitHandle.WaitOne now offers a timeout overload. WaitAny is still missing.
Thread.Abort and Thread.Join are now present.
Control.InvokeRequired makes an appearance, and the Control.Invoke overload that takes an array of object parameters to pass to the delegate is now present. However, the documentation still contains a note that for .NET Compact Framework, you must use the EventHandler delegate, which only offers an empty EventArgs. This appears to be a documentation error – passing other delegate types and parameters does now work! The new BackgroundWorker class of the desktop Framework is not offered.

Rumour has it that the marshalling’s better in this version too.

HOWTO: Program Symbol MC3000 function keys

2005-12-22T20:41:00.000Z

With my powers of intuition, I divine that someone out there needs to handle or program the function keys on the Symbol MC3000 device.

(Well, OK, it turned up in my referrer log – click the tracker icon at the bottom of the right-hand bar!)

The 28–key variant has F1 through F10 available through pressing the blue Func button and then the corresponding number key. The 38–key variant has F1 through F10 keys, then F11 through F14 on Func+F1 to Func+F4. The 48–key variant has F1 to F13 on Func+A to Func+M. (Nothing like consistency). When pressed, these will appear as VK_F1 through VK_F14 in your application’s WM_KEYDOWN and WM_KEYUP handlers or their equivalent in your environment (Control.KeyDown and Control.KeyUp events in the .NET Compact Framework, the value of KeyEventArgs.KeyCode will be Key.F1 through Key.F14).

That’s the default keyboard map. If you want to change the keyboard map, on any Symbol device, you need the Device Configuration Package for that device, which you can download from Symbol DevZone. In the package, you’ll find under Tools\kbtool a bunch of C++ files and a VC6 project. This is a program for your desktop computer which generates the keyboard map files. Open this with VC6 or later.

Each keyboard layout has (up to) three files associated with it – for the MC3000, MC3000KeynnM, R and S.cpp, where nn is the number of keys. The M file controls the multiple-press key operations, such as the mobile-phone-style input on the 28–key variant. The R file controls remapping key functions in different shift states. Finally the S file controls the scan code that’s output by each key – this is normally the one you want to edit. The available scan codes can be found in the accompanying ScanCode.h file.

Running the program generates the files that the keyboard driver is looking for. For the MC3000 these are named MC3000–{28,38,48}.{sct,rmt,mmt}. You can either overwrite these files in their normal location (\Platform\Keyboard) or modify the registry under HKLM\Hardware\DeviceMap\KEYBD to point to your version of the files.

We did this for New Look to make the two keys below the screen – with the red and green surrounds – perform ‘soft key’ functions in their applications.

Do I look old to you?

2005-12-20T20:02:00.000Z

I met up with some of the people from Channel 9 last Saturday – Dave Oliver, Barry Dorrans, Sarah Blow and Richard Peat (and his lovely wife Beth, who I don’t think has her own blog). Richard’s already written it up here.

Separately, Dave and Barry both told me that I looked older than my tender years (ahem) of 27, and suggested that I get rid of my beard. Barry also reckoned the ponytail should go (can’t remember what Dave thought but he did admit he’d had one when younger!)

For comparison:

What I looked like in 1998, the last photo I have to hand of me with short hair and no beard (digital snapshot of my passport photo!)

What I looked like last year (taken at my friend’s wedding in Toppenish, WA, USA).

The length of hair and beard varies. I was short-haired and clean-shaven at my graduation ceremony in 2001 and frankly I thought I still looked a bit too young in the photo (and very pale!) I’ve been letting the hair grow pretty much since then and it now just reaches the small of my back. I trim my beard every few weeks, normally when it starts itching too much – it’s quite short right now as you can see in Richard’s picture from Saturday where we’re attempting to play Barry’s card game Fluxx while waiting for food and/or Ian to arrive.

So, should I follow their advice?

Can the Windows Vista Network Map Control Panel detect a non-functioning switch?

2005-12-20T17:10:00.000Z

Adam Nathan blogged today about some new features in the Windows Vista December CTP. One of them is a new Control Panel applet: Network Map.

At work we commonly have problems with our switched network fabric. Sometimes after a power failure (a regrettably common occurrence at our location) the cheap 5-port switches at each desk will give all the signs of working, but not actually forward network frames – quite literally, “the lights are on, but no-one’s home.” Only power-cycling them gets them working again. Unfortunately there are chains of these in some places before reaching one of the wall sockets which connects to the main switch.

We also have a problem where, if one particular machine is switched on and connected, the entire LAN seems to go nuts, intermittently or permanently causing some routes to fail, unpredictably. We think this is due to the switches’ internal mapping of MAC addresses to ports somehow becoming incorrect, and forwarding packets to the wrong ports, or not at all. This is a mystery to all of us, especially since we’ve replaced the network card with a different model, tried it with different operating systems, tried it on a different wall socket, and repatched the wall socket/main switch connection. Nothing helps! If you have any ideas, leave a comment!

If this Network Map can tell you which device seems to be malfunctioning, that would be a serious boost to network administrators everywhere.

Idiotic things to do

2005-12-19T00:23:00.000Z

Look on Flickr for your own photos, from the machine you uploaded them from.

Doh!

The two-minute Xbox 360 and PGR3 review

2005-12-18T21:30:00.000Z

I’m underwhelmed.

OK, that evaluation is based on playing a guy for one race on multiplayer on a demo pod yesterday. The shop was pretty loud and the console quiet, which made it not very involving. The rumble didn’t feel that great. The graphics are better than the current generation but actually not that much better than PGR2 – but I had a one foot view, not a ten foot view since the controllers are hard-attached to the pod. I think the screen was set up correctly but I don’t actually know.

We were lost in the game’s UI. Very lost. We ended up having a race with one computer car which zoomed off into the distance – and that was only on Medium skill. I lost – I kept crashing into the corners due to the lack of feedback from the game (audio and rumble), and lost when I crashed into the last corner on the last lap. I lost so badly that I didn’t even reach the finish line before the 30 second timer ran out after the computer car crossed the line (hence ending the game).

Maybe I should go back when it’s quieter and have a go at Career Mode.

Coining a word

2005-12-14T22:44:00.000Z

horricious (adj): Mind-numbingly awful. Descriptive of the speed of Microsoft’s first attempt at a ‘real’ Windows CE emulator.

A colleague asked if there was an emulator for one of Symbol’s hand-helds – a customer wanted to evaluate it with some software. While eVC 4.0 and VS.NET 2003 do ship an emulation platform, and Symbol supplies an emulator image for this platform, it is pretty slow. In describing this emulator to my colleague, I inadvertently coined this word. I’d just used ‘horrible’ and ‘horrendous’ and was searching for a third – lists of three, you know – and somehow got ‘horrific’ and ‘atrocious’ mangled together.

Oops.

The Pocket PC 2000 emulator, and earlier Windows CE emulators, ran weirdly compiled x86 code inside an app running on top of Windows. Stable it was not, and it didn’t emulate a CE device at all accurately. This made it completely unusable for actually developing apps.

The Pocket PC 2002 and later emulators were a little better. Now they were based on Connectix’ Virtual PC technology, which virtualises your PC’s processor and other resources (Microsoft bought out Connectix soon after). Unfortunately that’s still a virtualised x86, which is different from the actual Pocket PC 2002 devices, which use ARM processors. The result is that you had to do a special build for the emulator, meaning again that you’re not testing the same code you’re going to ship – and it was very hard to get libraries built for the emulator.

Windows CE on the x86 has always been a slug – Microsoft have never really taken it seriously as a platform, most hand-helds using MIPS or SH3 to begin with, then later ARM. Virtual PC wasn’t really designed to run Windows CE. Putting some form of Additions on the image would have been a) hard and b) defeated the object a little.

Visual Studio 2005 (at least, some editions) includes the Microsoft Device Emulator 1.0. This, finally, emulates an ARM processor. And you know what? It’s quicker than the virtualised x86! At last, it’s possible to use virtually a real OS image and real binaries in an emulated environment.

It’s still quite a bit slower than a real device though, even a 400MHz PXA255.

A rant on programming books

2005-12-14T22:20:00.000Z

Matt Pietrek – he of the erstwhile MSDN Magazine column ‘Under The Hood’ – has an interesting post:

In a nut shell, there are a lot of books out there, and people are relying on them less and less as search engines become the dominant way to find information. It's pretty hard to pour massive amounts of work into something when you know most people just want the code snippet that gets them past their current problem. No actual learning required.

...and then they have no idea how the code works, so cannot adapt it to new requirements, or debug it when it doesn't work.

To me, a good tutorial is worth a thousand times a code snippet - although I do find that often MS Press books (at least, of a certain vintage) will print the entire source of a sample program which leads to the new stuff being lost among the boilerplate.

How new Windows programmers are supposed to learn what's going on when truly excellent books like Jeff Richter's "Programming Applications for Windows" go out of print I don't know. All abstractions leak, and I think you're better off understanding what leaks through the abstraction.

I only really ‘got’ MFC once I’d read “MFC Internals”, and understood the boilerplate that Visual Studio generates for a new MFC project.

How media companies root your system

2005-11-01T00:01:00.000Z

…and why you shouldn’t insert an audio CD into your computer if you’re running as an administrator: what Mark Russinovich found on one of his systems.

(If you’re not running as an administrator you, or software running on your behalf, cannot install a driver).

MetaBlog: CAPTCHA enabled

2005-10-11T20:03:00.000+01:00

Over the last week or so, comment spammers have found this blog. I’ve therefore turned on Blogger’s only security feature, word-recognition (sometimes known as CAPTCHA). It’s probably not enough to fool a determined comment spammer but I hope it will reduce it a bit.

I’ve also enabled ‘backlinks’ which I hope are just a Blogger name for referrers or some such.

Project Gotham Racing 3 videos

2005-10-01T00:42:00.000+01:00

Some shakycam footage of a presentation of Project Gotham Racing 3.

Xbox360 News has six videos. They link to another which is still downloading as I write this.

I have to admit that graphically, it’s very, very good. There’s no substitute for actually playing it though. Looking at these videos does make me think that the game may well actually be ready for launch day – for an idea of software manufacturing schedules, note that Visual Studio 2005 is due to RTM on 14 October with physical product supposedly available on 7 November. In both cases the manuals and boxes can be pre-prepared, with only the DVD duplication required. Bizarre Creations still have probably a month to fine-tune the game.

There’s something worrying me. In the first video (actually the fifth link, named ‘pgr3gp1.wmv’) the menu has ‘Buy A New Car’ and ‘Sell Car’. I’ve played many games where you have to earn money to buy new cars and they’re universally a pain – you can never, ever afford a decent car, and the sale price is a small fraction of what you paid to buy it. It doesn’t look like PGR3 will have ‘repair’ requirements though, where you have to pay to repair your car. This was one of the many, many things that made Sega GT (a freebie with my Xbox) completely annoying and unplayable (on the other hand, the other freebie game, Jet Set Radio Future, is one of my all-time favourites and one of the major reasons I bought the console).

I recall that PGR2 has ‘Kudos Tokens’ which you earn by gaining a level, which you do simply by earning Kudos (basically by racing well or doing tricks like sliding the back end out or doing donuts, although I hope that’s easier on the new game). You unlock cars by exchanging Kudos Tokens, and in PGR2 you can’t sell a pony that you’ve bought. Having said that in PGR2 it’s rare that you’ll buy a complete dud because the car stats are pretty accurate (indicating that the Porsche Cayenne Turbo is by far the best SUV for example). As long as you’re not penalised too heavily for making a bad car choice, I’m happy. PGR2 also had a test drive feature but I’ll confess I never used it! PGR3 is meant to have much more level car sets, so there isn’t one ‘right’ car to use for a section of the game.

Here’s hoping they skip the American Muscle section this time. Last time out you’d got used to the nice-handling-but-a-little-slow cars in the early stages of the game, only to suddenly encounter cars which accelerate fast but lurch all over the place, and have no brakes to speak of. These cars were uniformly awful, there was no good one in the bunch. Making it through this section was a tedious slog, and it was a definite relief when I finished that section and could move onto some cars which, you know, handled.

What does the (VB.NET) ComClass attribute do?

2005-09-04T05:17:00.000+01:00

A former colleague emailed me this week to ask about registering C# assemblies for COM Interop. He was convinced that you needed to do something special to advertise a COM object from C#, since we’d used the ComClass attribute in VB.NET. Also, Rockford Lhotka wrote (in July 2004):

“There are several COM interop features in VB that require much more work in C#. VB has the ComClass attribute and the CreateObject method for instance.”

I was able to assure him that for his scenario – late-bound use of an object – all you need to do is use regasm to register the assembly. That registers every public class in the assembly, except any that are marked [ComVisible(false)]. (A quick aside here – the registration time is longer the more public classes you have, so you should only make the classes you actually want to expose to COM public.)

But if regasm registers all public classes in an assembly, what is the ComClass attribute for? The documentation suggests it’s required but as we’ve just seen it isn’t, at least not for this scenario.

It’s a signal to the VB.NET compiler to do three things:

Emit an interface definition automatically generated from the methods and properties in the class – a so-called Class Interface.
Add this interface to the Implements list for the class (so the class appears to implement this interface).
Emit a <ClassInterface(ClassInterfaceType.None)> attribute to suppress auto-generation of a class interface.

That last one is interesting. The .NET Framework supports automatic generation of class interfaces at registration time using the ClassInterfaceAttribute attribute. That generated interface is either a dispatch-only or a dual interface depending on the type selected. So what’s the difference?

Let’s take a simple test:

Imports System.Runtime.InteropServices

<ComClass()> _

Public Class WithComClassAttribute

    Sub Test()

    End Sub

End Class

Public Class WithNoAttribute

    Sub Test()

    End Sub

End Class

<ClassInterface(ClassInterfaceType.AutoDual)> _

Public Class WithClassInterfaceAttribute

    Sub Test()

    End Sub

End Class

First, let’s look at the generated code with the help of Reflector (in C# mode):

namespace VbComTlbTest
{
      [ClassInterface(2)]
      public class WithClassInterfaceAttribute
      {
            // Methods
            public WithClassInterfaceAttribute();
            public void Test();
      }

      [ClassInterface(0), ComClass]
      public class WithComClassAttribute : _WithComClassAttribute
      {
            // Methods
            public WithComClassAttribute();
            public void Test();

            // Nested Types
            public interface _WithComClassAttribute
            {
                  // Methods
                  [DispId(1)]
                  void Test();
            }
      }

      public class WithNoAttribute
      {
            // Methods
            public WithNoAttribute();
            public void Test();
      }
}

Notice that VB has generated the automatic class interface as a nested type, inside the class it belongs to.

To see the difference from a COM perspective, we need to look at the type library. Export it using tlbexp, then use OLE View from the Platform SDK (also supplied with VS.NET) to load the type library (File/View TypeLib).

I’m not going to post the entire type library IDL shown by OLE View, but note that not setting an attribute generates an empty class interface definition (equivalent to setting ClassInterfaceType.AutoDispatch), and that setting AutoDual generates a class interface containing all inherited methods, including ToString, Equals, GetHashCode and GetType from System.Object.

So to duplicate VB.NET’s behaviour with the ComClass attribute in C# – if you need to, bearing in mind that this really only applies to early-binding scenarios – you need to define an interface then implement that interface, and use ClassInterfaceType.None. This (declaring interfaces) is, of course, what you have to do in traditional C++.

Answering the mail - anatomy of a solution

2005-08-29T01:14:00.000+01:00

I’m a bit slow in answering questions from the contact box. A couple of months ago, a commenter asked how the MC3000 application for New Look was implemented. Since we’re using this system as a case study it’s probably fair to post details if I don’t go into too much detail.

Actually I’m not sure where New Look’s name came up. I don’t think I’ve mentioned it here. It must have been some other forum.

As far as the MC3000 software goes, the main application is just Meteor Client. Meteor is our application server product which allows text-mode applications to be written using Windows-based desktop/server development tools – typically more rapidly than the equivalent handheld application. On this occasion, the four applications (Price Check, Stock Availability and the ancillary Menu and Head Office applications) were developed with Visual Basic .NET using a SQL Server 2000 data source. The total number of handhelds deployed is over 1200 I believe.

Meteor Client was originally written for Symbol’s Series 3000 DOS handhelds and the original/common code is written in C targetting a Symbol library called ADL and Symbol’s BiosXxx routines (wrappers around both the standard IBM PC BIOS interrupts and Symbol’s own BIOS features). It was ported to a Windows console application which is mainly used for development and testing although it has been used in deployment occasionally. It was also ported to PalmOS for the Symbol SPT 1700 and Monarch’s Pathfinder Ultra DOS device, although those ports are not currently maintained.

Meteor Client on Windows CE was originally targetted at Pocket PC-based devices such as the PPT 2700. I started the porting work about three years ago. Pocket PC devices don’t have console support and even if they did, the console would still – I think – have ended up with a certain amount of ‘furniture’ we didn’t want. Meteor Client is meant to be a locked-down application, no other applications can be used.

Without this console support, I had to provide all on-screen rendering. The only monospaced font I could rely on was Courier New, which I detest – it would be illegible at the screen sizes we need – and we needed the potential for custom font capabilities anyway. Therefore we use bitmapped fonts using bitmaps I drew, designed to use the maximum space available on the Pocket PC screen and others at two sizes, 8 rows by 20 columns (matching the PDT 3100) and 16 rows by 21 columns (matching the PDT 6800). Further fonts have been drawn to offer better displays on devices with different screen sizes – the MC3000 has a 320x320 screen which can all be used – and there are now 8 sizes ranging from a 9x9 pixel character cell up to a 15x39 pixel character.

The Windows CE port uses C++ for the Windows CE-specific code, and uses ATL for windowing support and a small amount of additional data structure support. MFC is overkill and I’d originally intended to do without it, but when it came to integrating barcode support I remembered that the code I’d written up to that point used bits of MFC since the new CE apps we were writing at that point were all written with MFC (the .NET Compact Framework wasn’t released until 2003). As part of the process to turn the port code into a separate library for porting DOS applications, I’ve eliminated this dependency.

Actually, the core of the porting library is really the keyboard buffer. Windows’ keyboard system requires that a window accept keyboard input messages. There’s a bit of a mismatch between the DOS behaviour, which made both scan code and character code available at the same time, and Windows, which generates a WM_CHAR message with the character code after the WM_KEYDOWN message with the virtual key code, but this wasn’t too hard to work around. Originally all the screen and keyboard code ran on the same thread as the application code, which meant that the screen didn’t update until you called either a keyboard wait function or a delay function, but I’ve recently – as part of the porting library effort – moved all the screen functions, and hence the keyboard functions, to their own thread. Cue some fairly complicated synchronisation code.

For barcode scanning I originally used a chunk of code that was, although slightly enhanced and made into a static library, some of the first code I wrote after graduating. Its main problem was that it only supported Symbol’s scanning APIs – it would run on a non-Symbol device but barcode scanning would be disabled. Over December to February 2002 we came up with a more abstract API that we call the Scanner Hardware Abstraction Layer, or ScanHAL for short. We now support Intermec and HHP’s APIs as well as Symbol’s, and have a package that can detect what device it’s running on and therefore install the right ScanHAL (there’s a null stub for devices with no scanner). This code is all C++ and, due to an error that was overlooked when building the first few versions, is tricky to use from C as all the entry point names are type-decorated (i.e. using the C++ linkage naming convention – we forgot to use ‘extern “C”’). We use this library, enhancing it as new features are required, in all our .NET Compact Framework applications too (if scanning is required). A colleague wrote the API and the initial version of the Symbol and Intermec HALs; I’ve enhanced both and wrote the HHP HAL, while Ian’s extended all the HALs at least once.

The New Look handheld software doesn’t just consist of Meteor Client though. There are a few other things too. Primary among them is support for firmware updates over-the-air (i.e. over the network). Symbol’s AirBEAM supports over-the-air upgrade but it doesn’t support scheduling an upgrade and for obvious reasons we didn’t want to push a firmware upgrade – which since they’re using the colour terminal is over 30MB – during work hours. Symbol do have a tool intended for making an update image on an SD card which we’ve adapted. Meteor Server pushes a notification of a new version to Meteor Client, which schedules the upgrade to occur at a random time within an administrator-specified time window. When that time occurs the updater program I wrote downloads the required files over HTTP (using the WinInet library), checks that the files downloaded correctly using the CRC32 algorithm, then runs our unpacker for Meteor Client and the other software, then Symbol’s updater. When the updater finishes successfully it cold boots the device which causes the new software to install. The install-after-cold-boot mechanism is another tool that’s evolved over my four years with the company – primarily my work.

To ensure that the update happens when the administrator wants it to, the device’s clock must be fairly accurate. Windows CE does supply a Simple Network Time Protocol client, but Symbol didn’t include that component in the MC3000 platform. We’d had trouble with its reliability on another device previously. I therefore wrote one myself, which took a couple of days – it’s less than 750 lines of C++ code (including blanks and comments) which includes converting from UTC to the local time zone since this part of Windows CE is often unreliable – it will tend not to take daylight saving into account, or do it twice. We therefore turn off the AutoDST setting in the registry – we update the clock every few hours anyway so being slightly wrong at the start and end of daylight saving doesn’t matter. It’s somewhat simplified code – the time is only computed to an accuracy of one second, which means no fixed point maths, and no roundtrip delay calculation is done. This provides plenty of accuracy for our purposes though.

The final part is the keyboard map. We do two things here: first, we map the two keys at the top of the MC3000’s keyboard which have, respectively, a green and a red surround so that they get key codes. The default keyboard map leaves them unmapped. These keys, being directly below the screen, are used as soft keys in the applications. Secondly, we found that since there’s no indicator light near the Alpha key, users could not tell when Alpha mode was active – we had a few support calls in the trial deployment asking why the handheld was simply beeping when any key was pressed, which was simply because Alpha was on. New Look use the 28–key keyboard. Style codes, colour codes and SKUs are all numeric, so there’s little need for alphabetic input, but the customer wanted to keep access to alphabetic characters for future applications. We therefore remap Alpha to a function key code and, in Meteor Client, toggle the on-screen keyboard when that key code is received.

That’s just the client side. This project, being the first to use multiple co-operating Meteor Servers to provide scale-out and failover capability, required some extensive work on the server side too.

Aaarrgh!

2005-08-14T00:59:00.000+01:00

In my last post I said Microsoft would be releasing some official information on a topic that had caused plenty of FUD on forums. They haven’t yet done so. This is kind of annoying. The guy I was talking to hasn’t responded to my query about where the statement would appear. So the world – or at least, that part of the world that discusses these things on forums – carries on consuming and spreading FUD.

Some parts of the Windows team reacted very quickly to the suggestion that Monad could allow the spread of viruses. This team needs a boot up the backside, IMO. Scoble, are you listening?

OK, enough beating about the bush – I asked about OpenGL, and specifically this topic. Now I come to re-read the topic, it’s clear – if the app asks for accelerated pixel formats, and there is an Installable Client Driver available, Windows Vista will switch off the desktop compositor while the ICD is in use. What effect this will actually have I don’t know – presumably it will temporarily revert to using software desktop rendering, i.e. directing each window to draw in turn, clipping the parts that are obscured, as Windows has done for about 18 years (Windows 1.0 only allowed tiled, not overlapping windows). And of course you’ll only notice this if you’re using a windowed OpenGL app – full-screen apps (like games) will no doubt want the desktop compositor turned off!

The stuff about taking a 50% performance hit is simply FUD, IMO. The presumption is that translating OpenGL calls to Direct3D calls is expensive, but I don’t really see that. Numerous sites have had posters claiming that OpenGL is a more performant API than D3D, but without a program released for both APIs with as much time spent on both implementations, benchmarked on a range of hardware to eliminate driver variability, there really is no way to tell. I think the basis for this is essentially iD fanboys – since iD use OpenGL for their Quake, Return to Castle Wolfenstein and Doom 3 engines, it must obviously be better, right? I honestly don’t know, except to say that the games I’ve played have always seemed ‘fast enough’ if my system’s been within recommended spec, whatever API was used.

If you want to know something, ask

2005-08-10T20:06:00.000+01:00

Conspiracy theorists: if you want to know why Microsoft are doing something, try asking them.

I recently saw some information on forums that seemed wrong, so I did a bit of research and found a presentation that seemed to apply. Microsoft presenters normally put their email addresses on the cover, and this was no exception, so I emailed the presenter asking him to clarify a few points. I got a nice reply back which covered what I was after.

Yes, I know that’s very wooly. I asked if I could blog the conversation and he said no. However, there should be some official information appearing later today. When it does, I’ll update (if I remember).

When in doubt, ask. If they can’t say, they’ll normally say so.

Where's the damn SDK?

2005-08-04T22:13:00.000+01:00

As I’m sure you’ve already seen, Windows Vista beta 1 was released about a week ago. This means I’m thinking about resurrecting my Changes to Win32 API in Longhorn series. Except, I can’t yet – because the SDK hasn’t been released yet!

Some kind of developer preview this is…

(Yeah, OK, the WinFX SDK has been released. But I’m still something of an unreconstructed Win32 guy.)

Loving the new monitor

2005-08-04T20:27:00.000+01:00

I finally decided I needed more desktop space: both computer desktop, and the real desktop my monitor sits on – and this computer sits under. So I decided it was time to replace my old faithful Iiyama VisionMaster Pro 400 17” CRT with a flat panel. I looked at a number of sites and picked the ViewSonic VP191b 19” panel. So I can now use 1280x1024 comfortably.

It took a little while to adjust it to how I like it – colours seemed a little washed out – and it’s very bright. For tips on adjusting your monitor, see Charles Poynton’s article on ‘Brightness’ and ‘Contrast’ controls.

It cost a lot – nearly £400 after VAT and delivery – but I reckon it’ll last. It’s got something to live up to – the Iiyama lasted seven years with me, and is still going strong.

The Little Button That Could

2005-07-14T23:28:00.000+01:00

Good story today on The Daily WTF.

The reason I posted this, though, was this fantastic comment: “It's like he tried to reinvent the wheel and some how came up with a Platypus or something.”

No2ID Flair

2005-06-21T21:26:00.000+01:00

Following Ian's example, I've added the No2ID PledgeBank petition flair to this site. It looks like this:

The counter updates with the number of people who have signed up. Meta: I need to think about moving off Blogger. Republishing the whole site (e.g. when changing the template) takes an age.

Check your signatures and use precautions

2005-06-14T18:49:00.000+01:00

I’ve had 24 copies of the I-Worm/Mytob.HI worm since 1am today. Either they’re sending to spam lists or there’s a lot of infections out there.

This one’s being sent with faked ‘From’ addresses, info@<domain>, admin@<domain>, service@<domain>, etc. Mostly the message is about account suspensions, password updates, etc. Don’t be fooled. Don’t open the attachments.

Mac & PC - will the performance comparisons end?

2005-06-12T01:33:00.000+01:00

From Paul Thurrott: a link to Java Rants, “Will partnering with Intel give Apple a Mac faster than a PC?”

No.

Let me expand. The theory goes that the processor is laden down with years of backwards compatibility that costs performance – starting afresh with a special customised ‘x86’ with fewer instructions, without this backward compatibility, could improve performance. Apple’s ‘special’ x86s could then beat ‘regular’ x86s.

The trouble with this argument is that it’s bullshit.

The Pentium 4 does feature compatibility right back to the 8086. If the platform support is right, you can, it is believed, boot the original MS-DOS 1.0. But that doesn’t slow it down.

It’s correct that the core of the P4 is RISC-like, but only in the sense that the aim of RISC was to have very simple instructions that could be decoded by simple logic circuits which would execute on the core in one cycle. Modern ‘RISC’ processors are the same, effectively, as the core of the P4 – there are multiple execution units that can execute operations concurrently, in the same cycle. On the P4 there are two Arithmetic-Logic Units [ALUs] which can actually perform two operations per cycle – one on the first half of the cycle, the other on the second. The key is in how the instructions are decoded.

Simple x86 instructions are decoded pretty much as a true RISC processor would – using logic to directly decode an x86 instruction into a core-compatible micro-operation [µop]. The resulting trace goes in the trace cache – the P4 attempts to only decode a stream of instructions once, performing loops directly from translated instructions. Anything more difficult than this – say, indexed indirect memory loads with autoincrement – goes to a microcode ROM which contains a ‘program’ of µops to implement that x86 instruction. If the microcode program is greater than 4 µops in size, the execution core has to execute directly from the microcode ROM rather than the trace cache. I think that pretty much all of the clever out-of-order stuff is suppressed when this occurs. Hit a large instruction from ROM and the processor slows to a crawl.

Anyway, after that brief technical interlude, let me explain why removing backwards compatibility won’t speed up the processor. Because most of it’s implemented in the microcode ROM. I don’t know if you recall when the P4 first came out. A lot of commentators, running their tests on Windows 98, suggested that the P4 was actually slower clock-for-clock than the PIII. Guess why? Partly that code was optimised for the PIII which had different characteristics from the P4 – but also that Windows 98 still contained a lot of 16–bit code, which hit the microcode ROM.

If you don’t use the expensive instructions, you don’t incur any cost, particularly, for them being there. Presumably there’s a little more complexity in the decoder logic to determine that the instructions are in microcode.

People often make the same mistake with respect to Windows XP. They think that the existence of the DOS and Win16–compatible subsystems slows it down. Nope. That code lives in NTVDM.EXE and WOWEXEC.EXE, which aren’t even loaded unless they’re being used. I also need to be clear that the console window is not DOS. cmd.exe, the Command Prompt, is a 32–bit Windows application that uses the console subsystem. The difference between a console app and a Windows app is that a console app has a console created for it if its parent was not running in a console. What is a console? It’s a window which is created and updated by the CSRSS process. There’s some code in there to handle back-compat graphics but again this almost certainly isn’t loaded unless it’s being used. A lot of this has been ditched for Windows x64, but that’s because the processor doesn’t support virtual-8086 mode, necessary for this support, in so-called Long Mode (64–bit mode).

cmd.exe supports (I believe) all of the DOS command interpreter, command.com’s feature set, and extends it greatly. This, plus the command-line environment, leads a lot of people to be confused. Particularly when some of the system utilities, like more, exist as files named more.com. If you look with dumpbin or depends, you’ll see that more.com is in fact a Win32 console executable.

Speaking of x64, I note that Apple aren’t going straight for x64. Their Universal Binary Programming Guidelines [PDF, 1.5MB] talk about the 32–bit register set, not the 64–bit extended set. I expect that this decision is because the Pentium M with EM64T – codename ‘Merom’ – isn’t due out until late 2006.

This whole argument presupposes that Intel will make Mac-specific processors. I think that’s highly unlikely. Apple haven’t told us their motivations, leading to a lot of speculation. My view is that they are trying to both save money, by using a more commodity processor, and enable quicker access to newer technologies, by using more commodity chipsets (oh, and save money). It took them years to get AGP in Mac hardware, and it looked like taking even more years to get PCI Express. I don’t therefore expect Intel to produce ‘special’ processors for Apple – it’ll be stock parts or nothing.

Another reason for using stock parts is so that stock software tools can be used. Apple can mostly wave goodbye to maintaining their own compiler. It’s no accident that their guidelines state they’re using the System V i386 Application Binary Interface [PDF, 1.0MB], with only minor modifications. Of course, that means that to begin with, Apple will be using the very compiler – GCC – which they used to ‘prove’ that PowerPC G5s outperformed P4s, in that contentious set of benchmarks. Unless they’ve improved the optimiser, or learned how to use it properly, ‘PCs’ may well still outperform Macs.

One tiny area in which Macs may have a temporary advantage is in exception handling. Today, on 32–bit Windows, each use of an exception handler pushes a handler frame onto the stack, a cost incurred whether or not an exception actually occurs. I believe GCC uses a table-based scheme, which in general incurs that cost only when an exception occurs. However, Windows on x64 – indeed on every architecture other than 32–bit x86 – also uses a table-based scheme.

Passing the word

2005-06-11T23:20:00.000+01:00

Want to keep your right, as a UK citizen[*], to privacy? Sign up for the No2ID pledge here.

Via Ian.

[*]Not a subject, dammit. I do not recognise the Queen as my ruler. Rulers are things you draw straight lines with – she’s a bit bumpy for that.

Watermark rendering bug in ListView

2005-05-28T20:40:00.000+01:00

Or at least, in the ‘My Pictures’ folder in Windows XP Explorer. Try it yourself – ensure the watermark is enabled. Then right click and select ‘New Text Document’. Click to remove the rename highlight. Then select it again and press Shift+Del. Hit Yes.

If this reproduces on your computer, you’ll see something like this:

I guess the ListView code simply calls ScrollWindowEx to move any remaining lines up, but doesn’t take account of the watermark.

New technology plug: since Avalon is a stored graph system, it should handle this kind of thing for you. That is, you tell it what components make up your rendering – it renders the appropriate parts as required.

More Mobile Developer Shafting: Compact Framework 2.0 won't run on CE 4.x

2005-05-28T16:57:00.000+01:00

I downloaded the Compact Framework 2.0 Redistributable yesterday as well, planning to do a little testing with one of our CF 1.0 apps. (I need to discover how to force using 2.0 if both are installed). In doing so I discovered that CF 2.0 only supports Pocket PC 2003 and Smartphone 2003, Windows CE 5.0 custom platforms, and Windows Mobile 5.0 (for Pocket PC and Smartphone).

That means that new devices, introduced this year, like Symbol’s MC3000 which runs a custom CE 4.2 platform, are not supported. Why aren’t Symbol using CE 5.0? Who knows. Maybe they didn’t want to revalidate all their components on the new OS yet.

Yesterday I was at the London Eye. They appear to use PPT 2800s with the optional trigger handle to scan the barcoded tickets. This device runs Pocket PC 2002.

Customers often want upgrades and/or new applications to run on existing hardware. We’d like to be able to use the newer tools to implement those upgrades or new applications. But we can’t, because those tools won’t run on the old hardware.

The full framework is compatible back to Windows 98. We can’t get three years’ worth of backwards compatibility in the mobile world.

.NET Framework 2.0 Compatibility

2005-05-28T14:04:00.000+01:00

Brad Abrams: Take the .NET Framework 2.0 Compatibility Challenge.

I was at a slightly loose end yesterday afternoon, so I took advantage of that time to check that Meteor Server, and the price checking/stock availability application we wrote for a large deployment at a UK retailer, still work correctly under .NET Framework 2.0 Beta 2.

Meteor Server itself is mostly VB6 components – a server executable, an application-host component executable (ActiveX EXE in VB6 terminology), a few application-interface components which have strict binary-compatibility requirements – with a few bits of C++ where the VB6 environment really ran out of steam (TCP/IP printing, since TCP is a pain with the Winsock control, and a ‘callback’ component which replaces the use of minimum-delay timers). Applications are written as COM components. Recently – in the last couple of years – we’ve been writing new applications in VB.NET or C#. We’ve generally found that more productive than VB6. To do this we’ve built Primary Interop Assemblies [PIAs] for the application-interface components.

Meteor Server isn’t really .NET-aware. We do ship configuration files for the server and application-host executables, to force the latest version of the PIAs to be used. This allows older applications to run on a newer version of the server, without having to distribute the PIAs with each app, or all versions that have ever existed with Meteor itself. Remember that .NET binds to a specific version of an assembly, not to the most recently installed. We did try using Publisher Policy but had problems with it – sometimes the Framework would fail to load an application object, often crashing with an Access Violation in fusion.dll. Due to a misunderstanding in version numbers we had to ship multiple policy DLLs when using that scheme – one for every major.minor version of the PIAs we’d released.

Using explicit configuration reduces the frequency of this problem, but not to zero – we still occasionally see this in deployment, with Framework 1.1 SP1. It’s something of a heisenbug – it never happens when you’re looking for it. And a particular nightmare when dealing with VB6 code, because the VB6 debugger does not run your code as it will be run in deployment. Instead it hosts it in-process in the VB6 environment. You’re not even running the compiled code, I believe it translates the code to p-code and interprets that. While you can turn on PDB generation and debug with VS.NET or another native code debugger such as WinDbg, the generated PDBs are poor quality and don’t, for example, often include your local variables or globals properly.

Since it’s not really .NET-aware, the default behaviour of the Framework when loading into an unmanaged host applies – the most recent version of the Framework is loaded. We support loading alternate applications under application control in the newest versions of Meteor (I think the version currently available on our website is very out-of-date). The aforementioned price-checking application is actually implemented as four applications: a menu application which loads the others, the price-check application, the stock-availability application, and a head-office application which has testing features. This does mean that all applications will need to support the latest version of Framework, or the admin will need to tweak the config files to force all applications to bind to the older runtime.

Anyway, I’m pleased to report that there were no obvious problems using Framework 2.0 with any of these applications.

StarCraft: Ghost

2005-05-19T22:57:00.000+01:00

How long have we been waiting for Blizzard’s StarCraft: Ghost? I can’t recall – I think they first announced it in 2001.

This week at E3 Blizzard have published new trailers and gameplay movies. I can’t help thinking it might be too little too late. What blew us away three or four years ago now looks a bit, well, ordinary. Halo 2 seriously raised the bar.

What’s been taking so long? I don’t know, but it looks like Blizzard have ditched their original partner Nihilistic Software, bought up a console developer, and brought the game back in-house.

The question for Blizzard: can they release this game before the next generation of hardware arrives and they look completely stupid?

Xbox Backwards Compatibility

2005-05-17T21:59:00.000+01:00

Xbox Backwards Compatibility

Now that it's been announced at E3, I can finally reveal that the Xbox 360 feature I work on is... Xbox backwards compatibility!
Yes, it's real. It's been fun to watch all the wild speculation over the past year or so as to whether the Xbox 360 would or wouldn't be backwards compatible. And all the wild ideas about how hard or easy it would be. It's clear that a lot of people have very little understanding of how these things work.

Were I forced to speculate, I’d expect some kind of JIT-compiler like the x86–on-Itanium IA32EL or the x86–on-Alpha FX!32 is involved. I can’t remember the old rule-of-thumb for emulation – was it that you should have a processor ten times as powerful as the emulated processor to reach parity? Of course that was for state-machine-based emulators, where essentially each instruction was executed in turn by emulating instruction fetch, decode, execute, retire.

Whether a particular Xbox game runs on Xbox 360 will depend on just how closely the assumed-JIT and its library emulate the original processor and, of course, GPU. Hopefully the fact that Jet Set Radio Future was a launch title and was bundled with some consoles (e.g. mine!) will be taken into consideration!

Dammit, more incompatibilities

2005-05-16T20:33:00.000+01:00

From Windows Mobile Platform Migration FAQ for Developers:

Starting with Windows Mobile 5.0, only one instance of the installer (wceload.exe) can be running at a time on Pocket PC devices. This restriction has always been in place on Smartphone. As a result, CAB files that start other CAB files from a custom setup DLL may not work on Windows Mobile 5.0. The workaround for this is to have a small executable in the CAB that a custom setup.dll starts in its Install_Exit entry point. This executable can get a handle to the running wceload.exe process, wait for wceload.exe to exit, and then restart wceload.exe on the additional CABs.

We have a pretty clever (says he who wrote it) technique of packing multiple CABs into one wrapper CAB file, which enables us to install Compact Framework, SQL CE, our barcode scanner hardware abstraction layer (ScanHAL), our simple large-key input method, and the application itself, all in one go. Looks like I’ll need to rewrite it slightly at the install-to-RAM stage.

Well, I hope it takes a while before devices appear

2005-05-10T21:50:00.000+01:00

Today, Microsoft announced Windows Mobile 5.0 (codenamed ‘Magneto’). They’ve made two bone-headed decisions from a development perspective:

ActiveSync 4.0 no longer supports TCP/IP synchronisation
According to this page, eVC 4.0 cannot be used to develop for Windows Mobile 5.0. You must use Visual Studio 2005.
Also, it seems you can’t use VS.NET 2003 to develop Compact Framework applications for WM5. Likewise, you must use VS2005.

The first can be worked around – you don’t need to use the ActiveSync transport or startup server for Platform Manager, but it’s a heck of a lot more convenient than using the TCP/IP Transport with the Manual startup server. With that combination you need to enter a horribly long command line (example: CEMGRC.EXE /T:TCPIPC.DLL /Q /D:192.168.1.2:5913) on the device. VS.NET 2003 and 2005 do not use Platform Manager, though – trying to get both hooked up to the same device – as I tried to do today to actually view the contents of a memory block allocated in C# code, because opening the debugger memory window kills the debugging session – is highly problematic.

The second and third are idiotic. Hint to MS: VS2005 is not yet released! Yes, there’s a beta. No, I wouldn’t trust it for production development.

And there’s a new development. Mobile device development is a feature in the Standard Edition and above. It is not a feature of the Express Editions. Microsoft claim that there will be a Compact Framework SDK. Previously eVC was free. This made a big difference.

It’s not as if you can mix-and-match that well, either, because the current VS2005 beta apparently requires ActiveSync 4.0 – not actually released yet – in order to deploy to a device. Gee, thanks </sarcasm>.

Yup. Mobile developers get the shaft again. I’ve spent some time today trying to work out a way of avoiding getting blocked when calling InternetCloseHandle when a dial-up networking link has gone down. We seem to have the choice of either rebooting when blocked (enterprise app, so not completely out-of-bounds but undesirable, especially when the hardware cold boots when it was supposed to warm boot) or leaking handles, or rewriting to use WinInet asynchronously. WinInet isn’t well documented for the synchronous case, for the async case it’s downright abysmal.

How does a New submenu Command know where to create the new item?

2005-04-24T03:45:00.000+01:00

On the command line, that’s how.

You must specify %1 in the command line somewhere, otherwise Explorer simply doesn’t do anything (it reports “Not enough storage to complete this operation”, an erroneous error message). The %1 is replaced by the full path to the file that would have been created.

Looking at Briefcase’s .bfc registry key shows that there’s also a second option passed to FormatMessage to create the command line, and it’s a number – the command line includes %2!d! which is indicative of the use of FormatMessage. It’s not clear what this second option means, however.

So that’s a step towards building our Burn Folders. Since our shell extension will be a DLL, we’ll probably include the New command as an entry point and use rundll32 to launch that. That keeps everything together.

Musings: How to produce a Burn Folder in Windows?

2005-04-24T03:11:00.000+01:00

One of the new features in Apple’s Mac OS X ‘Tiger’ is ‘Burn Folders’ – a folder which you can easily write to a CD by selecting the ‘Burn’ option. It contains links to the files or folders to burn, not copies of the files. In other ways it’s just like any other folder. This allows the user to stage multiple CDs at once, and keep frequently-written CD data in an easy-to-write folder. How hard would it be to create one of these folders in Windows?

The Windows (Explorer) shell is extensible. One of the things you can extend is the shell namespace. The Recycle Bin is a shell extension. So is My Documents. A Burn Folder might therefore be something like Recycle Bin. It’s even more like a Briefcase, a feature much touted back in Windows 95 – it’s still there in XP, but its functionality is largely replaced by Offline Files for the situation where you’re taking your computer away from the office.

Let’s assume that when you create a Burn Folder, we actually create a real folder with that name with some special properties, and in it, when you drop an item, we store a Shell shortcut to the item you dropped. We should also support copy and paste, pasting links. By using shortcuts we should be able to leverage the Distributed Link Tracking service to keep track of where items are if you move the original file. I’m not sure what we do if the original file was removed – presumably we have to try to resolve all the links before starting to burn, and if some are missing we need to report that.

So, we have a namespace extension, and it sounds like we’re probably going to use a folder as the junction point. We’ll have to extend the New submenu to allow such a folder to be created (I guess we’ll have to use the Command option). We’ll have to modify the Explorer toolbar to show the Burn button. To address a complaint Paul Thurrott made about ‘Tiger’s Burn Folders, we should probably also modify the status bar to show the size of the linked files (i.e. how much space is required).

That covers the UI, then we have to be able to write to the disc. Here we’re pretty much stuck with just being able to handle CDs and writing them using the Image Mastering API.

This might be a fun thing to do. Something to add to my stack.

Full-screen is a new feature??

2005-04-23T00:02:00.000+01:00

Paul Thurrott: Tiger math 2: When a feature isn't a feature

Apparently Apple are touting full-screen video playback as a new feature in QuickTime 7 in Mac OS X ‘Tiger’. Except that you actually don’t get it in ‘Tiger’, you have to pay for an upgrade to QuickTime Pro – the upgrade to the Pro edition of 6.5 cost $29.99.

In contrast, Windows Media Player has full-screen support out-of-the-box. I seem to recall that it has done ever since Media Player 6.4 on Windows 2000. Certainly the version of Media Player 6.4 that ships with Windows XP (run mplayer2 from Program Files\Windows Media Player), version 6.4.09.1130 on my computer, has this feature.

Scary...

2005-04-23T00:01:00.000+01:00

On top, Emperor Palpatine from the Star Wars trilogy. On the bottom, Pope Benedict XVI.

Or is it the other way around?

Tory posters defaced in Reading

2005-04-15T18:36:00.000+01:00

Seen on Microsoft Typography news: Anger after Tory ads vandalised.

It wasn’t me, but I can understand why they were vandalised. The Tories’ whole campaign is based heavily on ignorance – xyz can’t be that hard, can it?

Yes, usually it can. I’m surprised – well, actually I’m not – that Labour and the LibDems have not actually rebutted how hard it is to do xyz.

Bring Back Chris Brumme!

2005-04-07T19:16:00.000+01:00

Chris Brumme’s blog was one of the first I started to read, after Scoble, Chris Sells and Don Box. He’s managed a tricky task of explaining very complex subject matter (how the .NET CLR works) well.

Sadly he’s not posted in nearly a year, and the last entry was just announcing updates to existing articles.

So I’m joining Chris Anderson and Brad Abrams in saying:

Putting things in perspective

2005-03-11T19:22:00.000Z

Mozilla and all their fans, and FOSS people in general, are fond of saying that Firefox has had over 25 million downloads. Sounds like a lot, right?

I was on the Skype site earlier today. We use it at work in preference to calling mobile phones. (I was on the site because users were having trouble logging in – some kind of authentication server failure.) Skype are currently reporting 84 million downloads, and one million users using SkypeOut, the Skype-to-PSTN service.

Is Firefox really all that popular, then?

MS Press, what are you thinking?

2005-02-28T23:16:00.000Z

It looks like a couple of Jeffrey Richter’s classic and essential Windows programming books have gone out of print:

The .NET Framework and the C# programming language have huge productivity benefits, but there are times when you still need to get close to the metal. When you do, you need the best training possible. If you need to do it, those two books and Windows Internals, 4th Ed. are your best guide – IMO – to understanding what’s happening and how to make best use of the facilities at your disposal.

How to set the owner of an object in Windows XP

2005-02-28T01:14:00.000Z

Normally, in Windows NT-family security, you cannot set the owner of a secured object (such as a file or folder, or a registry key) to someone else – you can only take ownership for yourself or a group you’re a member of. If you’re unprivileged you can only take ownership if the Discretionary Access Control List (ACL) gives you permission to take ownership. There is also a privilege – SE_TAKE_OWNERSHIP_NAME – which allows you to take ownership even if you’re not permitted in the ACL. This privilege is normally and by default only assigned to the Administrators group.

This ability to take ownership is in marked contrast to Unix where administrators can set ownership, but users cannot take it.

But what if you want to set ownership to a different user? If you try to execute this without any additional privileges, under I think any security principal, you’ll get an error. If you (like me) were using ADsSecurityUtility, it’ll be encapsulated in an HRESULT, code -2147023589 (0x8007051b). Anyone familiar with HRESULTs will recognise the 0x8007 high word as representing SEVERITY_ERROR | FACILITY_WIN32, i.e. a Win32 error code encoded as an HRESULT, so we can look up 0x51b = 1307 in WinError.h which is ERROR_INVALID_OWNER, “This security ID may not be assigned as the owner of this object.”

Looking this error code up in a search engine reveals the problem, that other people have had this problem before, and the fix. If you have the SE_RESTORE_NAME privilege enabled you’re permitted to set the owner of a secured object. Which makes sense, really, as you want restored data to have the same owner as it did when it was backed up.

Privileges come in two types – those which are enabled by default if they are permitted to a user (or a group of which the user is a member), and those which are disabled by default. Of the latter type, some Win32 APIs enable the privileges they require automatically. SE_RESTORE_NAME is not one; you must call AdjustTokenPrivileges to enable the privilege if not already enabled.

By default the Administrators and Backup Operators groups only have the SE_BACKUP_NAME and SE_RESTORE_NAME privileges. You can modify this and other privileges through Group Policy – for your own machine use the Local Security Policy shortcut in Administrative Tools. It’s really not recommended to modify these settings unless you’re absolutely certain you know what you’re doing. If you must, see under Security Settings > Local Policies > User Rights Assignment.

What a lovely birthday present

2005-02-06T16:32:00.000Z

To whoever broke my driver’s door mirror, and didn’t leave a note:

Thanks for the wonderful birthday present. No, really. NOT!

I’m more annoyed because this time it’s not my fault.

My theory is that someone was going too fast down my road and got a bit close. It’s not as if they wouldn’t notice: the actual mirror is fine – still reflects the road behind me. It’s the fact that the only thing now holding it onto the car is a single clip on the facing plate – oh, and about two feet of gaffer tape. When I found it this morning it was basically hanging on by the three springs used to adjust the position. It might be possible to superglue it but I’ll try that at work tomorrow – I’m not standing out in the road doing it now. In future I’ll have to remember to tuck the mirrors in when I leave it. No-one else in the road seems to.

Part of the problem is that the end of what seems a fairly major road, just before it turns into the very major road, is in fact a bus lane – for all of about 50m – and is closed off to regular traffic, and the only warning of this is the sign a couple of metres before telling people to turn into my road. As a result this road is much busier than it should be.

(I’m 27.)

New Location

2005-01-28T21:23:00.000Z

Not for the blog, for me

After three and a half years living with my parents after graduation, I finally have my own flat. Well, I say my own - I'm renting. I could just barely afford to buy something around here, but only by completely bankrupting myself. It's furnished so I don't have that to worry about.

So far going OK but I'm back on the old modem again - my ADSL won't be connected until next Friday. I'd forgotten how slow this is! Actually I'm getting better connection speeds than ever before with a modem (got 52kbps yesterday) but it's still terrible.

I would have ordered the DSL before moving, but for some reason best known to themselves, BT decided to give me a different number from the previous occupants', which didn't appear in the DSL providers' databases until Tuesday.

Like most things in my life, I prevaricated for ages then sorted itself out pretty quickly. I was fairly happy living with my parents until recently. I think the trip to Seattle and environs in September, followed by a certain amount of independent living when my parents or my sister were away (pet-sitting in the latter case), plus a few weeks off before Christmas, reminded me a) that I could live independently and b) that I actually wanted to - I was going a little stir-crazy over Christmas.

It helped that I knew this flat was going to become free. The couple who lived here before are good friends of mine - indeed, it was their wedding I attended in Toppenish, WA back in September. I've known David since secondary school. He's now gone out to Bangladesh for Ph.D. fieldwork. By taking over the flat (new tenancy) I've saved myself the hassle of finding a place, the landlady's found a tenant easily, and we've both saved agent fees.

A lot of stuff is still in boxes - I only took last Friday off work, before moving in on Saturday - but it's getting unpacked slowly. As and when I can be bothered, mostly!

Reported Vulnerabilities in Pocket IE

2005-01-28T21:10:00.000Z

It's not too surprising that Pocket IE suffers from some of the same flaws as the desktop Internet Explorer - it shares a certain amount of the same code.

One 'fix' that has made it into Pocket IE - at least for Windows Mobile 2003 Second Edition - that didn't in the end go into XP SP2 is patent-related. Yup, the Eolas patent workaround - if the page contains any scripted ActiveX controls, it pops up a dialog reading 'Press OK to continue loading this page.' Pretty annoying.

I've published myself

2004-11-14T17:44:00.000Z

I got fed up with the mess of time zones on Channel 9's forums, and wrote a C# wrapper class for the registry time zone information and some useful Win32 APIs for converting times between UTC.

As a test application and hopefully something useful, I also wrote a World Clock application. It's basic but I hope functional.

Now I just hope I got the conversion code right!

CodeProject article (may move if it gets edited)

Inappropriate voting technology, part 2

2004-11-09T00:45:00.000Z

There are a number of persistent claims that the recorded results for the US presidential election don't actually match the voters' intentions.

What's worrying me is the number of people (look at the comments) talking about optical scanning machines. Sorry, but an optical scanning machine is just as open to fraud by its manufacturer or operators as a touch-screen voting machine. The counting needs to be as observable as the casting of the ballots. I can't see any way that a counting machine can be certified honest, on every day of the year, without actually checking the totals before and after scanning and confirming that the correct numbers are indicated, on the actual day of the election and for every ballot.

You'll note that this amounts to hand-counting. Why not get rid of the expensive, corruptable machinery, in that case? Use hand-counting alone. As I said last time, it doesn't actually take that long. What's more, the actual counting can be observed - the correct assignation of ballots to candidates can be verified, the correct number of ballots in a pile can be checked (normally ballots are placed in piles of 100 during a UK count), and the number of piles and the remainder can be easily counted. Also, the acceptance or otherwise of a voter's mark can be contested.

If you trust the machinery to give you not only the individual totals, but also the total number of papers counted, you have no true paper trail. You can only verify the results by counting the votes.

I've also seen the Open Voting Consortium site. They propose a touch-screen system which produces a paper ballot which can then be machine counted. The touch-screen system does not produce totals, it merely prints a paper ballot with the choices.

Unfortunately it suffers from an even greater flaw than optical-scan - it uses barcodes for the machine counting. There is absolutely no guarantee that the barcode actually contains the choices indicated on the human-readable face of the ballot. There's no guarantee that the barcode reader reports the correct information. There's no guarantee that the totalling machine records the right numbers.

In addition, there's way too much software and hardware to go wrong. Any time a UPS is brought into the discussion, you have to cringe.

My Dad (a 34-year computing industry veteran) pointed out something obvious with even Open Source voting software. There's no guarantee that the code running on the system is in fact compiled from the provided source. There's also no guarantee, as some have already pointed out, that the tools used to compile the source have not been subverted. The only way you could trust a compiled binary is to validate the object code. I've unpicked compiled code and believe me, that's not fun. I'm not sure I trust the security credentials of any organisation that states:

"The companies that produce voting machines have poured gasoline onto the smoldering embers of concern. Some of these products are built on Microsoft operating systems - operating systems that have a well earned reputation for being penetrable and insecure."

Yup, it's a bunch of ABMers.

Inappropriate uses of technology - voting machines

2004-10-30T02:47:00.000+01:00

I've been reading up on the US presidential election. For better or worse, we (the UK) are now in Iraq and would find it difficult to leave, for humanitarian reasons. For that among many other reasons, it would be good to have a US government we (the people) could get along with. Just a note for Bush supporters - Bush is not well-liked in the world. We generally do not want to see him elected. Under his presidency the world's opinion has shifted against the US. According to the PIPA poll, a majority of Bush supporters believe that the world support or are indifferent to the Iraq war, and a majority believe that people in the world would like to see Bush elected. It's not true.

But anyway, this article is about inappropriate use of technology. I'm not going to claim I know about a US election - I've only visited the US once, about a month ago (must write that trip report). From what I've seen through my browsing, it seems as though every locality has its own rules, and a combination of technical means for automated vote counting. We all know about the 'hanging chads' of the 2000 election, where punched cards were used to indicate the candidate voted for. I understand some locations have mechanical 'voting machines' where the voter sets levers according to the candidates they want to vote for, then pulls a big lever to increment the totals. Other places use optical mark readers, where the voter fills in a bubble or completes an arrow (JPEG, 1.6MB - seen on This Is Broken, note that the counting machine only counts the arrow even though the presidential arrows are misplaced!) It seems very few places do it the British way, and you have to vote in all elections at the same time with a whacking great ballot paper.

The British way is very different. For a start, we have no elections for boards of education or drain commissioner. We simply elect a local council. I live in Reading, which is a unitary authority, so there's only one local council - no county council, and no parish council. We also have a Parliamentary System - the head of government is (by convention) the leader of the largest (group of) parties - so no election for President, only for your local MP. We hold a local government election three out of every four years (back when Berkshire County Council existed, the fourth year was the county council election) and a general election whenever the Prime Minister feels like it. There's a mandatory five-year limit on a sitting Parliament (House of Commons, in fact, there is no democratic election to the House of Lords) but a Prime Minister can call it earlier. Normally they're every four years - waiting until the last possible moment, as John Major did in 1997, is a sign of weakness (which was duly punished). All of these elections are currently first-past-the-post systems.

Finally, every four years there are European Parliament elections. These are based on much larger multi-member constituencies and have proportional representation with the D'Hondt method on party slates - you vote for a party, the party decides what order their members will be elected in (a closed list).

The tradition in Britain is that voting is performed by marking a cross on a ballot paper in the box alongside the candidate (or party for European elections) you want to vote for, you then place the paper, folded once, in the ballot box. At the close of polls, the boxes are opened under the scrutiny of officials from each party, the returning officer (responsible for declaring the result, usually the Mayor [largely a ceremonial post for most UK boroughs, although chairing full council meetings]) and local government officers. The ballots are counted by hand, traditionally by bank tellers but they're a dying breed. Votes may be recounted upon request at the discretion of the Returning Officer. If there's a tie, the result is decided by drawing lots.

This year the local election was particularly odd here, because Reading's ward boundaries had been redrawn, and therefore all seats were up for re-election. There are now 15 three-member wards and one one-member ward; I don't live there so I could vote for up to three candidates on the same ballot paper. Interestingly two of the wards split two-to-one between two parties - either some voters had mixed minds or were not aware they could vote for more than one candidate. The European election was held concurrently. The candidates are indicated by their names, their party symbols (if they choose to use one) and the full name of their party affiliation. The European ballot showed the party name primarily but also the ten or so candidates from each party - that was a long ballot paper.

Given the complicated counting required this year, the counts took place the day after the election. Well, normally each election's ballots can be formed into piles for each candidate and a pile for spoiled ballots, then the piles can be counted, as there's a maximum of one cross on each unspoiled ballot. This could be done with the European election but was complicated by the large constituencies and the vote distribution arithmetic. The locals required each ballot to be counted up to three times, not a simple task, although I think they started by dividing the votes into piles of three-Labour, three-Conservative, three-Liberal Democrat and Other, which reduces the complexity quite a bit.

The way the local election worked out, by the way, was that the candidate with most votes has greatest seniority (the seat becoming up for election in four years), then the next for election in three years, and the final one in two years.

General Election ballots can be counted pretty quickly this way; the fastest in 2001 was Sunderland South in 44 minutes. Their task was obviously simplified by the fact they were counting how large the Labour victory would be. The small parties and independents care that their votes are counted fairly accurately - it could mean that they will lose their deposit. In one of the oddities of standing for Parliament, a candidate must pay a Â£500 deposit. If they win 5% or more of the vote, the deposit is returned. Otherwise the government keeps it. If the intent is to reduce the number of crackpot candidates, it's a failure - by-elections (elections outside the normal polling schedule) often have 20 candidates or more.

It seems to me that the US use of technology in elections runs a much stronger risk of tampering, incorrect recording of voter intent and inability to challenge results, with the benefits being solely in the cost of running the election and minor improvements in speed of result collation.

Halo 2 leak

2004-10-15T00:50:00.000+01:00

Halo 2 Piracy (bungie.net)

I haven't gone looking for it. If you see a link, please report it.

Microsoft generally have a problem with leaked versions. I remember seeing leaked Windows 2000 betas (and I think even RTM) when I was at University. My suspicion is that it's not anyone at Microsoft leaking, but someone at the CD/DVD mastering houses. No evidence, just supposition.

I think you'd have to have hacked/modded your Xbox in order to play this, though, since a genuine Xbox disc is dual layer, and it's pretty expensive to obtain a dual-layer burner (if a dual-layer DVD-R would even be Xbox compatible).