I have an 16GB iPhone 3G.
I have an iMac running 10.5.6
I have iTunes 8.0.2
I used Pwnage to create a custom restore firmware.
When I put my iPhone into DFU mode, iTunes fails to restore using custom firmware.
When I put my iPhone into Recovery mode, iTunes fails to restore using custom firmware.
I'm able to restore it at any point using the original firmware. It's almost like it won't accept the custom for some reason.
I had the same problem with DFU mode and i'm on 10.5.6 on OSX too...
long story short, the 10.5.6 osx update fuXXed the usb ports and stops it seeing the iphone/ipod when it's in DFU mode so it seems fine (black screen) but then fails to PWN...
I know all this cos I went through the same thing.. banging my head against a wall for a couple of days... (it's in the dev team documentation.... I just didn't read it.. obviously