I've posted this on the forums a few times, but may as well again.
Here are 2 shots out my front door, straight out of the camera (20D with Canon 10-22mm lens) and only resized for web. The only difference is the aperture. The first one linked is at F/8 and the second one linked is at F/22.
See how the first one is a lot sharper, and the second one really soft because of the distortion.
Before anyone says it, yes, I know I need to clean the sensor. It is why I took these photos in the first place...
I havn't noticed it as much with my 5D, and going by the diffraction calculator that bazz linked to, it doesn't seem to be as much of a problem the bigger the sensor size.