First I needed to remove all padding and margins from the page:

* { padding: 0em; margin: 0em; }

Then the CSS for the #nav container div:

#nav img { border: none; }
#nav li { display: inline; list-style:none; }
#nav li a { float: left; display:block; width: 37px; height: 50px; margin: 5%; }
#nav li a img {  border:none; vertical-align:bottom; /* for IE6 */}
#nav li a:hover { visibility:visible; /* for IE6 */ }
#nav li a:hover img { visibility:visible; /* for IE6 */ }
#nav li a:hover img/**/ { visibility:hidden; /* hide from IE5.0/win */}

Internet Explorer requires something to happen on a:hover as well as a:hover img.

The menu list items all use different images on rollover so I defined each with a different ID on the anchor element:

#one { background:url(http://www.alanwho.com/examples/css/rollover_menu/1over.png) no-repeat left bottom; }
#two { background:url(http://www.alanwho.com/examples/css/rollover_menu/2over.png) no-repeat left bottom; }
#three { background:url(http://www.alanwho.com/examples/css/rollover_menu/3over.png) no-repeat left bottom; }
#four { background:url(http://www.alanwho.com/examples/css/rollover_menu/4over.png) no-repeat left bottom; }

Lastly I created a div with an id of nav and applied the CSS into the final XHTML:

The Final Rollover CSS Image Menu XHTML.