Mendix has only tested Mendix deployments under minimal Windows Server 2003 with IIS 6, see the How-to. However, every web server can be run with Mendix, but there is one requirement: the web server has to support proxying some requests to the Mendix server port.
I think it is possible with IIS 5.1, and the simplest way to do that is using ASP.NET. Try to use the same manner described in the How-to for IIS 6. When this does not work, you can try do write your own proxy or sell one on the internet, for example ISAPI Rewrite 3 of HeliconTech which is suitable for Windows XP/IIS 5.1.