You cannot change the doctype attribute of a Mendix application, because the client depends on (the absence of) a DOCTYPE to preserve the correct behavior in all browsers.
But I've seen other mendix-built websites and examples that do have this declaration, even the mendix websites. Could there be another way the to have IE8 show these CSS attributes (outline: 0;) correctly? The problem is that without the DOCTYPE declaration, the browser goes into "Quirks" mode. This limits CSS handling (apparently, i'm no expert yb any means)