In the latest releases of Mendix the index page contains the following line:
<script> if (!document.cookie || !document.cookie.match(/(^|;)originURI=/gi)) document.cookie = "originURI=/login.html"; </script>
This will redirect the user to the login page if they aren't currently logged in.
If you remove this line from the index.html you should get the popup login again.
I registered ticket 102703 on this. After upgrade to 5.7.0 the inline login changed to full page login, with URL change. After deleting the deployment directory, it became inline again.
To make it even more weird, my colleague always gets the full page login, with the same project.
We want the inline login, which is displayed as a popup form over the application home page.
I had this issue again. Our custom theme does not have the script as indicated in Simon's reply. But I use different Mendix versions and projects with various themes, or no theme at all.
If any of the projects sets the cookie, the Mendix runtime will pick this up even when a given project has not set the cookie itself. Just a result of running many different projects at the same URL (localhost) I suppose.
So, if it happens not only clean the browser cache but cookies too.