Replace the favicon.ico in the folder <projectdir>/theme with your own custom favicon.ico
Mind you, caching might be playing tricks on you here. Press Ctrl-R in your browser to make sure it refreshes your favicon.
If you use the new Mendix 8.13 you can change your favicon directly in the Navigation Settings of your project, no need to change the layouts anymore.
Team lead | UI Content