I don't know if it affects the time to build and deploy, but it does increase response times within Studio / Studio Pro when having larger amounts of documents (microflows / pages etc).
It won't make the project unstable, but in my opinion it's a bad practice to keep all your document excluded in the project. You should wonder why you are saving al these documents, because if you do it to preserve them for future use: this is covered by GIT / SVN versioning.
Taking a look at this answer, excluded items are not part of a deployment package, so they wouldn’t negatively affect your deployment time and stability.
Unused items are part of the package. So having a lot of unused items could affect deployment speed. I don’t think unused items make a project unstable.
Hope this helps!