Remove and import from app store, might a good option if your jars are only used by the apps store modules. It could be that some of the custom Java Actions might depend on some others..
A good way to do this, without re-starting the project every time.
About dependency management
Applied in community commons modules, see https://github.com/mendix/CommunityCommons/
It also helps to check for duplicated names or higher versions, remove them and rebuild and check for errors. Also make sure you have a backup.
If you did not add any jarfiles manually, you can do a cleanup bij deleting everything from userlib and then do a fresh import from the appstore for all appstore components.
If you are not sure, you could also make a backup of the original userlib folder, do the cleanup and then compare the userlib with the backup.
Do note that reimporting all modules could still result in double jar files if that jar is used by multiple modules with different versions. So afterwards you still need to manually check.
A bit antiquated, but implementing something similar to loosejar with a runtime interface would be helpful so your mendix website can generate a list of which jar files are unused. Then you can make a java action that diffs the $PROJROOT/userlib/*.jar with the list to produce a difference, then backs that up to say $PROJROOT/userlibbak and removes them from $PROJROOT/userlib. So if any brainbox can come up with that I’d say that would be a useful module.