For debugging java actions, you must start the application from Eclipse in debugging mode and you can set the debug points in your java actions.
Go to the Project tab in Mendix Modeller and click on Deploy for Eclipse. If it is error free the project will be deployed successfully. Then open Eclipse and on the Project Explorer tab on the right, right click and import project. Here select existing project and navigate to your Mendix project to select it. Once this is done you can add custom java actions.
Use this tutorial for more detail on how to do that:
If you get an error saying java compilation failed in mendix, just go to project and click clean deployment directory.
Hope this helps :)