As Jouke said, this is a memory leak. For what this error means exactly, see the first reply of this thread.
What you should do is make a heap dump and inspect it to see what objects are clogging your memory. Then, once you know which objects are causing the problem, you should inspect your model and make sure these objects are cleaned up properly.
To make a heap dump, you can use jmap, see here how to.
To analyse the heap dump, you can use Eclipse, in combination with Memory Analyser
This sounds like you have a memory leak in your application, in which case only rewriting your project will be a solution. Normally the memory should be available again after a garbage collection cycle.