For future reference, and since this might be interesting to other readers:
We use the Eclipse Memory Analyzer (MAT) to look at heap dump files (hprof format). These can be generated automatically using JVM command line options (-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="C:\path\to\file.hprof") or manually using monitoring tools such as JConsole or JVisualVM.
MAT has some interesting automated analysis reports, the most useful of which is "leak suspects", which will look at which threads or objects keep the most memory active. In this (relatively simple) case, there was a single thread that kept 90% of memory (>3GB) in use.
MAT also allows you to look at the current state (stack) of all threads at the time of the dump. Looking at the reported thread showed that it was executing a custom java action, which did an XPath retrieve, which ultimately hit the database. By inspecting the local variables at each stack frame, we noticed that near the top of the stack a SQL resultset was being built which used all the memory.
We sent the action and XPath/SQL query that caused this to Brian, which allowed him to identify the cause of the issue.