lxx.Tomcat 3.55 vs cs.ags.Scarlet 1.1c
Are you using fasttrig? If so, how big are your arrays?
Are you using multi-dimensional arrays? It is very easy to use LOTS of memory. I have many multi-dimensional arrays, but use a lazy initialisation so that memory is only used if it gets written to first.
If you have your own kd-tree, check that it is splitting how you want it to. If it only splits one point off each time, it will get deep very quickly. This uses lots of memory, as each object needs a container and a bounding box.
Check if you have static arrays/lists that could initialised after each round.
Don't keep logs of data that won't get used again. Once it won't be used (eg at the end of the round) clear it explicitly by setting to null or calling .clear()
Check you don't have circular dependencies. Make sure you break the dependency if you delete something.
These are my first thoughts =)