Java process runs out of memory

There can be number of Reasons why your Java process runs out of memory, it could be just a matter of increasing -Xmx value, if it's too small. You can first analyze GC (keep -verbose:gc on command line) activity to determine your next action.

If you see that Java Heap increases over time and does not go down during Full GC / OC calls , then you most likely have a Memory Leak.

Heap Dump analysis is needed to figure out what is going on in your situation. Here are some Possible Memory Leak Scenarios.

HTTP Session
If you are dealing with Out of memory in your Web Application, it might be related to HTTP Sessions.

• Too many HTTP Sessions in your JVM. 
• Increase members in your Cluster to distribute users across more JVMs. 
• If you are using JSF, you may want to modify web.xml for "org.apache.myfaces.trinidad.CLIENT_STATE_MAX_TOKENS". Default for this value is 15, it is possible to set this value to lower value for example 5.
• Large Objects in HTTP Session
• Analyze what you have in HTTP Session and optimize your application to avoid having larger objects in HTTP Session. (There is overhead of Serializing HTTP Session in Clustered environment, so it is best to keep HTTP Session to a smaller size)

Object Cache
If you have a Cache (Java Map , Oracle Coherence near cache etc) in your process, Check for all your Key objects and it's usage.

• If your Key object does not implement hashCode() and equals() methods properly, it would lead to memory leak in cache implementation.
• Alway implement hashCode() and equals() methods properly.
• If your Key object has proper implementation of hashCode() and equals() method, but you reuse Key object for multiple Put calls. For example, you create Key object then put some object in cache, then modify same Key object and use it to put a different ojbect in same or different Cache.
• Create immutable Key objects (no setter methods on Key object) if possible. Or if this is not possible, take extra care while coding Cache calls to use a new / clone key objects.

Java process takes 100% of CPU

I see two main culprits, if your process takes 100% of CPU. You can see this by using "top" command on linux hosts. Sometimes you may see 200% if there are your process runs on 2 CPU machine.

First check GC logs. You must have "-verbose:gc" to get such output. If garbage collection is happening frequently and is running for long time, you most likely are dealing with low memory on this host. You process memory most likely is in swap and is taking long time. You can validate this by running "sar" command. Look at "%iowait" column, which would appear to be higher than normal. You can also run "free" command to see how much swap space is being utilized. Ideally, you want zero swap utilization for java processes.

Now if GC is not issue, you have a bad loop in code, either your code or vendor code. You can take thread dumps at regular interval to validate this. Generally "kill -3 pid" works to get thread dumps. But it may be different for you JDK. You can take 5 thread dumps at 10 seconds interval for example and then take few more after few minutes. Then you have to hunt for stacks that repeat in various dumps. Sometimes you may have infinite loop and it may appear in all the dumps, which would be easier to find. I have also seen HashMap methods run into infinite loop, if HashMap is used by multiple threads without any synchronization. There are some Thread dump analyzers for some JDK, which can also help.

There is definitely a chance that if your host is not big enough to handle work load , it will run very hot.