JPA : Troubleshooting

This section describes the most common problems found when using DataNucleus in different architectures. It describes symptoms and methods for collecting data for troubleshooting thus reducing time to narrow the problem down and come to a solution.

Out Of Memory error

Introdution

Java allocate objects in the runtime memory data area called heap. The heap is created on virtual machine start-up. The memory allocated to objects are reclaimed by Garbage Collectors when the object is no longer referenced (See Object References). The heap may be of a fixed size, but can also be expanded when more memory is needed or contracted when no longer needed. If a larger heap is needed and it cannot be allocated an OutOfMemory is thrown. See JVM Specification.

Native memory is used by the JVM to perform its operations like creation of threads, sockets, jdbc drivers using native code, libraries using native code, etc.

The maximum size of heap memory is determined by the -Xmx on the java command line. If Xmx is not set, then the JVM decides for the maximum heap. The heap and native memory are limited to the maximum memory allocated by the JVM. For example, if the JVM Xmx is set to 1GB and currently use of native memory is 256MB then the heap can only use 768MB.

Causes

Common causes of out of memory:

  • Not enough heap - The JVM needs more memory to deal with the application requirements. Queries returning more objects than usual can be the cause.
  • Not enough PermGen - The JVM needs more memory to load class definitions.
  • Memory Leaks - The application does not close the resources, like the EntityManager or Queries, and the JVM cannot reclaim the memory.
  • Caching - Caching in the application or inside DataNucleus holding strong references to objects.
  • Garbage Collection - If no full garbage collection is performed before the OutOfMemory it can indicate a bug in the JVM Garbage Collector.
  • Memory Fragmentation - A large object needs to be placed in the memory, but the JVM cannot allocate a continous space to it because the memory is fragmented.
  • JDBC driver - a bug in the JDBC driver not flushing resources or keeping large result sets in memory.

Throubleshooting

JVM

Collect garbage collection information by adding -verbosegc to the java command line. The verbosegc flag will print garbage collections to System output.

Sun JVM

The Sun JVM 1.4 or upper accepts the flag -XX:+PrintGCDetails, which prints detailed information on Garbage Collections. The Sun JVM accepts the flag -verbose:class, which prints information about each class loaded. This is useful to troubleshoot issues when OutOfMemory occurs due to lack of space in the PermGen, or when NoClassDefFoundError or Linkage errors occurs. The Sun JVM 1.5 or upper accepts the flag -XX:+HeapDumpOnOutOfMemoryError, which creates a hprof binary file head dump in case of an OutOfMemoryError. You can analyse the heap dump using tools such as jhat or YourKit profiler.

DataNucleus

DataNucleus keeps in cache persistent objects using weak references by default. Enable debug mode DataNucleus.Cache category to investigate the size of the cache in DataNucleus.


Resolution

DataNucleus can be configured to reduce the number of objects in cache. DataNucleus has cache for persistent objects, metadata, datastore metadata, fields of type Collection or Map, or query results.

Query Results Cache

The query results hold strong references to the retrieved objects. If a query returns too many objects it can lead to OutOfMemory error. To be able to query over large result sets, change the result set type to scroll-insensitive using the persistence property datanucleus.rdbms.query.resultSetType.

EntityManager leak

It's also a best practice to ensure the EntityManager is closed in a try finally block. The EntityManager has level 1 cache of persistence objects. See the following example:

EntityManager em = emf.createEntityManager();
EntityTransaction tx = em.getTransaction();
try
{
    tx.begin();
    //...
    tx.commit();
}
finally
{
    if (tx.isActive())
    {
        tx.rollback();
    }
    em.close();
}

Cache for fields of Collection or Map

If collection or map fields have large number of elements, the caching of elements can be disabled with the property datanucleus.cache.collections setting it to false.

Persistent Objects cache

The cache control of persistent objects is described in the Cache Guide

Metadata and Datastore Metadata cache

The metadata and datastore metadata caching cannot be controled by the application, because the memory required for it is insignificant.

OutOfMemory when persisting new objects

When persistent many objects, the flush operation should be periodically invoked. This will give a hint to DataNucleus to flush the changes to the database and release the memory. In the below sample the em.flush() operation is invoked on every 10,000 objects persisted.

EntityManager em = emf.createEntityManager();
EntityTransaction tx = em.getTransaction();
try
{
    tx.begin();
    for (int i=0; i<100000; i++)
    {
        Wardrobe wardrobe = new Wardrobe();
        wardrobe.setModel("3 doors");
        pm.makePersistent(wardrobe);
        if (i % 10000 == 0)
        {
            em.flush();
        }
    }
    tx.commit();
}
finally
{
    if (tx.isActive())
    {
        tx.rollback();
    }
    em.close();
}

Frozen application

Introdution

The application pauses for short or long periods or hangs during very long time.

Causes

Common causes:

  • Database Locking - Database waiting other transactions to release locks due to deadlock or locking contentions.
  • Garbage Collection Pauses - The garbage collection pauses the application to free memory resources.
  • Application Locking - Thread 2 waiting for resources locked by Thread 1.

Throubleshooting

Database locking

Use a database specific tool or database scripts to find the current database locks. In Microsoft SQL, the stored procedured sp_lock can be used to examinate the database locks.

Query Timeout

To avoid database locking to hang the application when a query is performed, set the query timeout. See Query Timeout.

Garbage Collection pauses

Check if the application freezes when the garbage collection starts. Add -verbosegc to the java command line and restart the application.

Application Locking

Thread dumps are snapshots of the threads and monitors in the JVM. Thread dumps help to diagnose applications by showing what the application is doing at a certain moment of time. To generate Thread Dumps in MS Windows, press <ctrl><break> in the window running the java application. To generate Thread Dumps in Linux/Unix, execute kill -3 process_id

To effectively diagnose a problem, take 5 Thread Dumps with 3 to 5 seconds internal between each one. See An Introduction to Java Stack Traces.


Postgres

ERROR: schema does not exist

Problem

Exception org.postgresql.util.PSQLException: ERROR: schema "PUBLIC" does not exist raised during transaction.

Troubleshooting

  • Verify that the schema "PUBLIC" exists. If the name is lowercased ("public"), set datanucleus.identifier.case=PreserveCase, since Postgres is case sensitive.
  • Via pgAdmin Postgres tool, open a connection to the schema and verify it is acessible with issuing a SELECT 1 statement.

Command Line Tools

CreateProcess error=87

Problem

CreateProcess error=87 when running DataNucleus tools under Microsoft Windows OS.

Windows has an (antiquated) command line length limitation, between 8K and 64K characters depending on the Windows version, that may be triggered when running tools such as the Enhancer or the SchemaTool with too many arguments.

Solution

When running such tools from Maven or Ant, disable the fork mechanism by setting the option fork="false".