Issue Details (XML | Word | Printable)

Key: NUCRDBMS-529
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Unassigned
Reporter: Klaus Malorny
Votes: 0
Watchers: 1
Operations

If you were logged in you would be able to see more operations.
DataNucleus Store RDBMS

Access to unsynchronized hash map causes lockup

Created: 23/May/11 09:12 AM   Updated: 18/Jul/11 08:18 PM   Resolved: 26/May/11 09:36 AM
Component/s: None
Affects Version/s: 2.1.3
Fix Version/s: 3.0.0.m5

Datastore: PostgreSQL


 Description  « Hide
We discovered that multiple threads are looping in the HashMap.put method:

   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.put(HashMap.java:374)
        at org.datanucleus.store.mapped.MappedStoreManager.setObjectIsInsertedToLevel(MappedStoreManager.java:608)
        at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:435)
        at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:154)
        at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:129)
        at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3273)
        at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249)
        at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1454)
        at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:382)
        at org.datanucleus.store.types.sco.SCOUtils.validateObjectForWriting(SCOUtils.java:1433)
        at org.datanucleus.store.mapped.scostore.ElementContainerStore.validateElementForWriting(ElementContainerStore.java:380)
        at org.datanucleus.store.mapped.scostore.FKSetStore.add(FKSetStore.java:309)
        at org.datanucleus.store.types.sco.backed.Set.add(Set.java:559)
        - locked <0x00007f1ce47041c8> (a org.datanucleus.store.types.sco.backed.Set)

Various sources in the Internet, for example[1], report that a parallel write access to an unsynchronized hash map may cause the internal linked list to degenerate to a cyclic list, which is not expected by the hash map implementation and thus resulting in an infinite loop. Please take the proper means to avoid this situation. Thanks.

[1] http://mailinator.blogspot.com/2009/06/beautiful-race-condition.html

Sort Order: Ascending order - Click to sort in descending order
Andy Jefferson added a comment - 23/May/11 08:48 PM
Suggest use 3.0 since that is the only place getting changes for non-commercial users

Klaus Malorny added a comment - 26/May/11 09:17 AM
That's a no-go. We would have upgraded to 3.0 if there wouldn't be a showstopper, which I now have reported as NUCCORE-714.

Andy Jefferson added a comment - 26/May/11 09:36 AM
Whether it's a no-go to you or not, its the only "solution" for the reason mentioned above.
SVN trunk now uses ConcurrentHashMap in MappedStoreManager; no test so left as an exercise to the reporter

Klaus Malorny added a comment - 26/May/11 09:45 AM
Thanks, we will test the fix on our benchmark system and see whether the problem reoccurs.

Clive Cox added a comment - 15/Jul/11 06:07 PM
The issue still exists.

protected Map<ObjectProvider, DatastoreClass> insertedDatastoreClassByStateManager = new HashMap();

I think this needs to be a concurrent hashmap as well.

Testing with latest DN 3.0.0.m6 build

I am seeing threads stuck like this:

  java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:374)
at org.datanucleus.store.mapped.MappedStoreManager.setObjectIsInsertedToLevel(MappedStoreManager.java:608)
at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:436)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:154)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:129)
at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3273)
at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249)

Andy Jefferson added a comment - 15/Jul/11 06:23 PM
If you think that then you can easily try it, and report your findings. Since nobody "here" can see it, it's in your hands ...

Clive Cox added a comment - 15/Jul/11 06:43 PM
OK. I can try to get things built from source and I will report back when/if I am able but it would be more efficient as I don't have a DN dev environment setup for this fix to be made if possible by a DN committer now for the reasons given below.

1. Given a previous attempt was made to fix this issue by making one of the maps concurrent and the issue marked as "fixed" that as the problem is still occurring the same "fix" is extended to the actual map that should have been made concurrent in the first place.
2. Code inspection shows there will be an issue in multithreaded access to the Map in this object.
3. Our tests see this happening under high load (500 concurrent calls per sec to our test server) with the latest 3.0.0.m6



Clive Cox added a comment - 15/Jul/11 09:01 PM
I've managed to setup a dev environ in Eclipse and make the change. Can someone point me towards how to create a jar as simply creating one naively seems to cause problems when I use it:

Caused by: org.datanucleus.exceptions.NucleusUserException: Persistence process has been specified to use a ClassLoaderResolver of name "" yet this has not been found by the DataNucleus plugin mechanism. Please check your CLASSPATH and plugin specification.
        at org.datanucleus.NucleusContext.<init>(NucleusContext.java:227)
        at org.datanucleus.NucleusContext.<init>(NucleusContext.java:166)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:333)

So I assume the datanucleus-core.jar needs to be created in a specific way with some extra stuff?

Clive Cox added a comment - 18/Jul/11 12:19 PM
OK. I have tested by making the simple change to:

protected Map<ObjectProvider, DatastoreClass> insertedDatastoreClassByStateManager = new ConcurrentHashMap();

in org.datanucleus.store.mapped.MappedStoreManager of the core project

The problem no longer occurs in testing.

Can this fix be added to DN please.


Andy Jefferson added a comment - 18/Jul/11 08:18 PM
SVN trunk has that line. thx