Issue Details (XML | Word | Printable)

Key: NUCMONGODB-80
Type: Improvement Improvement
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Unassigned
Reporter: Chris Rued
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
DataNucleus Store MongoDB

Add native support for query range using skip/limit

Created: 14/Jun/12 08:39 PM   Updated: 25/Jun/12 02:44 PM   Resolved: 15/Jun/12 09:21 AM
Component/s: Query
Affects Version/s: None
Fix Version/s: 3.1.0.m4

File Attachments: 1. Text File vcs-diff2495926650834641550.patch (4 kB)



 Description  « Hide
Currently all objects are loaded into memory and then resulting list is trimmed. For small collections, this is not a problem, but for larger result sets this significantly slows things down and leads to OutOfMemoryErrors.

A simple fix is to change getObjectsOfCandidateType() to take a skip/limit paramter and to use these to set the skip and limit on the resulting cursor. A problem with this is that it will not return the correct results for a query with an ORDER BY clause. This can be addressed by modifying JPQLQuery.java and JDOQLQuery.java to not pass the skip and limit parameters when there is an ORDER BY clause.

Alternatively (or perhaps in the future) it may be possible to translate simple ORDER BY clauses to a DBObject that can then be passed to the sort() method of the DBCursor if the ordering were passed to the getObjectsOfCandidateType() method in MongoDBUtils.java. Of course, this can't be used if the sort requires a join, so it would have to fall back to the current in-memory handling for those cases I would think.

Patch to follow...

Chris Rued made changes - 14/Jun/12 09:08 PM
Field Original Value New Value
Attachment vcs-diff2495926650834641550.patch [ 11664 ]
Chris Rued added a comment - 14/Jun/12 09:08 PM
Patch to implement skip/limit in MONGODB when there is no ordering specified

Andy Jefferson added a comment - 15/Jun/12 09:01 AM
A side comment to the issue you mention ... and the real solution is to enable use of MongoDBCandidateList, passing the DBCursor to that and it lazily loading any results, which is why it is there, just not enabled.

Andy Jefferson made changes - 15/Jun/12 09:21 AM
Summary Add native support for firstResult/maxResults to MongoDB queries using skip/limit Add native support for query range using skip/limit
Fix Version/s 3.1.0.m4 [ 11564 ]
Affects Version/s 3.1.0.m3 [ 11542 ]
Component/s Query [ 10290 ]
Andy Jefferson added a comment - 15/Jun/12 09:21 AM
SVN trunk has this patch (after formatting), and TODO for the real work in NUCMONGODB-81. Thx

Andy Jefferson made changes - 15/Jun/12 09:21 AM
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Andy Jefferson made changes - 18/Jun/12 10:05 AM
Status Resolved [ 5 ] Closed [ 6 ]
Andy Jefferson added a comment - 25/Jun/12 02:44 PM
Obviously the implementation of this is flawed where there are multiple candidate types (i.e subclasses which aren't stored in the same MongoDB document).