Inserting a persistent object containing a large byte array to MongoDB fails due to excessive document size. Even a simple object with an array of two megabytes causes Mongo driver to reject the insertion as it thinks the database object made by DataNucleus exceeds the BSON size limit:
>>> com.mongodb.MongoInternalException: DBObject of size 26151951 is over Max BSON size 16777216...
Just wondering if it would be correct to assume that the memory consumption of a database object should never be overly bigger than a byte stream representation of it? Considering however, that there's a reasonable max size limit in BSON, now the consumption just seems to way too high as we could easily run out of memory before we are even close to having 16 MBs of data.
This problem seems to have been first emerged in the DataNucleus Access Platform 3.1.0.release as the earlier milestone versions did not suffer from this issue. It looks like some changes might have been introduced into the array persistence scheme with that particular release. The earlier versions used to store the bytes in case of MongoDB with the BSON BinData type whereas the later versions seem to have resorted to storing the arrays with a plain value sequence. Is that correct?
Changing the Mongo driver version does not have any effect on the issue nor does this seem to be related to RDBMS persistence since persisting objects with a byte array of size up to 20 MBs caused no problems.
Our test configuration was as follows:
- DataNucleus Access Platform 3.1.0_m5, 3.3.8 (and 4.0.0_m2)
- MongoDB win32-i386-2.6.0 (and 2.4.5 on CentOS 6.4)
- Mongo Java driver 2.12.0 (and 2.9.3 where applicable)
- (H2 database 1.4.177)