Issue Details (XML | Word | Printable)

Key: NUCENHANCER-34
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Andy Jefferson
Reporter: Artem
Votes: 0
Watchers: 1
Operations

If you were logged in you would be able to see more operations.
DataNucleus Enhancer

Enchanced Scala class fails Java verification.

Created: 09/Apr/09 08:16 PM   Updated: 16/May/09 09:44 AM   Resolved: 13/Apr/09 02:47 PM
Component/s: None
Affects Version/s: 1.1.1
Fix Version/s: 1.1.2

File Attachments: 1. File Persistent.class (9 kB)
2. File Persistent.class (1 kB)

Environment: http://www.freshports.org/java/openjdk6/


 Description  « Hide
Here is a simple Scala class:

      @PersistenceCapable
      class Persistent {var name: String = null; var value: String = null}

which gives the following error after being enchanced:

java.lang.VerifyError: (class: fastcms/scalaAPI/Persistent, method: <init> signature: ()V) Expecting to find object/array on stack

here it is before and after enchancement: http://gist.github.com/92677

Sort Order: Ascending order - Click to sort in descending order
Artem added a comment - 09/Apr/09 08:17 PM
original version of the class

Artem added a comment - 09/Apr/09 08:17 PM
enchanced version of the class

Andy Jefferson added a comment - 10/Apr/09 07:18 AM
Obviously Scala is not supported since DataNucleus is for Java itself. The issue is likely related to Scala wanting to have variables initialised at construction, so can you not just add a default constructor (or whatever that equates to in Scala) so then the DataNucleus enhancer won't try to add one for you (and not know what values to assign to its variables)

Andy Jefferson added a comment - 10/Apr/09 08:14 AM
Also see http://www.jpox.org/servlet/jira/browse/ENHANCER-96 which was deemed impossible to implement due to no known way of separating static initialisation from constructor code etc. Obviously if you know of a way then you could contribute the code/ideas to do it

Artem added a comment - 10/Apr/09 09:34 AM
> Obviously Scala is not supported since DataNucleus is for Java itself.

DataNucleus is for Java bytecode, and Scala produces a legitimate one.
GAE for Java claims support for different languages, including Scala, and DataNucleus is their way of providing BigTable persistance. That makes this bug a real road-blocker.
I think DataNucleus should try to support any legitimate Java bytecodes, not just ones produced by the usual Java compilers.

> The issue is likely related to Scala wanting to have variables initialised at construction,
> so can you not just add a default constructor (or whatever that equates to in Scala)
> so then the DataNucleus enhancer won't try to add one for you
> (and not know what values to assign to its variables)

There is already a default constructor:

public fastcms.scalaAPI.Persistent();
  Code:
   0: aload_0
   1: aconst_null
   2: pop
   3: aconst_null
   4: putfield #13; //Field name:Ljava/lang/String;
   7: aload_0
   8: aconst_null
   9: pop
   10: aconst_null
   11: putfield #15; //Field value:Ljava/lang/String;
   14: aload_0
   15: invokespecial #19; //Method java/lang/Object."<init>":()V
   18: return

The problem, as I see it, is that DataNucleus incorrectly enchances it.

Andy Jefferson added a comment - 10/Apr/09 09:45 AM
Yes I saw the default constructor later; i'm more used to Java class display of bytecode.

The only ways the issue can move forward are :-
1. you define the equivalent Java code that equates to this Scala code. Then I can actually try it here.
2. you debug the enhancement process and identify what is wrong with the generated bytecode and what you think it ought to be.
Option 1 is the preferred choice.

DataNucleus supports what its developers have time to do, and I personally have little interest in getting into DSLs right now, hence why I say option 1. We are not "GAE for Java" as such, so Googles' policy and ours aren't necessarily the same thing.

Erik Bengtson added a comment - 10/Apr/09 10:25 AM - edited
There is an aload_0 missing in the stack before the jdoSet calls


it's missing because there is a strange

   aconst_null
   pop
   aconst_null

which should normaly be only aconst_null

Erik Bengtson added a comment - 10/Apr/09 10:59 AM
The solution is that the enhancer should check for crap operand stack instructions "pop"and "pop2" on the second operand, and so add an aload_a after it.

aconst_null <- these 2 lines will be ignored
pop <-
aload_0
aconst_null

Note that compiler (jdk or scala0 is probably generating this garbage, which still does not invalid the code.

Artem added a comment - 10/Apr/09 01:36 PM
Erik, forgive my ignorance, but it seems that "aconst_null; pop; aconst_null" is equal to "aconst_null".
I have tried your suggestion with Java Bytecode Editor, and it doesn't work.

It seems that the object is not "constructed enough" to be used with static methods until the "invokespecial java/lang/Object/<init>()V". Moving invokestatic after invokespecial works:

aload_0
invokespecial java/lang/Object/<init>()V
aload_0
aconst_null
pop
aconst_null
invokestatic fastcms/scalaAPI/Persistent/jdoSetvalue(Lfastcms/scalaAPI/Persistent;Ljava/lang/String;)V
return

Andy Jefferson added a comment - 10/Apr/09 01:42 PM
Why not ask the question, why isn't the superclass constructor called *first* ?
Standard java compilers arrange the bytecode like this.
The only thing the enhancer does to that constructor is replace field access with method call.

Andy Jefferson added a comment - 10/Apr/09 02:08 PM
JVM spec 2.12

"If a constructor body does not begin with an explicit constructor invocation and the constructor being declared is not part of the primordial class Object, then the constructor body is implicitly assumed by the compiler to begin with a superclass constructor invocation "super();", an invocation of the constructor of the direct superclass that takes no arguments."

Andy Jefferson added a comment - 10/Apr/09 04:29 PM
I'm marking as "Wont fix" since IMHO the input class is wrong. If you can convince me otherwise then it can be reopened ;-)

You don't say which version of Scala you used to create this. Scala bug register has various VerifyError issues. Is this https://lampsvn.epfl.ch/trac/scala/ticket/473 ? (marked as fixed in "Unreleased-2.8.x" whatever that is).

Artem added a comment - 10/Apr/09 04:44 PM
> You don't say which version of Scala you used to create this.

2.7.3-final.

> Scala bug register has various VerifyError issues. Is this https://lampsvn.epfl.ch/trac/scala/ticket/473 ?

No, the fix for this bug
https://lampsvn.epfl.ch/trac/scala/changeset/17118
does not change the order of field initialization in the contructor.

Andy Jefferson added a comment - 10/Apr/09 05:01 PM
I'll reopen this and leave it as one for the future. See
http://www.nabble.com/Object-initialization-order-in-Scala-td19104940.html

low priority for me since our prime audience is Java, but then its open source so you could look at a fix yourself. Go to "datanucleus-enhancer" src/java/org/datanucleus/enhancer/asm/JdoMethodAdapter.java which is like a SAXParser on the method. So if the method is the default constructor you'd need to make sure that superclass constructor is called first.

Artem added a comment - 10/Apr/09 05:05 PM
Thanks for digging into this!

Artem added a comment - 13/Apr/09 10:01 AM
For the time being, the following workaround works:
      @PersistenceCapable
      class Persistent {def NULL = null; var name: String = NULL; var value: String = NULL}
- here we initialize the fields with the result of a function instead of a constant value.

Andy Jefferson added a comment - 13/Apr/09 10:16 AM
Thx for the info. Something else you could try is in JdoMethodAdapter check if the the method is "<init>" and if so omit the enhancement of field access. My immediate reaction is that this should be perfectly valid.

Artem added a comment - 13/Apr/09 10:19 AM
Thanks, i'll give it a go soon.

Andy Jefferson added a comment - 13/Apr/09 11:01 AM
In fact code as follows is probably correct :-

            boolean fieldInThisClass = true;
            if (enhancer.getClassMetaData().getFullClassName().equals(ownerName))
            {
                // Same class so use the input MetaData
                cmd = enhancer.getClassMetaData();
            }
            else
            {
                fieldInThisClass = false;
                cmd = enhancer.getClassMetaData().getMetaDataManager().getMetaDataForClass(
                    ownerName, enhancer.getClassLoaderResolver());
            }

            // If the field access is in this class and this is the constructor then don't enhance it.
            // This is because this object is not connected to a StateManager nor is it detached.
            // Also languages like Scala don't necessarily initialise superclasses first and so
            // enhancing here would cause issues
            if (!fieldInThisClass || !(methodName.equals("<init>")))
            {
               ... (enhance the field access)
            }

So we omit the enhancement of field access *in this class* in constructors. We still enhance access of fields in other classes within the constructor. Works for Java cases anyway.

Artem added a comment - 13/Apr/09 02:40 PM
I have tried the following patch: http://gist.github.com/94447
It works! The JDO test passes with
      @PersistenceCapable
      class Persistent {var name: String = null; var value: String = null}

Andy Jefferson added a comment - 13/Apr/09 02:47 PM
Marking as fixed; I included just like my code 2 posts back in SVN a while ago (same as yours except I check the constructor bit first in a separate "if") - and the "nightly build" available on the DataNucleus site has this fix.
1.1.2 will be released in the next few days. Thx for your inputs