JVMs and kill signals

Ever wondered how a JVM reacts to various kill signals? The (intended) behaviour might be documented somewhere already, but I found having a table of the actual behaviour available quite useful. In particular I wanted to know which kill signals trigger the JVM to run registered shutdown hooks, and which kill signals don't actually terminate the JVM. So I decided to compile a table of that information.

I wrote up a small Java application that just registers a shutdown hook that I can detect whether it has executed or not, and then sleeps until I get a chance to kill it:

class Death {
  public static void main(String... args) throws Exception {
    Runtime.getRuntime().addShutdownHook( new Thread(){
      @Override
      public void run()
      {
        System.out.println("Shutting down");
      }
    } );
    for (;;) Thread.sleep(100);
  }
}

Then I ran the program in one terminal window (java Death; echo $?) while iterating through all kill signals (0-31) in another:

kill -$SIGNAL $(jps | grep Death | cut -d\  -f1)

signal	shutdown	runs hook	exit code	comment
default (15)	yes	yes	143	SIGTERM is the default unix kill signal
0	no	-	-
1 (SIGHUP)	yes	yes	129
2 (SIGINT)	yes	yes	130	SIGINT is the signal sent on ^C
3 (SIGQUIT)	no	-	-	Makes the JVM dump threads / stack-traces
4 (SIGILL)	yes	no	134	Makes the JVM write a core dump and abort on trap 6
5	yes	no	133	Makes the JVM exit with "Trace/BPT trap: 5"
6 (SIGABRT)	yes	no	134	Makes the JVM exit with "Abort trap: 6"
7	yes	no	135	Makes the JVM exit with "EMT trap: 7"
8 (SIGFPE)	yes	no	134	Makes the JVM write a core dump and abort on trap 6
9 (SIGKILL)	yes	no	137	The JVM is forcibly killed (exits with "Killed: 9")
10 (SIGBUS)	yes	no	134	Emulates a "Bus Error"
11 (SIGSEGV)	yes	no	134	Emulates a "Segmentation fault"
12	yes	no	140	Makes the JVM exit with "Bad system call: 12"
13	no	-	-
14	yes	no	142	Makes the JVM exit with "Alarm clock: 14"
15 (SIGTERM)	yes	yes	143	This is the default unix kill signal
16	no	-	-
17	no	-	145	Stops the application (sends it to the background), same as ^Z
18	no	-	146	Stops the application (sends it to the background), same as ^Z
19	no	-	-
20	no	-	-
21	no	-	149	Stops the application (sends it to the background), same as ^Z
22	no	-	150	Stops the application (sends it to the background), same as ^Z
23	no	-	-
24	yes	no	152	Makes the JVM exit with "Cputime limit exceeded: 24"
25	no	-	-
26	yes	no	154	Makes the JVM exit with "Virtual timer expired: 26"
27	yes	no	155	Makes the JVM exit with "Profiling timer expired: 27"
28	no	-	-
29	no	-	-
30	yes	no	158	Makes the JVM exit with "User defined signal 1: 30"
31	yes	no	134	Makes the JVM exit on Segmentation fault

This list was compiled using (a quite old) Oracle Hotspot Java 8 EA on Mac OS X:

java version "1.8.0-ea"
Java(TM) SE Runtime Environment (build 1.8.0-ea-b65)
Java HotSpot(TM) 64-Bit Server VM (build 25.0-b09, mixed mode)

Hope this is useful to more people than myself.

Java integration in future Jython

I've seen a lot of this lately, so I thought that it was time for an actual Jython developer (myself) to share some ideas on how Java integration in Jython could be improved. At the same time I'd like to propose some changes that could make the different Python implementations more unified, and even could lead to a common Java integration API in all of them.

The most basic part of the Java integration in Jython is the ability to import and use Java classes. This is impossible for other Python implementations to do in the same way, and thus breaks compatibility fundamentally. I therefore propose that we remove this functionality as it is in Jython today (!). Instead we should look at how IronPython enables using CLR (.NET) classes. In IronPython you first need to import clr before you can access any of the CLR types. The same is done in other languages on the JVM as well, for example JRuby where you need to require 'java' before using any Java libraries. I propose we require something similar in Jython, and what better package to require you to import than java?

An observation: The java package in Java does not contain any classes, only sub-packages. Furthermore all the sub-packages of the java package follow the Java naming conventions, i.e. They all start with a lowercase letter. This gives us a name space to play with: anything under the java package that starts with an uppercase letter.

What happens when you import java? The java Python module is a "magic module" that registers a Python import hook. This import hook will then enable you to import real Java packages and classes. In Jython many of the builtin libraries will of course import java, which means that this will be enabled by default in Jython. But writing code that is compatible across Python implementations would now be possible, by simply ensuring that you import java before any other Java packages.

The content of the java module

Most if not all of what is needed to utilize Java classes from Python code is provided by the import hook that the java module registers when it is loaded. This means that the content of the java module needs to deal with the other direction of the interfacing: defining and implementing APIs in Python that Java code can utilize. I propose that the Python java module contain the following:

JavaClass

A class decorator that exposes the decorated class as a class that can be accessed from Java. Accepts a package keyword argument for defining the Java package to define the class in, if omitted it is derived from the __module__ attribute of the class.
Possibly JavaClass should also be the Python type of imported Java classes.

Field

An object for defining Java fields in classes. Takes a single argument, the type of the field. Example usage:

@java.JavaClass
class WithAField:
    data = java.Field(java.lang.String)

Array

An object for defining Java arrays. This is used to define Java array types. Examples:

• Array[java.Primitive.int] corresponds to the Java type int[]
• Array[java.lang.String] corresponds to the Java type java.lang.String[]
• Array corresponds to the Java type java.lang.Object[]
• Array[Array[java.lang.String]] corresponds to the Java type java.lang.String[][]

Access

A set of Java access definition decorators. Contains:

• Access.public
• Access.package - this needs to be explicitly available since it does not make sense as the default in Python code.
• Access.protected
• Access.module - for the new access modifier in the upcoming module system (a.k.a. Project Jigsaw) for Java.
• Access.private
• The default access modifier should either be public or the absence of an access modifier decorator would mean that the method is not exposed in the Java class at all. This needs further discussion.

Primitive

The set of primitive types in Java:

• Primitive.void
• Primitive.boolean
• Primitive.byte
• Primitive.char
• Primitive.short
• Primitive.int
• Primitive.long
• Primitive.float
• Primitive.double
• These can be used as type parameters for Array but not for Generic types (Since primitives are not allowed as generic type parameters in Java).

Overload

Used to implement (and define) overloaded methods, several different methods with the same name, but different type signatures. Example usage:

@java.JavaClass
class WithOverloadedMethod:
    @java.Access.public
    def method(self, value:java.lang.String) -> java.util.List[java.lang.String]:
        ...
    @java.Overload(method)
    @java.Access.public
    def method(self, value:java.lang.Integer) -> java.lang.String:
        ...
    @java.Overload(method)
    @java.Access.public
    def method(self, value:java.lang.Iterable[java.lang.String]) -> java.Primitive.void:
        ...

Java classes and interfaces, when imported, are Pythonized in such a way that they can be used as bases for Python classes. Generics are specified by subscripting the generic Java class. Java annotations are Pythonized in a way that turns them into decorators that add a special attribute to the decorated element: __java_annotations__. Annotations on imported Java classes and methods would also be exposed through the __java_annotations__ property for consistency. Access modifiers would similarly add a __java_access__ property to the object they decorate.

Kay Schluer also suggested allowing decorators on assignments, to be able to support annotations on fields. I don't really have an opinion on this. Since I don't think fields should be exported in any public API anyway it's a bit useless, and for the the cases where fields are used (such as dependency injection systems) I think it suffices to have it all in the same assignment: dependency = javax.inject.Inject(java.Access.private(java.Field(JavaClassIDependOn))), the name will be extracted to be "dependency" when the class is processed by the JavaClass class decorator. But if others find assignment decorators useful, I am not opposed to them. If assignment decorators are added to Python, it might be worth considering having a slightly different signature for these decorator function, so that the name of the target variable is passed as a parameter as well. Then my example could look like this:

@java.JavaClass
class WithInjectedDependency:
    @javax.inject.Inject # This is JSR 330 by the way
    @java.Access.private
    @java.Field
    dependency = JavaClassIDependOn
    # could expand to: dependency = javax.inject.Inject(
    #     "dependency", java.Access.private(
    #         "dependency", java.Field(
    #             "dependency", JavaClassIDependOn)))
    # or to the same thing as above, depending on how
    # assignment decorators were implemented...

When defining methods in Java integration classes we use Python 3 function annotations to define the method signatures. These can be omitted, the default types in that case would of course be java.lang.Object. It is important that we support exposing classes that don't have any Java integration added to them from Jython, since we want to enable importing existing Python libraries into Java projects and use them without having to port them. These classes will not have the JavaClass decorator applied to them. Instead this will be done automatically by Jython at the point when the Python class first need to expose a class to Java. This is not something that the java module need to deal with, since it doesn't fit with other Python implementations.

Outstanding issues

There are still a few Java integration issues that I have not dealt with, because I have not found a solution that I feel good about yet.

Defining Java interfaces

Is this something we need to be able to do? If so, the proper approach is probably to add a JavaInterface decorator to the java module, similar to the JavaClass decorator.

Defining Java enums

This might be something that we want to support. I can think of two options for how to declare the class. Either we add a JavaEnum decorator to the java module, or we add special case treatment for when a class extends java.lang.Enum (I am leaning towards this approach). Then we need to have some way to define the enum instances. Perhaps something like this:

@java.JavaClass
class MyEnum(java.lang.Enum):
    ONE = java.EnumInstance(1)
    TWO = java.EnumInstance(2, True)
    THREE = java.EnumInstance(3, True)
    FOUR = java.EnumInstance(4)
    def __init__(self, number, is_prime=False):
        self.number = number
        self.is_prime = is_prime
    def __str__(self):
        return self.name()
    class SEVENTEEN(java.EnumInstance):
        """This is an enum instance with specialized behavior.
        Will extend MyEnum, but there will only be one instance."""
        def __init__(self):
            """This class gets automatically instantiated
            by the __metaclass__ of Enum."""
            self.number = 17
            self.is_prime = True
        def __str__(self):
            return "The most random number there is."

Defining generic types

I have discussed how to specify type parameters for generic types, but how would you define a generic Java type in Python? How about something like this:

@java.JavaClass
class GenericClass:
    T = java.TypeParameter() # default is "extends=java.lang.Object"
    C = java.TypeParameter(extends=java.util.concurrent.Callable)

This gets complicated when wanting to support self references in the type parameters, but the same is true for implemented interfaces, such as:

class Something implements Comparable<? extends Something> {
    ...
}

Defining Java annotations

I have dealt with supporting the use of Java annotations, but what about defining them? I highly doubt that defining Java annotations in Python is going to be useful, but I prefer to not underestimate what developers might want to do. I do however think we could get far without the ability to define Java annotations in Python, but if we were to support it, what would it look like? Defining the class would probably be a lot like how enums are defined, either by special casing java.lang.annotation.Annotation or providing a special java.Annotation decorator.

@java.JavaInterface
class MyAnnotation(java.lang.annotation.Annotation):
    name = java.AnnotationParameter(java.lang.String)
    description = java.AnnotationParameter(java.lang.String, default="")

java for other Python implementations

I mentioned that requiring the user to explicitly import java to make use of Java classes would make it possible for other Python implementations to support the same Java integration API. So what would the default implementation of the java module look like? There is a very nice standardized API for integrating with Java from other external programming languages: JNI. The default java module would simply implement the same functionality as the Jython counterpart by interacting with JNI using ctypes. Since ctypes is supported by all Python implementations (Jython support is under development) the java integration module would work across all Python implementations without additional effort. Right there is a major advantage over JPype and JCC (the two major Java integration modules for CPython today).

Integration from the Java perspective

I have not given as much thought to the area of utilizing Python code from Java. Still this is one of the most important tasks for Jython to fulfill. This section is therefore just going to be some ideas of what I want to be able to do.

Use Python for application scripting

This is possible today, and a quite simple case, but I still think that it can be improved. Specifically the problem with Jython today is that there is no good API for doing so. Or to be frank, there is hardly an API at all. This is being improved upon though, the next update of Jython will include an updated implementation of the Java Scripting API, and the next release will introduce a first draft of a proper Jython API, something that we will support long term after a few iterations, and that you can build your applications against.

Use Jython to implement parts of your application

We want to be able to write an polyglot applications, where parts of it is implemented in Python. This is more than just scripting the application. Applications generally work without scripts. We want to be able to write the implementation of parts of an application in Python with Jython. This is possible today, but a bit awkward without an official Jython API. This is being worked on in a separate project called PlyJy, where we are experimenting with an API for creating object factories for Jython. Jython object factories are objects that call into a Python module, instantiate a Python class, conforms it to a Java interface and returns it. So far this project is looking good and there is a good possibility that this will get included in the Jython API.

Directly link (Java) applications to Python code

This is where things are starting to get advanced. It would be nice if you could write a library in Python (or import an existing one) and link your Java code with the classes and functions defined in that library directly. This would require Jython to generate Java proxies, actual Java classes where the methods correspond to the actual signatures, with proper constructors and the things you would need to use it like any other Java code, while hiding away the dynamic aspects that make it Python. This could either be done through a compilation step, where some Jython proxy compiler generates the proxies that the Java code can link with, or through utilizing a ClassLoader that loads a Python module and inspects the content, automatically generating the required proxies. With the ClassLoader approach javac would need to know about and use it to load signatures from Python code. This is of course where the Java integration decorators described above fits in.

What do you think?

I would love to get feedback on these ideas. Either through comments to this entry, via Twitter or on the Jython-dev mailing list.

Please note that the ideas presented in this blog post are my own and does not reflect any current effort in the Jython project.

JavaOne 2009 from my perspective

It has been a week since I got back to Sweden after my trip to San Francisco for what I am sad to say was my worst JavaOne so far. Don't get me wrong, JavaOne was good. There were a lot of good sessions, and as always it was nice to meet my friends in the community. But the two previous years were better. Much of it can be blamed on the economy. And the rest can be blamed on me. To a large extent the reason I didn't get the JavaOne experience I would have liked to was because I was too busy. I had a deadline waiting for me on a project at work when I got home, and had to work quite a lot during my JavaOne week. But the fact stands, JavaOne was a lot smaller this year, only about 9000 attendees, and you notice that.

These are the highlight from the sessions I attended at CommunityOne and then JavaOne:

• Beyond impossible: How JRuby has evolved the Java Platform by Charles Nutter. Charles is a kickass developer and has become a really good speaker as well. It's amazing what he has done with JRuby, and this talk was a summary of the highlights from that. Including things such as the fact that the JRuby team have implemented their own regular expressions library and posix layer for doing system calls from the JVM. All of which adds up to making JRuby the fastest complete implementation of Ruby.
• Return of the Puzzlers: Schlock and Awe by Joshua Bloch and Neal Gafter. All I can say is that all the digging around on the low levels of the JVM that I have done has payed off, I had the right answer for 7 out of 7 Puzzlers, much better than the first time I attended a Java Puzzlers session.
• Toward a Renaissance VM by John Rose. John talked about Invokedynamic and Method Handles. It was all familiar stuff, but a good introduction for the odd people who do not spend most of their time implementing languages for the JVM.
• Meet the Java Posse. I've known the guys of the Java Posse for a while, but this is the first time that I was not doing something completely different during their BoF at JavaOne. Not much to say, listen to the podcast.
• JSR 292 Cookbook mostly by John Rose. This is what I think was the eye opening session about Invokedynamic and MethodHandles for most people. Hands on examples, showing all of the JSR 292 goodness in action. More people should have attended. Everyone not familiar with JSR 292 on beforehand that I talked to after the session realized that it is useful for far more things than just implementing languages.
• How to write a distributed garbage collector by Ari Zilka and Saravanan Subbiah from Terracotta. This was the last session of the last day and probably the best session at JavaOne this year. Ari and Saravanan talked about how garbage is detected and collected in the distributed environment that is Terracotta when all references to an object could be from a different node. Very interesting stuff, I need to take a deeper look at Terracotta at some point soon.

There were also some noteworthy sessions that I missed while I was working:

• Java Platform Concurrency Gotchas by Alex Miller. A smart guy talking about interesting stuff, too bad I missed that one.
• The Ghost in the Virtual Machine: A Reference to References by Bob Lee. I have a feeling that there are still a few things that can be done using Weak, Soft and Phantom references, that I don't know about yet. I would also have liked to hear what he had to say about Ephemerons. Are they coming to the JVM soon btw?

Then there are the two presentations I gave. A BoF on Interface Injection entitled "Language Interoperability on the JVM Made Simple" and a Technical session entitled "Exploiting Concurrency with Dynamic Languages".

Language Interoperability on the JVM Made Simple

Since this was a Birds of a Feather session I divided it into two main sections: The first half I gave an introduction to interface injection, the status of the implementation and some ideas as to what you can do with it. The second half of the session was spent on discussions about what people want from interface injection, how they want to use it and how it should work.

The short summary is:

• I have implemented the injection mechanism.
• I have designed a proposal API for how the injection process should work on the Java level.
• I have wired up interface injection to invokeinterface, there are still a few more places that would need to be able to call the injection mechanism.
• The Reflection API for injected interfaces is still unspecified.
• A lot of people want only explicit injection of interfaces. That is for interfaces to only be injected when an inject() method is invoked in some API. This can be emulated by setting a flag in the injector when this method is invoked, and unset it at the end of this method.
• There was also some concern about the fact that methods on an injected interface gets implemented automatically by methods with the same name and signature that exist in the target class. I agree that it would be better if interface implementations could be separate for each interface in the class file format. But I don't think this problem will be as big as people might fear since the methods have to match exactly on return type as well as parameter types.

The staff in room 102/103 was really helpful, they offered to rearrange the chairs and tables to better host the discussion I wanted to encourage about interface injection. This can not be said about the staff in room 104 where I had my second presentation. Although being very friendly, they were not very professional. First of all my session was supposed to be recorded, but after the session I was informed that they had forgotten to record the slides, and transitions. I gave them my slides on a memory stick and they said that they were going to match it to the best of their ability then send it to me for review, I have yet to hear from them. On top of that some members of the staff were constantly talking behind the stage, leaving me very distracted. Highly unprofessional of them. I was so surprised by this that I didn't even tell them to shut up.

Exploiting Concurrency with Dynamic Languages

I spent too much time at JavaOne preparing for my second talk about how languages other than the Java Programming Language on the JVM lend themselves better to expressing concurrent programs. I really should have prepared the talk much more before I left Sweden. I believe that I could have done a better presentation. The staff in the room is to blame to a large extent for the fact that I didn't feel satisfied by my performance, but half of the blame is on me. Although, the few reviews that I've heard have been good. Basically that the topic was good, that I had interesting ideas and made good points, but that it was too short, although with good Q&A. A fair review in my opinion, it will be interesting to see the review cards when they get sent out to the speakers.

What is more interesting however is what I was able to conclude in my talk. It should come as a surprise to no one that closures make it much easier to express concurrent programs, and to encapsulate best practices and patterns in libraries. Other higher level constructs are helpful here as well, please have a look at my slides for reference. I also found that Jython has a lot of room for improvement, but more on that in another blog post.

On Iterator composition in Java

In my last blog post I mentioned that I would like to see library support for manipulating iterators included in the JDK. I for one, and I know that more people than me, have a set of iterator manipulation classes that I bring along from project to project, usually by copy/paste. This post is an elaboration on what the things are that I think should be included in the JDK, all communicated through code (The actual implementation is not as interesting as the API, and thus left as an exercise for the reader):

package java.util;

import java.util.iter.*;

public final class Iterators {
    public <E> Iterable<E> empty() { ... }
    public <S,T> Iterable<T> convert(Iterable<S> iterable, Converter<S,T> converter) { ... }
    public <T> Iterable<T> upCast(Iterable<? extends T> iterable) { ... }
    public <T> Iterable<T> downCast(Class<T> target, Iterable<?> iterable) { ... }
    public <T> Iterable<T> append(Iterable<T>... iterables) { ... }
}

// These classes should preferably be reused from somewhere.
package java.util.iter;

public interface Converter<S,T> {
    T convert(S source);
}

public interface Filter<E> {
    boolean accept(E element);
}

And some sample uses:

package org.thobe.example;

import java.util.Iterators;
import java.util.iter.*;

class UsesIteratorComposition {
    private class Something {}
    private class Source extends Something {}
    private class Target { Target(Source s) {} }

    Iterable<Source> source = Iterators.empty();

    void conversions() {
        Iterable<Target> target = Iterators.convert(source, new Converter<Source,Target>() {
            Target convert(Source source) {
                return new Target(source);
            }
        });
    }

    void casts() {
        Iterable<Something> something = Iterators.upCast(source);
        Iterable<Source> back = Iterators.downCast(Source.class, something);
    }

    void append() {
        Iterable<Source> source = Iterators.append(source, new ArrayList<Source>(){
            {
                add(new Source());
                add(new Source());
            }
        });
    }
}

On Double Checked Locking in Java

You have probably all heard/seen the “Double-Checked Locking is Broken” declaration. And if you haven't I'll just tell you that before the improved memory model of Java 5, double checked locking didn't work as expected. Even in Java 5 and later there are things you need to consider to get it to work properly, and even more things to consider if you want it to perform well.

Joshua Bloch suggests in his book Effective Java, and keeps repeating in several presentations that there is one way to implement double checked locking and that this code snippet should be copied to every place where it's needed. I disagree. Not with the part about there only being one way of doing it, but with the copying part. In my eyes this is perfect use case for an abstract class:

package java.util.concurrent;

public abstract class LazyLoaded<T> {
    private volatile T value;
    // Get the value, guard the loading by double checked locking
    public final T getValue() {
        T result = value;
        if (result == null)   { // First check
            synchronized (this) { // Lock
                result = value;
                if (result == null) { // Second check
                    result = loadValue();
                    value = result;
                }
            }
        }
        return result;
    }
    protected abstract T loadValue();
}

This class would then be used like this:

package org.thobe.example;

import java.util.concurrent.LazyLoaded;

class UsesDoubleCheckedLocking {
    // The generics even makes it read well:
    private final LazyLoaded<Something>() {
        @Override protected Something loadValue() {
            return new Something();
        }
    };
    // ... your code will probably do something useful here ...
}

Just expressing my opinion. I hope someone important reads this and makes sure that it gets included in Java7. And while you're at it, make sure we get some standardized ways of composing Iterable/Iterators as well...

Call frame introspection on the JVM

One of the pain points of implementing Python on top of the JVM, and in my opinion the worst pain point, is the call frame introspection of Python, and how it is abused throughout Python frameworks. There is in fact a library function sys._getframe() that returns the frame of the current function, or a deeper frame in the call stack if passed an integer representing the depth.

For Jython this mean that we always have to keep a heap allocated call frame object around for each function invocation, and do all access to local variables through this object. The performance implications of this is absolutely terrible. So any improvement to call frame management would greatly improve performance.

Thinking about how to better implement call frame objects in Jython I thought “the JVM already manages call frames, and using a debugger I am able to get access to all the aspects of the JVM call frame that I am interested in to construct a Python frame.”

My first idea on how to implement such a thing was to get under the hood of the JVM and add these capabilities there. Said and done, I checked out OpenJDK and started to patch it to my needs. This actually got me to a working prototype, apart from the fact that I failed to tell the garbage collector that I created extra references to some objects, so they ended up not being equal to null but in every other sense behaving like null. I never tracked down where the problem in my code was, since I found another solution. While I was working on this I found code in the JVM that did things that were surprisingly similar to what I was doing. All of this code was located in a subsystem called JVMTI. I looked further into the documentation of the JVM Tooling Interface, and found that it had been around since JVMv1.5 (commonly referred to as Java 5). This was great news! If I could create a JVM frame introspection library using JVMTI I would have a library that works for all the JVM versions that we target for the next release of Jython.

It took a while before I actually started implementing the frame introspection library, there were other tasks with higher priority, I read up the documentation of JVMTI and there was also the issue with the build process for JNI-libraries being much more painful than for the nice Java stuff I'm used to, since JNI is C code. The last problem I solved over a weekend two weeks ago by implementing a generic make file and an ant task that feeds make with the required parameters of that makefile. This took me a full weekend, since my makefile skills were never that great to begin with, and even rustier than my C skills. I got it to work for my Mac though, and have yet to test it on other platforms (Linux will probably be tested and working before the weekend). This ant task is hosted on Kenai: https://kenai.com/svn/jvm-frame-introspect~svn/jnitask/.

Armed with a good way of building JNI libraries I met up with Charles Nutter in Malmö, where he was speaking at the Øredev conference, for three days. We managed to get the library working while hacking at various coffee shops in Malmö, it should still be polished a bit but I've published it in my personal svn repository on Kenai: https://kenai.com/svn/jvm-frame-introspect~svn/frame/ for everyone who wants to take a look. Trying it out should be as simple as:
svn co https://kenai.com/svn/jvm-frame-introspect~svn/frame/ javaframe
cd javaframe
ant test
The expected output is a few printouts of stack frame content.

The call frame introspection library gives you access to:

• Access to one call frame at a given depth.
• Access to the entire stack of call frames for the current thread.
• Access the stack of call frames (without local variables) for any set of threads, or all threads, in the JVM.
  
• Get the reflected method object that the frame is a representation for.
  
• Local variables (if this information is added to the class file by javac)
  
  • The number of local variables.
  • The names of the local variables.
  • The signatures of the local variables.
  • The values of the (currently live) local variables.
  • Locals may be accessed either by name or by index.
    There is also a getThis() method as a shorthand for getting the first local variable, or null if the method is static.
    
• Get the current value of the instruction pointer.
  
• Get the current line number (if this information is added to the class file by javac).

Update 2008-11-23: I've added support for getting stack traces of call frame snapshots from multiple threads in the JVM. The JVMTI guarantees that the stack traces of all threads are captured at the exact same point of execution. There are method calls for getting traces for a given set of threads or for all JVM threads. These methods do not provide access to the local variables in the frames, since there is no way (apart from suspending the threads) to guarantee that the frame depth is the same at the point of capturing the stack trace as at the point of acquiring the local variables.
I've also made sure that the build scripts work under Linux as well as Mac OS X, and added licensing information (I use the MIT License for this). Also, by request, cleaned up the paragraphs of this entry.

It would be great to get input from my peers on what more information you would like to access from the call frames. The Java bytecode of the method perhaps?
Charles and I also talked about what else we can use JVMTI for, and he was quite enthusiastic about the ability to walk the entire Java object graph, for implementing the objectspace feature of Ruby. One idea would be to write a library that brings the entire functionality of JVMTI to the Java level. The only problem with this would be that a lot of the JVMTI operations don't make much sense unless they are invoked in conjunction with other operations, and that many of the operations are not callback safe, meaning that we cannot allow the execution of arbitrary Java code in between the JVMTI operations. But it should be possible to create an abstraction with a more Java-esque API that performs multiple JVMTI operations at the JNI level.

Another interesting aspect of using JVMTI from Java code is that we can prototype a lot of the functionality that we want the JVM to expose to us directly, and thereby vote with code on what we want the JVM of the future to look like.

I hope you will find this useful!

Update: I have move this project to Kenai, the links have been updated accordingly.

My JVM wishlist, pt. 1 - Interface injection

If you've been following what's going on in the JVM-languages community you have probably stumbled across John Roses blog. One of his entries was about interface injection. In short wordings interface injection is the ability to at runtime add an interface to a class that was not precompiled as implementing that interface.
Interfaces are injected at one of 3 situations:

• When a method of the interface is invoked on an object.
• When an instanceof check for the interface is performed on an object.
• When the interface is queried for via reflection (I can see this working with Class#isAssignableFrom, but I have my doubts when it comes to Class#getInterfaces, although I'm sure someone smart will be able to solve this without having to know about all injectable interfaces beforehand).

When any of these occur a class can either have the interface implemented already (in the regular way) or a special static injection method on the interface class is invoked. It is up to this injection method to determine if the given class can implement the interface or not. If it determines that the class can implement the interface any missing methods have to be supplied at that time. John suggests that these methods are to be supplied as method handles. Since method handles, according to the EDR of the InvokeDynamic proposal (JSR 292), can be curried this would make it possible to attach an extra state object to the returned method handle, or to return a different implementation of the interface depending on the class they are injected to. Once an injection method of a specific interfaced has been invoked for a specific class, the injection method will never be invoked for that class again, meaning that once a class has been found to not implement an interface, that information will be final, and once an implementation of an interface has been supplied, this implementation can never be changed. This is important since it will allow the VM to perform all optimizations, such as inlining, as before.

What can this be used for?
As a language implementer on the Java platform I think interface injection would be a blessing. In fact I think it is the one thing that would simplify the implementation of languages on the JVM the most. Any language probably has a base interface (as Java has java.lang.Object and Jython has org.python.core.PyObject), let's be unbiased and call it "MyLangObject" for the sake of the continued discussion. There are two things that make the Java platform great:

1. There are a lot of really good toolkits an libraries implemented for the Java platform.
2. There are a lot of great languages for the Java platform in which even more great libraries and toolkits will be developed.

Therefore, if you are implementing a language for the Java platform you would want to interact with all of these libraries and toolkits. Problem is that most of them haven't been designed with your language in mind, and they shouldn't be. If MyLangObject was an injectable interface all you would need to do to be able to integrate with any object from another language would be to just interact with it through the MyLangObject interface, and the injection mechanism would take care of the rest.
The injection mechanism could even be used with the classes within your language. Instead of having a base class supplying the default implementation of the methods of MyLangObject you could let the injection method return the default implementation for your methods.
Or why not use interface injection to support function invocation with different argument counts. Each function in your language would implement a set of call methods, one for each argument count it can be invoked with. Your language would then have a set of injectable Callable interfaces one for each argument count that any function in your language can be invoked with, each with only one call method, with the appropriate number of arguments. These interfaces could be generated at runtime if your language supports runtime code generation. The default implementation of the call method in each Callable interface would of course raise an exception, since the function obviously doesn't support that argument count if it doesn't implement the appropriate method.
Interface injection really does provide a huge set of opportunities.

How could interface injection implement a Meta Object Protocol?
There is a great project initialized by Attila Szegedi of the JVM-languages comminuty to create a common meta object protocol (MOP) for all languages on the JVM. With interface injection this would be a simple task.

1. Let all objects in your language (that supports the MOP) implement the (probably not injectable) interface java.dyn.SupportsMetaObjectProtocol. An interface with only one method:
   java.dyn.MetaObjectProtocol getMetaObjectProtocol();
   This would return the implementation of the java.dyn.MetaObjectProtocol interface for your particular language.
2. The java.dyn.MetaObjectProtocol contains methods for getting method handles for each dynamic language construct that the community have agreed to be a good common construct, such as getters and setters for subscript operations. These method handles would come from the actual implementation of them for your particular language, and would therefore benefit from every imaginable optimization you have cooked up for your language.
3. When the main interface of my language is being injected into a class from your language it finds that your class implements java.dyn.SupportsMetaObjectProtocol and uses that to get the method handles for all dynamic language constructs supported by my language, rebinding them them to the method names used in my language.

And as simple as that interface injection has been used to implement a common ground for all languages on the Java platform with absolutely no overhead. I'm not saying that this is the way to implement a meta object protocol for the Java platform, I am just suggesting one way to do it, someone a lot smarter than me might come up with a much better implementation.

To sum things up: I can't wait until the JVM supports interface injection.

Edit: this post has also been re-posted on Javalobby.

Jython - Summer of Code, finally getting started

Disclaimer: Even though this is an announcement of the fact that I am getting started on my SoC project, I have to admit that there might be some rants about courses at my university and ScaMPI.

As most of my friends already know, this summer I am doing a Google Summer of Code project. The project I applied for, and have been assigned to work on, is the Jython project. My part in Jython will be back end compiler work, i.e. bytecode generation. I have a great mentor, Jim Baker, and that makes me quite sure that I will be able to accomplish something during this project.
My original plan was to be able to start working on this project a week and a half ago, but that didn't turn out as I wanted it to. A week and a half ago I had my final exam, in cryptology (I actually managed to get quite good scores on that exam), and I had thought that I would be free from my studies after that. I had never expected that the final lab assignment in my programming of parallel computers course would take as much time as it did. I thought I would have completed that assignment before my exams. I was wrong. Before focusing on my exams I knew that the code segfaulted in a call to MPI_Sendrecv, that was quite easy to find out using some trace printouts. But figuring out why was a whole other story. It turns out that the MPI library on our super computer (ScaMPI) didn't adhere to the MPI standards. In order to get the rapid development cycles I am used to, instead of having to wait for CPU time to run the application, i installed OpenMPI on my MacBook (hey, it has multiple cores, it can do parallel computing), and manged to get everything working fine there, but it still segfaulted with ScaMPI. In the end it turns out ScaMPI doesn't support the MPI_STATUS_IGNORE parameter, a limitation that I haven't been able to find in any of the documentation I have been looking at. That took me about a week to track down. This experience has made me even more thankful to Open Source, since trying to use a parallel debugger (that was a pain to start due to license enforcement algorithms) on assembly code that you really don't know what it was compiled from is NOT a pleasant experience.
Anyhow, back to the main topic, my Jython SoC project. For those of you who might want to follow the progress of my work I will try to publish posts here about the project quite regularly (I am aiming at once a week). If you are interested in my project notes and source code for the stuff I am playing around with in the scope of this project, these will be hosted on my personal server and available at https://projects.thobe.org/jython/. The final code and documentation will be hosted at the Jython repository over at SourceForge and the official Jython wiki respectively.
Well then. I think that was all I wanted to announce in this post. Wish me happy hacking. Even though I suppose that tomorrow wont be as productive as I would want it to be, it is the Swedish midsummers holiday tomorrow, an occasion Swedes (ab)use to get really drunk. I am going to be the designated driver for my girlfriend and her family during the day, and then my girlfriend and I are going to have our own party at the evening. Well, I am sure that all of my hacking after that will be of the happy kind.