Embedding the Scala Interpreter

Recently, I was given an application at work that needed some extra functionality, and basically needed to be made more flexible. As this is an internal tool, it currently requires 'expert' usage only. Thinking through the design, I decided that perhaps I should try an idea I've heard others use:

Let's write the configuration for the application in a Scala Domain Specific Language.

To do so, I need to make the application "modular" in the sense that various components can be configured through some sort of "builder" pattern DSL. They are then assembled together (again in a DSL), which should be as readable as possible, and easy to change. The DSL creation went great, the app is looking good. Here's a pseudo example of what I'm talking about.

val input = new InputChannel with ReconnectOnFailure {
    inputUrl = "http://suereth.blogspot.com"
}
val output = new OutputChannel with FileWriter {
   outputUrl = "file:///home/josh/test.out"
}

processingBuilder from input to output and { p =>
      p hasModule new ThemeExtractor {
           maxThemes = 10
           //Define more variables
      }
      //Define more modules       
}
processingBuilder.startProcess()

The above is a hypothetical DSL for processing inputs and placing the results in an output. The idea being you can generate "filter chains" via the processingBuilder object. Ideally, this configuration should be in some kind of file we can pass into our program, rather than having to recompile the program every time we want to run. Imagine however, that we combine these two solutions.

Let's compile the DSL and all the internals of the program and place them into a jar file. Let's then take the "configuration" portion of the program and compile it at runtime, just before execution. To do this, we can simply embed the Scala interpreter. I'm going to make use of an open-source "library" (really just one class) I wrote one evening while testing this theory. The library (Scala Embedded Interpreter) is located on github and is BSD licensed.

To start off with, we need some kind of main class that will take a "configuration" file as an argument, and pass it into our embedded interpreter. Let's try something like this:

object App {
   def main(args : Array[String]) {
        //Let's assume all arguments are file locations for configuration

        val interpreter = new InterpreterWrapper {
            for(fileName <- args) {
              addScriptFile(fileName)
            }
        }
        interpreter.startInterpreting()
   }
}

So, the above is fairly simple. We construct and InterpreterWrapper and call "addScriptFile" for every configuration file we find on our command line. So... how is InterpreterWrapper working? Let's look into the details:

import scala.tools.nsc._
import interpreter._
import java.io._
import scala.reflect._
/**
 * A trait to ease the embedding of the scala interpreter
 */
trait InterpreterWrapper {
  private var files = List[String]()
  ...
  protected def addScriptFile(fileName : String) {
    files = fileName :: files
  }
  ...
  def startInterpreting() {
    //Make a place for output to go
    val out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(System.out)))
    //Create a new Interpreter settings object with the given output method
    val settings = new GenericRunnerSettings(out.println _)
    //For every file, add it to the interpreter settings
    files.foreach(settings.loadfiles.appendToValue)
    
    //Create a new interpreter loop and start running!
    val interpreter = new InterpreterLoop(out)
    interpreter.main(settings)
    ()
  }
  ...
}

As you can see, the trait is fairly simple. It creates a "Settings" object (which is normally created by parsing off the command line arguments) and passes it into "InterpreterLoop". InterpreterLoop is the class which wraps "Interpreter" by continually prompting and executing input. This allows us to do live configuration of our program (which can be very fun).

The next thing we need to do is promote our DSL into the interpreter, so our configuration files don't begin with unnecessary "import foo.bar._" statements. Let's modify our App and the InterpreterWrapper.

object App {
   def main(args : Array[String]) {
        //Let's assume all arguments are file locations for configuration

        val interpreter = new InterpreterWrapper {
            autoImport("my.awesome.dsl._")

            for(fileName <- args) {
              addScriptFile(fileName)
            }
        }
        interpreter.startInterpreting()
   }
}

So the main code doesn't change much, simply adding the "autoImport" method call when constructing the InterpreterWrapper. However, we now need to some more magic in the interpreter wrapper. Here's the (relevant portions of the) code:

import scala.tools.nsc._
import interpreter._
import java.io._
import scala.reflect._

trait InterpreterWrapper {

  private var packageImports : List[String] = List()  
  
  protected def autoImport(importString : String) {
    packageImports = importString :: packageImports
  }
  
  /**
   * This class actually runs the interpreter loop.
   */
  class MyInterpreterLoop(out : PrintWriter) extends InterpreterLoop(None, out) {    
    
     override def bindSettings() {       
       super.bindSettings()      
       interpreter beQuietDuring {
         //This will silently execute our import statements when starting!
         for( importString <- packageImports) {
          interpreter.interpret("import " + importString)
         } 
       }
     }   
  }
  
  def startInterpreting() {
    val out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(System.out)))
    val settings = new GenericRunnerSettings(out.println _)
    ...
    val interpreter = new MyInterpreterLoop(out)
    interpreter.main(settings)
    ()
  }
}

As you can see, we're using a subclass of MyInterpreterLoop and extending the "bindSettings" method to additionally execute our import statements. We can now almost use our DSL directly. The only remaining issue is that we'd like to have our processingBuilder object be automatically bound and in scope for the interpreter session. This allows the configuration to just start directly talking with the object. Let's add "bind" functionality to the InterpreterWrapper with the following API:

object App {
   def main(args : Array[String]) {
        //Let's assume all arguments are file locations for configuration

        val interpreter = new InterpreterWrapper {
            autoImport("my.awesome.dsl._")
            bind("processingBuilder", new MyProcessingBuilderImplementationClass())
            //OR if I want to restrict access to an interface
            bindAs("processingBuilder2", classOf[ProcessingBuilder], new MyProcessingBuilderImplementationClass())
            for(fileName <- args) {
              addScriptFile(fileName)
            }
        }
        interpreter.startInterpreting()
   }
}

We now need to update the InterpreterWrapper. We'll be doing a similar trick as autoImport and adding some "silent" execution during the bindSettings method of InterpreterLoop. Here's the relevant code:

trait InterpreterWrapper {
  ...
  private var bindings : Map[String, (java.lang.Class[_], AnyRef)] = Map()
  ...
  protected def bind[A <: AnyRef](name : String, value : A)(implicit m : Manifest[A]) {
    bindings +=  (( name,  (m.erasure, value)))
  }

  protected def bindAs[A <: AnyRef, B <: A](name : String, interface : java.lang.Class[A], value : B) {
    bindings += ((name,  (interface, value)))
  }
  ...  
  class MyInterpreterLoop(out : PrintWriter) extends InterpreterLoop(None, out) {
    
     override def bindSettings() {       
       super.bindSettings()      
       interpreter beQuietDuring {
         for( (name, (clazz, value)) <- bindings) {
           interpreter.bind(name, clazz.getCanonicalName, value)
         }
         ... //auto-imports
       }
     }
  }
  
  def startInterpreting() {
    ...
  }
}

Here we're using the "bind" method of the underlying Interpreter object. This allows us to bind arbitrary objects and specify their interface. For the InterpreterWrapper we've two bind methods, one which uses the experimental Manifest feature of scala to automatically use the visible type at the bind call, and another method where you can explicitly specify a type.

We now have our configuration file being loaded and executed in a nice convenient DSL. The last bit we should do is "brand" the interpreter, so that it looks like we intend to use it (and not like scala suddenly shows up in our app). To do so we're going to override the help message, the welcome message and the prompt. Here's what our main object should look like:

object App {
   def main(args : Array[String]) {
        //Let's assume all arguments are file locations for configuration

        val interpreter = new InterpreterWrapper {
            def prompt = "Processor> "
            def welcomeMsg = "Welcome to the InterWeb Text Processing Magician! Please set up a processing stream to be executed."
            def helpMsg = "To set up a processing stream, please create an input and output and wire them together using the processingBuilder.  :help will print this message, :quit will exit the processor"

            autoImport("my.awesome.dsl._")
            bind("processingBuilder", new MyProcessingBuilderImplementationClass())
            //OR if I want to restrict access to an interface
            bindAs("processingBuilder2", classOf[ProcessingBuilder], new MyProcessingBuilderImplementationClass())
            for(fileName <- args) {
              addScriptFile(fileName)
            }
        }
        interpreter.startInterpreting()
   }
}

As you can see, using the InterpreterWrapper allows you to quickly and easily embed the scala interpreter and use it to configure your application (at start up or during execution). Remember, though, that as with an user-interface you need to worry about threading issues when attempting to run concurrent threads (or use actors).

Finally, here's the entire code for the InterpreterWrapper (under BSD license BTW):

import scala.tools.nsc._
import interpreter._
import java.io._
import scala.reflect._
/**
 * A trait to ease the embedding of the scala interpreter
 */
trait InterpreterWrapper {

  private var bindings : Map[String, (java.lang.Class[_], AnyRef)] = Map()
  private var packageImports : List[String] = List()  
  private var files = List[String]()
  
  /**
   * Binds a given value into the interpreter when it starts with its most specific class
   */
  protected def bind[A <: AnyRef](name : String, value : A)(implicit m : Manifest[A]) {
    bindings +=  (( name,  (m.erasure, value)))
  }
  /**
   * Binds a given value itnot he interpreter with a given interface/higher-level class.
   */
  protected def bindAs[A <: AnyRef, B <: A](name : String, interface : java.lang.Class[A], value : B) {
    bindings += ((name,  (interface, value)))
  }
  
  /**
   * adds an auto-import for the interpreter.
   */
  protected def autoImport(importString : String) {
    packageImports = importString :: packageImports
  }
  
  /**
   * Adds a file that will be interpreter at the start of the interpreter
   */
  protected def addScriptFile(fileName : String) {
    files = fileName :: files
  }
  
  def helpMsg : String
  def welcomeMsg : String
  def prompt : String
  
  /**
   * This class actually runs the interpreter loop.
   */
  class MyInterpreterLoop(out : PrintWriter) extends InterpreterLoop(None, out) {
     override val prompt = InterpreterWrapper.this.prompt
    
     override def bindSettings() {       
       super.bindSettings()      
       interpreter beQuietDuring {
         for( (name, (clazz, value)) <- bindings) {
           interpreter.bind(name, clazz.getCanonicalName, value)
         }
         for( importString <- packageImports) {
          interpreter.interpret("import " + importString)
         } 
       }
     }
    override def printHelp {
       printWelcome()
       out.println(helpMsg)
       out.flush()
     }
     
     override def printWelcome() {
       out.println(welcomeMsg)
       out.flush()
     }
  }
  
  def startInterpreting() {
    val out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(System.out)))
    val settings = new GenericRunnerSettings(out.println _)
    files.foreach(settings.loadfiles.appendToValue)
    //Below for 2.8.0-SNAPSHOT
    //settings.completion.value = true
    
    val interpreter = new MyInterpreterLoop(out)
    interpreter.main(settings)
    ()
  }
}

I must say, I'm very happy with how my project turned out at work. It actually made things *much* easier to debug during development as I could interact directly with the program during execution. The only complaint so far from co-workers is that they don't know enough Scala to used some of the more advanced features (for-expressions actually). However, the sample configurations I provided were enough for them to get up and running with the application and create slightly varied configurations. We have slightly more verbosity in configuration now, but the flexibility/power is incredible.


Below is the output of the exeuction of my very silly "test" usage of the InterpreterWrapper (included in the github project). This sample is using Scala 2.8.0 nightly and Paul Phillips's interpreter improvements. You can see the auto-complete suggestions below.

josh@suereth-desktop:~/projects/blog/embed-the-interpreter$ java -jar target/embedded-interpreter-0.1-SNAPSHOT-jar-with-dependencies.jar
Welcome to Awesomeness!
This is my version of the Scala interpreter
TestInterpreter> j

java javax jline
TestInterpreter> j

java javax jline
TestInterpreter> josh.is
res0: java.lang.String = Brillant!!!
TestInterpreter> josh.

getClass is name toString
TestInterpreter> josh.

getClass is name toString
TestInterpreter> josh.name
res3: String = josh
TestInterpreter>