Rants, Raves and Ridicule

Migrating to new site

2011-12-21T22:16:00.002-05:00

Hey guys, I'm finally moving off of blogger. The new blog will be located at http://jsuereth.com and the source code is available at http://github.com/jsuereth/jsuereth.github.com. Feel free to fork what you want, and send back pull requests if you do something nice :)

Scala Fresh is alive

2011-12-07T13:59:00.001-05:00

It seems a lot of posts I've meant to write for a *long* time have been able to be used optimally in response to other blogs.

Todays post is brought to you by David Pollak's fragility post.

So let's talk about the problem of "Binary Compatibility(TM)" in Scala.

First off, David raises a big issue in Scala. One that most of us have seen. Scala's new language features necessarily break binary compatibility of the bytecode. Adding things like Specialization change Scala's standard library enough that you cannot use code compiled against older versions with the new version. This is a two-edged sword.

The Scala experience should be as smooth as possible for all customers.
Things like java.util.Date shouldn't survive endlessly our standard library.

This requires a careful balance between breaking binary compatibility and advancing the language. So far, I'm pretty happy with the way this has been done, but would like to see things stabilize in the future. Let's take a look at what's happened:

Scala's new binary compatible releases

Scala 2.9.1 is binary compatible with Scala 2.9.0. If you compile code against Scala 2.9.0, you can use it with the standard library from Scala 2.9.1. This will hold for *all* 2.9.x releases. In Scala, binary compatibility will be at the bug-fix release. All 2.10.x releases will be binary compatible with previous 2.10 releases. This means that as long as your dependencies are compiling for a given Scala minor version, then you can continue to enjoy binary compatibility of libraries.

This was made possible by the freely available Migration Manager tool. A lot of us at typesafe use the publicly available version when developing to ensure binary compatibility of libraries.

Note: This was *not* true for a lot of David Pollak's Scala experience, and is the result of many of us petitioning for better binary compatibility. It's my opinion that this guarantee solves 50-80% of the problem.

What happened to Scala fresh?

I've begun work on what can only be called "Scala Fresh 2.0". That is, a place where community libraries will be built and deployed against the latest version of Scala. This can serve two purposes:

Ensure that future versions of Scala do not break community libraries
Ensure that the core libraries of Scala are available for every major version of Scala.

You can find the project publicly available here. All of the work is public and will be migrate to the "scala" user on github once complete and ready for more contributions. Feel free to contribute or offer suggestions.

People may not realize but Scala Fresh failed because *I* failed Scala Fresh. I had very little time between a more than full time commitment at Google, writing commitments, and kids. This should not be an issue in the future, thanks to Typesafe taking binary compatibility seriously. I now have a prototype build that you can migrate your projects into.

My coworker likens this idea to linux distribution repositories. I think it's a decent way to think of it. I, and others, are working hard to ensure the ecosystem for every major Scala release is easy to use and stable. Migrating major Scala versions should be as difficult as finding deprecation warnings and removing them.

SBT cross releasing

Mark Harrah, recognized that Scala remains generally *source* compatible between major releases (2.8 to 2.9). A *long* time ago, Mark met me at a Panera in Anne Arundel, MD to discuss a mechanism of cross deploying libraries so that they were available for every release of Scala. This was well before the 2.9.x binary compatibility days and is still in heavy use within the community. It's been pretty widely adopted, but can only go so far. It's a good stop-gap solution that we, the community, can improve on.

Binary compatibility is a community effort. I know Typesafe is doing what it can with its resources, and I'm personally tackling as much as I can (probably trying to juggle too many balls). However, if you want to help, please email me!

The JVM is amazing, don't doubt its powers
Finally, I want to clarify a confusion that a lot of people have about binary compatibility. That is, that traits are *HUGE ISSUES FOR BINARY COMPATIBILITY ZOMG!!!*. In actuality, traits are only slightly more dangerous than interfaces + implementation pairs.

In the code below, the method colored in red was added in a bug-fix release. It is *binary compatible* with the previous version.

trait Foo {
def foo = "foo"
def bar = "NEW BAR IS NEW BAR!"
}

That's right, adding methods, even with implementation, does *NOT* break linkage. Think about it. How did Java support JDBC 4.0 interfaces running JDBC 3.0 drivers? The trick is that trait-linkage errors happen at *runtime* when *calling* a method that has no linked implementation. This is the kind of magic that Java can get away with to ensure binary compatibility and it's just as useful in Scala.

Now, there is an issue where you can break binary compatibility. Again, red italics will denote new code

trait Bar {
def foo = {
bar + " foo"
}
def bar = "NEW BAR IS NEW BAR!"
}

NOW, the implementation of a previous method calls a new method. While this new method has an implementation in the trait, compiled code against the trait does not. This is akin to the following Java scenario:

interface Foo {
public String foo();
public String bar();
}

abstract public class AbstractFoo implements Foo {
@Override public String foo() { return bar() + " foo" }
}

The abstract method implements one of the trait's methods, but not the new one. However, the implementation is modified to call the new method.

Scala's collections underwent a minor change in 2.10 to improve both class file size *and* unintentionally binary compatibility. That is, the collections now follow this pattern:

trait Traversable[A] ....
abstract class AbstractTraversable[A] extends Traversable[A] {}
class Vector[A] extends AbstractTraversable[A] with Traversale[A]....

In this scenario, if your collection extends an Abstract collection (rather than its own parent class), you can remain binary compatble *and allow* new methods to be added to a trait *and be called* be the implementation. Vector is now "binary resilient" to changes in the Traversable trait. There are no guarantees that a trait is not used directly, so we can't take advantage of this when fixing bugs in collections in the Scala library. It's still a nice trick to use.

You *do* lose some flexibility doing this. Specifically, you can't have multiple class parents unless they're linear (like Java).

So, my point is that Binary Compatibility is a community issue. The JVM and Scala do what they can. I hope to see the compiler do more in the future. The Migration Manager tool from Typesafe can detect a lot of binary compatibility issues and has been crucial to ensuring 2.9.x Scala releases are binary compatible.

I'm trying to organize projects to help ensure *all* libraries for Scala can be released against all the major versions and remain binary compatible. The story is changing, and has done so rapidly over the past two years.

I'd love to see Lift, and others start adopting binary compatibility standards for there releases as well. This is not just a scala-the-library or scala-the-langauge issue. Libraries matter, just as they do in Java.

So while David's post highlights an issue, my response is:

Help us (the Scala community) out! We're going there, we're doing that.

Macro vs. Micro Optimisation

2011-11-29T19:59:00.001-05:00

So there's recently been a bit of hype about another Colebourne article: http://blog.joda.org/2011/11/real-life-scala-feedback-from-yammer.html

I'd like to respond to a few points he makes.

First - You should evaluate Scala and pay attention to its benefits and flaws before adopting it. Yes, there are flaws to Scala. Working at typesafe makes you more aware of some of them. We're actively working to reduce/minimize/get rid of these. In my opinion, the negatives of using Scala are peanuts compared to the postives of choosing Scala over Java. I think everyone should make up their own mind about this. Not everyone is going to choose Scala. I feel bad for those who don't, but I make no effort to convince you further than showing you the 40+ Open source Scala projects I have on github. It's a language with a lot to like and a bit to dislike.

Now, to the meat of what I want to say. Don't get lost in micro optimization when discussing programming.

The blog article discusses writing high-performance code in the critical path that has crazy performance needs. This is not your every day development. Scala loses a lot of benefits in this world, because features like closures have overhead on the JVM. Hopefully when Java adopts closures, this overhead can be alleviated, but it is there right now. The set of rules from the email is known to a lot of us Scala devs when writing a performance intensive section of code.

I'll reiterate a few to agree with:

(1) Avoid closure overhead. If you can't get the JVM to optimize away and need to allocate a closure constantly in a tight loop, this can slow down a critical section. This is correlated with

(1a) Don't use a for loop. For expressions (and loops) in Scala are implemented as closures. There are some efforts underway to *inline* these closures as an optimization. This performance hit isn't a permanent one. As the compiler matures, you'll see a lot of optimization work happen.

(2) Use private[this] to avoid an additional method. Scala generated methods and fields for val/var members. Using private[this] informs the compiler that it can optimize away the method. Again, while hotspot can optimize this away if you're in a very critical section, it may be a good idea to optimize. In fact, the whole promotion to fields and methods aspect of Scala classes deserves attention from anyone writing performance critical code. The rules are also there in Java, it's just that you probably have people that already know them.

(3) Avoid invokevirtual calls. (Coda lists this as avoid Scala's collections). The true issue here is that invoke-virtual can make a difference in performance critical sections of code. Again, this is one I think we can improve with a few final declarations and maybe an annotation or two.

Here's the big *missing* point in all that feedback. This is for performance critical sections of code, not for general purpose applications. I think when it comes to performance bottlenecks, you need to pull out the stops and optimize the heck out of your apps. Look at the Akka framework from typesafe. Akka uses a lot of "dirty" scala code to achieve high non-blocking concurrent performance. It's one of the most amazing libraries, and the code is pretty low level. It also supports a very high level of abstraction when writing your application. It uses PartialFunctions (which are sort of a combination of pattern matching and closures) and traits, both of which have some overhead. However the resulting *application* is fast. Why? The inner loops are fast and optimized and the application *architecture* can be optimised. In a high level language, you can take advantages of designs that you would *never* execute in a lower-level language because the code would be unmaintainable.

You see, most of us like to be able to read and understand what our code does. Scala has some features where you can write very expressive code in a few lines. After getting over the initial hump, this code is pretty easy to maintain. If I were to write that same code in Java, it would look odd, confusing as hell, and no one would want to maintain it.

I learned this lesson pretty hard core at Google.

Google has a pretty stringent C++/Java style guide. One that is tuned for high performance servers. The C++ style guide is public. The style guide frowns on things like the use of 'smart' pointers that 'garbage collect' because you can't be sure that GC will happen at a critical moment in the code.

Google also has a high performance Map-Reduce library. This thing is pretty amazing, with all sorts of crazy cool features like joining data on the *map* part of the map-reduce. The library followed all the google coding conventions and is generally held up as a piece of awesome software, which its.

However, writing applications with the Map-Reduce library was less than idea. I could write pretty clean code with the MR library. My Mappers were pretty light weight and minimalistic. Same with my reducers. You'd then string together a series of map-reduces in this crazy guitar inspired "patch it together" configuration and the thing would be off to the races. The downside is that the throughput of this kind of processes was *sub optimal*.

It wasn't anything inherent in the libraries. The libraries were fast and good. The APIs were optimised for high speed performance. However, writing optimal *architectures* in the framework was tough. If I wanted to do any crazy performance features, like combining map functions and reduce functions in a single map reduce run, the code got ugly *fast*. Not only that, it was very difficult to maintain because of all the odd bookkeeping. I have 10 outputs ordered by key, this is the one writing to that file right? KRAP.

I spent 6 months writing applications of this nature and feeling like there had to be something better. I started on a venture to write something, and as usual found that someone else already had. That's when I found out about Flume and met Craig Chambers.

FlumeJava was a map reduce library that was gaining a lot of traction at Google, but I had heard a lot of complaints about its API. Around the time I was looking at it, Flume C++ was coming into existence. My team was one of the alpha users of the C++ variant.

The C++ API was a thing of beauty. Like its Java cousin, it treats data a set of Parallel Distributed Collections. You can see Daniel and my talk on the Scala equivalent here. Converting Mappers and Reducers into this API was pretty simple. You could even do it directly by just annotating your Mappers and Reducers with types.

My team saw a 30-40% reduction in code size for some of our map-reduce pipelines. We had an 80% reduction in code for unit tests (which was by far the most amazing benefit). Not only that, the Flume library optimizes the pipeline by performing all sorts of dirty map-reduce tricks for you. In some cases we dropped a map-reduce call or two. In the worst case, we had the same number we had started with.

What was wrong with the library? It violated *almost every* style rule for C++ at google. That's right, smart pointers, classes with inline member definitions the works. I loved every second of it. Why? Because I was getting stuff done *faster* than before with *less* code and the pipelines ran *faster*. It was a crazy win.

The startup time might not have been as optimal as it could be. Flume ran an on-the-fly optimization before running your pipeline. That was being improved all the time with neat tricks. Things I didn't have to write to watch my app speed up, both runtime and startup time.

The key here is that the designers of Flume weren't focused on micro optimization but *macro* optimization. The inner guts of the library used the very fast and efficient Map-Reduce library and the wrappers they had were as efficient as they could make them. My code did not have to follow these rules, because the core loops were fast. When it came down to it, my code used high level concepts. Something akin to a closure and for-expressions in Scala (Note: The Scala equivalent *did* use for-expressions and closures with no noticeable performance hit).

There are times when writing Scala code requires care and optimization in the micro level. However, don't lose the forest for the trees. Think about the entire application architecture. Scala will open up possibilities for writing code that you'd never dream of trying to maintain in C++ or Java. Take Akka as a shining example.

And when you need the performance, listen to those techniques. Viktor Klang can probably give you a *ton*more. I know I learn more every time I talk with him.

SBT and Plugin design

2011-09-16T12:44:00.000-04:00

Sbt 0.10 brings *a lot* of power to the table. SBT 0.10 switched from a class/inheritance based build system into a more functional approach. For those who aren't familiar, here's the quick spiel on SBT.

Basics of SBT

In SBT, a project is composed of Setting[_] values. A Setting is sort-of a name-value pair (more of a name *computation* pair). In the SBT command-line you can type the name of a setting and get its value (or computed value). For example, test is a task in SBT that you can type in the command line. The setting's computation is executed and the value returned. This setting may depend on other settings for its value.

SBT provides a simple way to construct a project. In the root directory, any *.sbt file is compiled to Setting[_] values. A Setting[_] is two things: A name (Key + Scope) and a Value (or computation, called Initialize in SBT). One can construct a Setting[_] via the SBT dsl:

sourceDirectory in Compile <<= baseDirectory apply { dir =>
  dir / "src" / "main" / "scala"
}

In this example, the sourceDirectory Key (name) is assigned an Initailzation (value/computation). The <<= operator is used to construct a Setting[_] by joining a Key and an Initialiation. In the above example, the Initialization is constructed to pull the current value of the baseDirectory key and modify it for the value of the sourceDirectory key.

Note: The apply method is used on baseDirectory because both baseDirectory and sourceDirectory are SettingKey[_]s. SBT distinguishes between three types of Setting[_] values: Setting, Task and InputTask with corresponding SettingKey, TaskKey and InputKey "name" types. The three are distinguished as follows:

SettingKey - Something that is computed *once* on project load (or reload). like a val.
TaskKey - Something that is computed each time it is called, like a def.
InputKey - Something that takes user input to perform its task.

Configurations

SBT uses a configuration matrix to define the same task against different configurations. For example, SBT defines a task for compiling Scala code called compile. This mechanism has a bunch of required settings. However, it wouldn't be DRY to repeat all these settings for compiling *test* code as well. So instead, SBT defines the *same* settings in two different configurations, on called Test and another called Compile. To compile just tests in SBT, you can prefix a task with its configuration, e.g. test:compile.

Plugin Design

So what does this have to do with plugin design? SBT plugins need to integrate Setting[_] values into a build without conflicting with SBT default settings and other plugins. To complicate matters, SBT imports all the members of plugin classes into scope of a project using a wildcard import. This means all the plugins you use could have conflicting names that step on each other. Combined with potentially conflicting key names, plugins need to be very careful with how they define things.

Having worked on several plugins recently, I'd like to outline a strategy that I think achieves a certain elegance in definition and usage, as well as the safety one wants from a plugin.

The basic pattern is as follows. Define an object with the name you want for your plugin *inside* the Plugin class. For example, if I want a xsbt-suereth-plugin, I would define the following:

import sbt._
import Keys._

object SbtSuerethPlugin extends Plugin {

  object suereth {
     // Your code here
  }
}

Inside of the suereth object I hide all my definitions and code. This isolates my plugin from other sbt plugins, as long as no one names their methods "suereth".

Next, let's define a new Config object that we can use to protect our keys from other plugins.

  object suereth {
     val Config = config("suereth")
     // Your settings here
  }

The Config also shares the name of the plugin, so in the command line tasks and settings can be run using suereth:<your-task-here>.

The next step is define whatever custom keys your plugin will use. Let's create a blog key.

  object suereth {
     ...
     val blog = SettingKey[String]("blog", "location of the blag") in Config
  }

The key is automatically placed into the suereth configuration using the in method. This has two benefits:

When defining the Initialization for a Setting[_], there's no need to continue writing blog in Config.
Users of the plugin can directly access suereth.blog without needing to specify suereth.blog in suereth.Config.

Note: You can also reference SBT keys in your configuration by writing: val sources = Keys.sources in Config.

Finally, we can provide default values/computations for tasks and settings in our plugin. By convention, calling these settings is a good idea.

  object suereth {
     ...
     lazy val settings: Seq[Setting[_]] = Seq(
        blog := "http://suereth.blogspot.com"
     )
  }

Notice how the keys are access directly but are actually in the appropriate config matrix. This helps defining your plugin source code, but will also help users of your plugin. Let's look at what a build.sbt file will be for this plugin.

seq(suereth.settings:_*)

suereth.blog := "http://blog.typesafe.com"

Notice how the settings for this plugin are completely namespaced by the suereth object. We've tied the concept of a "configuration" axis for keys with accessing values in an object.

I find this mechanism of defining plugins both helpful from a development perspective and a user perspective. Curious to hear what others think.

Leaving Google for Typesafe

2011-07-22T15:46:00.001-04:00

So, as some of you may have found out already, I'll be leaving Google effective July 22nd and moving to a new role at Typesafe. Some of you may be wondering why, so I'm offering my reasons in a blog post for the curious:

Scala has been gaining lots of traction in the market space recently. I think Scala is and has been reaching its tipping point. I'm hoping to contribute everything I can to sure it remains a viable language for everyday development.
While Google does have 20%time to work on individual projects, this correlated to about a day a week to work on all things Scala within Google. Rare was the opportunity to do anything significant or to contribute back to the community, due to time constraints. While I was able to accomplish one pretty good thing, aligning my work with my passions seemed like the best thing to do.
It's part of a new weight loss program called "pay for your own food". This was tempered by my wife's purchase of a new grill for my birthday, which has seen heavy use in the past few weeks.

So, while I see Scala's future as bright and rosy and I can't explain how excited I am to start at typesafe, I'd also like to do a little reflection on Google and some of what makes it a great company.

Google cares about employees
This is not a lie. Google, as best as it can, tries to mean what it says. At my previous companies you'd hear "I'd love to do this, but my hands are tied". At Google, you often hear "I don't know, but let's ask someone who does" which usually turns into "Yes, go ahead". It was rare that something I asked for was denied. My *perception* of what was acceptible is an entirely different thing. At Google, there are people who think what you think (at over 10k engineers, it's guaranteed one will probably agree with you about something), and that engineer might have also tried to accomplish the same task as you. Go ask about it and find like-minded individuals. You'll be surprised how much Google tries to do what they say.
The other side to this coin is that the *corporate entity* and by that I mean the higher ups like to do things for all Google employees. Like bonuses. They really do happen, and they really do try to treat you like a human being.

The other side to this coin is the interview process. I've seen so many negative posting about the interview process. Well, I'm here to dispell a few of those. Google's interviews will challenge your technical abilities. If this annoys you and you don't take the job, then good. You're probably not the type to ~~herd cats~~ deal with politics/relationships involves with a large developer base. If you can't cut the interview, then it gives you a reason to go back and practice. However, more importantly, Google would rather turn away a good candidate than hire a bad one. This mindset is very impressive. It really makes a huge difference on the ability of teams to accomplish things. I knew every employee I worked with at google was reliable to get work done, not something that's always the case externally.
In the future I know my interview style will change. No more will interviewees be allowed to just talk about experiences without showing some code. It's amazing some of the depths you can learn here. It's even great when an interviewee fails to answer the question correctly because you *learn their thought process*. I know for me, a few candidates who struggled were the ones I wanted sitting beside me coding, more so than the ones who blazed through a question but had a more 'better than thou' aura.
So, the point here is care about your employees and care about who you hire. Your company will be in far better shape if you do this.

Culture is faster than process
Google tries to instill a culture of 'doing the right thing' its employees, rather than outlining software process to a T. There are a few 'inconveniences' that exist, but other than mandatory code reviews, a lot of the process is up to the team to do what's best. The other side to this coin is the corporate culture tries to help define and change what's best. It's amazing how one executive making a statement in an all hands meeting can suddenly alter the perceived "best way to code" and get the company to move. It's also surprisingly hard to change culture once it has been really embedded into the engineers, which is the danger. A sense of 'right' in the ways of writing software can be beneficial, but can also turn into anti-patterns if not tempered. I'd love to go into more details here, just feel free to bug me.

A Generic Quicksort in Scala

2011-06-16T23:41:00.002-04:00

So, I decided to create a quicksort algorithm in Scala that showcases how to write 'generic' collection methods. That is, how can we write an external method that works across many types of collections *and* preserves the final type.

Well, here's how you do it:

import scala.collection.SeqLike
import scala.collection.generic.CanBuildFrom
import scala.math.Ordering

object QuickSort {
  def sort[T, Coll](a: Coll)(implicit ev0 : Coll <:< SeqLike[T, Coll],
                             cbf : CanBuildFrom[Coll, T, Coll],
                             n : Ordering[T]) : Coll = {
    import n._
    if (a.length < 2)
      a
    else {
      // We pick the first value for the pivot.
      val pivot = a.head
      val (lower : Coll, tmp : Coll) = a.partition(_ < pivot)
      val (upper : Coll, same : Coll) = tmp.partition(_ > pivot)
      val b = cbf()
      b.sizeHint(a.length)
      b ++= sort[T,Coll](lower)
      b ++= same
      b ++= sort[T,Coll](upper)
      b.result
    }
  }
}

I've chosen a somewhat imperative approach to the problem. The quick sort algorithm is split into two parts: The first checks for small collections and returns them, the second picks a pivot and decomposes the collection into three pieces. Theses pieces are sorted (if necessary) and pushed into a builder, cleverly named "b". This "b" is given a hint to expect the entire collection to eventually wind up in the built collection (hopefully this helps performance). Finally, after passing the three partitions to the builder, the result is returned.

The magic here is in the rather confusing type signature:

def sort[T, Coll](a: Coll)(implicit ev0 : Coll <:< SeqLike[T, Coll],
                             cbf : CanBuildFrom[Coll, T, Coll],
                             n : Ordering[T]) : Coll

Let's decompose this a bit. T is the type parameter representing elements of the collection. T is required to have an Ordering in this method (the implicit n : Ordering[T] parameter in the second parameter list). The ordering members are imported on the first line of the method. This allows the < and > operations to be 'pimped' onto the type T for convenience.

The second Type parameters is Coll. This is the concrete Collection type. Notice that *no type bounds are defined*. It's a common habit for folks new to Scala to define generic collection parameters as follows: Col[T] <: Seq[T]. Don't. This type does not quite mean what you want. Instead of allowing any subtype of sequence, it only allows subtypes of sequence that *also* have type parameters (which of course, is most collections). Where you can run into issues is if your collection has (a) no type parameters or (b) more than one type parameter. For example:

object Foo extends Seq[Int] {...}
trait DatabaseResultSetWalker[T, DbType] extends Seq[T] {...}

Both of these will fail type checkking when trying to pass them into a method taking Col[T] >: Seq[T].

To get the compiler to infer the type parameter on the lower bound, we have to defer the type inferencer long enough for it to figure this out. To do that, we don't enforce the type constrait until implicit lookup using the >:> class.

The type parameter:

Coll <:< SeqLike[T, Coll]

Ensures that the type Coll is a valid Seq[T]. You may be asking why this signature uses SeqLike rather than Seq.

GOOD QUESTION!

SeqLike differs from Seq in that it *retains the most specific type of the sequence*. This one of the magic tricks behind Scala's collections always returning the most specific type known. That type is embedded in SeqLike. To ensure that we can return the most sepcific type, we can Capture Coll as a SeqLike with Coll as the specific type. This means that filters, maps, flatMaps, partitions should all try to preserve the type Coll.

The last implicit parameter is the cbf CanBuildFrom. Because we don't know how to construct instances of type Coll (because we don't know the type Coll at all), we need to implicitly receive evidence for how to construct a new Coll with sorted data.

Let's look at the result:

scala> QuickSort.sort(Vector(56,1,1,8,9,10,4,5,6,7,8))
res0: scala.collection.immutable.Vector[Int] = Vector(1, 1, 4, 5, 6, 7, 8, 8, 9, 10, 56)

scala> QuickSort.sort(collection.mutable.ArrayBuffer(56,1,1,8,9,10,4,5,6,7,8))
res1: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 1, 4, 5, 6, 7, 8, 8, 9, 10, 56)

scala> QuickSort.sort(List(56,1,1,8,9,10,4,5,6,7,8))
res18: List[Int] = List(1, 1, 4, 5, 6, 7, 8, 8, 9, 10, 56)

scala> QuickSort.sort(Seq(56,1,1,8,9,10,4,5,6,7,8))
res: Seq[Int] = List(1, 1, 4, 5, 6, 7, 8, 8, 9, 10, 56)

You may be asking why I've chosen Seq instead of GenSeq or GenTraversable or even GenIterable. No particular reason, besides I wanted a reasonable assurance that the collection author expects the .length and indexed access methods to be called.

So, what are the lessons to be learned here?

(1) Use *Like subclasses to preserve the specific collection type
(2) Defer inference using <:< to give the type checker a hope of succeeding
(3) Provide @usecase comments for scaladoc so users won't get distracted by typesafe details.

At lest, IMHO, this is the current way of creating generic collection code.

Scalatypes Podcasting Starting!

2011-06-12T00:48:00.000-04:00

Well, Daniel, Yuvi and I started podcasting interviews and discussions on Scala. Here's the first interview with Paul Phillips. We have a *lot* more content to publish, but mastering audio still takes a backseat to day jobs and writing Scala in Depth. Let us know what you think!

Scaladays 2011 Talk: Parallel Distributed Collections API

2011-06-03T16:48:00.002-04:00

Annotate your type classes (@implicitNotFound)

2011-03-15T23:39:00.000-04:00

This is a real quick post saying that as of 2.8.1, everyone should be annotating their type class traits for better error messages. Obviously, those of us who are supporting 2.8.0 or earlier are left out.

Here's an example REPL session:

Welcome to Scala version 2.9.0.r24384-b20110305105029 (OpenJDK 64-Bit Server VM, Java 1.6.0_20).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import annotation.implicitNotFound
import annotation.implicitNotFound

scala> @implicitNotFound(msg = "Cannot find Serializable type class for ${T}") trait Serializable[T]
defined trait Serializable

scala> def foo[X : Serializable](x : X) = x
foo: [X](x: X)(implicit evidence$1: Serializable[X])X

scala> foo(5)
:11: error: Cannot find Serializable type class for Int
       foo(5)
          ^

The annotation has very little documentation, but apparently supports ${TypeName} templates to inject the names of types the compiler is looking for. I believe you can also use this implicit on types with no type parameters.

Happy hacking everyone!

Slides for today's NEScala talk

2011-02-18T11:40:00.002-05:00

Here's the slides for todays talk for those who wish to follow along on their laptops.

On Macro-Composition vs. Micro-Composition and Social software development.

2010-10-15T22:49:00.001-04:00

I was talking to a co-worker today about using C++ templates to compose functionality in objects. I also mentioned how you can encode type traits using templates. My Coworker, conveniently aliased as "Bob", asked why you would want to use type traits rather than defining an interface directly and wrapper objects.

In response, I gave rather lame answers, which I intend to amend here. The basic gist is that I feel a lot of Object Oriented languages don't pay enough attention to macro composition of components in a system.

First, I'd like to state that In a lot of my code, there are a set of classes that define the format of data, or the "data layer". This layer has a certain "mind-share" around it. It is "owned" be a group of individuals and the portion of code they own. This data is passed around to other teams of people.

Note that I specifically state "other teams" and not "other components". In a lot of software shops, we think of software being comprised of sub-components of a system, but I tend to find that the sub-components of a system are also defined by the teams that work on them. If a particular team works on several sub-components, these components tend to be cohesive and reuse a lot of code. The boundaries between these teams are important to pay attention to.

It's the boundaries where we tend to do a lot of conversions from one data definition to another. Each team wants to wrap its own "needs" around the ontology of classes. So in an OO system, this usually means wrapping data from one set of classes to another, or bunch of external methods that manipulate the data from another team. Because of this natural bent in programming, it's sometimes necessary to be able to adapt other people's specific classes into your own ontology. This is where type classes come in.

I've blogged about type classes/type traits before, so I won't bore you again. The basic gist of the idea is to define an interface for your method. For example, I'm writing something that works with tree-like data structures where each parent node could have multiple children. I define a type trait as follows:

trait TreeLike[NodeType] {
     def isParent(node : NodeType) : Boolean
     def children(node : NodeType) : Iterator[NodeType]
  }

This allows me to take any "node" type and traverse it. Now, Let's write an algorithm where we want to see if a condition holds true down one "branch" of the tree to a leaf for every Node of this path. We can use our TreeLike type trait.

final def hasValidDepthPath[T : TreeLike](node : T, validator : T => Boolean) : Boolean =
  if(validator(node)) {
    val helper = implicitly[TreeLike[T]]
    !helper.isParent(node) || helper.children(node).exists(hasValidDepth(_, validator))
  } else false

The hasValidDepthPath method uses the TreeLike type trait rather than a different class. This lets us use non-treelike structures as trees. For example, we can turn a list into a tree where each branch is the same list but missing one item of the index.

scala> def dropIndex(x : List[Int], idx : Int) = x.zipWithIndex.filter(_._1 != idx).map(_._2) 
dropIndex: (x: List[Int],idx: Int)List[Int]

scala> dropIndex((1 to 10).toList, 2)
res2: List[Int] = List(0, 2, 3, 4, 5, 6, 7, 8, 9)

implicit val decomposeList[T] = new List[T] {
  def isParent(node : List[T]) = node.length != 1
  def children(node : List[T]) = (1 to node.length).map(dropIndex(node, _))
}

Now we can use a List of items as an input to our hasValidDepthPath method.

Now to my coworkers question: Why would I want to add this extra layer?

My answer is that it helps when working with "ontologies" from different teams. That is, each team has their own way of describing data. This mechanism allows our functions to mostly ubiquitously deal with these differing items, writing adapters for each ontology. This makes more sense when your algorithms deal with many different input types. In fact, I think it makes the most sense for 'peripheral' code. That is code that interacts with other teams.

Now, for the second part of the rant.

I feel a lot of languages don't support macrocomposition well. A great example of this is the existence of the Spring Framework. Spring allows you to define interfaces, objects and their dependent components. You can then build configurations where you create specific objects, bind them to names and inject them into other objects. You are essentially defining the components of your system at a micro->macro level. Spring configuration allows you to focus on building the various components as needed at an object-by-object level until you are configuring objects that comprise major subcomponents of your system. Google Guice allows you do to a similar thing in Java using annotations.

The important thing here, IMO is that there is a distinction between defining components/classes and composing them. We define a component that uses others via some interface in one section, and we combine these components together in another.

In C++ we can also accomplish this to some extent using type traits. In C++ type traits usually take the form as templated structures. Composition is then done using template arguments. The best example of this is the STL and its usage of traits for allocation, comparison, hashing, etc.

In C++, we could define a service as follows

template<typename Logger, typename Threading>
class Service {
 public:
   typename Threading::ScopeLock ScopeLock;
   void MyMethod() {
     Logger log;
     Threading threading;
     {
       ScopeLock scoped(mutex_);
       log.info("OMG!!!");
     } 
  }
 protected:
   typename Threading::Mutex mutex_;
};

This service is really really simple, but shows how to compose with type traits. The Threading interface is any class that defines a Mutex typedef (for mutexes), a ScopeLock typedef (for locking mutexes within a scope). This class could be reused across various threading libraries, by providing something with the correct type definitions. I would expect to see, somewhere else in code, the following "Compositional" Code:

// This code replaces a Spring XML configuration.
typedef Service<PthreadThreading, DefaultLogging> MyService;
typedef SomeHigherLevelService<...., MyService> MyHigherLevelService;

The nice aspect here is that I can use typedefs to compose "micro" components of my system and aggregate those into "macro" components. It's the same technique used for both.

In Scala, I've used the following tactic: Define a trait which implies composition. This method has a lot of boilerplate, so bear with me.

I want to define a Logger service that traits can use. To do this I define the generic Logger trait and a generic HasLogger trait.

trait SimpleLogger {
  def info(msg : String) : Unit
  def warn(msg : String) : Unit
  def fatal(msg : String) : Unit
}

trait HasSimpleLogger {
  val logger : SimpleLogger
}

This SimpleLogger provides no advanced features, just three levels of logging. The HasSimpleLogger trait defined an abstract value for the Logger. It does not instantiate the logger, it merely denotes that a class will have one (i.e. needs to be composed with one). Later, we will define some service:

trait MyService extends HasSimpleLogger {
  def serviceMethod() {
    logger.info("MyService: Someone called my method")
  }
}

This is how we define each "abstract" component. We extend each composition trait (i.e. HasFoo) and then utilize the components as necessary. Again, there will be code somewhere that composes these components together. In Scala, the mechanism of composition is the trait *not* a class instance:

trait MyComposedService extends MyService with HasDefaultLoggerImplementation

trait HasMyComposedService extends HasMyService {
  val mySerivce = new MyComposedService {}
}

Why do we have to compose in this manner? Because we'd like to have more control when composing large systems together. This mechanism allows us to "reconfigure" a MyComposedService with additional behavior or alternative subcomponents without having to redeclare *all* the subcomponents.

The downside to this approach is that it is full of boilerplate. However, notice again the distinction between defining components and aggregating behavior into our final system.

I'm of the opinion that these two functions are key to creating reusable software and they need to be simple and elegant to be used. I've yet to see a large system where I felt the composability ever reached the ideals touted by J2EE of old. In fact, I feel EJBs are horrible for microcomposition. The Spring framework further proved this by provided a microcomposition layer below Session beans that became quite popular.

In any case, these are my ramblings and musings on development. I'd love to hear back from you. I'm of the opinion that after becoming a proficient software developer, the solutions I write solve three issues:

(1) A complex new algorithm to solve a problem "never before solved" (at least by the company I work for), or some foundational library.
(2) Munging data between social components of software, or differing ontologies
(3) Composing components to form a complex system.

I feel a lot of languages focus on (1). The languages I tend to prefer allow me to do (2) and (3) with ease.

What's the state of Jigsaw?

2010-08-17T22:41:00.000-04:00

I hadn't been paying close attention, but recently a few emails questioning the future of the jigsaw project have been sent to the jigsaw-dev mailing list. The last check-in posted to the mailing list was on June 8th, and was the tail-end of a flurry of check-ins from the jigsaw team.

The whole thing is rather curious, but without any specific word from Oracle, it's all speculation. I have my own guesses, but it's probably best I keep those to myself for now.

Monkey Patching, Duck Typing and Type Classes

2010-07-13T00:17:00.000-04:00

Somewhat recently, there have been a bunch of articles written about Scala and how you can create something that looks like Haskell's type class feature. I'm not going to duplicate these posts, but I am going to go beyond just the mechanics of creating a type class and discuss how to utilise some of Scala's advanced features with type classes to create some pretty robust code. A lot of the post borrows ideas from my upcoming book, Scala In Depth. The examples come from the Scala ARM library.

First off, let's assume you haven't read anything at all about Scala's type classes, but you do have a Ruby background. In Ruby, you really loved duck typing. It's the best. I can just use methods and pass objects around and things work. If a class doesn't have a method I need, I can either rely on some method missing magic, or just monkey patch it. To those who don't know Ruby, monkey patching is the ability to add a method to an existing class on the fly.

What if I told you that in Scala, for a small penalty, you could create methods that are flexible, like duck typed methods, and 100% compile-time type safe. Not only that, you're also able to monkey-patch types as needed. Don't believe me? Let's look at some of the fun in the Scala ARM library.

The motivation behind the ARM library is to create a flexible, robust (from exceptions) and simple-to-use automated-resource-management library that can greatly simplify things like JDBC or I/O in Scala. The standard introduction to ARM in Scala is the withResource method outlined below.

def withResource[A <: { def close() }, B](resource : => A)(f : A => B) = {
  val r = resource
  try {
    f(r)
  } finally {
    r.close()
  }
}

The with resource method takes a by-name argument that returns an opened resource, and a function that manipulates the resource. We ensure that the close method on the resource is called after the manipulation function f. We've declared the type of the resource using structural typing. This means the compiler will accept anything that defines a close method. We can then utilize withResource as follows:

withResource(new FileInputStream(new File("myFile.txt"))) { stream =>
  // Code using stream and returning something
}

So, this code is already giving us a form of duck-typing (called structural typing in Scala). What about monkey patching? This is where using the type class pattern can help. Also note, that I prefer calling the type class pattern from haskell the "type trait" pattern in Scala. This could be confusing, as C++ has a type trait pattern as well, however the fact that I use traits to define type classes is just as confusing. In any case, please think of type traits and type classes, for the rest of this blog article as the same thing: A manifestation of type classes in Scala.

Let's modify the withResource method to use a type class instead of a structural type.

trait Resource[R] {
  def close(r : R) : Unit
}

Here is our Resource type class. It defines a single method close. We can now modify our withResource method to utilise this:

def withResource[A : Resource, B](resource : => A)(f : A => B) = {
  val r = resource
  try {
    f(r)
  } finally {
    implicitly[Resource[A]].close(r)
  }
}

Notice two things here. First, we use the context-bounds constraint on the resource type parameter (A : Context). This means that we have to look up our implicit type trait using the implicitly method. We could, instead, give our method a well-defined implicit. For now, we'll use the context-bound syntax as it's quickly starting to denote type traits in Scala.

If we attempt to call this method as things stnad, we'll get an implicit lookup error:

scala> withResource(new java.io.StringReader("HAI")) { input => () }
:9: error: could not find implicit value for evidence parameter of type Resource[java.io.StringReader]
       withResource(new java.io.StringReader("HAI")) { input => () }
                                                     ^

Now let's think through what we've done. Any type can be used in the withResource method as long as an implicit is available for the given type in the implicit lookup resolution. The simplest way to create an implicit for usage is to provide one directly. This is easy to do in the REPL:

scala> implicit object stringResourceTrait extends Resource[java.io.StringReader] {
         override def close(r : java.io.StringReader) = r.close()                    
       }
defined module stringResourceTrait

scala> withResource(new java.io.StringReader("HAI")) { input => () }

You can see that we've satisfied the type system for the particular type java.io.StringReader. Our old method was duck typed to use anything that had a close method. We can actually get this same functionality with type traits as well. Let's provide a default type trait in the Resource Companion object. To do so, let's create a resource.scala file that looks like the following:

trait Resource[R] {
  def close(r : R) : Unit
}
object Resource {
  implicit def genericResourceTrait[A <: { def close() : Unit }] = new Resource[A] {
    override def close(r : A) = r.close()
  }
}

What we've done is provide a method "genericResourceTrait" that is available on *every implicit search* for any Resource[_] type. We've now gotten back to our previous functionality, and we can take this a bit farther. What if we wanted to provide a specific implementation for a type? In the case of resources, Java provides a java.io.Closeable interface that most IO classes implement. Creating a type trait implementation for this will give us the benefit of not needing to use reflection when we know a specific type. Let's take our first crack at it.

trait Resource[R] {
  def close(r : R) : Unit
}
object Resource {
  implicit def genericResourceTrait[A <: { def close() : Unit }] = new Resource[A] {
    override def close(r : A) = r.close()
    override def toString = "Resource[{def close() : Unit }]"
  }
  implicit def jioResourceTrait[A <: java.io.Closeable] = new Resource[A] {
    override def close(r : A) = r.close()
    override def toString = "Resource[java.io.Closeable]"
  }
}

We now have a duck typed, reflection based implementation and an implementation for the java.io.Closeable interface. We can prove that this works via the REPL by creating a mechanism of checking what type trait is used:

scala> def showTypeTraitUsed[A : Resource](a : A) = println(implicitly[Resource])
showTypeTraitUsed: [A](a: A)(implicit evidence$1: Resource[A])Unit

scala> showTypeTraitUsed(new java.io.StringReader("HAI"))
Resource[java.io.Closeable]

scala> showTypeTraitUsed(new { def close() : Unit = () })
Resource[{def close() : Unit }]

We see that when passed a java.io.StringReader, something that subclasses java.io.Closeable, it uses the type trait defined for closeables. When we pass an anonymous object that has a close method, it uses the structural-typed type trait instead. What about monkey patching? Well, using type traits, you can achieve a form of monkey patching. Let's pretend an integer is a resource. All we have to do is create a new type trait implementation for it. Let's make that happen:

scala> implicit object intAsResource extends Resource[Int] {
     | override def close(r : Int) = println("closing: " + r)
     | }
defined module intAsResource

scala> withResource(5)(println)
5
closing: 5

We define our close method to print out a string, so we can make sure it happens. Finally, we can use an integer in our withResource method as if it were a resource, hence monkey-patching a close method onto it. In fact, we could also use this mechanism to change the implementation of the Resource type trait in a particular context. The key to this, as described in my book, is to ensure that you place your default implicitly available type trait implementation into the correct scope, such that

They can be overridden
They do not conflict

We can also use certain Scala tricks like having method return types be based on implicits available within scope. Combining these all, you can create some rather interesting libraries, such as what is available in scala arm's typeclass-fix branch. This branch contains a rather interesting flatMap that can detect if you are mapping to a Traversable type, and return a Traversable. It will also detect if you are flatMapping to another Resource and return a new ManagedResource that will open/close that nested resource. Finally, it also supports the standard flatMap implementation.Working with type classes in Scala adds a bit of overhead in amount of code, and some time to toy with the type system but can provide great flexibility in design. I'm working on a library for interactive with Nexus, using webDAV protocol. Using a "FileLike" type trait has been helpful in the design, especially since certain methods can be optimised in particular algorithms where we know both types. It's also useful because I've adapter several different WebDAV-enabled libraries to the API before finally deciding on the library I disliked the least.So in recap, let's look at why the typeclass pattern is so powerful in Scala:

Can provide implementations for many differing types, in hierarchical fashion
Can provide new type trait implementation from any scope
Allows at-usage-site overriding of implementation details

Comcast Privacy Fail

2010-06-21T23:16:00.000-04:00

Wow, Comcast just amazed me again. I haven't had a good rant post on here in a *long* time, but this is totally deserving of the biggest rant ever. This is the transcript from an ONLINE chat my friend participated in (props goes to my friend for sharing). I tried to scrub this somewhat clean, but perhaps there's enough here for Comcast to figure out internally who made the blunder and who's account was compromised.

user John_ has entered room

John(Mon Jun 21 2010 21:15:04 GMT-0400 (Eastern Daylight Time))>

I need a password for my comcast.net address

analyst Ricky has entered room

Ricky(Mon Jun 21 2010 21:15:06 GMT-0400 (Eastern Daylight Time))>

Hello John_, Thank you for contacting Comcast Live Chat Support. My name is Ricky. Please give me one moment to review your information.

Ricky(Mon Jun 21 2010 21:15:17 GMT-0400 (Eastern Daylight Time))>

My pleasure to have you on this chat, John! I always remain committed and focused to provide you quality customer service at my fullest effort. Before anything else, I want to extend apologies for any trouble, inconvenience and frustration login issue has brought along your way. I still honestly hope you're fine.

Ricky(Mon Jun 21 2010 21:15:17 GMT-0400 (Eastern Daylight Time))>

A comcastic day to you! How are you doing today?

John_(Mon Jun 21 2010 21:15:56 GMT-0400 (Eastern Daylight Time))>

I've had better evenings Ricky. The comcast system is a bit screwed up

Ricky(Mon Jun 21 2010 21:16:22 GMT-0400 (Eastern Daylight Time))>

I am very sorry for that, John.

Ricky(Mon Jun 21 2010 21:16:27 GMT-0400 (Eastern Daylight Time))>

I understand that you need your user name and password to access your account online, am I correct?

John_(Mon Jun 21 2010 21:16:58 GMT-0400 (Eastern Daylight Time))>

sort of.

John_(Mon Jun 21 2010 21:17:14 GMT-0400 (Eastern Daylight Time))>

i have a user name and password that i have used to log in and pay my bill for 2 years now

John_(Mon Jun 21 2010 21:17:25 GMT-0400 (Eastern Daylight Time))>

apparently that's not good enough to log in and download norton antivirus

John_(Mon Jun 21 2010 21:17:38 GMT-0400 (Eastern Daylight Time))>

so now i need *another* one. I understand it's supposed to be a comcast.net address

Ricky(Mon Jun 21 2010 21:17:16 GMT-0400 (Eastern Daylight Time))>

I see, let me go ahead and assist you with that.

Ricky(Mon Jun 21 2010 21:17:19 GMT-0400 (Eastern Daylight Time))>

I do apologize for the inconvenience this issues has brought you,John. No worries as your service repsentative I will make sure that we get this issues resolved today.

Ricky(Mon Jun 21 2010 21:17:29 GMT-0400 (Eastern Daylight Time))>

As your Comcast service repsentative, I want you to know that issue resolution and your satisfaction are my top priorities for today. Together, we can work this out, John.

Ricky(Mon Jun 21 2010 21:17:30 GMT-0400 (Eastern Daylight Time))>

Before we proceed resolving your issue, I need to verify some information in order for me to pull up your account. That is to insure the integrity of your account and for me to be able to assist you further. May I please have the account holder's full name, the account number, the complete billing address, and the last four digits of the account holder's SSN?

John_(Mon Jun 21 2010 21:18:08 GMT-0400 (Eastern Daylight Time))>

I was able to get the email address from the system, but since i don't have a secret question, it wouldn't let me reset the password

Ricky(Mon Jun 21 2010 21:18:34 GMT-0400 (Eastern Daylight Time))>

Let me go ahead and provide you your log ins so that you will not have issue accessing the account, okay?

Ricky(Mon Jun 21 2010 21:20:11 GMT-0400 (Eastern Daylight Time))>

John_(Mon Jun 21 2010 21:21:20 GMT-0400 (Eastern Daylight Time))>

John C Awesome 123 Amazing ST S Cool Town, USA 13124. xxxx

John_(Mon Jun 21 2010 21:22:00 GMT-0400 (Eastern Daylight Time))>

xxxxxxxxxxxxxxx is an

Ricky(Mon Jun 21 2010 21:21:49 GMT-0400 (Eastern Daylight Time))>

Thank you for helping me verify the account.

Ricky(Sun May 21 2010 21:21:52 GMT-0400 (Eastern Daylight Time))>

Please allow me two to three minutes to pull up your account information so that we can resolve it in the most efficient way possible. Would that be okay with you?

Ricky(Mon Jun 21 2010 21:25:11 GMT-0400 (Eastern Daylight Time))>

By the way, as a Comcast customer, I would like to tell you about the Service Protection Plan (SPP) we are offering. For a few dollars per month, you can have worry free protection for any in-home wiring problem. This is also simple and convenient, any problem with any Comcast service will be taken care of without any additional fees. Would you like to add it on your account today?

Ricky(Sun May 21 2010 21:26:03 GMT-0400 (Eastern Daylight Time))>

Thank you for patiently waiting.

John_(Mon Jun 21 2010 21:26:29 GMT-0400 (Eastern Daylight Time))>

def not Ricky

Ricky(Mon Jun 21 2010 21:26:25 GMT-0400 (Eastern Daylight Time))>

I see here that you have a comcast user name: johnawesome@comcast.net

Ricky(Mon Jun 21 2010 21:26:43 GMT-0400 (Eastern Daylight Time))>

May I ask if you have the password.

John_(Mon Jun 21 2010 21:28:24 GMT-0400 (Eastern Daylight Time))>

hmm, that doesn't look right.

John_(Mon Jun 21 2010 21:28:44 GMT-0400 (Eastern Daylight Time))>

I got the email address from the system and it said it was something else...let me doublecheck that

Ricky(Mon Jun 21 2010 21:28:38 GMT-0400 (Eastern Daylight Time))>

Yes please.

John_(Mon Jun 21 2010 21:29:55 GMT-0400 (Eastern Daylight Time))>

I went to this link: http://www.idhelp.comcast.net/display/4n/search.asp?tab=search and clicked on "I do not remember my Comcast user name or email address"

John_(Mon Jun 21 2010 21:30:22 GMT-0400 (Eastern Daylight Time))>

It then asked for my phone number. When i put that in, it said my email address was JohnAEWsome@comcast.net

Ricky(Mon Jun 21 2010 21:30:03 GMT-0400 (Eastern Daylight Time))>

I see, that is the registered e mail on your account, John.

John_(Mon Jun 21 2010 21:31:22 GMT-0400 (Eastern Daylight Time))>

what is the email address?

Ricky(Mon Jun 21 2010 21:31:19 GMT-0400 (Eastern Daylight Time))>

I see here that you have a comcast user name: johnawesome@comcast.net

John_(Mon Jun 21 2010 21:32:01 GMT-0400 (Eastern Daylight Time))>

your system says that's not correct

John_(Mon Jun 21 2010 21:32:15 GMT-0400 (Eastern Daylight Time))>

it's telling me it's johnaewsome@comcast.net

Ricky(Mon Jun 21 2010 21:32:25 GMT-0400 (Eastern Daylight Time))>

I am very sorry but that is the e mail address that I am seeing on your account.

John_(Mon Jun 21 2010 21:34:18 GMT-0400 (Eastern Daylight Time))>

wonderful, well i don't have the password to that one either

Ricky(Mon Jun 21 2010 21:34:19 GMT-0400 (Eastern Daylight Time))>

Just to set your expectation, we do not have the old password on your email,John what can I do here on my end is to reset it into a new password, would that be okay?

John_(Mon Jun 21 2010 21:35:43 GMT-0400 (Eastern Daylight Time))>

that's fine

Ricky(Mon Jun 21 2010 21:36:46 GMT-0400 (Eastern Daylight Time))>

Thank you!

Ricky(Mon Jun 21 2010 21:36:47 GMT-0400 (Eastern Daylight Time))>

By the way, there will be a question survey after you click on the END SESSION button, please do take the survey this will be my supervisors basis of my performance, Thank you

Ricky(Mon Jun 21 2010 21:37:27 GMT-0400 (Eastern Daylight Time))>

Would you mind if I ask you the security question?

John_(Mon Jun 21 2010 21:38:00 GMT-0400 (Eastern Daylight Time))>

sure, but i can't guarantee i know the answer

Ricky(Mon Jun 21 2010 21:38:07 GMT-0400 (Eastern Daylight Time))>

Thank you!

Ricky(Mon Jun 21 2010 21:38:08 GMT-0400 (Eastern Daylight Time))>

Where did you go on your honeymoon?

John_(Mon Jun 21 2010 21:38:40 GMT-0400 (Eastern Daylight Time))>

i'm not married

Ricky(Mon Jun 21 2010 21:38:49 GMT-0400 (Eastern Daylight Time))>

I am very sorry but that is the registered security question on your account.

John_(Mon Jun 21 2010 21:39:20 GMT-0400 (Eastern Daylight Time))>

ricky i don't know what to tell you

Ricky(Mon Jun 21 2010 21:39:03 GMT-0400 (Eastern Daylight Time))>

May I ask for the four digit phone security pin instead.

John_(Mon Jun 21 2010 21:39:45 GMT-0400 (Eastern Daylight Time))>

i've never been asked that ricky

Ricky(Mon Jun 21 2010 21:39:31 GMT-0400 (Eastern Daylight Time))>

I see.

Ricky(Mon Jun 21 2010 21:39:31 GMT-0400 (Eastern Daylight Time))>

Since you are not able to provide me the security pin, a comcast repsentative will call you for verification. Please stay online while the call is in progress.

John_(Sun May 21 2010 21:40:06 GMT-0400 (Eastern Daylight Time))>

are you really sure you want me to take that survey ricky?

John_(Mon Jun 21 2010 21:40:22 GMT-0400 (Eastern Daylight Time))>

cause i'm getting real tired of spending my night dealing with comcast's crap systems

Ricky(Mon Jun 21 2010 21:40:49 GMT-0400 (Eastern Daylight Time))>

I am very sorry for that, John.

Ricky(Mon Jun 21 2010 21:41:00 GMT-0400 (Eastern Daylight Time))>

Please advise me if you have received the call.

John_(Mon Jun 21 2010 21:41:59 GMT-0400 (Eastern Daylight Time))>

i ahven't

John_(Mon Jun 21 2010 21:42:02 GMT-0400 (Eastern Daylight Time))>

*haven't

Ricky(Mon Jun 21 2010 21:41:57 GMT-0400 (Eastern Daylight Time))>

We are calling you on your comcast phone right now, John.

John_(Mon Jun 21 2010 21:42:32 GMT-0400 (Eastern Daylight Time))>

my comcast phone?

John_(Mon Jun 21 2010 21:42:35 GMT-0400 (Eastern Daylight Time))>

I don't have a comcast phone

John_(Mon Jun 21 2010 21:42:38 GMT-0400 (Eastern Daylight Time))>

i only have internet

Ricky(Mon Jun 21 2010 21:42:18 GMT-0400 (Eastern Daylight Time))>

Yes, John,

John_(Mon Jun 21 2010 21:42:50 GMT-0400 (Eastern Daylight Time))>

why is your system so wrong?

Ricky(Mon Jun 21 2010 21:43:02 GMT-0400 (Eastern Daylight Time))>

I am very sorry for that, John.

Ricky(Mon Jun 21 2010 21:43:23 GMT-0400 (Eastern Daylight Time))>

Ia m very sorry for the confusion.

John_(Mon Jun 21 2010 21:43:55 GMT-0400 (Eastern Daylight Time))>

I'm not confused at all Ricky

John_(Mon Jun 21 2010 21:44:15 GMT-0400 (Eastern Daylight Time))>

In fact, i'm seeing things quite clearly at the moment.

Ricky(Mon Jun 21 2010 21:44:06 GMT-0400 (Eastern Daylight Time))>

Please take note of your password, John. You will be needing this regularly.

Ricky(Mon Jun 21 2010 21:44:11 GMT-0400 (Eastern Daylight Time))>

Here is your password: xxxxxxxx (all in lower case)

Ricky(Mon Jun 21 2010 21:44:16 GMT-0400 (Eastern Daylight Time))>

To make sure your issue is resolved, can you check for me if the log ins work, John?

Ricky(Mon Jun 21 2010 21:44:17 GMT-0400 (Eastern Daylight Time))>

Please advise me if you were able to log in successfully.

John_(Mon Jun 21 2010 21:45:52 GMT-0400 (Eastern Daylight Time))>

Ricky, you gave me access to someone else's account.

Ricky(Mon Jun 21 2010 21:45:57 GMT-0400 (Eastern Daylight Time))>

That is your account, John.

John_(Mon Jun 21 2010 21:46:23 GMT-0400 (Eastern Daylight Time))>

Now, why would you ask my address and account info if you were just going to ignore all of that

John_(Mon Jun 21 2010 21:46:39 GMT-0400 (Eastern Daylight Time))>

Ricky(Mon Jun 21 2010 21:46:24 GMT-0400 (Eastern Daylight Time))>

let me double check that, John. I am very sorry for that.

John_(Mon Jun 21 2010 21:46:49 GMT-0400 (Eastern Daylight Time))>

don't be sorry

John_(Mon Jun 21 2010 21:46:52 GMT-0400 (Eastern Daylight Time))>

for me

John_(Mon Jun 21 2010 21:46:57 GMT-0400 (Eastern Daylight Time))>

be sorry for this other John Awesome

John_(Mon Jun 21 2010 21:46:59 GMT-0400 (Eastern Daylight Time))>

who lives at ervice address
Mr John Awesome
12345 xxxxxxx rd
xxxxxxx, NOT MY STATE xxxxx-xxxxx

John_(Sun May 21 2010 21:47:05 GMT-0400 (Eastern Daylight Time))>

and account number is : xxxxxxxxxxx

John_(Mon Jun 21 2010 21:47:15 GMT-0400 (Eastern Daylight Time))>

none of that is close to what i provided above

Ricky(Mon Jun 21 2010 21:47:36 GMT-0400 (Eastern Daylight Time))>

May I ask for the phone number on the account please.

John_(Mon Jun 21 2010 21:48:06 GMT-0400 (Eastern Daylight Time))>

Whose account?

John_(Mon Jun 21 2010 21:48:12 GMT-0400 (Eastern Daylight Time))>

Mine or this other guys?

John_(Mon Jun 21 2010 21:48:24 GMT-0400 (Eastern Daylight Time))>

Who is extremely lucky i'm a nice person and am not going to screw his account up

Ricky(Mon Jun 21 2010 21:48:06 GMT-0400 (Eastern Daylight Time))>

May I ask for the account number on your account please.

John_(Mon Jun 21 2010 21:48:33 GMT-0400 (Eastern Daylight Time))>

Scroll up and look at it

Ricky(Mon Jun 21 2010 21:48:52 GMT-0400 (Eastern Daylight Time))>

I am very sorry but I was not able to pull up your account using the information you have provided.

Ricky(Mon Jun 21 2010 21:49:02 GMT-0400 (Eastern Daylight Time))>

May i ask for the phone number on the account pleas.e

John_(Mon Jun 21 2010 21:49:32 GMT-0400 (Eastern Daylight Time))>

XXX-XXX-XXXX

Ricky(Mon Jun 21 2010 21:49:36 GMT-0400 (Eastern Daylight Time))>

Thank you!

Ricky(Mon Jun 21 2010 21:50:02 GMT-0400 (Eastern Daylight Time))>

Can you please check the address, for the address that appears here on my end is 123 AMAZING ST S COOL TOWN, USA 13124

John_(Mon Jun 21 2010 21:51:19 GMT-0400 (Eastern Daylight Time))>

yes, that is remarkably similar to the one i provided up above. An Exact duplicate one could say.

Ricky(Mon Jun 21 2010 21:51:24 GMT-0400 (Eastern Daylight Time))>

Thank you for verifying,

Ricky(Mon Jun 21 2010 21:51:44 GMT-0400 (Eastern Daylight Time))>

I see here that your user name is JOHNAEWSOME@comcast.net

John_(Mon Jun 21 2010 21:52:09 GMT-0400 (Eastern Daylight Time))>

No shit

John_(Mon Jun 21 2010 21:52:12 GMT-0400 (Eastern Daylight Time))>

that's what i told you

Ricky(Mon Jun 21 2010 21:51:55 GMT-0400 (Eastern Daylight Time))>

Let me go ahead and reset the password, okay?

Ricky(Mon Jun 21 2010 21:52:21 GMT-0400 (Eastern Daylight Time))>

By the way, I see here that Comcast Digital Voice is available in your area. Comcast's Digital Voice® uses advanced fiber optic network to provide reliable home phone service. Would you like to take advantage of this service?

John_(Mon Jun 21 2010 21:52:42 GMT-0400 (Eastern Daylight Time))>

my question to you is what are you going to do for this other John Awesome? Just wait for him to call up and say that he can't access his account?

Ricky(Mon Jun 21 2010 21:52:26 GMT-0400 (Eastern Daylight Time))>

Please take note of your password, John. You will be needing this regularly.

John_(Mon Jun 21 2010 21:52:47 GMT-0400 (Eastern Daylight Time))>

hell no

John_(Mon Jun 21 2010 21:52:50 GMT-0400 (Eastern Daylight Time))>

damnit ricky pay attention

Ricky(Mon Jun 21 2010 21:52:34 GMT-0400 (Eastern Daylight Time))>

Here is your password: xxxxxxxx (all in lower case)

Ricky(Mon Jun 21 2010 21:52:42 GMT-0400 (Eastern Daylight Time))>

I am very sorry for that, John.

Ricky(Mon Jun 21 2010 21:52:47 GMT-0400 (Eastern Daylight Time))>

I am, John.

John_(Mon Jun 21 2010 21:53:15 GMT-0400 (Eastern Daylight Time))>

i've been screwing with comcast for 30 minutes now, you've compromised a user, and your'e trying to upsell?

John_(Mon Jun 21 2010 21:53:18 GMT-0400 (Eastern Daylight Time))>

what's the matter with you?

Ricky(Mon Jun 21 2010 21:53:25 GMT-0400 (Eastern Daylight Time))>

I am very sorry John, there was a system glitch and I was able to pull the wrong account.

John_(Mon Jun 21 2010 21:54:00 GMT-0400 (Eastern Daylight Time))>

a system glitch?

John_(Mon Jun 21 2010 21:54:06 GMT-0400 (Eastern Daylight Time))>

that's crap

John_(Mon Jun 21 2010 21:54:26 GMT-0400 (Eastern Daylight Time))>

there were 4 opportunities to catch the mistake. there were not system glitches

Ricky(Mon Jun 21 2010 21:54:07 GMT-0400 (Eastern Daylight Time))>

To make sure your issue is resolved, can you check for me if the log ins work, John?

Ricky(Mon Jun 21 2010 21:54:21 GMT-0400 (Eastern Daylight Time))>

I understand you are frustrated, however I need to ask you to please refrain from using abusive language so that I may continue assisting you. Keeping our dialogue professional and courteous will allow me to better serve you. If you continue to use inappropriate language, I will be forced to end the chat session.

John_(Mon Jun 21 2010 21:55:07 GMT-0400 (Eastern Daylight Time))>

actually, what's a number i can use to call your manager

Ricky(Mon Jun 21 2010 21:55:30 GMT-0400 (Eastern Daylight Time))>

You can call my manager at this number: 1-888-266-2278

Ricky(Mon Jun 21 2010 21:56:04 GMT-0400 (Eastern Daylight Time))>

Just to recap. I gave you your Comcast email user name and password. Would there be anything else I can assist you with?

John_(Mon Jun 21 2010 21:57:51 GMT-0400 (Eastern Daylight Time))>

No, there's nothing that you can personally do Ricky.

Ricky(Mon Jun 21 2010 21:57:47 GMT-0400 (Eastern Daylight Time))>

Were you able to log in, John.

John_(Mon Jun 21 2010 21:58:42 GMT-0400 (Eastern Daylight Time))>

Yes i was able to. Also, the number you gave me is the main comcast line

John_(Mon Jun 21 2010 21:58:54 GMT-0400 (Eastern Daylight Time))>

is your manager a automated phone system?

Ricky(Mon Jun 21 2010 21:58:55 GMT-0400 (Eastern Daylight Time))>

That is correct, John. For that is the only way you can contact our manager.

John_(Mon Jun 21 2010 21:59:26 GMT-0400 (Eastern Daylight Time))>

so how do i reference this chat?

John_(Mon Jun 21 2010 21:59:32 GMT-0400 (Eastern Daylight Time))>

or speak to a person?

Ricky(Mon Jun 21 2010 21:59:26 GMT-0400 (Eastern Daylight Time))>

I will provide you my ID for your reference, John.

John_(Mon Jun 21 2010 21:59:55 GMT-0400 (Eastern Daylight Time))>

that would be useful

Ricky(Mon Jun 21 2010 21:59:54 GMT-0400 (Eastern Daylight Time))>

Here is my ID: XXXXXXXXXXXXXXXXXXXXXXXXXXX

John_(Mon Jun 21 2010 22:00:32 GMT-0400 (Eastern Daylight Time))>

that should be fun to communicate over the phone

Ricky(Mon Jun 21 2010 22:01:03 GMT-0400 (Eastern Daylight Time))>

Would there be anything else, John.

Ricky(Mon Jun 21 2010 22:01:17 GMT-0400 (Eastern Daylight Time))>

I am very sorry for the inconvenience, John.

John_(Mon Jun 21 2010 22:01:44 GMT-0400 (Eastern Daylight Time))>

i *highly* recommend resetting johnawesome@comcast.net's password

Preserving Types and Differing Subclass Type Parameters

2010-06-12T10:50:00.000-04:00

My last post was on keeping your type annotations as simple as possible to solve your problem. It is important to keep your type system h4x0r-y as simple as possible to solve the given problem, but no simpler. This gives users the greatest flexibility. So, when doing some kind of query from a Traversable type that does *not* return the original type, you can simply take type Traversable[T], where T is a parameter to your function as well.

I started off the last post attempting to solve a problem I did not have. However, what if I did have this problem? That is: I want to preserve the type of a subclass of traversable for the return of my function. The subclass may have a differing type parameter then the traversable, or in my contrived example: TestClass[X] extends Traversable[Y]. Well, let's start digging into a solution.

Let's ignore things that did *not* work in the last post.

def foo[A, M[T] <: Traversable[T]](m : M[A]) = ... // M has differing type parameter, or multiple
def foo[A, M[_] <: Traversable[_]](m : M[A]) = ... // A is not the parameter to Traversable..
def foo[A : Manifest, M <: Traversable[A]](m : M) = ... // Type Inference does not work, but this *does* abide by the type system.

Let's see if we can fix the type inference problem on the last option. To do so, we need to find a way to help the type system out (as it's not very fond of inferring types out of a subclass). Let's take a quick crack at this:

def foo[C <: Traversable[_]](c : C) = new {
    // We now know the type of C from inference.  Let's see if we can infer the type
    // associated with the Traversable *correctly*
    def apply[X : Manifest](t : Traversable[X]) = {
      // We're going to print the inferred Traversable type to make sure it comes out correctly
      println(implicitly[Manifest[X]])
      // The resulting type should be the infered C
      c
    }
  }

What we've done is created a curried function that *takes the same parameter twice*. The first time we infer it's full type C and ensure it's a subclass of Traversable. The second function (apply) will infer the type parameter to the Traversable subclass and reify it into a Manifest. We print this value to ensure we've done everything correctly. Now it's time to test and see if we've achieved our goal!

// We create a subclass where the type parameter to Traversable is *different* from the subclass.
class Temp[X] extends Traversable[Int] {
    def foreach[U](f : Int => U) : Unit = f(5)
}

// This is now interpreter output
scala> val x = new Temp[String]
x: Temp[String] = line2(5)

scala> foo(x)(x)
Int
res0: Temp[String] = line2(5)

So we have the unfortunate ugliness of having to send the parameter to *two* function, since we split the type inference in half, However, notice that we've inferred the type parameter to Traversable correctly! Also notice that the return type is Temp[String], which is exactly what we desired.So, we had to jump a few hoops to get what we want. The unfortunate truth is that this kind of inference can't be done all in one go (And if you figure out a way, please let me know!!!). However, it's still possible to manipulate the type system to do what we want, albeit at the cost of some extra ugliness.Why might I want to perform this kind of trickery? Let's suppose Temp[String] is an immutable type that acts like a Traversable on Ints. I have a method that wants to iterate those Ints and create some kind of new Temp[String] based on what it found. To do this, I'm using a builder pattern with implicit builders. Let's take a look at the new signature I can use to accomplish my goal:

def foo[C <: Traversable[_] : Builder](c : C) = new {
    def apply[X : Manifest](t : Traversable[X]) = {

      // We're going to use the traversable, and we need the manifest of X to do some
      // specialized calculations, maybe some Array fun.
      val result = doSomethingWithTraversableUsingXManifest(t)

      // Construct a new C using its Builder with a modified Attribute
      implicitly[Builder[C]].copyWithModify(c, ModifyAttribute(MyAttribute, result))
    }
  }

As you can see, this is now possible! I'm making the Builder / Modify attribute magic purposefully hazy as I hope to cover this in a blog article in the near future.

On higher kinded and existential types,...no stereos

2010-04-27T21:05:00.001-04:00

So I was recently bit by a type issue with existential types until someone forced me to start thinking again. Let's start with a good example of what I was trying to do:

def foo[A, M[T] <: Traversable[T]](m : M[A]) : A = m.head
def foo2[A, M[_] <: Traversable[_]](m : M[A]) : A = m.head

foo2 will fail to compile with a type error. What gives! Aren't these the same? Well, no they are not. Let's see if we can reason through the type signature we've written. For foo, A is any type and M is a higher kinded type. M has a type parameter T. M is also some kind of subclass of Traversable. Also, the type parameter T on M is *the same* type parameter used for the Traversable parent (i.e. defined trait/class M[T] extends Traversable[T]). Now for foo2, it's a bit different. A is any type. M is some higher kinded type (takes a type parameter) but in this case we don't acre which. M is also a subclass of Traversable, and once again we don't care about the type parameter to Traversable. The big difference here is that we don't care about the type parameters to M or Traversable, meaning they could be different types. Now when we define parameter m, we're looking at M[A] <: Traversable[_] or, in other words, we still know nothing about the type in the traversable. Let's try to prove our intuition about foo2 by capturing the manifest of type parameter A and see if it has any correlation to the type in the traversable:

def foo[A : Manifest, M[_] <: Traversable[_]](m : M[A]) : A = {
  println(implicitly[Manifest[A]])
  m.head.asInstanceOf[A]
}

class Temp[X] extends Traversable[Int] {
  override def foreach[U](f : Int => U) : Unit = f(5); ()
}

foo(new Temp[String]) // Prints java.lang.String -> NOT Int

As you can see, the compiler has capture the String type parameter, not the Int as we really wanted to capture in this method. Let's see if we can correct our mistake:

def foo[A : Manifest, M[X] <: Traversable[X]](m : M[A]) : A = {
  println(implicitly[Manifest[A]])
  m.head.asInstanceOf[A]
}

class Temp[X] extends Traversable[Int] {
  override def foreach[U](f : Int => U) : Unit = f(5); ()
}
foo(new Temp[String]) //THis line errors ->
:9: error: inferred type arguments [String,Temp] do not conform to method foo's type parameter bounds [A,M[X] <: Traversable[X]]
       foo(x)

Ok, so I've seen a lot of code with this M[X] <: Traversable[X]... This should work right? Well, let's think it through: M[X] <: Traversable[X] implies that the type M is higher-kinded (takes a type parameter). It also implies that it extends Traversable, and finally that The type parameter to Traversable should be *the same* as the type parameter to M. It's this last piece we've messed up. Our Temp class does take a type parameter, but it has nothing to do with the type parameter in its traversable parent. This distinction is important, especially with collections. M[X] <: Traversable[X] can capture an awful lot of collection types, however it will miss any collection where the type is known... In other words, if I defined something like

IOByteStream extends Stream[Byte]

then

M[X] <: Stream[X]

will fail to capture this type! Let's take one more attempt, this time trying to capture a subclass of Traversable where the Traversable's type parameter is captured in a manifest.

def foo[A : Manifest, M <: Traversable[A]](m : M) : A = {
  println(implicitly[Manifest[A]])
  m.head
}

class Temp[X] extends Traversable[Int] {
  override def foreach[U](f : Int => U) : Unit = f(5); ()
}

foo[Int,Temp[String]](new Temp[String]) //We need to use our types here

Well the method compiles, but we have the issue that the typer cannot infer our types! At least we managed to express what we wanted... or did we? Our method does not care about the actual subtype of Traversable, and does not need to capture it. If you look, the return type is A, not M. Therefore we should be able to take in just a Traversable, like so:

def foo[A : Manifest](m : Traversable[A]) : A = {
  println(implicitly[Manifest[A]])
  m.head
}

class Temp[X] extends Traversable[Int] {
  override def foreach[U](f : Int => U) : Unit = f(5); ()
}

foo(new Temp[String])  //Prints Int and returns 5

I purposely made me initial example odd so I could make this point: When using the type system, try to keep things as simple as possible. Expand your definitions as needed, or when assumptions are proven invalid. Here, we had no need for the actual subtype of Traversable and yet we were focused on getting the signature right to capture a subtype. This is important for developers to learn, and even *more* important for library authors. When using type parameters, make sure to reason through the signatures to be sure they capture what is desired - no more or no less. Just for kicks, let's try a method were we *do* care about the original subtype, and wish to preserve it, yet also need to use the Traversable's type parameter. What can we do? Let's attempt to capture both type parameters:

def foo[A : Manifest, B, M[X] <: Traversable[A]](m : M[B]) : M[B] = {
  println(implicitly[Manifest[A]])
  m
}

class Temp[X] extends Traversable[Int] {
  override def foreach[U](f : Int => U) : Unit = f(5); ()
}

foo[Int,String, Temp](new Temp[String]) //We need to use our types here again

You can see we've captured both type parameters in A and B, however once again the Scala type inferencer cannot distinguish our meaning without some help. Is there anything we can do here?

Unfortunately... I don't actually know of a way to help the type inferencer here. I also realized that in my situation, that kind of a requirement was completely unnecessary.... If anyone has any ideas for the academic excercise, please let me know.

Since the new collections API starts throwing more type issues in the face of every day developers, expect a new series from me: "Enough Type magic to work with Scala Collections". As I figure out ideal ways to deal with new type signatures required in abstract methods, I'll try to post findings.

Leaky Monads Talk

2010-04-15T18:19:00.000-04:00

Here's my talk from ScalaDays 2010 on how I completely ruin a theoretical concept:

The source code is available on github

Thanks to everyone who was there! Special thanks to Adrian for pointing out my misinterpretation of the implicit spec (ignore slides 15/16).

How you should think about Delimited Continuations in Scala

2010-03-10T22:56:00.014-05:00

Scala 2.8.x is going to have a new delimited continuations plugins. Delimited continuations are interesting animals, and I've seen some decent articles about them. I'm going to portray here how you should think about delimited continuations in a way that will help you know how to make use of them in your coding.

Let's take a look at the wikipedia definition of delimited continuations:

a composable continuation, delimited continuation or partial continuation, is a "slice" of a continuation frame that has been reified into a function.

This is basically saying that a delimited continuation is marking a particular section of a program in such a way the compiler can "reify" it into a Function (or Closure if you will). In fact, the compiler can use similar semantics to how a closure is made, the difference here is in how you make arguments to functions. Let's take a look at a simple Scala delimited continuation example and try to de-mystify:

import scala.continuations._
import scala.continuations.ContextControl._

object SimpleCont {

  def main(args : Array[String]) : Unit = {
    val result = reset {
       1 + shift { k => k(k(5)) } + 1                   
    }
    Console.println(result)
  }
}

This simple example prints 9 to the screen. So what is happening? Well, let's look at the definition: A "slice" of a continuation frame that has been reified into a function....

Well, we know reify essentially means to make it a first-class citizen (or first-class-function). So what is our "slice of continuation frame"? It's everything inside the reset block. This is where the "delimited" portion comes in. We're using reset to as an outer-bounds for the compiler to capture code and reify it into a function. In doing so, the compiler essentially takes all shift keywords and turns them into arguments for the reified function. Let's look at what our reified function is equivalent to in the above example:

reset {
       1 + shift { k => k(k(5)) } + 1                   
  }
  val reifiedFunction = { shiftVal1 : Int => 
       1 + shiftVal1 + 1
  }

Notice how the entire shift {..} expression is converted into an argument of the reified function. That's where the continuation comes in. All code inside the reset, minus the shift expressions, is turned into continuations... Something that could be called later and/or multiple times. For the real magic, let's take a look at this shift operation.

The shift command takes a function as an argument. This function takes another function as its only parameter (higher-order) and that function *is* the reified continuation. So... strangely enough, the shift expressions are turned into arguments on functions *passed into the functions defined in the shift expression*. Yes, this blew my mind at first, but becomes very interesting and handy. Not only that, it's a true inversion of control.

In this simple example, the portion of code defined in shift *becomes* the main execution path, and calls the reified continuation. Let's look at the translated code side-by-side

object SimpleCont {

  def main(args : Array[String]) : Unit = {
    val result = reset {
       1 + shift { k => k(k(5)) } + 1                   
    }
    Console.println(result)
  }
}

object SimpleCont {

  def main(args : Array[String]) : Unit = {   
     // Reified function
     val k = { shiftVal1 : Int =>
       1 + (shiftVal1) + 1
     }
     val result = k(k(5))
     Console.println(result)
  }

As you can see, we've completely reversed the shifts and the resets. Let's try to understand an example which involves two variables. The Continuations paper includes an example that takes the cartesian product of two lists (i.e. List(A,B) and List(C,D) => List((A,C), (A,D),(B,C),(B,D)) ). First the example code straight from the paper (with a few modifications for simplicity):

implicit def reflectiveList[A](xs:List[A]) = new {
      def reflect[B]() : A @cps[List[B], List[B]] = shift { k:(A => List[B]) =>
        xs.flatMap(k)
      }
    }
    
    reset {
      val left = List("x","y","z")
      val right = List(4,5,6)
      List((left.reflect[(String,Int)], right.reflect[(String,Int)]))
    }
  }
// result: cartesian product

Let's decompose this one a bit. The implicit reflective stuff is used to enabled the reflective method on lists. We'll analyze reflective soon, but for now, notice that it calls shift. This means that calls to reflective are equivalent to calls to shift. Let's try to figure out what the reified continuation is by first "inlining" all the code:

reset {
  val left = List("x","y","z")
  val right = List(4,5,6)
  List(Tuple2(
      left.reflect[Any],


      right.reflect[Any]


  ))
}

reset {
  val left = List("x","y","z")
  val right = List(4,5,6)
  List(Tuple2(
        shift { k:(A => C[B]) =>
          left.flatMap(k)
        },
        shift { k:(A => C[B]) =>
          right.flatMap(k)
        }
  ))
}

First of all, Notice the new type signatures on the shift expressions (without the generics). We'll discuss these more in depth later, but if hand-reify the continuation, we need to make sure we match the signatures expected from the shift expressions. Let's take our first crack at it:

val reified : = { (shift1, shift2) =>
  List(Tuple2(
        shift1,
        shift2 
  ))
}

Well, that seems simple enough... except we're not creating the lists in the reified function! Ah... remember, we're only reifying the continuation piece. We can directly execute the portion of the code that are directly executable. Let's try to completely unwind the code and translate it:

// Original Code
reset {
      val left = List("x","y","z")
      val right = List(4,5,6)
      List((left.reflect[(String,Int)], right.reflect[(String,Int)]))
    }
val reified : (String, Int) => List[Any] = { (shift1, shift2) =>
  List(Tuple2(
        shift1,
        shift2
  ))
}

//Original reset was here...
val left = List("x","y","z")
val right = List(4,5,6)
left.flatMap( l => right.flatMap(r => reified(l,r))

Well that's interesting.... This is the effect of apply delimited continuations and should help you begin to reason about them and how they operate, however the actual truth of the matter is, your code is transformed differently. Let's take a look at what is really happening, just so we can dig deeper if needed. Technically, there are two reified continuation (for the two shifts) so let's make our hand-reified code do the same:

//Original reset was here...
val left = List("x","y","z")
val right = List(4,5,6)
val reified1 : (String => List[Any]) = { shift1 =>
  val reified2 : (Int => Any) = { shift2 =>
        Tuple2(shift1,shift2)     
  }
  right.flatMap(reified2)
}

left.flatMap(reified1)

So this is where it gets strange. We create a function "refiied1" which is passed to the first shift expression (left.flatMap). This contains the outer continuation. We then have a second call to shift. This means there is a second continuation generated inside the first continuation. The second function is responsible for generating the tuples. This is based to the second shift expression "right.flatMap". As you can see, every time you use the shift function, you're causing the compiler to generate a new continuation to defer the next set of processing. You can also see how in "pure" form, this has a very nested structure, whereas the original code was very "flat". Also, because work is automatically defered into continuation functions, you can use this style for concise and performant asynchronous work.

Let's take a mini-dive into a continuations for automated resource management. First, let's make our sample file:

HAI!

O
MY!

Now, let's right a continuation-based program to read each line and give us the size of it. I'm purposely making the code slightly confusing here to demonstrate the inversions. Well, let's start off with a mechanism of "reflecting" or "inverting" a resource for delimited continuations.

trait Resource[+R] {
      def flatMap[U](f : R => U) : U
      def reflect[B]() : R @cps[B,B] = shift { k : (R => B) => flatMap(k) }
}

def resource[R <: Closeable](acquire : => R) = new Resource[R] {
      override def flatMap[U](f : R => U) : U = {
        val x = acquire
        try {
          f(x)
        } finally {
          System.out.println("Closing Resource!")
          x.close()
        }
      }
}

We define a simple Resource trait that gives us a flatMap (some may call this "using") and a reflect (or inverse) method. This will let us "flatten" resource usage inside a reset block. The next component is the implementation of resource for any java.io.Closeable implementation. This will take a function to acquire a resource and implement the flatMap method correctly. This should look *very* similar to the reflect method defined on lists... One might even say this method works to flatten any monad ;)

Now, let's get really dirty and define an inversion on java.io.InputStream to deal with lines:

def reflect[A <: InputStream](input : A) = new {
      def each_line[B] : String @cps[B,List[B]] = shift { 
          k : (String => B) =>
        val b = new BufferedReader(new InputStreamReader(input))
        var line = b.readLine
        var list = ListBuffer[B]()
        while(line != null) {
          list += (k(line))
          line = b.readLine
        }
        list.toList
      }
    }

We've defined our reflect method to work on every line within the file. It builds up a list and returns it as the final result. You'll notice the shift call takes a function requiring a String and returning a "B". The fun type-system issue we have to deal with here is you need to know the result of the reified continuation in the return value for each_line. We'll get to that more in a moment, but for now make note of this.

Alright, now let's define our simple function to manipulate every line in a file:

reset {
      val input = resource(new FileInputStream("test.txt")).reflect[Any] //We cheat on the return value here!
      val line = reflect(input).each_line[Int] //We need to know the continuation returns an Int Here!!!
      System.out.println(line)
      line.size
}

Wow... so really all we're doing is grabing our input, inverting to the "each_line" call and calling line.size after printing the line to the console.

So what's the output look like?

scala> Main.resourceTest()
HAI!

O
MY!

Closing Resource!
res0: Any = List(4, 0, 1, 3, 0)

This is great! It's doing what we wanted. Notice how we've "flattened" the code substantially compared with other methods. Contrast the above with the following:

val input = new scala.io.magic.File("test.txt")
for { 
  stream <- file.managedInput
  line <- input.lines
} yield line.size

You'll notice a few similarities (in my made up library). For-expressions are another area in scala where you're reifing portions of the code into closures and passing them to flatMap blocks. However, delimited continuations open the doors to way more than flatMap. Let's go back to that typing issue though.If you remember, I was returning Any because I was being lazy with my type annotations when reflecting the input stream. Let's use the correct annotation:

reset {
      val input = resource(new FileInputStream("test.txt")).reflect[List[Int]]
      val line = invert(input).each_line[Int] //We need to know the continuation returns an Int Here!!!
      System.out.println(line)
      line.size
}

So... what do these type annotations mean? In the case of the resource's reflect method, the type annotation is the ultimate result from the reset call (i.e. the result of the reified continuation) because of how we defined flatMap on Resource. The type annotation for each_line describes the type the resulting List of processed lines should contain... i.e. the result of the last portion of the continuation. If we were to hand-code this reset piece we'd have:

val rcont1 : (FileInputStream => List[Int]) = { 
  inputStream =>
     val rcont2 : (String => Int) = { line =>
        System.out.println(line)
        line.size
     }
     val b = new BufferedReader(new InputStreamReader(inputStream))
     var line = b.readLine
     var list = ListBuffer[B]()
     while(line != null) {
        list += (rcont2(line))
        line = b.readLine
      }
      list.toList
  }
  resource(new FileInputStream("test.txt")).flatMap(rcont1)

So... delimited continuations certainly use far less code here. The understanding of what's going on under-the-covers perhaps needs some good terminology/consistent paradigms to be useful in library-form (i.e. if I make each_line or reflective part of a standard library). Also, it's my opinion that the type-inference barrier for the @cps annotation should be looked into. Perhaps we can invent creative ways to infer the types in an inverted fashion when using delimited continuations. In the meantime, the compiler will complain until you figure out the correct types, so it could be worse ;)

I plan to post a bit more about delimited continuations after I have some useful I/O related tools built up around them. Let me know if you find this post helpful. I'd also like to clarify that my "hand-reification" is not really how scala does reification, but close enough for understanding the concepts.

Also, there's some interesting continuation-based actors code in the paper that I'm going to try to dig into here when I have the time!

Actor Styles... OO, Functional or Blended...

2010-02-28T10:12:00.001-05:00

I had the privilege of a free cup of coffee with David Pollak when I was in the bay area recently. He was describing his new goat-rodeo library when he showed me the interface for his "Worker" class (which is similar to an Actor class). He made a comment about "being annoyed with writing all those case statements" in actors, and so figured out a method using reflection and method overloading to define an actor that merely responds to all messages it receives. I'm sure there was more to it than this, but I believe he's partially solved an OO issue I complained about in an earlier blog about partial functions and inheritance. Let's take a bit to remember that solution to the issue.

The crux of the problem is the blend of OO and functional. Functions are composable. You can have one function take another and compose them all day long. Objects are also composable. One object can take another object and utilize it all day long. Methods (functions attached to objects) are not as composable. Scala traits give you means to compose with them, but all your composition is done at instantiation time (i.e. mixing in traits, inheritance, etc.). Now we come to the main issue. An Actor is an abstract class that you subclass. This leads one to compose behavior through traits (similar to other OO-style programs in Scala), but the method of doing so is a bit strange. We start with our "FinalMixinActor" class

trait BaseActor extends Actor {

   def makeMessageHandler() : PartialFunction[Any,Unit] = {
      case x => //Unhandled message, assume design flaw!
            error("Unknown message: " + x)    
   }
}
trait FinalActor extends BaseActor[T] {
   lazy val messageHandler = makeMessageHandler()
   override def act() {
       Actor.loop {
          react messageHandler
       }
   }
}

These two classes provide us with a way to compose actors. We make sure all behavior extends the BaseActor, and we make sure when creating an actor that "FinalActor" gets mixed in *last* in the inheritance linearization. This means that when FinalActor calls "makeMessageHandler", all the standard OO inheritance has a chance to kick and pull the behaviors you've composed. Let's see an example of composing an actor with two functionalities....

case class Ping()
case class Pong()

trait PingPongBehavior extends BaseActor {
   def makeMessageHandler() : PartialFunction[Any,Unit] = {
      val myBehavior : PartialFunction[Any,Unit] = {
         case Ping() => sender ! Pong()
      }
      return myBehavior orElse super.makeMessageHandler()
   }
}

This is our "PingPong" behavior actor for sending/receiving Ping/Pong messages. As you can see in the makeMessageHandler... we're manually converting functional composition into inheritance composition. It does give us flexibility if we'd like our behavior to run last. Let's create our second piece of behavior...

case class Shutdown()

trait RemoteControlBehavior extends BaseActor {
   def makeMessageHandler() : PartialFunction[Any,Unit] = {
      val myBehavior : PartialFunction[Any,Unit] = {
         case Shutdown() => Actor.exit
      }
      return myBehavior orElse super.makeMessageHandler()
   }
}

This is a simple actor that just lets us send a shutdown command to an actor instead of relying on the actor knowing when to shut itself down (or linking it). Not necessarily useful, but it will help illustrate our next point. Let's create an actor that mixes in both these functionalities.

val myActor = new PingPongBehavior with RemoteControlBehavior with FinalActor
myActor.start

Notice how we *have* to mix in FinalActor last so the composition of makeMessageHandler is done correctly. This also prevents other actors from adding behavior in their act methods....

Anyway, it's a decent method of allowing mixin behavior of actors. It doesn't have much type safety, but the Scala standard actors library actually makes you jump through some hoops to get type-safety anyway (plus all their samples are against Any... so it seems they tend to encourage this...)

Now for a look at how Goat-Rodeo helps solve this. First, Goat-Rodeo uses a type-safe message passing API, including knowing whether a message is allowed a response, and the type of that response. Let's take a look at a boiled down version of the twitter-clone sample on goat-rodeo's wiki. I've used elipses to ignore implementation details of Goat Rodeo:

class UserWorker(...) extends WorkerImpl[..., UserMsg](...) {
  /**
   * handle the Follow message
   */
  def doFollow(msg: Follow) {
     ...
  }
}

As you can see, it's just a method. Goat-Rodeo looks for methods with "do" or "handle" in the name and constructs a PartialFunction for the actor-behavior. The

Follow

class must be a subclass of

UserMsg

as Goat-Rodeo is also strongly typed. Because

doFollow

returns Unit, goat-rodeo knows that there is no return expected with this message. If you'd like to handle different messages, simple use method overloading. The best part, if we'd like to use OO inheritance to compose behavior, we can do so directly now...

class UserWorkerPlus(...) extends UserWorker(...) {
  /**
   * handle the Follow message
   */
  def doFollow(msg: Follow) {
     println("ZOMG!!!!!!!!!!!!!!!!!!")
     super.doFollow(msg)
  }
}

That should be a lot more comfortable to OO developers. With some type trickery you could even create the "management" behavior traits and mix them into "workers" as needed. It still leaves some things to be desired. e.g. when I compose partial functions in a purely functional sense, I can have the type-checker automatically infer what the new signature of a partial function should be after an orElse. With both of these methods, that is not possible. For the pure functional users, I'd recommend creating actors using something akin to this function (or you should just use scalaz, as their actors library is very well done):

import scala.actors.Actor
import scala.actors.Channel

class ChanneledActor[T](f : PartialFunction[T,Unit]) extends Actor {
  val typedChannel = new Channel[T](this)
      def act() {
         Actor.loop {
            typedChannel react f
         }
      }
}

def makeActor[T, U](f : PartialFunction[T,U]) : Channel[T] = {
   val a = new ChanneledActor(f andThen ( x => () ))  //Ignore results...
   a.start
   a.typedChannel
}

This will let you compose partial functions in a functional style and let the type-checker figure out the signature. Finally when you create an Actor with the makeActor method, it will return the type-safe channel to send messages with. Here's an example interpreter session:

scala> makeActor[Int,Unit] { case x : Int => println("HAI") }                   
res3: scala.actors.Channel[Int] = scala.actors.Channel@154f77b

scala> res3 ! "HAI"
<console>:10: error: type mismatch;
 found   : java.lang.String("HAI")
 required: Int
       res3 ! "HAI"
              ^

scala> res3 ! 5    

scala> HAI

Anyway, if you truly desire a functional approach to actors, you should look into Scalaz, as they've done it right. If you desire a hybrid or an OO approach, I hope I've helped outline the details for you. I'm very interested to see how all these libraries evolve over time, and kudos to David Pollak for goat-rodeo.

Knowing your faults

2010-01-27T21:34:00.000-05:00

Knowing your faults is great advice. It's important to know where your own flaws lie, so that you can find ways to work around them. If you aren't aware of your flaws, you can wind up making some serious mistakes, like letting your emotions make your decisions for you.

I think the same can be said of software. It's important as an engineer to know where your software is weak. Every project I've ever been on has had at least ten things that "could be improved" if we had more time/money. Knowing this list is one thing, tracking it and adapting for it is another.

When designing software, it's important to make sure that the weaknesses aren't critical to the success of your business, however sometimes this is inevitable (or it was designed by someone else ;) ). Recognizing these flaws lets you mitigate them until you have a chance to remove them.

In real life, I have a set of close friends who are very honest with me. If I'm doing something stupid, they'll let me know. Imagine these friends as my profilers, loggers and monitors.

There was a recent blog post about Monitoring Driven Architecture. Finding and determining useful monitoring metrics can be the key differentiated between quickly solving architectural issues and shooting at the wall hoping you hit your own zombie-infested hand.

In the future, I will not start a new project without identifying weak areas and monitoring points. These things need to be monitored as soon as possible. Also, only real deployments can truly tell you which metrics are useful, so make sure you update your monitoring as time goes on.

To my readers who still subscribe even though I've been really lame about making useful posts recently:

I normally don't write these rambling though-stream posts, so send me some feedback. If you'd rather like more technical blogs, I'm working on a few (The "Leaky" Monad should hopefully be done soon ;) ).

Hudson Slave agent startup script

2010-01-20T20:39:00.001-05:00

Last night, I got really annoyed at our hudson build slaves not automatically starting on our build nodes. I found a few init.d scripts online, and I pieced this one together with some additional functionality.

Still needs a bit more features, specifically:

Ability to run as hudson user or root user
Ability to change hudson port
Ability to have different Credentials

Anyway, I hope others find this helpful. It's meant to be run on a hudson build node to automatically connect to hudson at startup.

#!/bin/bash
#
# Init file for hudson server daemon
#
# chkconfig: 2345 65 15
# description: Hudson slave

. /etc/rc.d/init.d/functions


RETVAL=0
NODE_NAME="centos-executor"
HUDSON_HOST=1.2.3.4
USER=hudson

pid_of_hudson() {
   ps auxwww | grep java | grep hudson | grep -v grep | awk '{ print $2 }'
}

start() {
        if [ ! -e /var/log/hudson ]; then
  touch /var/log/hudson
  chown $USER /var/log/hudson
 fi
        echo -n $"Starting hudson: "
        COMMAND="java -jar /opt/hudson/slave.jar -jnlpUrl http://${HUDSON_HOST}/hudson/computer/${NODE_NAME}/slave-agent.jnlp 2>/var/log/hudson.err >/var/log/hudson"
        su ${USER} -c "$COMMAND" &
        sleep 1
        pid_of_hudson > /dev/null
        RETVAL=$?
        [ $RETVAL = 0 ] && success || failure
        echo
}

stop() {
        echo -n "Stopping hudson slave: "
        pid=`pid_of_hudson`
        [ -n "$pid" ] && kill $pid
        RETVAL=$?
        cnt=10
        while [ $RETVAL = 0 -a $cnt -gt 0 ] &&
                { pid_of_hudson > /dev/null ; } ; do
                        sleep 1
                ((cnt--))
        done

        [ $RETVAL = 0 ] && success || failure
        echo
}


status() {
        pid=`pid_of_hudson`
        if [ -n "$pid" ]; then
                echo "hudson (pid $pid) is running..."
                return 0
        fi
        echo "hudson is stopped"
        return 3
}


#Switch on called
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        status
        ;;
  restart)
        stop
        start
        ;;
  *)
        echo $"Usage: $0 (start|stop|restart|status}"
        exit 1
esac

exit $RETVAL

Too Busy

2010-01-13T16:18:00.000-05:00

I've been crazy busy between work and writing a book. I have been writing a few blogs for the company, the first of which is complete, if you'd like to view it.

You can also see advertisements for my upcoming book (name misspelled, yay) on Manning's under contract page.

Using Maven to Develop Scala

2009-11-10T19:18:00.000-05:00

Here's my presentation for the Scala LiftOff (East Coast) Conference.

The cost of dogfood.... (or brocolli)

2009-09-02T23:35:00.004-04:00

So, I've been busy lately. My blog has suffered. I apologize guys. Pretty soon I'll post yet another rant against EJB. Honestly, I feel people who love EJB have not used it extensively, or have only programmed in Java.

Anyway, I've been trying to help blog at work, and wrote an article you can find on our company website here

Enjoy!

Avoid the Applicaiton trait in Scala

2009-08-17T18:14:00.004-04:00

All, It's been a *long* time since I've been writing this blog. Today I felt I'd try for a very short article.

It seems to me that about every 2 weeks, someone comes on the Scala #irc or mailing lists and asks about performance issues. It also appears there is no end to people who want to use Scala's Application trait.

Do not be so tempted.

The application trait is a very neat feat of scala-mojo for automagically creating a main method, however it has one serious flaw:

Your entire application is run inside a static initializer block.

This having another serious flaw:

Hotspot will not optimize your application.

Anyone who wishes to have a real "scala" application like trait and doesn't want to write an ugly

def main(args : Array[String]) : Unit = {}

should run down to their local github and look at DRMacIver's Optional library.

Update: You should look at paulp's github fork of optional. I was unaware of this before I wrote the blog article. I've modified appropriately

You still have to write a main method, but this team the arguments *mean* something, not just to the compiler, but to the command line!

Here's the readme from paulp's Optional repo:


optional is a command line option parser and library.

YOU WRITE:

object MyAwesomeCommandLineTool extends optional.Application {
  // for instance...
  def main(count: Option[Int], file: Option[java.io.File], arg1: String) {
    [...]
  }
}

THEN YOU DO:

  scala MyAwesomeCommandLineTool --count 5 quux

AND YOUR MAIN METHOD WILL BE INVOKED SUCH THAT:

  count = Some(5)
  file = None
  arg1 = quux
  
See the example programs for many more features.

HOW IT WORKS:

  Reflection, man.

CREDITS:

  Idea and prototype implementation: DRMacIver.
   Fleshing out and awesomification: paulp.

In Other news

I've recently been doing a bit of Scala + EJB and (funny enough) PowerShell scripting. Hopefully I'll have good things to write about and (more importantly) the time to do so in the near future.

Why ARM is not quite enough

2009-06-12T20:14:00.008-04:00

This post is for you project coin! I would rather not see any ARM in the Java language, than a subdued ARM (ARM = Automated Resource Management). Here's why:

ARM is trying to solve the age old problem of ensuring closure of resources when finished with your business logic. In C++ we would use RRID (Resource Release Is Destruction) to accomplish this. e.g.



template<typename T>
class LockEnforcer : public boost::noncopyable<LockEnforcer>{
     T& lock;

  public:
      LockEnforcer(T& lock) : lock(lock) { lock.lock(); }
      ~LockEnforcer() { lock.unlock(); }      
}


//Later in a method....

{
   LockEnforcer(someLock);

   //Do my business logic

}
//Lock is shut down here by nature of Destruction

This suffered from a few problems:

Destructors should not throw exceptions! (Read Effective C++). This could leave you in a dangerous state potentially if you cared whether or not "close" threw some exception.

The destruction behavior ALWAYS happens. If you wanted to change whether or not cleanup occurs, you have to put state into the class and some more complex logic in the destructor. You also need to make calls on the class to inform it *not* to do the usually cleanup.

Java steps in lacking destructors, and forces developers to take a new paradigm. Let's look at a typical Java JDBC connection usage paradigm.



Connection conn = null;
try {
   conn = getConnection();
   //Business Logic
} finally {
    try {
        if(conn != null) {
            conn.close();
        }
    } catch(SQLException e) {
       //Ignore - that way if we already had an exception, we don't lose it on close.
    }
}

Now, I have potentially buggy code everywhere I need to use connections. I'm also not able to re-use code easily. Now, the Spring framework truly does solve this problem (see any of the *Template classes), but we'll get to how and why it's useful later. For now, assume we're stuck with pure java (or we're in a domain that isn't covered by Spring yet). Coin proposes the following ARM syntax:


try (Connection conn = getConnection()) {
   
   //Business Logic
}

This would be equivalent to the above block. It is much nicer, and certain helps in the general case of resource management. Using multiple resources would of course imply multiple try blocks.

The problem is that this doesn't solve the more general problem. The general problem is that you have code that is duplicated across several methods. This code needs to run before and after some non-duplicated section. Take for example the following:



try  {
  if(isNewMessage(msg)) {
        handleMessage(msg);
  }
  sendMessageAcknowledgement(msg); //Note this is *not* in the finally
} catch (Exception e) {
  //Do something with exceptions
}

I'm unable to use ARM because I want the final code (before the catch) to be run if the above code passes without exception. The other problem here is that I have 5 methods with this same boiler-plate (except same isNewMessage function), and different handleMessage functions. Typically, Java encourages you to create a base class, with abstract handleMessage. Then you extend classes with different handleMessages. That's pretty much the only way to reuse code here. What if I don't really want other classes?

Well, Scala comes to the rescue (slightly). Scala will let me write the above code using functional composition, and behind the scenes it's desugaring to multiple classes that inherit the "Function" interface.
Note: That the Spring *Template classes take this approach, only with less generic interfaces. Instead of "Function1", it's SqlRowMapper.

Here's an example in scala.


//Helper function
def processMessage(msg : Msg)(handlerFunction : Msg => Unit) = {
   try {
      if(isNewMessage(msg) {
           handlerFunction(msg)
      }
      sendAcknowledgement(msg)
   } catch {
           case e => //TODO - Handle
   }
}

//My actual method
processMessage(msg) { 
   msg =>
       //Business Logic
}

As you can see, I've removed duplicate code, and without a lot of boilerplate. ARM would not be necessary if Java had closures, and the entire set of problems (not just resource management) revolving around repeated code at start/end of blocks would also be solved. Personally, I would rather just wait for Java to have closures.

Demo Oriented Programming (and Project Jigsaw)

2009-05-18T21:23:00.003-04:00

So recently I've been amused watching the OpenJDK Jigsaw project mailing list. Basically there have been a few emails that look like the following:

I have this working in this way. It has these flaws. I'd like to ignore them so we can finish in time for JavaONE

Sounds good

I have an even bigger hack I'd like to introduct temporarily for JavaONE, it will make the demo run smoothly, just don't use Windows or Mac

Ok, but make sure you fix it after JavaONE

The fun part about "open" source is that this kind of back-and-forth is just that: open. Things that every company does for demos is now public domain and I can create a link showing one of the conversations.

We've all had times in development where you need to push some hacks through to get something done in time for a demo. The key is to cleanup your code afterwords and make sure that it's ready for production in time. This is where powerpoint can be your friend and your enemy. It is great to *show* people what you mean, but sometimes they need to get the feel of it to really see what it is you want to do. That's why I believe wholeheartily in Demo-Oriented programming. Get something out there that looks enough like the real thing so your users can complain about it. Until you hand them something that they can use, you won't get very articulate/useful feedback.

However, Demo-Oriented Programming relies heavily on the ability to refactor quickly and/or rewrite sections of code. You need a good management chain that will allow you to take the time and fix "demo-related" items in time for actual release. This means you can get good intiial feedback, and continuing good feedback from users. I definitely see Sun as being a great DOP shop.

On a slightly unrelated note, it looks like project-jigsaw is really coming along. I'm still very unsure if packaging modules "natively" (i.e. rpm, deb, etc) is the right approach, but at least some of the underlying "jvm bootstrap" issues appear to be getting some attention. I'm also *very* curious to see what happens with jigsaw + OSGi. I do not see a lot of mention of OSGi from the jigsaw-dev lists, but I see lots of investment in OSGi from other companies, and a lot of strength in the platform. I would not use EJB anymore (by choice) and I believe an OSGi runtime + carefully chosen libraries will be my new "enterprise application server stack". Hopefully jigsaw will not slow down the momentum of the OSGi community. I'm very much looking forward to being able to choose which orthogonal, cross-cutting "concerns" I want in my application. Perhaps just transactions and scalability? Maybe I like terracotta. Maybe I need something more custom?

Embedding the Scala Interpreter

2009-04-28T21:39:00.006-04:00

Recently, I was given an application at work that needed some extra functionality, and basically needed to be made more flexible. As this is an internal tool, it currently requires 'expert' usage only. Thinking through the design, I decided that perhaps I should try an idea I've heard others use:

Let's write the configuration for the application in a Scala Domain Specific Language.

To do so, I need to make the application "modular" in the sense that various components can be configured through some sort of "builder" pattern DSL. They are then assembled together (again in a DSL), which should be as readable as possible, and easy to change. The DSL creation went great, the app is looking good. Here's a pseudo example of what I'm talking about.



val input = new InputChannel with ReconnectOnFailure {
    inputUrl = "http://suereth.blogspot.com"
}
val output = new OutputChannel with FileWriter {
   outputUrl = "file:///home/josh/test.out"
}

processingBuilder from input to output and { p =>
      p hasModule new ThemeExtractor {
           maxThemes = 10
           //Define more variables
      }
      //Define more modules       
}
processingBuilder.startProcess()

The above is a hypothetical DSL for processing inputs and placing the results in an output. The idea being you can generate "filter chains" via the processingBuilder object. Ideally, this configuration should be in some kind of file we can pass into our program, rather than having to recompile the program every time we want to run. Imagine however, that we combine these two solutions.

Let's compile the DSL and all the internals of the program and place them into a jar file. Let's then take the "configuration" portion of the program and compile it at runtime, just before execution. To do this, we can simply embed the Scala interpreter. I'm going to make use of an open-source "library" (really just one class) I wrote one evening while testing this theory. The library (Scala Embedded Interpreter) is located on github and is BSD licensed.

To start off with, we need some kind of main class that will take a "configuration" file as an argument, and pass it into our embedded interpreter. Let's try something like this:


object App {
   def main(args : Array[String]) {
        //Let's assume all arguments are file locations for configuration

        val interpreter = new InterpreterWrapper {
            for(fileName <- args) {
              addScriptFile(fileName)
            }
        }
        interpreter.startInterpreting()
   }
}

So, the above is fairly simple. We construct and InterpreterWrapper and call "addScriptFile" for every configuration file we find on our command line. So... how is InterpreterWrapper working? Let's look into the details:



import scala.tools.nsc._
import interpreter._
import java.io._
import scala.reflect._
/**
 * A trait to ease the embedding of the scala interpreter
 */
trait InterpreterWrapper {
  private var files = List[String]()
  ...
  protected def addScriptFile(fileName : String) {
    files = fileName :: files
  }
  ...
  def startInterpreting() {
    //Make a place for output to go
    val out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(System.out)))
    //Create a new Interpreter settings object with the given output method
    val settings = new GenericRunnerSettings(out.println _)
    //For every file, add it to the interpreter settings
    files.foreach(settings.loadfiles.appendToValue)
    
    //Create a new interpreter loop and start running!
    val interpreter = new InterpreterLoop(out)
    interpreter.main(settings)
    ()
  }
  ...
}

As you can see, the trait is fairly simple. It creates a "Settings" object (which is normally created by parsing off the command line arguments) and passes it into "InterpreterLoop". InterpreterLoop is the class which wraps "Interpreter" by continually prompting and executing input. This allows us to do live configuration of our program (which can be very fun).

The next thing we need to do is promote our DSL into the interpreter, so our configuration files don't begin with unnecessary "import foo.bar._" statements. Let's modify our App and the InterpreterWrapper.


object App {
   def main(args : Array[String]) {
        //Let's assume all arguments are file locations for configuration

        val interpreter = new InterpreterWrapper {
            autoImport("my.awesome.dsl._")

            for(fileName <- args) {
              addScriptFile(fileName)
            }
        }
        interpreter.startInterpreting()
   }
}

So the main code doesn't change much, simply adding the "autoImport" method call when constructing the InterpreterWrapper. However, we now need to some more magic in the interpreter wrapper. Here's the (relevant portions of the) code:


import scala.tools.nsc._
import interpreter._
import java.io._
import scala.reflect._

trait InterpreterWrapper {

  private var packageImports : List[String] = List()  
  
  protected def autoImport(importString : String) {
    packageImports = importString :: packageImports
  }
  
  /**
   * This class actually runs the interpreter loop.
   */
  class MyInterpreterLoop(out : PrintWriter) extends InterpreterLoop(None, out) {    
    
     override def bindSettings() {       
       super.bindSettings()      
       interpreter beQuietDuring {
         //This will silently execute our import statements when starting!
         for( importString <- packageImports) {
          interpreter.interpret("import " + importString)
         } 
       }
     }   
  }
  
  def startInterpreting() {
    val out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(System.out)))
    val settings = new GenericRunnerSettings(out.println _)
    ...
    val interpreter = new MyInterpreterLoop(out)
    interpreter.main(settings)
    ()
  }
}

As you can see, we're using a subclass of MyInterpreterLoop and extending the "bindSettings" method to additionally execute our import statements. We can now almost use our DSL directly. The only remaining issue is that we'd like to have our processingBuilder object be automatically bound and in scope for the interpreter session. This allows the configuration to just start directly talking with the object. Let's add "bind" functionality to the InterpreterWrapper with the following API:


object App {
   def main(args : Array[String]) {
        //Let's assume all arguments are file locations for configuration

        val interpreter = new InterpreterWrapper {
            autoImport("my.awesome.dsl._")
            bind("processingBuilder", new MyProcessingBuilderImplementationClass())
            //OR if I want to restrict access to an interface
            bindAs("processingBuilder2", classOf[ProcessingBuilder], new MyProcessingBuilderImplementationClass())
            for(fileName <- args) {
              addScriptFile(fileName)
            }
        }
        interpreter.startInterpreting()
   }
}

We now need to update the InterpreterWrapper. We'll be doing a similar trick as autoImport and adding some "silent" execution during the bindSettings method of InterpreterLoop. Here's the relevant code:


trait InterpreterWrapper {
  ...
  private var bindings : Map[String, (java.lang.Class[_], AnyRef)] = Map()
  ...
  protected def bind[A <: AnyRef](name : String, value : A)(implicit m : Manifest[A]) {
    bindings +=  (( name,  (m.erasure, value)))
  }

  protected def bindAs[A <: AnyRef, B <: A](name : String, interface : java.lang.Class[A], value : B) {
    bindings += ((name,  (interface, value)))
  }
  ...  
  class MyInterpreterLoop(out : PrintWriter) extends InterpreterLoop(None, out) {
    
     override def bindSettings() {       
       super.bindSettings()      
       interpreter beQuietDuring {
         for( (name, (clazz, value)) <- bindings) {
           interpreter.bind(name, clazz.getCanonicalName, value)
         }
         ... //auto-imports
       }
     }
  }
  
  def startInterpreting() {
    ...
  }
}

Here we're using the "bind" method of the underlying Interpreter object. This allows us to bind arbitrary objects and specify their interface. For the InterpreterWrapper we've two bind methods, one which uses the experimental Manifest feature of scala to automatically use the visible type at the bind call, and another method where you can explicitly specify a type.

We now have our configuration file being loaded and executed in a nice convenient DSL. The last bit we should do is "brand" the interpreter, so that it looks like we intend to use it (and not like scala suddenly shows up in our app). To do so we're going to override the help message, the welcome message and the prompt. Here's what our main object should look like:


object App {
   def main(args : Array[String]) {
        //Let's assume all arguments are file locations for configuration

        val interpreter = new InterpreterWrapper {
            def prompt = "Processor> "
            def welcomeMsg = "Welcome to the InterWeb Text Processing Magician! Please set up a processing stream to be executed."
            def helpMsg = "To set up a processing stream, please create an input and output and wire them together using the processingBuilder.  :help will print this message, :quit will exit the processor"

            autoImport("my.awesome.dsl._")
            bind("processingBuilder", new MyProcessingBuilderImplementationClass())
            //OR if I want to restrict access to an interface
            bindAs("processingBuilder2", classOf[ProcessingBuilder], new MyProcessingBuilderImplementationClass())
            for(fileName <- args) {
              addScriptFile(fileName)
            }
        }
        interpreter.startInterpreting()
   }
}

As you can see, using the InterpreterWrapper allows you to quickly and easily embed the scala interpreter and use it to configure your application (at start up or during execution). Remember, though, that as with an user-interface you need to worry about threading issues when attempting to run concurrent threads (or use actors).

Finally, here's the entire code for the InterpreterWrapper (under BSD license BTW):


import scala.tools.nsc._
import interpreter._
import java.io._
import scala.reflect._
/**
 * A trait to ease the embedding of the scala interpreter
 */
trait InterpreterWrapper {

  private var bindings : Map[String, (java.lang.Class[_], AnyRef)] = Map()
  private var packageImports : List[String] = List()  
  private var files = List[String]()
  
  /**
   * Binds a given value into the interpreter when it starts with its most specific class
   */
  protected def bind[A <: AnyRef](name : String, value : A)(implicit m : Manifest[A]) {
    bindings +=  (( name,  (m.erasure, value)))
  }
  /**
   * Binds a given value itnot he interpreter with a given interface/higher-level class.
   */
  protected def bindAs[A <: AnyRef, B <: A](name : String, interface : java.lang.Class[A], value : B) {
    bindings += ((name,  (interface, value)))
  }
  
  /**
   * adds an auto-import for the interpreter.
   */
  protected def autoImport(importString : String) {
    packageImports = importString :: packageImports
  }
  
  /**
   * Adds a file that will be interpreter at the start of the interpreter
   */
  protected def addScriptFile(fileName : String) {
    files = fileName :: files
  }
  
  def helpMsg : String
  def welcomeMsg : String
  def prompt : String
  
  /**
   * This class actually runs the interpreter loop.
   */
  class MyInterpreterLoop(out : PrintWriter) extends InterpreterLoop(None, out) {
     override val prompt = InterpreterWrapper.this.prompt
    
     override def bindSettings() {       
       super.bindSettings()      
       interpreter beQuietDuring {
         for( (name, (clazz, value)) <- bindings) {
           interpreter.bind(name, clazz.getCanonicalName, value)
         }
         for( importString <- packageImports) {
          interpreter.interpret("import " + importString)
         } 
       }
     }
    override def printHelp {
       printWelcome()
       out.println(helpMsg)
       out.flush()
     }
     
     override def printWelcome() {
       out.println(welcomeMsg)
       out.flush()
     }
  }
  
  def startInterpreting() {
    val out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(System.out)))
    val settings = new GenericRunnerSettings(out.println _)
    files.foreach(settings.loadfiles.appendToValue)
    //Below for 2.8.0-SNAPSHOT
    //settings.completion.value = true
    
    val interpreter = new MyInterpreterLoop(out)
    interpreter.main(settings)
    ()
  }
}

I must say, I'm very happy with how my project turned out at work. It actually made things *much* easier to debug during development as I could interact directly with the program during execution. The only complaint so far from co-workers is that they don't know enough Scala to used some of the more advanced features (for-expressions actually). However, the sample configurations I provided were enough for them to get up and running with the application and create slightly varied configurations. We have slightly more verbosity in configuration now, but the flexibility/power is incredible.

Below is the output of the exeuction of my very silly "test" usage of the InterpreterWrapper (included in the github project). This sample is using Scala 2.8.0 nightly and Paul Phillips's interpreter improvements. You can see the auto-complete suggestions below.

josh@suereth-desktop:~/projects/blog/embed-the-interpreter$ java -jar target/embedded-interpreter-0.1-SNAPSHOT-jar-with-dependencies.jar 
Welcome to Awesomeness!
This is my version of the Scala interpreter
TestInterpreter> j

java    javax   jline
TestInterpreter> j

java    javax   jline
TestInterpreter> josh.is
res0: java.lang.String = Brillant!!!
TestInterpreter> josh.

getClass   is         name       toString
TestInterpreter> josh.

getClass   is         name       toString
TestInterpreter> josh.name
res3: String = josh
TestInterpreter> 

Partial Functions + Inheritance

2009-04-13T22:33:00.010-04:00

Recently, I've been doing more work with Scala actors, and a particular problem arose with attempting to define 'abstract' actors and extend them with traits. My goal was to implement something like the "chain of command" pattern, but using partial functions and messages. To demonstrate, imagine the following system:

We're going to simulate network traffic on a machine. A machine may have any number of servers on it, each of which can attempt to handle an incoming network connection. Let's picture an web server process on that server. Let's take a first cut at modeling a "network daemon" in Scala.


trait NetworkMessage //This would be some complex object
import scala.actors.Actor
import Actor._
trait NetworkDaemon extends Actor {
   def trafficHandlers : PartialFunction[Any,Unit]
   def act() {
       loop(react(trafficHandlers))
   }
}

So why not just create the partial function for react inline? Why have a separate method that defines the function? Remember, we're trying to implement a chain of command pattern with our actor. This means our subclasses need to attempt to handle message first, and defer to us if unable. The main reason to use a "method returning a partial function" vs. actually method overriding/extending is so that we preserve the partial-function's "isDefinedAt" semantics. Because we're using actors, it is beneficial if our "handler" function does *not* work against all input, but allows things to pass through. This helps us not catch internal actors communication *and* allow things like linking actors together.

Now, let's start with modeling a web server that hosts static content.


//Extractor for staticly served content
object HttpRequestForStaticResource {
   def unapply(msg : Any) : Option[StaticResource] = None //TODO - Implement
}

//WebServer basic model
trait WebServer extends NetworkDaemon {
   def trafficHandlers : PartialFunction[Any,Unit] = {
      case HttpRequestForStaticResource(resource) => reply(resource.getContents)
   }
}

Now imagine we want two types of web servers, a 'static' server that *always* server static content, and a subclass of that which can also proxy connections to other web servers.


//Define a new extractor for proxying
object ProxyableConnection {
   def unapply(...)...
}

//WebServer basic model
trait ProxyWebServer extends WebServer {

   def trafficHandlers : PartialFunction[Any,Unit] = {
       val myHandlers : PartialFunction[Any,Unit] = {
             case ProxyableConnection(...) => forward(...)
       }
       myHandlers orElse super.trafficHandlers //Use our partials *first* before others
   }
}

Notice the boiler-plate code here. We're chaining our partial functions in a way that makes this work like chain-of-command. All partial functions from "myHandlers" should be checked before our super-class trafficHandlers. However, to make this work, we end up with a not-quite-so-good-looking syntax. This should just show itself when defining the class hierarchy, code that uses this pattern shouldn't really be able to tell how we're implementing our chain of command pattern, e.g.


//Imagine setting up a complex simulation item that has various roles mixed in.
val webServer = new ProxyWebServer with ModularWebServer {
   addModule(new CgiModule());  //In our hypothetical simulation, this simulates the overhead of making CGI calls.
}

val results = for( msg <- simulateMessages) yield {
  webServer !! msg 
}
//TODO - Loop through results and analyze simulation

As you can see, this can lead to some nice functionality and composability of your actor-entities.
The question remaining in my mind is "Can the boilerplate be reduced in scala". Ideally I would love to be able to do the following (for this situation):



trait NetworkDaemon extends Actor {
   partial def trafficHandlers(msg : Any) : Unit  //Abstract partial function
   def act() {
       loop(react(trafficHandlers))
   }
}

trait WebServer extends NetworkDaemon {
   //Implements the partial def with an implementation
   partial def trafficHandlers(msg : Any) : Unit = {
       case HttpRequestForStaticResource(resource) => reply(resource.getContents)
   }
}

trait ServletWebServer extends WebServer {
  //Completely overrides our parents partial function
   override partial def trafficHandlers(msg : Any) : Unit = {
      case ProxyableConnection(...) => forward(...)
   }
}


trait ProxyWebServer extends WebServer {
   //Adds our implementation before our parents
   override partial def trafficHandlers(msg : Any) : Unit = {
      case ProxyableConnection(...) => forward(...)
      super //This means delegate to the partial function defined on our super class (could be qualified)
   }
}


trait DevNullDaemon extends NetworkDaemon {
   //Overrides the partial definition with a *complete* definition
   //This could be useful for enforcing completeness at a given layer of the hierarchy.
   override def trafficHandlers(msg : Any) : Unit = {
         //ignore
   }
}

I'm still debating the syntax on delegating to a superclass partial function, but the general gist is to allow similar mechanisms to method over-riding/chaining but instead with partial functions, as well as the ability to define partial functions as methods on a class. Currently, I only see this feature covering a niche of use cases, but it could be quite handy for those. Behind the scenes, the compiler would be desugaring the syntax into something like the original implementation.

If anyone else is interested in seeing this kind of functionality in scala, perhaps we should get together and produce a SIP (after fixing up the "delegating to super" syntax...). I've debated trying to accomplish this with a compiler plugin and annotations, and I still may go that route for a proof of concept. (Granted, this all requires enough free time to actually finish one of my side projects!).

Tricksy JBoss AS

2009-03-28T08:53:00.002-04:00

So, initially when I heard that JBoss AS 5.0 was going to modular, of course I thought "OSGi". This has been the precedence for Glassfish and Spring DM. However, the next set of news I had was from some OSGi advocates railing on JBoss for rolling their own module system. However, I ran across this link that makes things more apparent Most Awaited JBoss 5.0 Soon to come.

If you read between the lines, it appears like JBoss went with their own module framework so that:

they could find a way to appease the OSGi community (as they propose that their implementation is compatible)

support legacy MBean components

inter operate between various implementations (conjecture: jigsaw?)

It's my opinion that this was actually a wise move, assuming the effort was not very high. This is because Project Jigsaw is still an unknown in the modularization market, and perhaps JBoss is positioning themselves to be one of the first to support it.

As long as I can use some nice OSGi module-based solutions in the meantime, I think this is a big win for JBoss as an App server choice. I'd prefer however for modularization to make into Java via a JCP (and therefore into the JEE spec), so at least there's some kind of standard implementation that can be used (with servers like JBoss able to provide proprietary extensions if desired).

Why you want to learn Scala!

2009-03-19T20:31:00.007-04:00

Why you want to learn Scala!

Below are the slides from the talk I gave to the Northern Virginia Java user's group.

A few random errata from the meeting:

A few minor introductions to Scala's type system/generics may be helpful

I messed up my description of Nothing and Unit. Page 130 of the Scala Spec has a diagram of the type system. Nothing + Null are at the "bottom" meaning they psuedo-inherit from every type.

I need to flesh out my examples a bit, especially pattern matching. I'll try to update the slides as I find time

Apache MINA + Scala - blog posting

Planet Scala - A scala news aggregator. I recommend checking out old posts from scala-blogs and graceless failures

Update: Here's a picture of me giving the presentation.

Thanks to all who attended! I thought interaction was really good, and was excited to see so much enthusiasm!

-Josh

Way to go Spring Source!

2009-03-06T08:51:00.002-05:00

<rant>

It's good to see some critical feedback on the JEE spec. I'm glad I'm not the only one questioning some of the merits of the JSRs. I'm also disappointed that the blog-o-sphere picked up on Apache's "the JCK is not really open" complaints, but seem to have completely missed (or don't have a response to) SpringSource's "too much, in the wrong places" complaint.

Granted, Seam looks to be very interesting. The biggest killer (IMHO) is JSF. Now, if the new JSF spec fixes that... perhaps it could be viable. The question remains, are Spring's complaints of "too much risk" valid? (Seam appears to be a pretty good working prototype of this JSR). Yes it would be nice to have way to easily piece together simple data driven web-apps. I personally believe JEE is still overkill for this.

Regardless, I think it is most important that the entire Java stack makes it easy to "compose" software. The easier it is to "compose" the more it encourages people to write modular code. I don't think dependency injection => modular code, but it sure encouraged *some* folks to start writing better code.

Although spring is a *rather large* codebase, it still feels to be one of the better designed library stacks for the Enterprise Development. I often find classes inside it highly useful. More so than the JEE spec. Possibly it's because they took a closure-like approach to their apis. Hmmm..... closures + java? what a novel idea.

</rant>

How should traits in Scala be used?

2009-02-14T09:35:00.018-05:00

This blog is a survey of the way I've seen traits used in various scala projects (including my own). The purpose of this article is to explore the new-found (at least to those coming from java) power of traits and determine what works well and what the cost/benefits are of each. The taxonomy used in this Blog is non-standard. I believe if I read more of the academic papers, I'd know the real names attributed to these usages, but for now I believe the names chosen will suffice to describe the concepts. (If anyone knows appropriate references to articles/papers please comment with links!)

Traits as Interfaces
This is the most-natural usage of traits for Java developers. This is where you define a trait that has no method definitions (only declarations). The traits then act merely as interface definitions, and allow developers to code to the interface of a class, rather than any specific implementation of it. I like to use these when defining entry points for third-party implementers to use when entering the "domain" of a given library. It allows my library to interact with the users classes without having to know anything about their implementations. I also like to use this for high-level constructs I pass out of my library, as it keeps the user of the library from knowing too much about the specifics of the implementation.

An example of this comes from my earliy post about writing a Scala DSL for Database Migrations. Here's the trait that was passed to all migration scripts in the DBMigrationEngine:


trait MigrationContext {
  
  def createTable[A](name : String)(creatorScript : TableCreator => A) : A
  def executeScript(queries : Function[DbVendor,String]) : Unit
  
}

I'm not sure what all to say about these, except that they provide nice clean interfaces between larger modules of your system. They can also be compilation barriers (assuming they don't change frequently and you manually specify them in return types... I know sometimes I get lazy here with scala's nice type inference).

Traits as Abstract Classes
This usage comes naturally to those of us coming at Scala from C++. A trait can be treated as an abstract virtual class with all virtual methods. This means we can proivde a very rich interface for a class, while only requiring subclasses to implement one or two "pure virtual" (or undefined) methods. The Scala collections libraries have good examples of traits as Abstract classes. Let's look at the Seq class in Scala.

The Seq class in scala implements all sorts of nice collections functions, flatMap, map, foreach, isEmpty, etc. However to create your own sequence, you only need to override the following methods:


  def length: Int
  def elements: Iterator[A]

This is perhaps the style of traits that I find myself developing most for my various projects. It has all the familiarity of a Java abstract class, but all the power of real multiple-inheritance.

Traits as Bolt-Ons
This is where trait self types really find their use. The goal of a bolt-on trait is to add functionality (or "bolt on") to an existing class. Bolt-on traits are very similar to decoraters in that they "decorate" existing classes with functionality. As you can create mixin traits when instantiating objects, it has almost as much flexibility in construction as Decorators, only methods don't need to pass down to delegates. Bolt-on traits serve more purposes than just decoration though (as the above example doesn't quite fit the decorator pattern).

In my own projects, I've created two very simple bolt-on traits that I think show case the idea. These two traits are "InitializingSpringActorBean" and "DisposableSpringActorBean". The basic of just of these traits are that they are to be mixed in *only* with Actor subclasses. They implement the "InitialzingBean" and "DisposableBean" interfaces of Spring in the context of actors. Here's the source code for the InitializingBean:


/** 
 * This is a trait which will start an actor after a bean's properties have been set.
 */
trait InitializingSpringActorBean extends InitializingBean { 
  self : Actor =>  
  override def afterPropertiesSet() {
    start
  }
}

Notice the usage of the self type. This ensures that the trait can only be mixed into an actor class, and (IMHO) is what makes it a "bolt-on" trait. This particular trait implements an interface from spring that enables the Spring BeanContext (or IoC Container) to call "start" on an Actor at the correct point in the Spring lifecycle. Typical usage of this would be:


/**
 * Some kind of bean the emulates business logic.
 */
@Service("CoffeeBean")
class CoffeeBean extends Actor with InitializingSpringActorBean {
  ... Injected properties...
  override def act() {
        ...
  }
  //No need to call start as spring knows how/when to start us now!
}

The thing to always remember about traits, is that when editing one that defines methods, you must recompile all the *subclasses* of the trait as well. This is because the mangling Scala uses to implement traits duplicates the methods in each of its concrete subclasses. So far, most of my bolt-on classes have been rather simple, and do not need to be constantly tweaked/recompiled, but this is a valid concern for a large project. As in C++, I would reserve bolt-on traits for cross-cutting concerns or specific applications. (The most commonly used "bolt-on" from my C++ days was boost::non_copyable, which luckily for me, hardly ever changed! However the issues are the same in Scala as they were in C++ [assuming all code resides in header files])

Traits as Class Ecosystems
This is perhaps one of the neatest (and dangerous) usage of traits I've seen so far in Scala. This is where a trait contains one or more definitions of classes that operate together. These classes are all used to solve a particular problem. If trait includes some high level methods to call into it ecosystem, it really starts to resemble a facade pattern. However, there are many other uses of this, such as the Virtual Class pattern (shown off in the eclipse plugin). This use of traits (once again in my opinion) has the potential for pure awesome and the greatest potential for pure evil. Here is actual Ecosystem trait code with the code removes to help protect the innocent (or guilty?)


trait Files {
   class FileDependencies{
        class Tracker extends OpenHashMap[File, Set[File]]{...}
        ...
   }
   object FileDependencies {...}
   case class File ... {...}
}
object Files extends Files

You can see that the above ecosystem consists of a FileDependencies class, a FileDependencies Object and a File class. The ecosystem trait also provides convenience methods for using the above classes (not shown above). In the above example, the trait actually provides some implicit conversions for "files" form another ecosystem (namely the java standard library) to be automatically converted into this ecosystem. Finally (and important to me) is there is a static Object provided from which one can gain access to the ecosystem *without* mixing in the trait (i.e. import Files._ works somewhat similarly to extends Files). Ecosystem traits also take form in two interesting ways in the Scala Compiler and the Scala Eclipse Plugin.

I'm going to hold off discussing the Scala compiler because it deserves its own blog entry. The Scala Compiler uses what is known as the "Cake" pattern. Martin Odersky (creator of Scala) outlines the Cake Pattern in his description of how the scala compiler was creating in his paper "Scalable Component Abstractions." The paper is quite worth the read, and you should all google for it and read it RIGHT NOW. However, the pattern isn't without critics. I've heard it mentioned a few places as the "Bakery of Doom" used in the compiler. Why do some developers love this pattern and others hate it? Well, you'll have to wait for my next entry, as I'd like to really spend some time delving into this.

For now, let's look into an easier (and smaller) code base for trait ecosystem examples: The Scala Eclipse Plugin.

The Eclipse Plugin and Virtual Class pattern.
The Scala Eclipse Plugin makes heavy use of "Virtual Classes". The original author of the plugin (Sean McDirmid) outlines a simple skeleton of the pattern/idiom used when creating Virtual classes in scala here. Virtual Classes originated (AFAIK) in the programming language BETA, and is described here. BETA is an interesting language (for those interested in languages), but the pattern itself holds interest to us in Scala, and is used a great deal in the Eclipse Plugin.

The main benefit of the Virtual Class "pattern" is that you can layer not only your application, but your "types" or classes". At the lower levels, the class only contains methods needed by lower levels, and as you move up the inheritance chain, the Class "fills out" with all the necessary method. This helps create a minimal interface for the lower levels, while providing a robust interface for the higher levels.

The virtual class pattern is great when you need to slowly build a type in a variety of layers, and slightly change behavior in each layer. However, it does have some downsides. One of which is that unit testing partial types isn't a walk in the park. You actually need to create a "final" layer for the virtual class and unit test that. You end up doing this for every layer in the pattern. As creating a new layer means implementing the entire ecosystem, you're no longer really unit tests, as you are integration testing. I believe as long as you *do* testing, it shouldn't affect the overall stability of your project.

The real issues show up in terms of sprawl. I believe an inherit weakness in using virtual classes is the desire to shrink your layers into minimal sets of functionality (a good thing) and then re-use layers in many many different places (perhaps not so good a thing). The virtual class layers really need well-defined boundaries in the system and preferably low-level (or facade-like) entry points to keep them loosely coupled. If you decide to create virtual-classes for things like lists and strings, it would be very hard to loosely couple this code. (Reminds me of private inheritance from C++) The eclipse plugin really showcases the issues involved in designing with virtual classes (as well as showcasing the benefits). The plugin actually implemented a much faster (but not 100% correct) algorithm to improve recompilation speeds for itself. This is because of the massive amount of trait-inheritance sprawl in the plugin.

To illustrate my point, I've tried to make a few Ven diagrams of the "ecosystem traits" in the eclipse plugin. The names of each "bubble" are real trait names. I've left out showing the internal ecosystem classes, as the diagram gets very cluttered very quickly. In terms of a Ven Diagram, you can assume that any trait (or bubble) encapsulating another bubble mean "inheritance", as inheritance of "ecosystem" traits involves merging the two, well, ecosystems of objects. Here's the diagram just for the lampion.core package:

Notice how most of the ecosystem traits extend each other before reaching the RangeTrees trait. This is a side effect of using the Virtual Class pattern. Each trait partially defines a type with the outer ecosystem traits refining the ones from the inner ecosystem, as well as adding new types. Things get slightly more confusing when we bring in one more scala eclipse plugin package (and this is the furthest I'll go for clarity).

The next diagram not only has the above ven diagram features, but I've also added thick black lines to denote "is-the-same-as" and dashed lines to denote package boundaries. As scala promotes multiple inheritance this idea of one ecosystem encapsulating another gets interesting, as it's possible to merge the same ecosystem from different parent ecosystems. You'll see interesting code spattered around the eclipse plugin to handle this. Here's my representation of the lampion.core and lampion.compiler plugin:

As you can see besides a horrible color selection, things in the plugin are really getting interesting. The Parsers trait inherits every ecosystem class from lampion.core (except for Plugin). The consequences are any change to any class (even internal to an ecosystem) involve recompiling parsers (and therefore typers and tokenizers... not to mention their subclasses). Once again, realize that the eclipse plugin defines its own dependency analysis to recompilation to attempt to bring its cost down to one that is acceptable for development. This seems to me a like a "bad smell" to on of the pieces in the plugin, and I'm pointing at the over-use of virtual-classes (and traits in general) in the plugin.

My biggest complaint (as stated before) with the design (as it exists in the eclipse plugin) is there are no boundaries for the Virtual Class pattern. As you can see, if I were to ven diagram the eclipse plugin, almost all classes would be encompassed be a few high-level classes, with minor amounts of utility code/standalone classes here and there. Not only is the barrier to learning the pattern high (without appropriate documentation), the maintenance (and recompilation) costs are also high. The virtual-class pattern is useful in the right contexts (and a portion of the eclipse plugin might even have that context), but it seems like more of a golden-hammer approach as it stands, and tends to dissuade me from using virtual classes in plugin enhancements/bug fixes that I try to contribute.

I also believe one reason the eclipse plugin gets far more criticism than scalac (besides lack of design documentation) is the lack of automated testing. Although unit testing (as mentioned above) is rather difficult, integration testing is not, and there should be a very strong integration test suite accompanying the plugin. This would not only prove the virtual class pattern is viable for its current usage, it could also help new developers see how the layers work, and what is expected of each. I would love to see a project that uses the virtual class pattern with a strong set of integration tests to prove my theory correct (that integration testing can make up for the pain of unit testing virtual classes).

Conclusions

If you have any examples of traits being used (either efficiently or horribly) please make some noise! I'd love to get a better sampling to color the descriptions in this blog.

To the Java developer, Scala provides new-found power, and requires a modified approach to design. Coming from C++ to Scala, I've been able to apply quite a bit of knowledge on software architecture, however there are still some things that Functional Programming brings to the table that I need to wrap my head around. I'd like to challenge those that have real-world experience in designing large-scale scala systems to start sharing some knowledge of helpful practices/tips and architectures.

Levaraging Annotations in Scala Part 2: scalac Plugins

2009-02-05T19:41:00.012-05:00

This post began as a discussion on #scala about how people define a public "getter" method and private "setter" method using only defs and vars in scala. The usual method in scala is:


class MyClass {
   private var x_private : Int = _
   def x = x_private
}

The downside to this is that you have two names to express one concept. I'm at the point where I'm no longer caring too much about public/private parts of classes, but I thought I'd tackle the problem to exercise my new-found compiler skills. It turns out this feat is pretty easy to accomplish via a scalac plugin.

To create your first plugin you should follow the guide here. We'll be using the Maven as our build tool, since to do otherwise would be blasphemy (for me).

To start off with, here's the POM file for our plugin:


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>org.scala-lang</groupId>
 <artifactId>private-setter-scalac-plugin</artifactId>
 <packaging>jar</packaging>
 <version>1.0-SNAPSHOT</version>
 <name>Var definition extensions for the scala compiler</name>
 <url>http://suereth.blogspot.com</url>
 <repositories>
  <repository>
   <id>scala-tools.org</id>
   <name>Scala-tools Maven2 Repository</name>
   <url>http://scala-tools.org/repo-releases</url>
  </repository>
 </repositories>
 <pluginRepositories>
  <pluginRepository>
   <id>scala-tools.org</id>
   <name>Scala-tools Maven2 Repository</name>
   <url>http://scala-tools.org/repo-releases</url>
  </pluginRepository>
 </pluginRepositories>
 <dependencies>
  <dependency>
   <groupId>org.scala-lang</groupId>
   <artifactId>scala-compiler</artifactId>
   <version>2.7.3</version>
  </dependency>
 </dependencies>
 <build>
  <plugins>
   <plugin>
    <groupId>org.scala-tools</groupId>
    <artifactId>maven-scala-plugin</artifactId>
    <executions>
     <execution>
      <goals>
       <goal>add-source</goal>
       <goal>compile</goal>
       <goal>testCompile</goal>
      </goals>
     </execution>
    </executions>
    <configuration>
     <jvmArgs>
      <jvmArg>-Xms64m</jvmArg>
      <jvmArg>-Xmx1024m</jvmArg>
     </jvmArgs>
    </configuration>
   </plugin>
   <plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>shitty-maven-plugin</artifactId>
    <executions>
     <execution>
      <goals>
       <goal>clean</goal>
       <goal>install</goal>
       <goal>test</goal>
      </goals>
     </execution>
    </executions>
   </plugin>
  </plugins>
 </build>
</project>

There's a few things in this pom, mostly to ensure that the scala-tools repositories are available too us. Also, ignore the shitty (Super-Helpful-Integration-Testing-ThingY) plugin for now. The most important part is that we're compiling to a jar file, and we're depending on the scala-compiler. The version of the compiler we depend on is the *only* version of scala your plugin should be used with. In all reality, the scala-compiler should probably be deifned as a 'provided' dependency and the scala-library should explicitly be depended on, but for now we'll cheat on completeness.

Next we need to make sure there is a scalac-plugin.xml file in the created JAR. This is simple in maven, just place one in the src/main/resources directory. Here is what my src/main/resources/scalac-plugin.xml looks like:


<plugin>
  <name>private-setters</name>
  <classname>org.scala_lang.privateSetter.internal.privateSetterPlugin</classname>
</plugin>

You'll notice I'm naming my plugin "private-setters" and placing it in a "internal" package. This could be my eclipse plugin development rubbing off, but this helps me know what only the compiler should see.

Next we need a way for clients of our plugin to "notify" us that they want a var with a private setter method (varname_=), but public getter (varname). Here's my initial cut at client syntax:


import org.scala_lang.privateSetter._
class TestWidget {
  @privateSetter
  var myVar = 5  
}

Pretty simple really, but effective. (Also I already know how to look up annotations from my scala-mojo-support project, so it only took me a few hours to work out the scalac plugin details. This post actually took the most amount of time in the whole venture).

Now we need to define the privateSetter annotation we can use in our classes. It's a fairly simple file:


package org.scala_lang.privateSetter
/** This annotation is used to make a var's setter method be private */
class privateSetter extends StaticAnnotation {
}

Next we need to write the plugin itself. All plugins contain some boiler-plate code, so we'll ignore that for the time being. (see the documentation mentioned above). The truly interesting part of this plugin is the newly defined phase.

For some history, the Scala Compiler (scalac) is composed of various "phases". Each phase has a responsibility it performs. Some phases are easy to identifier e.g. "icode" which converts the AST to icode for each "compilation unit". In the Scala Compiler a compilation unit corresponds to a source code file and may produce multiple class files. Here's the scalac -Xlist-phases output on my machine:

$ scalac -Xshow-phases
namer
typer
superaccessors
pickler
refchecks
liftcode
uncurry
tailcalls
explicitouter
erasure
lazyvals
lambdalift
constructors
flatten
mixin
cleanup
icode
inliner
closelim
dce
jvm
sample-phase

On to our implementation! The basic structure of our phase looks like this:


/** The actual phase the removes units from being compiled that are up-to-date */
    class MakeSettersPrivatePhase(prev: Phase) extends Phase(prev) {
      
      override def name = VarAccessChanger.this.name
      import global._
      /**
       * Called when our plugin is running.
       */
      override def run {
 for (unit <- global.currentRun.units; if !unit.isJava) {
           unit.body = TreeTransformer.transform(unit.body)      
 }
      }
      ...
}

I'll mention that this class is nested inside an outer "Plugin" class which is passed the 'global' object. For those of you unfamiliar with the compiler, the "Global" object is the outer layer of the "cake" pattern used by the Scala Compiler. I'm reserving judgment on Global, but I have noticed that it's very hard to unit test any "module" you write for Global (as most of them have the "self:Global =>" syntax).

The Transformer is a very nice class for doing AST manipulation (thanks DRM for suggesting it). This class simply transforms the AST from one form to another. It's perhaps the easiest way to implement our plugin. Let's set up a Tree Transformation that does absolutely nothing useful:


/** 
       * This object transforms the AST of a compilation unit apply our PRIVATE metamorphosis for vars.
       */
      object TreeTransformer extends Transformer {
        /**
         * This method transforms individual nodes of the tree.
         */
        override def transform(tree: Tree) = tree match {
         case t =>         
           super.transform(t)
       }
      }

The structure here is we override the transform method. This method takes a tree and returns a tree. We want to transform *only* the setter part of a var method if it contains the privateSetter annotation. Let's apply our pattern matching skills to the test with an extractor: The "AnnotationSetterShouldBePrivate" extractor.


/** 
 * Selector to remove annotated var setters 
 */
object AnnotatedSetterShouldBePrivate {
  /** 
   * Pattern matcher for privateSetter annotated setter methods 
   */
  def unapply(node : Tree) : Option[DefDef] = {
    /** 
     * Helper method to determine if a public setter method has the privateSetter annotation. 
     */
    def hasPrivateSetterAnnotation(annotations : List[Annotation]) : Boolean = {
      for {
           annotation <- annotations
           if annotation.tpe.safeToString == classOf[privateSetter].getName
      } {
        return true
      }
      false
    }
    node match {
      case x @ DefDef(mods,name,_,_,_,_) if name.toString.endsWith("_$eq") =>
         if(hasPrivateSetterAnnotation(mods.annotations)){
            Some(x)
         } else {
            None
         }
      case _ => None
    }
  }
}

First, note that our extractor (unapply method) takes in a tree and returns a DefDef. DefDef is the AST class for a "def" node. All var's are parsed into dual "def" methods (setter and getter). We define a helper method that takes a list of annotations and looks for our "privateSetter" annotation. The actually implementation of the extractor matches against the tree node, checks to see if it's a DefDef and has a tailing name of "_$eq". "_$eq" is the mangled form of "_=" which is the convention for scala setter methods. Note that I probably could move the if/else statement into the pattern match on the DefDef, but wasn't feeling adventurous enough this evening. In the case where we find a valid annotated setter DefDef method, we return it, otherwise return None.

Now that we have our extractor, writing the tree transformation becomes fairly simple:


/** 
 * This object transforms the AST of a compilation unit apply our PRIVATE morphosis for vars.
 */
object TreeTransformer extends Transformer {
    
  /**
   * This method transforms individual nodes of the tree.
   */
  override def transform(tree: Tree) = tree match {
      case AnnotatedSetterShouldBePrivate(d @ DefDef(mods,name,tparams,vparams,tpt,impl)) => 
          import symtab.Flags._
          val tree = copy.DefDef(d, mods | PRIVATE,name,tparams,vparams,tpt,transform(impl))
          tree.symbol.setFlag(PRIVATE)              
          tree
      case t =>        
          super.transform(t)
  }
}

As you can see, we're combining our "AnnotatedSetterShouldBePrivate" extractor with the natural extractor for the DefDef case class so that we can pull out all the constructor variables (along with the DefDef itself using the "d @" syntax). Initially I tried returning just a newly constructed DefDef with "mods | PRIVATE" instead of "mods" in the constructor. You'll find this blows up horribly. The main issue is that the AST nodes contain *more* than just their constructor values (types and symbols being the two things I found). The Transformer class provides a "copy" value/object that you can use to "copy" various parts of the tree. The copy class contains a method for every tree node that takes an original tree and overriding constructor values. For our purposes, we're applying the PRIVATE flag to the "mods" attribute of the DefDef and two the symbol flags for the DefDef. The symbol flags are what are eventually used in the icode->bytecode conversion code.

Next we should choose what phase to run this plugin after. I've chosen the "typer" phase, as this ensures we at least have an AST and the types are correct. When defining a class in isolation, this is working perfectly. However when defining the class and using it with other classes, I'm running into the difficulty where the methods I'm modifying are eventually being replaced with public methods of a differing names. To look into this, we should set up some integration tests. NOW we can use the SHITTY plugin!

The maven shitty plugin allows you to execute "integration" projects that depend on the currently "building" project. You simple create a directory in src/it, add a pom.xml that depends on your project (with a version of "testing"), and a goal.txt that describes which maven goals should be executed. If the integration project's maven build succeeds, the overall project's maven build continues. If an integration project's maven build fails, the entire build fails. This works great for "positive" tests (or test where you make sure that things compile with your plugin). Let's define a relatively simple positive test. First, we take the class outlined earlier as our example syntax:


package org.scala_lang.privateSetter
/** This annotation is used to make a var's setter method be private */
class privateSetter extends StaticAnnotation {
}

Next we create a pom file for this integration test:


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>org.scala-tools</groupId>
 <artifactId>testPrivateSetter</artifactId>
 <packaging>jar</packaging>
 <version>1.0-SNAPSHOT</version>
 <name>test-privateSetter-scalac-plugin</name>
 <dependencies>
  <dependency>
   <groupId>org.scala-lang</groupId>
   <artifactId>private-setter-scalac-plugin</artifactId>
   <version>testing</version>
  </dependency>
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>4.5</version>
   <scope>test</scope>
  </dependency>
 </dependencies>
 <build>
  <plugins>
   <plugin>
    <groupId>org.scala-tools</groupId>
    <artifactId>maven-scala-plugin</artifactId>
    <executions>
     <execution>
      <goals>
       <goal>add-source</goal>
       <goal>compile</goal>
      </goals>
     </execution>
    </executions>
    <configuration>
     <compilerPlugins>
      <dependency>
       <groupId>org.scala-lang</groupId>
       <artifactId>private-setter-scalac-plugin</artifactId>
       <version>testing</version>
      </dependency>
     </compilerPlugins>
     <args>
      <arg>-verbose</arg>
     </args>
    </configuration>
   </plugin>
  </plugins>
 </build>

 <repositories>
  <repository>
   <id>scala-tools.org</id>
   <name>Scala-tools Maven2 Repository</name>
   <url>http://scala-tools.org/repo-releases</url>
  </repository>
 </repositories>
 <pluginRepositories>
  <pluginRepository>
   <id>scala-tools.org</id>
   <name>Scala-tools Maven2 Repository</name>
   <url>http://scala-tools.org/repo-releases</url>
  </pluginRepository> 
  <pluginRepository>
   <id>snapshots.scala-tools.org</id>
   <name>Scala-tools Maven2 Snapshot Repository</name>
   <url>http://scala-tools.org/repo-snapshots</url>
  </pluginRepository>
 </pluginRepositories>

</project>

You'll notice we're making us of the "testing" version of our plugin AND the "compilerPlugin" configuration option of the maven-scala-plugin. This option is new to the (not yet released except as a SNAPSHOT) 2.10 version, and allows you to depend on any number of scalac plugins during your build. We're using it now to depend on our build. Our goals.txt simply consists of "clean compile".

Another thing that the SHITTY plugin lets you do is provide a 'validate.groovy' file with your pom.xml and goals.txt. This file is run after a build to ensure things were successful. We can use this to ensure our generated classfiles have private setters. We'll tackle that problem another day (I'm currently being lame and running javap on the .class files).

I think I've typed as much as I can for one night, I'll try to cover the remaining pieces (after I code/finish them) later. Once again, there are some issues with the plugin as I'm confusing the hell out of some of the compiler phases (not to mention being confused myself as to where things happen in some cases). If you'd like to look at the source (and perhaps contribute? ), it's available on github: private-setter-scalac-plugin

The Protocopter Story

2009-01-20T20:45:00.021-05:00

To steal the words of Chesterton... This posting is about a yachtsman, who thinks he's discovered a new island off the south seas, only to realize he had discovered England. The man lands (armed to the teeth) and plants the British flag in a barbaric temple which turns out to be the pavilion in Brighton. As Chesterton states, I will reiterate "I am that man."

The Beginning

The story begins over a year ago with Adam Lindsay's invention of LOLCode. I was shown this language sometime in the summer of 2007, and decided I would try my hand at writing an Eclipse plugin for it. The plugin turned into an interpreted runtime, which flowed to my more active involvement in the shaping of the language. While the language itself is rather humorous and very readable, the most interesting part to me were the concepts we were describing.

LOLCode is designed by committee, which although allows for many various interesting ideas to get discussed, also leads to very-long debates and slow decision making. However, a few major pieces of LOLCode were decided early that shaped it into a very interesting language. Of primary importance was the appeal to feline programmers. This untapped market could revolutionize the software business for decades to come. It was later agreed that other fuzzy and/or exceptionally cute lifeforms should also be allowed to participate in this revolution. The most exciting decision (for me) was the decision to use Prototype-OO as the basis for the language. Most other design decisions fell from this.

While designing LOLCode, not only were we having fun, we were also toying with some very interesting programming language designs. The choice of prototype-OO lead to several other interesting choices, that eventually got a few of us excited about the possibilities what the language could express. Not only could you perform acts for others, you may also be able to do so succinctly and elegantly. This lead to the idea for "Protocopter"

A Language that can fly

Protocopter is a language runtime (currently very buggy + implemented on top of Scala) that we were using to test out the ideas of LOLCode in a very minimal syntax. The idea was to reduce parse complexity (as most of us weren't compiler designers by profession) while ironing out details in the runtime. So far, we've had some very exciting results, and I'd like to go over the general design/implementation of Protocopter/LOLCode for you. For future reference I will only use protocopter, but the designs are interchangeable at this point. Protocopter is almost a "vm" for LOLCode.

SLOTS IN YR OBJECT

The primary design of Protocopter revolves around its object construct. An object is a "thing" that may contain any number of "slots". Each slot has a name, and contains any number of objects. The concept is very similar to Java's Map<String,Object> and almost exactly the same of a JavaScript object.

The most interesting part about Protocopter objects are that they have prototypes. A "prototype" is essentially a parent class only it remains an object. In Ruby, you can access a class and inject methods into it, but in Protocopter there is no need. Everything is an object. IF you want to create a "class", you simple make an object (places all methods/fields you want inside) and then prototype it to create "instances". The concept shouldn't be new for any AJAX developer, but it can be rather unintuitive for C/Java programmers.

The hard part in dealing with prototypes is defining how to look up a slot name on an object using its prototypes. In Javascript, when looking for slot 'x', the object itself is initially queries for slot 'x'. Then it proceeds to the prototype of the object, then the prototype of the prototype and so on. In Protocopter we made the decision to allow a list of prototypes per object. This means we needed a more sophisticated algorithm. At the writing of this article, we've implemented a Depth-First search on the prototype list. This means we first check the first prototype, then its first prototype, etc, until we've check as far down the tree as possible (the entire list), then recurse backwards searching for slot names. The algorithm also marks objects that have been visited to prevent cyclic references from causing infinite lookup times.

Now that we've defined objects, and slot lookups, what do prototypes really look like? The prototype list needs to be modified at runtime, therefore it should fit in a slot. Not only that, the prototype list is a LIST, ideally it should prototype the "ListPrototype" object, so we can re-use list functions when working with it. All of these decision were made, although we haven't thoroughly defined the "ListPrototype" object, or how one would work with it.

Putting FUN back into FUNction
Function are the next interesting piece to Protocopter. There was much debate over how to create them, and what they should look like. It was decided that the language should support blocks of code. Blocks of code operate with some "scope object". A scope object is the object used to look up local variables when a "block" is executing. The block references these slots on the scope object without any qualifiers.

Note: The initial script run by Protocopter is run in the scope of an anonymous prototype of the "Object" object. Other potential prototypes may be implementation defined, but are non-standard. "Object" is the only global inside Protocopter and the prototype of *every* object in the system.

It was then decided to turn Functions into first-class objects. A function consists of a code block and a list of arguments. When calling a function, the function object is prototyped, and the arguments/self slots are assigned to the prototype. The anonymous prototype is then used as the "scope object" for the code block of the function. To turn a function into a closure, simply insert the "scope object" for the scope a function is created, into the function's prototype list. Not only was this approach elegant, it made implementation much simpler (as it uses the already existing slot-lookup algorithm + prototype system).

WE HAZ A FLAVORZ
The next part of protocopter was defining enough operations to make it a viable "VM" for lolcode. The operations are listed below in the "Protocopter" syntax. Later in the article I attempt list the relative LOLCode->Protocopter conversions.
Slot Access

 expr '.' identifier

This access the value of a slot from an object

Call Code Block/Function

 '!' expr expr expr*

This calls a given code block with the scope/arguments

Assign Slot

 l-value '<-' expr

This assigns a value to a given slot

Create Prototype

 '~' expr

This creates a prototype of an object

Append Prototype

 l-value '<<' expr

This appends a prototype to a given object

Code Block

 '{' statement* '}'

This creates a code block which will execute a series of statements.

Literal Translation

 '@' expr

This treats a string expression as a code-literal (for slot lookup)

The above form the basis for Protocopter syntax (and allow mapping from LOLCode). I've also (somewhat successfully) tried to define a few convenience extensions in the current implementation. These are not needed in the language, but helped me write some test cases quickly.

Function Definiton

 
'{'  ('|' arg-list '|')?  statement* '}' 
arg-list := arg (',' arg)*

This creates a Function object that takes a series of arguments.

Call-function inline syntax

expr '!' identifier call-arguments

This syntax calls a function identified by a given slot on an object, using the object as "self" for the function.

Function Closure syntax

'[' ('|' arg-list '|')? statement* ']'

This syntax creates a function that automatically prototypes the current scope.

Code-Block Closure syntax

'[' statement* ']'

This syntax creates a code-block that automatically prototypes the current scope.

IM IN YR LOOP WHILE YR NOT!

Now that we have some syntax available, it's easier to describe some issue's we're running into. The first is how to deal with while loops. Ideally, we want to implement tail-recursion optimizations and define while purely inside Protocopter like the function shown below (assuming 'if' is a valid function):


Object.while <- { |condition, loop|  
  ! if current (!condition) [ ! loop;  ! Object.while current condition loop ]
}

Note that I'm using some lazy syntax here with the || arguments, but let's look at the one-liner implementation. First, you should note that code blocks return the last value executed in their scope. Second you should note that "current" always refers to the current scope object. I could accomplish this same code in more lines, but I'm preferring brevity here. It might be more apparent what's going on if I write it without the syntax extensions.



Object.while <- ~Function
Object.while.block <- {
   trueBlock <- { 
      !loop 
      ! Object.while current condition loop
   }
   trueBlock << current
   ! if current (!condition) trueBlock
}
list <- ~ Object.libs.collection.List
list.0 <- 'condition'
list.1 <- 'loop'
Object.while.args <- list

Notice first, that we're creating an anonymous prototype of "Function" and placing a code-block and argument list inside it. Then notice to create a closure, we're making a code-block and then appending the current scope as a prototype. Also notice that we're using a very low-level mechanism of creating a list (assigning numbered slots). Although it gets into the details, I hope that's detailing some of what is going on at the lower-levels of protocopter.

The goal with the protocopter language is to provide a very simple low-level base that LOLCode can be implemented on, while provided nicer higher-level language features that make writing boiler-plate test-code easier (like functions/closures, etc.).

My next goal is to provide a few layers of niceties on top of the existing language extensions so you don't always have the verbose: "! function scope arg1 arg2" nonsense every time.

CAN HAZ CHEEZBURGER?
The most important part of Protocopter is the ability for LOLCode to compile into Protocopter instructions. Currently they're both interpreted, so it's a mere matter of source->source translation, but hopefully in the future Protocopter could be a form of byte-code.

Let's take a look at some LOLCode 1.3 spec examples and how they would look in Protocopter.


I HAS A var ITZ 0 
---------------------------------------
var <- 0


HOW DUZ I foo YR bar
 VISIBLE bar
        FOUND YR NOOB
IF U SAY SO
---------------------------------------
foo <- { |bar|
     ! print current bar
     null 
}


I HAS A thingy ITZ A Bukkit
---------------------------------------
thingy <- ~ Bukkit


O HAI IM tired IM LIEK sleepy
   I HAS A thingy ITZ 0
KTHX
---------------------------------------
tired <- ~sleepy
! { thingy <- 0 } tired


O HAI IM pile
     I HAS A length IZ 0

     HOW IZ I pushin YR item
          ME HAS SRS length ITZ item
          length R SUM OF length AN 1
     IF U SAY SO

     HOW IZ I popin
          length R DIFF OF length AN 1
          BTW Do bounds checking...
          I HAS A mom ITZ ME'Z SRS length
          ME'Z SRS length R NOOB
          FOUND YR mom
     IF U SAY SO
KTHX
---------------------------------------
pile <- ~Object
! {
   length <- 0
   pushin <- { |item|
      self.(@length) <- item
      self.length <- ! + current self.length 1
   }
   poppin <- { 
      self.length <- ! - current self.length 1
      mom <- self.@length
      self.@length <- null
      mom
   }
} pile


I HAS A fishBarrel ITZ A pile
fishBarrel IZ pushin YR "halibut" MKAY
fishBarrel IZ pushin YR "trout" MKAY
fishBarrel IZ pushin YR "salmon" MKAY
VISIBLE fishBarrel IZ popin MKAY  BTW prints "salmon"
---------------------------------------
fishBarrel <- ~ pile
! fishBarrel.pushin current "halibut"
! fishBarrel.pushin current "trout"
! fishBarrel.pushin current "salmon"
! visible (! fishBarrel.popin current )

NOOOO HEZ STEALIN MAH BUKKIT!
We've almost discovered England, and dug our flag into Brighton. The only thing left is to look into a language known as IO. I discovered this language a few weeks ago and it made me very excited and also very sad. Not only is it almost exactly what we've been trying to implement with Protocopter, it even ups the ante with actor-based objects. In I/O all objects are actors and all slot-access/function calls are messages between actors. Most of the other work is already in-place in I/O. While this gives me the satisfaction of knowing we were on the right track, it also begs the question: What should we do with Protocopter?

Currently the I/O language is a little too far removed for LOLCode to compile into its world. However, would it be easier to finish Protocopter implementation, or re-define LOLCode 1.3 spec to fit the I/O language design? I leave the debate up to the readers and the LOLCode forums.

The unfortunately reality is that most of us trying to hack on Protocopter and LOLCode have more-than-full-time jobs and kids. I also try to place the needs of the scala-maven-community before some of my other nerdy hobbies (which are way below family/work), so Protocopter ends up being pretty low on the totem pole. Perhaps given enough insight, caffeine and hilarity to leverage, you'll see a 0.10 release of Protocopter (and 1.3 of LOLCode) out soon.

Anyone willing to help, please email me and we'll set you up with access to the project and whatever else you need. This blog now stand as the greatest single document of accumlated LOLCode-thought for the 1.3 spec. If you'd like to see the thought process that arrived at this point, please check out the LOLCode forums.

The current major issues in Protocopter are:

Sequence/List Prototype Definiton

Using Sequences for Prototype lists on objects, and runtime manipulation

self/this references in functions. How can we pass "scope" + "self" without being incredibly verbose

actor/continuation implementation + cleanup

parser refinement

Code-Block slot/metadata definitions

Leveraging Annotations in Scala

2009-01-12T20:06:00.010-05:00

This post attempts to explain how to leverage Scala annotations in the Scala language.

I'm currently working on the ability to create Maven 2 plugins (Mojos) in the Scala language. Maven plugins are pretty easy to create, they basically involve creating a class that extends AbstractMojo and annotating fields on that class that should be injected with configuration/runtime information from maven. Bringing this feature to Scala however, required me to make some trade-offs. Should I use the "annotations in the documentation" approach that maven uses for java + groovy mojos? Or should I leverage Scala's annotation support (which means I don't have to write a parser)? I chose the annotation method, and below is how to make it work.

First, we need a few annotations to leverage. In the case of Maven Mojos, they're almost all defined for me. Here's a look at the "goal" annotation:


class goal(val name : String) extends StaticAnnotation

Scala defines two types of annotations (both extending from Annotation): Static and Classfile. Classfile annotations are retained in the .class file (i.e. the byte code) and are available at runtime (I believe?). Static annotations, on the other hand are 'available to the compiler'. I haven't read through the SLS (Scala Language Spec), so I'm not quite sure exactly what this means, but I know they're at least available in the Abstract Syntax tree (what we care about).

Those of you who use Java annotations will notice that Scala annotations do not provide as many options as Java annotations. I'm hoping this changes with time, but currently Scala takes the standpoint that Java does a well enough job of defining annotations and annotation processing tools. While I agree, I'd still like to be able to define annotations in Scala with a few more options. However, defining annotations in Scala is more elegant than with Java (That's right Mr. @interface....)

Next, let's make our first Maven mojo (something like the following):


@goal("echo")
class EchoMojo extends AbstractMojo {
 
  @throws(classOf[MojoExecutionException])
  override def execute() {
    getLog.error("HAI WURLDZ!")
  }
}

Next, we need to parse this file and create a "mojo description" xml file. Maven provides us with an interface that is used to look for mojo-annotated classes. We'll assume in your case (if you're leveraging annotations) that you already have hooks for where you will parse, and what you need to parse out. To actually perform the parsing, we're going to steal a trick from scaladoc + the eclipse plugin called a "presentation-only" compiler.

A "presentation-only" compiler is fairly easy to construct, you simply subclass the Scala "global" object and override its "onlyPresentation" method (as shown below). The "Global" class is the amalgamation of all the various modules of the Scala compiler, plus a few methods to execute compilation.


val compiler = new Global(settings, reporter) {
    override def onlyPresentation = true
}

The Global class takes a settings object and a reporter. The settings object contains all of Scalac's command-line settings in code form. This is where you'll have to set the classpath and other fun flags (luckily for me, I get those for free from my maven mojo extractor interface). The Reporter is simply a "strategy" for where to pipe error/compilation output to. Once again, my choice is simple: The maven logger.

After creating a compiler, you actually need to execute it against a list of source files (hopefully the ones containing your annotations) like so:


val run = new compiler.Run
run.compile(sourceFiles.toList)

The "run" of the compiler should parse through all the files and generate an Abstract syntax tree for you. After which, you can easily go rip through them all and process them, like so:


for(unit <- run.units if !unit.isJava) {
  extractMojos(unit)
}

I'm purposefully ignoring Java files, as they're only parsed for mixed java-scala projects, and the maven Mojo extractor for Java already exists.

Now for the real fun... What does extractMojos look like? Well, it has the following signature (give or take a few revisions in git):


def extractMojos(unit : compiler.CompilationUnit) : Unit

A CompilationUnit is a .scala or .java file that has been compiled. A .scala file may contain many packages and classes/traits/objects. We gain access to these by calling "unit.body". The body is the Abstract Syntax tree for the given file.

A little more context is needed for what we're going to do next. The Scala Compiler uses something called the Cake pattern. The gist of it is that you carve up functionality into modules (in this case traits) and mix them together as needed. One unfortunate problem with the Scala Compiler itself is that a lot of the modules have nested dependencies (i.e. trait XYZ { self : Global => } ), and therefore have to be mixed into a global. Therefore we're going to define our tree extraction methods on a trait (in a similar manner) and mix them into the compiler variable when we instantiate. The module will look something like this:



import scala.tools.nsc.ast._
import scala.tools.nsc.symtab.{Flags, SymbolTable}
import org.scala_tools.maven.mojo.annotations._
trait MojoAnnotationExtractor {
  self: Global =>
    
  /** Pulls all mojo classes out of the body of source code. */
  def parseCompilationUnitBody(body : Tree) : List[MojoParsedInfo] = {
       //TODO - Implement
  }

What " self : Global => " portion of code means, is that the trait *must* be mixed into the Global class. Assuming the above layer (trait) is fully implemented, the entire "annotation extraction compiler" looks like the following:


  val compiler = new Global(settings, reporter) with MojoAnnotationExtractor {
    override def onlyPresentation = true
  }
  val run = new compiler.Run
  run.compile(sourceFiles.toList)       

  //Extract mojo description
  def extractMojos(unit : compiler.CompilationUnit) = {
     import compiler._
     for(info <- compiler.parseCompilationUnitBody(unit.body)) yield {
       extractMojoDescriptor(info)
     }
  }
  def extractMojoDescriptor(info : MojoParsedInfo) : MojoDescriptor = error("TODO - Implement")
import scala.tools.nsc.ast._
import scala.tools.nsc.symtab.{Flags, SymbolTable}
import org.scala_tools.maven.mojo.annotations._
trait MojoAnnotationExtractor extends CompilationUnits {
  self: Global =>
    
  /** Pulls all mojo classes out of the body of source code. */
  def parseCompilationUnitBody(body : Tree) = {
  for(unit <- run.units if !unit.isJava) yield {
    extractMojos(unit)
  }

I've added a few things in there. First is the "with MojoAnnotationExtractor" as part of the Global class initialization. This is mixing our our new extraction function (parseCompilationUnitBody), and allows us to call it inside the extractMojos method. We're finally able to get to the meat of our application, actually pulling in annotations and doing something with them.

The first thing we notice when attempting to parse the AST of a compilation unit body, is that we *must* keep track of package nesting ourselves... (there's no figuring it out after the fact, at least not cheaply that I found). I ended up with the following helper to ensure I knew the package for each class:


 /** Pulls all mojo classes out of the body of source code. */
  def parseCompilationUnitBody(body : Tree) = {

    //List of all the info about mojos we've found
    var mojoInfos : List[MojoClassInfo] = List()

    /** Parses down into packages... */
    def parsePackage(pkgName : String, pkgDef : PackageDef) {

      pkgDef.stats foreach { _ match {
          //We found a package, recurse into it, keeping track of where we are
          case subPkgDef : PackageDef =>
            parsePackage(pkgName + pkgDef.name + ".", subPkgDef)

          //Find mojo classes
          case classDef : ClassDef => parseMojoClass("", classDef)
          case _ => //Ignore
        }
      }
    }
    //Pull top-level body if it's a package, or a class definiton.
    body match {
      case pkg : PackageDef => parsePackage("", pkg)
      case classDef : ClassDef => parseMojoClass("", classDef)
      case _ => Console.println("Error! Unexpected source file format for Scala Mojo Extractor");
    }
    
    mojoInfos
  }

The parsePackage code above is used to recurse into package tree nodes, and pull out any classes defined on them. Scala converts the following code:


package x.y.z;
class A {}

into the following Tree:

Package X <- Package Y <- Package Z <- Class A

.
This is because the above code is syntactically equivalent to


package x {
  package y {
     package z {
         class A {}
     }
  }
}

Now that we at least have a method of finding classes, let's start looking into annotations. We want to filter the classes we inspect based on whether they are executable Maven Mojos. All maven mojos need to define a "goal" with which to run, corresponding to our "goal" annotation. Let's throw this check before out parseMojoClass method:


for { 
  annotation <- classDef.mods.annotations
  if annotation.tpe.safeToString == classOf[goal].getName
} {
  mojoInfos = parseMojoClass(pkgName + pkgDef.name + ".", classDef) :: mojoInfos
}

The "ClassDef" node contains a "mods" branch which (among other things) holds A list of the annotations defined on a class. Every annotation has a Type "tpe" field. The "type" corresponds to the class of the annotation. In this case I'm being lazy and calling the 'safeToString' method on type which should give me the full class name of that type (in this case org.scala_tools.mojo.annotations.goal). I'm then comparing that to the runtime string of the class name, to ensure that the class does have a "goal" annotation. Note: This was the path of least resistance for me. Types are not guaranteed to be class names, so I'm hoping to figure out a better way of doing this in the future.

The Scala syntax tree is very nice for doing manipulations and compilation like things, but for my circumstances, I really just want a map of annotation to attribute lists for that annotation. Let's try to make an abstract method that will pull an annotation's name and all the "arguments" to it's constructor (i.e. @annotation(arg)). I wound up with something like this:


 
  /** Parses a list of annotations into a list of MojoAnnotationInfo classes */
  def parseAnnotations(annotations : List[Annotation]) = {
      for {
        annotation <- annotations 
        Annotation(constr, _) <- annotation
      } yield {
        // TODO - In the future the second arg to Annotation may be needed... (as it's what's inside the
        //anonymous instantiation of an annotation, i.e. @xyz {}
        //TODO - Do we need to match on constructor?
        val argVals = for { 
          Apply(_, args) <- constr
          arg <- args
          argVal <- extractStaticValue(arg)
        } yield argVal
        new MojoAnnotationInfo(annotation.tpe.safeToString, argVals.toList)
      }
    }
  /** Attempts to pull a static value from the given tree.
   * 
   * @returns
   *         Some(value) if a value is found, None otherwise
   */
  def extractStaticValue(tree : Tree) = tree match {
    case Literal(c) => Some(extractConstantValue(c)) 
    case _ => None
  }
 /**
   * Extracts a constant variable's runtime value
   */
  def extractConstantValue(c : Constant) = {
    //TODO - handle other values
    c.stringValue
  }

The for loop defined in 'parseAnnotations' is looking at every annotation and pattern matching to rip out their constructor (constr member). If you need to support @annotation { val bodyVal = "someval" }, then you would also need to match against the second member of the "Annotation" AST node.

The annotation constructor is an instance of the "Apply" AST node (which I believe means 'function call', but don't quote me). Since I'm unconcerned over which "this" is being called, we are only extracting the 'args' field from the Apply node. We then iterate through the list of argument AST nodes, and pass them to our "extractStaticValue" function. Given that all annotation arugments must be statically accessible values, and that we implement extractStaticValue correctly, this should work just fine.

Next the 'extractStaticValue' method is pulling "Literal" AST nodes and extracting their values. A Literal AST can only hold a constant, so it pairs well with the 'extractConstantValue' method. As you can see, I'm getting away with murder because non of my mojo annotations take anything but Strings. In the "real world" you would need to handle static references as well constant values.

The 'extractConstantValue' method takes an AST constant and pulls its string value. The Constant AST class also has methods for pulling various other types of constants, however I'll leave a real implementation of 'extractConstantValue' as an excercise to the reader. Once again, I only care about string constants.

The next processing I'd like to perform is to determine if there are any 'vars' in the MojoClass that I can inject maven resources into. We'll assume we've defined a no-argument annotation called 'parameter' that you can place on var's like so:


  @parameter
  @expression("${project.build.directory}")
  var outputDirectory : File = _;

The above should inject the Maven Project's build output directory into the variable "outputDirectory" at runtime. To do this, we need not only the name of the variable, but also it's type. let's weed through the class AST's "impl" node and look for var definitions that have a @parameter annotation.

You'll quickly notice that THERE IS NO VAR AST NODE!!! Scala's very slick in that it converts the syntax:


  var xyz = _;

Into


  def xyz = _
  def xyz_= = _;

Therefore we need to look for the "Def" AST Nodes. Not only that, we need to look for "setters" (i.e. methods ending with the string "_$eq", where $eq is the mangled form of = in a method name)

Using the above code to parse through all annotations in a list, our "parameter" parser method looks like the following:


 /** Pulls out information about all injectable variables based on mojo annotaitons. */
    def parseMojoInjectedVars(classImpl : Tree) = {
      for { node @ DefDef(mods,name,tparams,vparams,tpt,_) <- classImpl
            if name.toString.endsWith("_$eq")
            annotation <- mods.annotations
            if annotation.tpe.safeToString == classOf[parameter].getName
            argument @ ValDef(_,_,tpt,_) :: Nil <- vparams //setter should only have ONE argument!
      } yield {
        val argName = name.toString.slice(0, name.length - "_$eq".length)
        //DO a real extraction of the type!
        val argType = tpt.toString
        val argAnnotations = parseAnnotations(mods.annotations)
        new MojoInjectedVarInfo(argName,argType,argAnnotations)
      }
    }

Notice the DefDef (Def AST Node) is made up of modifiers (annotations + others), a name (e.g. outputDirectory_$eq), type parameters, v? (regular) parameters, a return type and a body. We use the name to filter out only var setter methods, and then look through the annotations for the parameter name. Finally we filter out only methods that have ONE regular parameter, and we extract the type of that parameter with the pattern match "ValDef(_,_,tpt,_)". We can then parse the annotations, and create our "Mojo Info" class.

One thing I should mention is that all types I've been parsing so far have been in *external* libraries, and are therefore referencing class files in a jar somewhere. I'll try to update the post as I get more robust in what my annotation processor can handle.

Viola, we've extracted all the info for the beginnings of a scala mojo annotation processor. The really fun part is generating a test case for it. The test case involves running a maven integration test, where a new Scala Mojo project defines a mojo in scala, and the above code is used to generate the scala definition. This integration-test project then has its *own* integration test where it creates another project and calls itself, verifying that it echoed "HAI" correctly to the output.

Anyway, I hope this post was not too long for most of you. The topic is fairly complex, and I'm not the best to be writing about it, however I hope that the few minutes it takes to read can save you the several hours of frustration I had trying to get things to work initially.

Initial thoughts on JavaFX

2009-01-03T07:44:00.005-05:00

So recently I've been reading a little bit about the new "JavaFX" platform Sun has released. Specifically I've been looking into the Fx Script language. When I first heard about it, I kept thinking "OH no, *another* JVM language. I hope they don't make it too similar to java!". However, I've been pleasantly surprised by the features so far.

Based on a few tutorials, JavaFX script appears to have chosen nice features from languages like Scala and Javascript, while adding one languages feature that truly distinguishes it (and IMHO makes it a worthwhile language to learn/use for its domain). Here's a list of features I saw in the tutorials that I believe make the core of the Fx Script language:

Declarative Constructors (like javascript) -
```
{ fieldName: value }
```

Truly Useful Type inferences (with types on the right) [like scala] -
```
var radius = 5.0
```
Note: types can be specified

Three main types of constructs (similar to scala) - mutable references, immutable reference and functions
```
var mutableValue = 5.0
def someStaticValue = 2.0
function doSomething() {}
```
These made me think scala's def=> Fx's function, Scala's var=> Fx's var and Scala's val=> Fx's def

Value Binding - (this is a truly compelling feature for JavaFX)
```
var x = 0;
def y = bind x;
```
The above code means that x and y will always have the same value, even though y is immutable. This is useful for synching your model to your view. I'm excited to start toying with this. I've only shown on possible syntax, there are actually a few ways to bind variables.

Bound functions -
```
var x = 0;
def foo = bound function(...) {...};
```
The above code means that the function foo will see any changes made to "x". I.e. foo is no longer a closure in the sense that it retains the value of x during its call.

Anyway, I'm pleasantly surprised with JavaFX and the Fx Script language. I think a few good ideas from other JVM languages are finally really starting to take hold. Most importantly, I'm glad Scala isn't the only viable JVM language with really useful type-inference. I'm hoping the other statically typed languages will follow suit (This means you java!).

Cora May Suereth

2008-12-17T21:13:00.000-05:00

Subclassing in ExtJS

2008-12-12T19:42:00.006-05:00

It's been a while since I posted about ExtJS, so I thought I'd take a crack at another post here. This is going to cover the

Ext.extend

method, and how you go about sub-classing an Ext component.

The basic just is that we are going to create a "constructor" method for the widget and bind it to a location (e.g. Ext.ux.data.MyDataStore). Then we're going to use Ext.extend to plug in the methods of the parent objects, and override some at the same time.

For our example, I'll be using the Ext.data.DataProxy class as what we're extending.

So... first, let's define our constructor in a location:


//Ensures there are at least dummy objects leading into Ext.ux.data
Ext.namespace('Ext.ux.data');

Ext.ux.data.MyProxy = function(config){

    Ext.ux.data.MyProxy.superclass.constructor.call(this);

};

The code doesn't do a whole lot, but we have our constructor. However when executing new Ext.ux.data.MyProxy() we get an error, superclass is *not* defined. So... where does superclass come from? It will actually be injected into our prototype via the Ext.extend method. In fact, Ext.extend will also inject the methods "override" and "extend" into our class. Here's our new class with the Ext.extend call:


//Ensures there are at least dummy objects leading into Ext.ux.data
Ext.namespace('Ext.ux.data');

Ext.ux.data.MyProxy = function(config){

    Ext.ux.data.MyProxy.superclass.constructor.call(this);

};

Ext.extend(Ext.ux.data.MyProxy, Ext.data.DataProxy);

Now we can make use of DataProxy methods from within MyProxy using "this.methodName(args)". This isn't very exciting yet because DataProxy is mostly an empty class (besides configuring the Observable class for us).

The convention for adding methods into a subclass is to use the override parameter from Ext.extend. This parameter is essentially an anonymous class containing methods to inject "over-top" of the parent class. In the case where the method did not exist before, it is merely inserted. Let's implement the load method (required for DataProxy subclasses) using this approach:


//Ensures there are at least dummy objects leading into Ext.ux.data
Ext.namespace('Ext.ux.data');

Ext.ux.data.MyProxy = function(config){
    //Calls our super-class (Ext.data.DataProxy) constructor for full initialization
    Ext.ux.data.MyProxy.superclass.constructor.call(this);

};

Ext.extend(Ext.ux.data.MyProxy, Ext.data.DataProxy, {

   //Load function as defined in HttpProxy, and needed for the "DataProxy interface..."
   load: function(params, reader, callback, scope, callbackArg) {
         params = params || {}; //Ensure params exists as an object
         var result;
         try {
            //Use record reading to read "stock" data
            result = reader.readRecords({ zombieKid: ['i', 'like', 'turtles'] });
         } catch(e) {
           //We had an issue pulling in stock data, fire appropriate events
           //use callback in fail mode and bail.  This function is actually coming from
           // the Observable class which is our "grandparent" class
            this.fireEvent("loadexception", this, callbackArg, null, e);

            callback.call(scope, null, callbackArg, false);

            return;
         }
         //Inform the callback of the result of parsing our "stock" data
         callback.call(scope, result, callbackArg, false);         
   }

});

As you can see... our load method is basically taking the reader (Record reader) passed in and attempting to read some stock data (calling appropriate callbacks as necessary).

Notice the use of the "this.fireEvent" inside our method. Because of how JavaScript handles resolution, this will refer to our fully constructed class at the time of invocation ( as opposed to definition), therefore the fireEvent method from Observable will be accessable from our object.

The next important thing in defining your own widget is adding information to take in during construction. Our superclass constructor does not take any arguments, hence the "Ext.ux.data.MyProxy.superclass.constructor.call(this);" in our constructor. If perhaps our superclass constructor took a config object, we would have to change it. Luckily our current widget constructor expects a config object, so we can just pass it through to our super (assuming the configs are complementary).


...

Ext.ux.data.MyProxy = function(config){
    //Calls our super-class (Ext.data.DataProxy) constructor for full initialization
    Ext.ux.data.MyProxy.superclass.constructor.call(this, config);
}

...

Since the DataProxy class doesn't take anything in its constructor, we'll remove this, but it's an important feature to make note of. Also remember you can make subclasses which could add/change configuration information sent to parents. That's the nice part about having control of the config obejct before passing it to the superclass constructor.

Now onto taking in our own information. Let's mimic the "MemoryProxy" and take in some data during construction that we using during our "load" method. We'll call this element 'data' in the config and our object.


//Ensures there are at least dummy objects leading into Ext.ux.data
Ext.namespace('Ext.ux.data');

Ext.ux.data.MyProxy = function(config){   
    //Pull out our "data"
    this.data = config.data

    //Calls our super-class (Ext.data.DataProxy) constructor for full initialization
    Ext.ux.data.MyProxy.superclass.constructor.call(this);

};

Ext.extend(Ext.ux.data.MyProxy, Ext.data.DataProxy, {

   //Load function as defined in HttpProxy, and needed for the "DataProxy interface..."
   load: function(params, reader, callback, scope, callbackArg) {
         params = params || {}; //Ensure params exists as an object
         var result;
         try {
            //Use record reading to read "passed in" data
            result = reader.readRecords(this.data);
         } catch(e) {
           //We had an issue pulling in stock data, fire appropriate events
           //use callback in fail mode and bail.  This function is actually coming from
           // the Observable class which is our "grandparent" class
            this.fireEvent("loadexception", this, callbackArg, null, e);

            callback.call(scope, null, callbackArg, false);

            return;
         }
         //Inform the callback of the result of parsing our "stock" data
         callback.call(scope, result, callbackArg, false);         
   }

});

The above is an almost exact duplicate of MemoryProxy.js from the ExtJS source. I thought about having a more clever tutorial about a custom widget, but I'm trying to avoid assuming too much knowledge of ExtJS internals. Anyway, I hope this makes you feel more comfortable creating your own classes and extending ExtJS widgets with custom functionality. As always, let me know if you find any errors or typos.

Debugging inside the Scala Eclipse Interpreter

2008-12-02T17:16:00.004-05:00

So today we didn't have a babysitter, so I took the day to spend time with my daughter. It was quite fun, and I actually had a chance to get to some of the scala development work I've been meaning to during her nap.

Specifically, there have been a bunch of bugs with the Scala Eclipse plugin that really fall under contributions I've submitted. Primarily relating to the Interpreter.

After spending quite a long time getting my SDT development workspace working again (seriously... I have no idea why it's so hard to make it work on windows, but it is), I was up and running. In about 20 minutes I managed to convert the original "roll-my-own jvm executor" to use Eclipse JDT's "Launch Configuration" support for a JVM. Not only does this let people specify nice things like JVM arguments, it also rolls in the ability to actively debug.

Here's a screenshot of a workspace working off my locally installed scala plugin:

Hopefully this will make it into the trunk in a few weeks, depending on how much free time I can find. There's still a few things I need to work out such as:

Wiring the old menu-style invocation to work with the new LaunchConfiguration

Fixing/Tweaking the Launch Configuration Dialog to get rid of wonky behavior and allow configuration of things that matter

Allow "bootstrap" code to send to the interpreter

Ensure jline works correctly and eclipse passes down "interesting" keystrokes, OR roll my own

Figure out how to wire into the syntax highlighter for a console

As you can see... it's already an extensive list. I figure I'll at least get the basics done for the next release (2.8 I guess?)

Anyway, if anyone wants to try out the code locally, let me know and I'll send you a .patch.

Using Partial Functions and Pattern Matching

2008-11-16T09:21:00.017-05:00

As I dig further into scala, I begin to see the utility of certain language features when designing a library API.

For example, partial functions are a language feature in which you can declare functions that may or may not be defined for a given input. A partial function (in scala) inherits from Function1 (one-argument function) and adds two interesting methods:

isDefinedAt - Check to see if the function "is defined" for a given input

orElse - Apply another partial function if this one is "undefined" for a given input

As you can see (by the orElse method), partial functions can make a chain-of-command pattern easy to implement using only language-level features. The Lift web framework puts this to great use with their URL rewriting API.

So how does one define a partial function? It's as easy as defining a function made purely of case statements. Here's a function that is only defined for a particular string:


scala> val myPartial : PartialFunction[String,String] = {
     |     case "mymatchedstring" => "my output string"
     | }
myPartial: PartialFunction[String,String] =

Notice how I only define one case that does *not* cover all potential strings. Here's some examples of using "isDefinedAt" to check if a partial function can apply:


scala> myPartial.isDefinedAt("mymatchedstring")
res1: Boolean = true

scala> myPartial.isDefinedAt("someotherstring")
res2: Boolean = false

What happens if you attempt to call a partial function on a string for which it is not defined?


scala> myPartial("mymatchedstring2")
scala.MatchError: mymatchedstring2
 at $anonfun$1.apply(:4)
 at $anonfun$1.apply(:4)
 at .(:6)
 at .()
 at RequestResult$.(:3)
 at RequestResult$.()
 at RequestResult$result()
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMe...

This is a "MatchError" runtime exception. I'd recommend using the isDefinedAt function and avoiding a runtime exception, but the choice is yours.

Let's try adding the string "mymatchedstring2" to this partial function using orElse:


scala> val partial2 : PartialFunction[String,String] = {
     |   case "mymatchedstring2" => "Delegation FTW"      
     | }
partial2: PartialFunction[String,String] = 

scala> val doublePartial = myPartial orElse partial2
doublePartial: PartialFunction[String,String] = 

scala> doublePartial.isDefinedAt("mymatchedstring")
res5: Boolean = true

scala> doublePartial.isDefinedAt("mymatchedstring2")
res6: Boolean = true

scala> doublePartial.isDefinedAt("mymatchedstring3")
res7: Boolean = false

scala> doublePartial("mymatchedstring")             
res8: String = my output string

scala> doublePartial("mymatchedstring2")
res9: String = Delegation FTW

scala> doublePartial("mymatchedstring3")
scala.MatchError: mymatchedstring3
 at $anonfun$1.apply(:4)
 at $anonfun$1.apply(:4)
 at scala.PartialFunction$$anon$1.apply(PartialFunction.scala:38)
 at .(:8)
 at .()
 at RequestResult$.(:3)
 at RequestResult$.()
 at RequestResult$result()
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ...

Neato, now we can very easily create a chain-of-command pattern without defining any new classes (at least *I* don't define any new classes, the compiler may do whatever it needs to do).

Let's try to make use of PartialFunctions for the Database Migration API I posted about several blog postings back. The goal here is going to be creating a method of executing raw SQL queries during a migration. We're aiming for something close to the following syntax:


class Migration1 extends Migration {
 import MigrationDSL._


 def upgrade(implicit ctx : MigrationContext) {
  
   execute_script {
       case "MySQL" => """do mysql query;"""
       case "Derby" => """Do derby query;"""
   }  

 }
 ...
}

The signature for the above execute will look something like the following:


trait MigrationContext {  
  def executeScript(queries : PartialFunction[String,String]) : Unit
  ...  
}

object MigrationDSL {
  def execute_script(queries : PartialFunction[String,String])(implicit ctx : MigrationContext) = ctx.executeScript(queries)
  ...
}

Things are working pretty well so far. We can try to detect the case when a migration is not defined for a given database vendor, and throw some sort of exception which will make the migration fail. What if we'd like to give feedback to the user on whether or not they've left any supported databases "out" of their function?

The preferred way to do that is via sealed traits,shown below:


sealed trait DbVendor {

}

case class MySQL() extends DbVendor;

case class Derby() extends DbVendor;

case class Oracle() extends DbVendor;

trait MigrationContext {  
  def executeScript(queries : PartialFunction[DbVendor,String]) : Unit
  ...  
}

object MigrationDSL {
  def execute_script(queries : PartialFunction[DbVendor,String])(implicit ctx : MigrationContext) = ctx.executeScript(queries)
  ...
}

Now with the following code...


execute_script {
       case MySQL() => """do mysql query;"""
       case Derby() => """Do derby query;"""
   }

... you don't see the desired error message. Why? Because we're taking a 'partial' function, which by definition could be undefined in certain locations. To fix this we need to instead take a plain old vanilla function like below:



trait MigrationContext {  
  def executeScript(queries : Function[DbVendor,String]) : Unit
  ...  
}

object MigrationDSL {
  def execute_script(queries : Function[DbVendor,String])(implicit ctx : MigrationContext) = ctx.executeScript(queries)
  ...
}

This informs the scala compiler that we expect the function to work for *all* inputs. If we give the compiler a little encouragement (i.e. sealed classes/traits), it can now help us avoid "missing" possibilities in our case statements.

Now with the following code...


execute_script {
       case MySQL() => """do mysql query;"""
       case Derby() => """Do derby query;"""
   }

.. will generate this error...


[WARNING] /home/josh/projects/blog/scala-db-migrate/src/main/java/com/blogspot/suereth/dbmigrate/test/Migration1.scala:16: warning: match is not exhaustive!
[WARNING] missing combination         Oracle
[WARNING] 
[WARNING]     execute_script({
[WARNING]                    ^
[WARNING] one warning found

Now our users can tell if they are handling every defined database vendor. If they wish to ignore the message (or define a "generic" query to try to use, they can simply match on "_" (or wildcard) like so:


execute_script {
       case MySQL() => """do mysql query;"""
       case Derby() => """Do derby query;"""
       case _ => """Do generic query;"""
   }

Now we're no longer explicitly using partial functions, but our API can still make use of pattern matching.
In a future post I plan to discuss how we can use extractors to do slick things with defining queries for specific versions of database vendors (or just all versions).

Maven For Beginners

2008-10-31T15:38:00.009-04:00

Not this content is also being submitted to the scala-lang.org site. It is duplicated here mostly for peer review and for those of my readers who are *not* scala-centric.

This article is a more in-depth look at what maven is and how a new user can make use of it. If you'd like a getting-started-quickly guide, please check out: maven for scala

What is Maven?

Maven is modern build/project management tool. According to its website, the core purposes are:

Make the build process easy

Provide a uniform build system

Provide quality project information

Provide guidelines for "best practices" in development

Allows transparent migration to new features

In practice, maven can greatly simplify builds for "standard" projects, and makes it easier to add new libraries/modules to existing projects. Maven builds tend to consist mostly of "declarations" and configuration, with very little scripting. Maven also promotes the "convention over configuration" idea to reduce the length of build files.

Downloading Maven

The first thing you need to get started with maven, is maven itself. It can be downloaded from http://maven.apache.org/download.html. Unzip/untar it into a directory of your choice and make sure ${MAVEN_HOME}/bin is in your PATH (even on windows). For some operating systems their are maven-packages you can download using your package manager. There are also IDE integrations for most Java IDEs. Some of these integration come with a version of maven "bundled" with the plugin (e.g. Eclipse IAM + Eclipse M2E).

Creating your first project

After downloading maven, you should create a simple project to ensure that everything is working correctly. This easiest way to do this is by using the Maven Archetype Plugin. In maven an "archetype" is the general skeleton structure (or template) of a project. You can run the archetype plugin as follows:

mvn archetype:generate

You may notice that maven has started downloading jar files onto your computer at this point. Maven uses a dependency resolution mechanism that will download dependencies as needed (and only once). If you're using a fresh install of maven, it may need to resolve the archetype plugin.

Maven should give you a list of available project templates (scala was #30 on my list). Select one that may interest you most (and if it's not scala... well....).

Next maven will begin asking you for a groupId, artifactId, versionId and packaging. We'll cover these later, but for now just use an inverted domain name for groupId (e.g. org.scala-lang) a project name for artifactId (e.g. test-project) and the default for veresionId/package. Next cd test-project and run mvn package. Once again you should see maven automatically downloading dependencies (such as scala-library, scala-compiler, junit, etc.) Congratulations, you've built your first maven project!

Build File Structure

Maven projects are defined by their "Project Object Model" or pom. This file is located in the base directory of a maven project and is called pom.xml. The Project Object Model includes a lot of different information:

The name/description of the project

The style/packaging of the project

The version of the project

The dependencies (libraries or otherwise) of the project

The configuration for the build (plugins, directories, etc.)

The configuration for reports (test coverage, static analysis, etc)

The developers/contributors for the project (name, email, etc.)

The Infrastructure (Source Control Repository, Continuous Integration Server, Issue Tracker) for the project

The maven repositories used for the project

As you can see, this is a lot of information! In addition to all this, POM files are object-oriented and may inherit from one parent. By default, all maven projects inherit from the maven "master pom". The master pom defines the standard build layout/configuration for maven projects. You can override/extend this behavior in your project pom if desired.

POM File

Here's a standard maven pom file for a scala project:


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.scala-lang.demo</groupId>
    <artifactId>scala-test</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>Demo of maven for Scala Lang website</name>
    <url>http://scala-lang.org</url>
    <repositories>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>
    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.7.2-rc2</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <sourceDir>src/main/java</sourceDir>
                    <jvmArgs>
                        <jvmArg>-Xms64m</jvmArg>
                        <jvmArg>-Xmx1024m</jvmArg>
                    </jvmArgs>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Let's analyse this a bit. First off, all pom files consist of a "project" xml element. This encompases the entire xml document. The modelVersion element is always the same and is required for maven2 projects. Here's some simple descriptions of the other elements:

groupId - This corresponds to the inverted domain-name of the project. Although you could use any arbitrary value, it's recommended to use your company domain name here
artifactId - This is a unique id for the project (within the group). This name will be used when creating jar/war/ear files.
version - This represents the current version of the project. More details later
name - This is the human-readable name of the project.

url -This is the url for the project website.

packaging - This defines the "style" of project your building (e.g. ear, jar, war, etc.). For more information on packaging, check out the respective plugins (maven-jar-plugin, maven-war-plugin, etc.)

dependencies - Specifies the dependencies of the project. This will be materialized from any defined maven repositories. More details later

repositories - Specifies alternative locations for maven to look when materializing dependencies.

pluginRepositories - Specifies alternative location for maven to look when materializing build plugins

build - Specifies configuration on *how* to build the project

reports - Specifies configuration on *what* reports to generate for the project

Default Directory Layout

Maven projects have default layouts defined by a combination of the maven super-pom and any build-plugins you have declared. These defaults are customizable, but it is generally recommended to use the convention. (The more you deviate the more XML you need to write). Here is a sample project layout with links to descriptions on which maven plugins are responsible for which directories and how you can override the defaults.


project/
   pom.xml   -  Defines the project
   src/
      main/
          java/ - Contains all java code that will go in your final artifact.  
                  See maven-compiler-plugin for details
          scala/ - Contains all scala code that will go in your final artifact.  
                   See maven-scala-plugin for details
          resources/ - Contains all static files that should be available on the classpath 
                       in the final artifact.  See maven-resources-plugin for details
          webapp/ - Contains all content for a web application (jsps, css, images, etc.)  
                    See maven-war-plugin for details
     site/ - Contains all apt or xdoc files used to create a project website.  
             See maven-site-plugin for details       
     test/
         java/ - Contains all java code used for testing.   
                 See maven-compiler-plugin for details
         scala/ - Contains all scala code used for testing.   
                  See maven-scala-plugin for details
         resources/ - Contains all static content that should be available on the 
                      classpath during testing.   See maven-resources-plugin for details

Although the above represents the default layout for most scala projects, every maven plugin used could bring in its own "convention" for where it would like to see code. An example of this is that the gmaven plugin for groovy expects to find groovy source code in src/main/groovy and src/test/groovy (similar to the scala plugin).

Build Lifecycle

The maven build lifecycle is made up of "phases" and "goals". A "goal" is a various action defined by a plugin. Examples include scala:compile, jetty:run-war, jboss:deploy and (the one you should be familiar with) archetype:generate. Goals are specific to a particular maven plugin and perform a single operation. Phases on the other hand are more abstract ideas, and represent particular points in a standard build system. Here's a listing of some of the phases for a maven build (taken from here):

validate - validate the project is correct and all necessary information is available

compile - compile the source code of the project

test - test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed

package - take the compiled code and package it in its distributable format, such as a JAR.

integration-test - process and deploy the package if necessary into an environment where integration tests can be run

verify - run any checks to verify the package is valid and meets quality criteria

install - install the package into the local repository, for use as a dependency in other projects locally

deploy - done in an integration or release environment, copies the final package to the remote repository for sharing with other developers and projects.

In reality, there are many more phases defined. For a complete listing, read here.

When operating maven on the command line you can type "mvn" plus a list of goals and/or phases. These will be executed in the order defined. Some phases may depend on other phases, such that they do not need to be specified. For example, package depends on test. This means you cannot package your software unless all your tests pass, and packaging will always attempt to build. When specifying goals on the command line, the goal will execute immediately with no dependencies.

Build Plugins

One of the great features of maven is that there's a build plugin for almost any activity required in a Java/Scala project. These plugins are defined in the project/build/plugins section of the pom. Most build plugins automatically attach their goals to appropriate build phases when used, however this is configurable at runtime. For example, here is how we ask the scala:compile and scala:testCompile goals to automatically bind to their respective phases:


    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                ...
            </plugin>

Inside the execution block there's a (sometimes) option id element and a phase element. The phase element can be used if you'd like to bind a goal to a different phase then it defaults too. As an example, I sometimes bind the jboss:undeploy goal to the package phase and the jboss:deploy goal to the install phase. This way if my unit tests succeed, my application gets undeployed/redeployed right after packaging, and I only to call mvn install on the command line.

Dependencies

One of the powerful features of maven is declarative dependencies. Maven allows projects to declare what dependencies they have, and will automatically materialize those dependencies (including transitive dependencies). Maven dependency declaration looks like this:


   <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.7.2-rc2</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

Each dependency consists of several items:

groupId - The group of the dependency to rely on

artifactId - The artifact in the group to rely on

version - The version of the dependency to rely on

scope - The "scope" of the dependency. Defaults to compile (more details later)

packaging - The packaging for the dependency. Defaults to jar (e.g. jar, war, ear)

The default maven repository has a website that allows you to search for dependencies called http://mvnrepository.com. Most open source projects are available either via the "central" maven repository and a separately hosted maven repository (e.g. jboss host's their own repository).

Dependencies in maven have scope. The scope determines when/where the dependency will show up in a classpath. Here are the commonly used scopes:

compile - The dependency is available when compiling/running/testing the project and as a transitive dependency

runtime - The dependency is available when running/testing but not for compilation. Runtime dependencies show up as transitive dependencies.

test - The dependency is available during testing phases, but not compilation or (non-testing) runtime. Test dependencies do not show up as transitive dependencies

provided - The dependency is available during compilation, but not runtime. e.g. You expect the servlet-api will be provided by a servlet container and therefore should be a "provided" dependency

More information (and scopes) is available here.

Pom Inheritance
The Second nicest feature of maven is pom-file inheritance. Pom file inheritance works similarly to how single-inheritance works in an OO language. The child POM inherits all configuration items of its parent, and can override/extend them with its own configuration.

This is done through specifying a pom with a packaging of "pom", and then using the "parent" tag in a child project. This can greatly reduce the amount of configuration done in "child" projects (Often times my child project poms only specify their parent and their artifact/group/version triplet).

I've posted about this before when talking about GMaven

A child groovy project to the above pom would look like the following:


<project xmlns="http://maven.apache.org/POM/4.0.0"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.blogspot.suereth</groupId>
 <artifactId>groovy-child</artifactId>
 <version>1.0-SNAPSHOT</version>
 <packaging>jar</packaging> 
 <parent>
  <groupId>com.blogspot.suereth</groupId>
  <artifactId>groovy-parent</artifactId>
  <version>1.0-SNAPSHOT</version> 
 </parent>
</project>

This behavior is usually associated with multi-module projects, but that discussion is beyond the scope of this article.

Going Further

This article is only designed to help you take your first steps towards using maven as your build environment. There are more advanced topics that can greatly help your efforts of creating a build system. Some of these include

Multi-module projects

Profiles

Settings.xml

Setting up your own Maven Repository (and/or Proxy)

Deploying artifacts

Project reporting

I'm hoping to provide some more new-user documentation for the following in the near future. For now, I'd recommend what most good maven-build-experts use for information: google.com and maven.apache.org

Virtual Appliance - Not a Toaster on Second Life

2008-10-23T22:14:00.005-04:00

This posting is going to outline what a virtual appliance is. This touches on what I do at work enough to give you a hint of some of the neat new things coming down the line.

No really, It's not a toaster... or a coffee maker

So what is a virtual appliance? A virtual appliance is a virtual machine that can be distributed (like any file) and deployed onto some hypervisor (a server that runs virtual machines). Because a virtual appliance is an entire virtual machine, the appliance developer has control of the entire aspect of the machine, from operating system to window themes to startup procedures. ***Not only could one use some custom operating system, one could perform more interesting OS integration without worrying about "cross-platform compatability". Imagine buying a server that could initally be running a VM with your DNS and Email, and then repurpose it to dedicated indexing/search web interface.

A few other interesting things to note about virtual appliances involve web hosting and choice of operating systems. It is now feasible for a web host to simply provide some hypervisor (virtual server) and give you a virtual machine on this server. You can then customize the machine however you want (under the resource restrictions the hosting service provides). The host can control how much CPU/RAM/Network they allocate to your machine and enforce it in the hypervisor. You can control almost every other aspect of your machine. I know a lot of web hosting services have already started adopting this model.

Another intersting thing you do on a virtual appliance is install any kind of (or no) operating system. Many Linux distributions are starting to offer "virtualization optimized" packages (e.g. Ubuntu's JeOS, rPath's JeOS and Red Hat's (not released) Appliance Operating System [AOS]). [JeOS => Just Enough Operating System, a term coined (AFAIK) by VMWare). These JeOS products could greatly improve performance of your virtual appliance. I predict that in the future we'll see more custom distributions, kernels or entire operating system stacks.

The Early Brew

In the 90's MIT was doing research into something they called the "Exokernel". The idea behind an exokernel was one step beyond micro-kernels: The kernel should only multiplex resources on a machine. The idea was neat, and they experimented with a webserver they called "Cheetah". Cheetah implemented its own memory management to reduce the os-abstraction-overhead of server static content. Cheetah was able to gain an order-of-magnitude in performance by marking and passing memory loaded with file contents directly to the NIC without having to be copied to and from user-space. The exo kernel failed in other areas, mainly in that it tried to multiplex host resources *separately* for all "processes". I remember watching the ExOS library (a POSIX-as-a-shared-library implementation) fly together, but the file system really started caused some trouble. It was hard to share a disk across operating systems without imposing some kind of filesystem on them all.

Virtualization is very similar to the ExOS idea. It multiplexes physical resources across many virtual machines. The big difference here is in the fundamental unit of execution: For ExOS it was still a Process, however for the Hypervisor it's a "Machine". This difference in fundamental unit is what has made virtualization so popular, however it doesn't mean we have to give up some of the things we've learned from the exokernel. We can still implement an OS that is highly tuned for some specific purpose (almost like an embedded device). However, instead of compiling for a particular architecture, we compile to the x86-virtualized-architecture. We can back our order-of-magnitude performance gain for hosting static content. The question remains, what is the cost of the extra-layer of abstraction that virtualization brings? With no hard data, I can't answer this question.

The Burnt Edges

Because using virtual machines give so much new flexibility to how applications are developed, a whole new host of issues also creep in. The biggest of these is how to generate these virtual appliances in the first place. What I've seen most often in "How To" documents (Amazon EC2's for example) is that you should start with a base OS install and begin to customize your appliance until it suits you. Being a virtual machine, you can then clone this machine to distribute to clients (or yourself). This works great, but is tougher to automate, and upgrade.

One thing that can help in this process is the "snapshot" feature provided by some vm vendors. This means you can set up a virtual machine how you like it, but without most of your software installed. Then when doing a distribution, you install your software, clone your machine, and revert the "template" back to a pre-install state.

Although this gives you a way of creating nearly infinite numbers of virtual appliances, it still leaves something to be desired (automation). I've noticed that as virtual appliances become more prevalent a few companies have begun offering tools to help manage this. rPath rBuilder is one such product. It uses a completely new packaging system (Conary) and its own JeOS Linux Distro.

If you're more into completely free (well, pay for service), Ubuntu also recently posted some documentation for their open-source JeOS"JeOSVMBuilder".

Anyway, The virtualization market is pretty exciting times. Migrating to virtualization can provide a *lot* of flexibility to your infrastructure. Use completely open-source software, I've been able to set up a "development server" complete with source control, project website, issue tracking, etc. AND nightly backups are as simple as copying a file to a USB disk. If you're interested in doing the same, I'd recommend looking into VirtualBox. If I can find the proper place to host (and time to clean it up), I may host my Ubuntu JeOS development server (Apache2 + Tomcat + Archiva + Hudson + Trac + SVN).

As a side note, the more I've seen of GIT the more I like it. Given some time (once again, hard to come by), I'm thinking of making the dev server be flexible enough to create new projects with svn, Hg or git.

References

[1] MIT Exokernel Operating System
[2] Red Hat Appliance Operating System
[3] rBuilder
[4] JeOSVMBuilder

Singletons are for Simpletons (Statics => Fail)

2008-10-06T18:21:00.001-04:00

Clarification: After having many IM chats about this post, I thought I'd write a minor clarification. In this post I outline a very dogmatic approach to avoiding Singletons in the attempt to ease unit testing. In real life, I'm far too practical to completely avoid singletons. As a rule of thumb, I think they are avoidable and you can simplify development by not having them. Next time you're thinking of using singletons, think: "If statics did not exist and I needed this functionality, would I pass this class into every class in my application?". Hopefully it'll help you reach a common ground between over-use and never-use. This post is an attempt to sway statics/singleton-heavy users over to a middle-ground of sorts.

Some of you may be wondering why I'm writing about fundamentals of mechanical engineering. Actually, This posting is about statics in programming. Other words sometimes used for statics are globals, singletons or pure evil. The goal of this post is to show why you should avoid statics/globals/singletons in your own code, and what you can gain by doing so.

Statics are pretty tempting when first learning Java. Sometimes it's easier to just make something static then figure out how/who needs to receive functionality from a particular class. Singletons are even more dangerous in that they grant the illusion of have a real class while not really buying you an amazing amount of power. (I guess inheritance counts...)

Let's see a typical Java Singleton in action:


package com.blogspot.suereth.statics.suxorz

public class SimpletonSingleton {

   private static SimpletonSingleton instance = new SimpletonSingleton();
   //This appears like a non-static variable huh?  Even looks testable I bet...
   private ImportantInterface memberOne;

   private SimpletonSingleton() {
         memberOne = lookUpImportantInterface(memberOne);
   }

   //Helper methods, all synchronized of course
   ...
   public static SimpletonSingleton getInstance() {
        return instance;
   }
}

Rather than diverge into the issues surrounding java singletons and classloaders etc., let's point out a few real issues with this code. I see two major issues

It appears clever
It's not

In terms of 1, I've seen and done a lot of "clever" things with singletons. It's amazing how "easy" you can make developing an application. Sure, create a database abstraction that lets you pull in all sorts of things from your database into your application. I remember my days of CORBA in C++ where I created an "OrbAccess" singleton. That was my genius way of not having tons of references passed in the constructors of every class. In C++ the overhead of managing that much information became painful, even with multiple inheritance. (I toyed with the idea of a "ORBHelper" virtual base class that just took in an ORB and gave you helper methods). The problem here is that once you make something a static/global/singleton you lose control of access of the object, but more importantly you lose control of composition of your software.

The second point revolves around this. Object-Oriented software is much more about composition than inheritance. More importantly re-usability and testing become FAR easier with composition.

Look at the above code. Now image how we go about testing it. Here's a few scenarios:

"There's Only one of these per application so I'll make it a Singleton" Singleton - If only I hadn't seen these in the wild... I wouldn't even mention them. Just because you only want one instance of a class is *NO* reason to make it a formal singleton (like the above). The main problem with this is you lose access to when the object is created, and therefore the *arguments* to the singleton. It is now harder to compose your application. Good Job!

Data-Layer Access Singleton - We want a singleton that will manage all our access to the database. This way all our JDBC and queries are in one place! great. The "lookUpImportantInterface" method actually pulls in some .properties file and wires in some way of grabbing connections or hibernate sessions, or :::insert framework here:::. Slick huh? Alright, so now let's say I want to test this singleton. I now have to abstract out either the entire JDBC API or some other layer on top of it (ugh). WORSE! If I want to test classes that *use* the singleton, I STILL have to abstract out the entire JDBC API, or bootstrap a database, or some other nonesense. Also, this singleton winds up getting changed any time I want to grab some new type of data out of the database. Isn't this one of the code smells from Martin Fowler's "Refactoring"?

Factory Singleton - Perhaps this is cheating (since the Data-Layer Access Singleton could be considered a Factory singleton). The basic premise is I want a factory that will generate instances of classes. This way I don't have to remember all the crazy sub-classes, I can just change implementations based on arguments and overloaded methods. YAY! The underlying problem here is how you go about testing *users* of the Factory singleton. You need to be careful about how/when you go about creating and using the objects. If you've designed your software such that a 3rd party does the composition (ala dependency injection), then you can get away with a Factory Singleton AND have testability. If you make the factory calls yourself in a "user" class, you can no longer isolate your testing of that user class, and have to mock whatever framework the Factory Singleton uses when testing its users. This can be difficult, if not impossible if you are not the author of the Factory Singleton.

Resource Caching Singleton (or the Flyweight Singleton) - Suffers from all the same issues as the Factory Singleton PLUS you have to write massively ugly threading code with statics...

Event Queue Singleton - As these are usually used in GUI applications where you can only have one thread interacting with widgets, I don't have much negative here. It's already a GUI so testing is hard. Go crazy and write bad code. I'll go work on the server :) As an aside, I think the days of single-threading GUI event loops are coming to an end. Multi-core is going to force their hands. Perhaps when the languages catch up, the core libraries will be next. For other uses, I would rather see instantiations of event queues that get passed into objects than singletons personally.

X-Pool Singleton - Seems great right? Need a socket, grab one from the pool. Need an inflatable alligator? It's in the pool. Anyone can access the pool, but it enforces fairness right? Oh... you mean it doesn't know who's using it? Well, I'll just have my class call the "get_low_priority" method. Wait, nevermind, *my* class is important. *YOU* call the low priority method. How does the pool get instantiated? OH, it needs to know how to start itself. What about testing? Well I'll need to have a mock-configuration for every test I run on users of the pool... Not so fun anymore... MAYBE I'll just make "get_object" interface and pass it in to the constructor of my object. That way I can change how it works when I compose my application (for testing or production or otherwise).

Utility Singleton - So this is perhaps the most useful of singletons. What you're actually doing is creating a bunch of pure functions. In C++ you wouldn't even need to place them in a class, just a namespace. As long as the functions are pure (no mutating state), This singleton becomes easily testable.

As you can see, although there are some *ok* usages of singletons, the vast majority of usages detract from locality of code and testability. I've worked on a lot of projects where automated unit testing is thrown out, not because there's no time, but because the design/architecture of the code makes automated unit testing hard/costly. My argument is that (at least in modern languages), there's no reason to throw testability out for these static/singleton usage patterns. I can use this newfangled "Dependency Injection" technique and *still* have slick code.

Am I interfacing with a JmsQueue? Make a simple abstraction:


interface JmsQueue {
   public void sendMessage(Serializable msg) throws CommunicationError, FatalError;
   public void addMessageListener(MessageListener ml);
   public void addErrorHandler(KeepAliveErrorHandler el);
}

Now I can pass a mock instance of this interface into all my classes that send JMS Messages. I can even make sure the messages sent are correct using my mock. I can even test communication errors using my mock, and make sure my business logic is designed to handle it.

For JMS Receivers implementing the MessageListener interface I should be able to test that interface directly.

DON'T Fall into the trap of wiring "Listener" classes into the "Listened" objects during the "Listener" class's constructor. Although it may seem clever, it means that you now need to abstract out the *entire* "listened" object, instead of just calling methods on the listener class. It's better to give yourself the freedom of composition for later enhancements. What if you wanted to load-balance your listeners? What if you wanted to consolidate several listeners?

In conlusion... Avoid singletons if possible. Save "wiring" your classes until later. You can get something working quicker if you can test it out IMMEDIATELY vs. waiting for your infrastructure/architecture to fall in place. Think of coding as two phases:

Single Class Design, Implementation and Testing
Composition/Architecture of classes and Integration Testing

In practice don't separate these too much, as you'll design some really bad classes. However it's a good logical distinction.

Also keep this in mind. Dependency injection does not mean Spring or Guice or J2EE. Dependency injection can be accomplished via "Composer" classes as I like to call them. These are the "Main" classes of an application, that can just instantiate objects and pass them into constructors. This is the guy who sets up an environment for the classes to run in. Your JUnit tests are 'composer' classes. They set up a much smaller execution environment, but they are setting one up. Your main application could have multiple or one large composer class, or it could hide and use a "generic" one that reads "xml" or "annotations". In any case there *is* a composer class that creates your execution environment for you. Break you application into "users" and "composers" and testing will be much easier on you. Make sure your composer classes don't contain too much logic and you should be on your way to an easier-to-maintain code base.

Using Scala for Domain-Specific-Languages

2008-09-28T10:49:00.001-04:00

In this post, I'm going to attempt to create something similar to Ruby on Rails (or rake's) db:migrate scripts in Scala. I'm not going to actually implement features, just show how one can go about constructing a DSL in the Scala language. This is a learning process for me, so please excuse some mistakes I may make, and hopefully I can correct them in future posts.

For a quick over-view of what the db:migrate scripts provide in Rails:

Convienient language-based syntax for creating/modifying/deleting tables

Versioning of the Database API

Convention for defining scripts to perform migrations between various versions

I've seen proprietary methods like db:migrate implemented at various companies, so I don't think it's anything new. The nice thing about rails db:migrate is that you can remain db-agnostic for simple cases AND creating a new migration is pretty easy. I'd like to keep the same conventions in scala.

To begin with, we're going to define a parent class (actually a trait) for all database migration scripts to inherit from. In rails, I believe it's called ActiveRecord::Migration. For our purposes, we'll call it Migration.


package com.blogspot.suereth.dbmigrate

/**
* This trait is a common parent for all migration scripts.  Each migration represents a version.
*/
trait Migration {
 def upgrade(implicit ctx : MigrationContext)
 def downgrade(implicit ctx : MigrationContext)
 def version : Int  //TODO - Figure out how we're really going to do versions.
}

The above trait is very similar to the Rails version. The three methods do the following:

upgrade - Migrate the database from the previous version to "this" version

downgrade - Migrate the database from "this" version to the previous version

version - "this" version for the migration. i.e. upgrade should leave me at version X

Already I'm introducing the implicit argument keyword here. We'll get to why this is useful later, but for now just pretend like we're passing in a MigrationContext object that can be used to perform database manipulations. We can assume the MigrationContext will handle all of our database transactions in a reasonable fashion.

For now, let's focus on defining tables in our DSL. What we'd like to see is something like the following:


class Migration1 extends Migration {
 import MigrationDSL._


 def upgrade(implicit ctx : MigrationContext) {
  
   create_table('user) {
     tbl =>
       tbl has_column 'user_id as classOf[Int] default 0 primaryKey()
       tbl has_column 'user_name as classOf[String] not_null()
   }   

 }
 ...
}

So how do we get here? We'll let's do a top-down approach. First, there's the syntax:

create_table('user) { tbl => ... }

This looks like (and is) a function call. We have two options for where to place it. We can place it in the Migration trait (which I'm reluctant to do) or we can place it into the MigrationDSL object (see the import above). We're going to place it in MigrationDSL. Here's our first cut at the create_table function:

object MigrationDSL {
  def create_table[A](name : Symbol)(creatorScript : TableCreator => A) = {
    //TODO - Implement
  }

Alright, so now after importing MigrationDSL._ we can write create_table('symbol) { tbl=> } anywhere we want.

There's two things I don't like about this. One is that we're placing implementation into our MigrationDSL. I'd rather keep the DSL abstract, so we can implement it however we feel like. We should be using the MigrationContext trait/interface here. The other complaint is that we can write create_table from anywhere as long as we import MigrationDSL._. We want to only be able to use this in the context of a migration. Let's try to have MigrationDSL forward its method call to a migration context like so:

object MigrationDSL {
  def create_table[A](name : Symbol)(creatorScript : TableCreator => A)(implicit ctx : MigrationContext)  = ctx.createTable(name.toString)(creatorScript)

Once again I'm using the implicit argument keyword. What's it doing here? Well this time I'm using it so that when you call the create_table method, you either need to explicitly pass in a MigrationContext or have one available for implicit usage. If you remember when we defined the Migration trait, the upgrade/downgrade methods both take in an implicit MigrationContext. This means that within an upgrade/downgrade method, there is a MigrationContext implicitly available (and therefore you don't have to specify it). It's the last parameter of the last parameter-set just so the compiler can use it implicitly. We're back to having our desired syntax, AND we're still delegating all calls through the MigrationContext.

Now we need to define what the MigrationContext trait looks like (at least so that it has the createTable method. It's going to look very much like the earlier create_table that was on our MigrationDSL object. Here's the new MigrationContext:

trait MigrationContext {
  def createTable[A](name : String)(creatorScript : TableCreator => A) : A
}

Notice that the method is completely abstract. We've now divorced our DSL in such a way that someone simple needs to implement a MigrationContext for our DSL.

The next item we'd like to implement are the column definition lines:

 tbl has_column('user_id) as classOf[Int] default 0 primaryKey()

As we haven't defined the TableCreator type yet (which is what our creatorScripts are accepting), let's define it and add the "has_column" method:

 trait TableCreator { 
 def has_column(name : Symbol) : ColumnCreator
}

Once again I'm keeping everything as pure-abstract Traits. We'll leave implementation as an exercise to the reader. As you can see the has_column method take in a symbol and returns a ColumnCreator. We're going to have to put the rest of our column DSL into the ColumnCreator class. First, let's tackle the "as" method. All columns need to have a type associated. This is something we'd like to require, however we'll leave that as an implementation detail for the time being. If anyone has a suggestion for how to require this method be called at compile time, please let me know!

Let's look at the ColumnCreator trait:

trait ColumnCreator {
 def as[A](clazz : Class[A]) : ColumnCreator2[A]
}

This trait defines one method (as) and returns a completely different (typed) ColumnCreator2 type. This is our way of guaranteeing that (a) users always define types for column and (b) we have a valid type before allowing the default function of our DSL to be called. So after this call, we have a valid column that could be created in a database table. However for all our configuration options, we'll have to do something with the ColumnCreator2[A] trait. Here's what it looks like for me:

trait ColumnCreator2[A]
{
 def default(value : A) : ColumnCreator2[A]
 def primaryKey() : ColumnCreator2[A]
 def not_null() : ColumnCreator2[A]
}

Because ColumnCreator2 is typed, we can now enforce type-safety for the column's default value. That means

 tbl has_column('MyCol) as classOf[Int] default ""

will fail to compile.
The only downside to this DSL (which I'm trying to determine a way around) is that primaryKey and not_null attributes of a column both require a () to compile within scala. This is a very minor downside, but perhaps I'll determine a clever way around it in the future.

The next bit I'd like to show is some syntax to add foreign keys to our tables. Here's another potential migration usage example:


class Migration2 extends Migration {
 import MigrationDSL._


 def upgrade(implicit ctx : MigrationContext) {
  
   create_table('roles) {
     tbl =>
       tbl has_column 'user_id as classOf[Int]
       tbl has_column 'role as classOf[String] not_null()
       tbl has_foreign_key 'fk_user_id on 'user_id references 'user_id on 'user
   }
   ...   
 }
 ...
}

Notice we've added the "has_foreign_key" syntax here. The above syntax should create a foreign key titled "fk_user_id" from column user_id on the roles table to the column user_id on the user table.
Once again, this looks like (and is) a function call on the TableCreator trait. Here's our new Trait definition:

trait TableCreator { 
 def has_column(name : Symbol) : ColumnCreator
 def has_foreign_key(name : Symbol) : FkConstraintCreator1
}

Now to define the FkConstraintCreator traits. Because we need to ensure specific ordering of the DSL, we're basically defining a series of traits the expose the correct methods in the appropriate order. Here they are:


trait FkConstraintCreator1 {
 def on(name : Symbol) : FkConstraintCreator2
}
trait FkConstraintCreator2 {
 def references(columnName : Symbol) : FkConstraintCreator3
}
trait FkConstraintCreator3 {
 def on(tableName : Symbol) : FkConstraintCreatorLast
}
trait FkConstraintCreatorLast {
 def on_delete(action : Symbol)
 def on_update(action : Symbol)
}

With this series of FkConstraintCreators, we can now enforce a particular ordering of language in our DSL. The nice part is that we can still define one FkConstraintCreator (or Builder) implementation that inherits from ALL the FKConstraintCreator traits. This could help simplify our implementation while leaving the DSL staticly-type-safe.

In conclusion it was amazingly easy to define a somewhat rich DSL in Scala. Perhaps in the future I'll write an implementation for the MigrationContext and contribute this code to Lift and/or Scala proper.

Using Maven-Ant-Tasks instead of Ivy.

2008-09-25T23:18:00.000-04:00

So recently I had the (clever?) idea to attempt to use the maven-ant-tasks to bootstrap an entire build of scala. This could be a good way to jump-start existing ant users into the scala environment. If not, at least it proved to me I have at least a basic understanding of what I'm using.

So basically the build file is pasted below in its entirety. The build file will:

Download the maven-ant-tasks jar file (2.0.9)

Create 2 files, one scala and one java

Download the scala-library and scala-compiler from the maven-central repository

Compile the scala and java code

Create a jar file with the compiled code. This jar file will have its Main-Class and Class-Path attributes set to allow easy usage

Copy the scala-library into a local path that the "project" jar file expects.

When finished, type: java -jar build/test-maven-in-ant.jar (or use \ in windows).

The fun part about this build file was having it bootstrap ant-tasks via the maven task. It wouldn't be hard to pull in the groovy task as well (which I plan to attempt given more free time). Thanks to the latest scala feature of parsing java files, scala-groovy-java projects are now more easily accomplished.

Anyway, without further ado... Here's the buildfile:



<project name="test-maven-in-ant" default="package.done" xmlns:artifact="urn:maven-artifact-ant">

 <property name="root.dir" value="." />
 <property name="sources.dir" value="${root.dir}${file.separator}src" />
 <property name="build.dir" value="${root.dir}${file.separator}build" />
 <property name="build.classes.dir" value="${build.dir}${file.separator}classes" />
 <property name="lib.dir" value="${root.dir}${file.separator}lib" />

 <property name="maven.ant.tasks.jar" value="${lib.dir}${file.separator}maven-ant-tasks-2.0.9.jar"/>
 <property name="maven.ant.tasks.bootstrap.location" value="http://apache.inetbridge.net/maven/binaries/maven-ant-tasks-2.0.9.jar"/>
 <available property="maven.ant.tasks.jar.exists" file="${maven.ant.tasks.jar}"/>

 <property name="package.name" value="${ant.project.name}.jar"/>
 <property name="package.location" value="${build.dir}${file.separator}${package.name}"/>
 
 
 <!-- This will download the "latest version" of the maven-ant-tasks if needed -->
 <target name="bootstrap.maven.tasks" unless="maven.ant.tasks.jar.exists">
  <mkdir dir="${lib.dir}"/>
  <get src="${maven.ant.tasks.bootstrap.location}" dest="${maven.ant.tasks.jar}"/>
 </target>

 <!-- This will initialize all the maven ant tasks -->
 <target name="init.maven.tasks" depends="bootstrap.maven.tasks">
  <path 
   id="maven.ant.tasks.classpath" 
   path="${maven.ant.tasks.jar}" />
  <typedef 
   resource="org/apache/maven/artifact/ant/antlib.xml" 
   uri="urn:maven-artifact-ant" 
   classpathref="maven.ant.tasks.classpath" /> 
 </target>
 
 <!-- This bootstraps the scala ant tasks from the maven repository and initializes them -->
 <target name="init.scala.tasks" depends="init.maven.tasks">

  <artifact:dependencies pathId="scala.tasks.classpath">
   <dependency groupId="org.scala-lang" artifactId="scala-compiler" version="2.7.2-rc2" scope="compile" />
   <remoteRepository id="scala-tools.repository" url="http://scala-tools.org/repo-releases" />
  </artifact:dependencies>

  <taskdef resource="scala/tools/ant/antlib.xml">
   <classpath refid="scala.tasks.classpath" />
  </taskdef>

 </target>

 <!-- Initializes all the directories we need to build -->
 <target name="init.build.dirs">
  <mkdir dir="${build.classes.dir}" />
 </target>

 <!-- Initializes the classpath for the project dependencies-->
 <target name="init.project.dependencies" depends="init.maven.tasks">
  
  <artifact:dependencies pathId="project.classpath" filesetId="project.dependency.fileset">
   <dependency groupId="org.scala-lang" artifactId="scala-library" version="2.7.2-rc2" scope="compile" />
   <remoteRepository id="scala-tools.repository" url="http://scala-tools.org/repo-releases" />
  </artifact:dependencies>
 </target>

 <!-- Initialize source code for compilation -->
 <target name="init.sources">
  <mkdir dir="${sources.dir}/test"/>
  <echo file="${sources.dir}/test/IDoStuff.java">package test;

   public interface IDoStuff {
    public void doStuff();
   }
  </echo>
  <echo file="${sources.dir}/test/ScalaDoStuff.scala">package test

   class ScalaDoStuff extends IDoStuff {
     override def doStuff() : Unit = {
       Console println "HAI"
     }
   }

   object ScalaMain {
     def main(args : Array[String]) : Unit = {
       val tmp : IDoStuff = new ScalaDoStuff();
       tmp.doStuff();
     }
   }
  </echo>
 </target>
 
 
 <!-- Dummy target so we can add arbitrary init tasks -->
 <target name="init.done" depends="init.build.dirs,init.project.dependencies,init.scala.tasks,init.sources">
 </target>

 <!-- Initial step for all compilation.  Marker -->
 <target name="compile.init" depends="init.done" />
 
 <!-- compiles Scala Code -->
 <target name="compile.scala" depends="compile.init">
  <scalac srcdir="${sources.dir}" destdir="${build.classes.dir}" classpathref="project.classpath" force="changed">
   <include name="**/*.scala" />
   <include name="**/*.java" />
  </scalac>
 </target>
 
 <!-- Compiles java code, depends on scala -->
 <target name="compile.java" depends="compile.scala">
  <javac srcdir="${sources.dir}" destdir="${build.classes.dir}" source="1.5" target="1.5">
   <include name="**/*.java"/>
   <classpath refid="project.classpath"/>
   <classpath>
    <fileset dir="${build.classes.dir}">
     <include name="**/*.class"/>
    </fileset>
   </classpath>
  </javac>
 </target>
 
 <!-- Marker for finishing the compilation phase -->
 <target name="compile.done" depends="compile.scala,compile.java" />
 
 <!-- Marker for package phase -->
 <target name="package.init" depends="compile.done"/>
 
 <!-- copy our dependencies from maven's struture into our lib directory -->
 <target name="package.copy.dependencies" depends="package.init">
  <copy todir="${build.dir}/lib">
    <fileset refid="project.dependency.fileset" />
    <!-- This mapper strips off all leading directory information --> 
    <mapper type="flatten" />
  </copy>
 </target>
 
 <!-- package our jar file -->
 <target name="package.jar" depends="package.init">
  <jar destfile="${package.location}" basedir="${build.classes.dir}">
   <include name="**/*.class"/>
   <manifest>
    <!-- Hacked for now -->
    <attribute name="Main-Class" value="test.ScalaMain"/>
    <attribute name="Class-Path" value="lib/scala-library-2.7.2-rc2.jar"/>
   </manifest>
  </jar>
 </target>
 <!-- marks the end of the package phase -->
 <target name="package.done" depends="package.jar,package.copy.dependencies"/>
</project>

Masochism

2008-09-24T21:35:00.004-04:00

Note: Due to excessive comments about my grasp of the english languages, I've done a second round of editing for this post. It may show up on the RSS feed again, but there isn't any new content.

I've recently had to jump into a bit of (very minor) C++ development at work. This started bringing back memories of all the ?fun? it was to play around in the C++ type system. Mostly you use the C++ type system in an attempt to prevent yourself from doing bad things. Sometimes though, it was a way of doing conditional compilation in a type-safe manner.

I recently decided to outline how I tend to over-engineer my C++ code. I'll give a few ground rules/goals first:

I won't use boost. In real life, I do use boost whenever I can. For the purposes of illustration I'm doing a lot by hand. If you're a C++ developer and you don't use boost, go learn it. Now. Really, I mean right now go to www.boost.org and enter the 21st century of coding. No really, GO
I'm not doing detailed explanations (although I do some explanations) of the C++ language or its compiler workings.
This could be an attempt to ramp myself to attack learning scala's (and maybe haskell's) type sytem. C++ is the closest typed language I know that could prepare me
C++/Java feels more like massochism the more new languages I learn. I'm hoping this "simple" post about using the C++ type system in a "simple" way will prove this to you. At some point I hope to post a Scala/Haskell equivalent to this post (or perhaps someone else would like to?) just so you can see the amount of work C++ really requires (and the level of understanding required to write even a simple line).

Alright! so first off, the premise: We're trying to create a class (DynamicLibrary) that will let us dynamic load a shared library and access symbols. I'm only including code for linux (although porting to windows or "cross-platforming" isn't too bad. look into the LoadLibrary function in windows and dlopen in linux).

Before we begin, I'm going to construct something (available in boost) that should be in the utility kit of every C++ developer... a base class that removes the copy-constructor and operator= from visibility (effectively making it non-copyable)

#ifndef NONCOPYABLE_H_
#define NONCOPYABLE_H_
//============================================================================
// This class prevents copying on subclasses
//============================================================================
template
class NonCopyable
{
public:
inline NonCopyable() {}
inline ~NonCopyable() {}
private:
NonCopyable(NonCopyable& other) {}
NonCopyable& operator=(NonCopyable& other) {}
};


#endif

It's a very simple class. Most of its functions are purposefully inlined. It should get compiled out during optimization phases, AND it can reduce typing for me later when I don't want a class copied around on the stack.
(Note: I could have saved one line of typing by changing to a struct and getting rid of the public: line. or putting all private functions first. Sue me, but I prefer this look).

Ok, next we're going to create a very small Meta-Programming Library. This library will be used to tell us all sorts of information about the types in C++. For now, please assume all these examples are surrounded in a namespace mpl {} block.

Here's our first bout of magic:

struct true_;
struct false_;  //Not really needed, but easier to understand if we have it

WTF? Why make two completely silly classes? This is so I can represent the notion of a boolean in the type system. From now on, I want you to think of true_ as "true in the meta-programming library" and similarly for false_. Note: In actuality I could probably use bool as a template parameter and get away with it, but this is more fun

Ok, now let's make a simple method just so we can print the boolean value of a type.

template<typename T>
struct is_true
{
enum { value = 0 };
};
template<:>
struct is_true<true_>
{
enum { value = 1 };
};

So..... what is that? That, my friends, is a C++ meta-programming function. The first template applies to every type and "returns" a value of 0 (or false). The second is a specialization for the true_ type that will return 1. This means if we ever use the is_true structure with true_ as the argument, the "member" value will be 1. Its a little convoluted (or convolved?), but it works! How do you call it? Simply like this:

is_true<some type>::value.

That will return an actually value we can assign, test in an if statement or display on the console. We're going for displaying on the console here (console = stdout).

Ok, so let's get more interesting meta-functions. The most important for what I'd like to do later is the "is_ptr" function. Here's a look at it:

template<typename T>
struct is_ptr
{
typedef false_ value;
};
template<typename T>
struct is_ptr<T*>
{
typedef true_ value;
};

So before we saw "template specialization". That is, template<> with all types completely specified in an alternative implementation of the function. This time, we see "partial template specialization" (YAY!!?). This is where we have an alternative implementation that still has some wildcards (i.e. partially specialized). When used, the compiler will select the function from most specialized to most general based on our types, so we can get away with this.

Now, what's that funny type-def doing and why isn't "value" part of an enum? Well this meta-function takes a type as argument and returns a type (not a value). You can string together meta-functions of this kind to make even stranger (and horrendous to look at) meta-functions.

As a simple example, let's make an "if" meta-function:

template<typename COND, typename T, typename F>
struct if_
{
typedef F value;
};
template<typename T, typename F>
struct if_<true_, T, F>
{
typedef T value;
};

The above is a structure that will select either type T or F based on the valiue of COND (much like an if statement).

For now, let's test our first two meta-functions on a few combinations of the various incarnations of ints in C++.

cout << "mpl::is_ptr<int*>::value = " << mpl::is_true<mpl::is_ptr<int*>::value>::value << endl;
cout << "mpl::is_ptr<int>::value = " << mpl::is_true<mpl::is_ptr<int>::value>::value << endl;
cout << "mpl::is_ptr<const int>::value = " << mpl::is_true<mpl::is_ptr<const int>::value>::value << endl;
cout << "mpl::is_ptr<const int*>::value = " << mpl::is_true<mpl::is_ptr<const int*>::value>::value << endl;
cout << "mpl::is_ptr<volatile int>::value = " << mpl::is_true<mpl::is_ptr<volatile int>::value>::value << endl;
cout << "mpl::is_ptr<const volatile int>::value = " << mpl::is_true<mpl::is_ptr<const volatile int>::value>::value << endl;
cout << "mpl::is_ptr<volatile int*>::value = " << mpl::is_true<mpl::is_ptr<volatile int*>::value>::value << endl;
cout << "mpl::is_ptr<const volatile int*>::value = " << mpl::is_true<mpl::is_ptr<const volatile int*>::value>::value << endl;

And the output is:

mpl::is_ptr<int*>::value = 1
mpl::is_ptr<int>::value = 0
mpl::is_ptr<const int>::value = 0
mpl::is_ptr<const int*>::value = 1
mpl::is_ptr<volatile int>::value = 0
mpl::is_ptr<const volatile int>::value = 0
mpl::is_ptr<volatile int*>::value = 1
mpl::is_ptr<const volatile int*>::value = 1

Neat-o huh? Yeah I probably could change the output to really represent things, ie mpl::is_true<mpl::is_ptr<....>::value>::value ? "true_" : "false_". However it was already way to much typing.

Ok, now onto why we were making meta-functions to begin with. I want a somewhat type-safe way of importing symbols from a dynamicly loaded library and casting them to their appropriate types. This idea came from an old colleague at APL. I hope this solution is different enough, as I don't recall his solution.

Let's start off with our basic class structure for DynamicLib:

//ASSUME LIBRARY_HANDLE_TYPE and SYMBOL_POINTER_TYPE are defined somewhere above (os-specific)
class DynamicLib : public NonCopyable<DynamicLib> {
public:
DynamicLib(const char*const name); //implementation is OS-specific
~DynamicLib();                     //implementation is os-specific
static SYMBOL_POINTER_TYPE get_symbol_internal(const char*const name, LIBRARY_HANDLE_TYPE handle); //implementation is OS-specific.  Didn't feel like figuring out to privatize this.
private:
LIBRARY_HANDLE_TYPE handle; //handle to library (os-specific)
};

First off, notice the use of NonCopyable. This ensures the class cannot be copied (i.e. you have to use pointers to share it). This helps us ensure that the library is loaded once and destroyed once. You could still construct it twice for the same library, but if you do that... well a maniacal penguin dressed in a business suit will come steal your fish.

Next, the get_symbol_internal method is static. This is because the public (local) method has yet to be written and needs to use the MPL library (YAY!!!?). What we'd like to do is find a way to return primitives by value, and everything else by pointer. I'm going to call anything a "primitive" if it has a valid copy constructor (boy would this be wrong in a large app). Instead of pulling in Matthew Wilson's "must_be_pod" meta-function-constraint, I think I'll just assume proper usage (like a fool). Note: A maniacal penguin did show up at my door, but I was able to dissuade it with a tennis racket and tuna fish.

Ok, so onto the symbol loading... Here's what our public symbol loading function looks like:

//function for external use
template<typename T>
inline T get_symbol(const char*const name) {
//Call appropriate function based on whether type is pointer or not.
typedef get_symbol_<T, typename mpl::is_ptr<T>::value> get_symbol_functor;
get_symbol_functor functor; //need an instance to use operator()...
return functor(name, handle);
}
private:
template<typename T, typename isptr>
struct get_symbol_;

Well, that's simple. Too bad it won't compile when someone actually tries to use it. What's going on here? Well the get symbol function is templatized on some type. Then we try to call the get_symbol_ functor struct. What is a functor struct (besides a lame excuse for a closure/real functor)? It's a structure that (sort of) looks like a function in code (as shown above). We declare (but not define) the structure. The structure is where we will specialize our implementation for pointer types vs. non-ptr types. Since we're using the "is_ptr" meta function above, all we need are two partial specialization for the get_symbol_ functor. The first specialization will take "true_" as its second type parameter. The second specialization will take "false_" as its second type parameter. The false variant should perform a copy of the value pointed at by the symbol. The true variant can just cast and return the symbol pointer. Here's the code:

//Repeated for clarity
template<typename T, typename isptr>
struct get_symbol_;

template<typename T>
struct get_symbol_<T, typename mpl::true_> {
T operator()(const char*const name, void* handle) {
//This should fail to compile for types that don't have copy-constructors
return reinterpret_cast<T>(DynamicLib::get_symbol_internal(name, handle));
}
};
//Get symbol for non-ptr types
template<typename T>
struct get_symbol_<T, typename mpl::false_> {
T operator()(const char*const name, void* handle) {
//TODO - Fix const...
return *static_cast<T*>(DynamicLib::get_symbol_internal(name, handle));
}
};

So now you can see how a struct-functor is created. Basically you just define an operator() overload inside the structure, and it will start acting like a function (or at least looking like one...).

You can see how on the one hand we're de-referencing the pointer, and on the other we're just casting. I'm also going to assume here (wrongly) that the get_symbol_internal will throw an exception on error, thereby ensuring that I will never dereference an invalid pointer (feel free to laugh, or send the penguin again).

Ok, and now for the main method that uses this mess of code (note I'm using getopt.h):

int main(int argc, char*argv[]) {
int c;
extern char* optarg;
extern int optind, optopt;
string library = "";
string symbol = "";
string type = "";
bool usage = false;
while ((c = getopt(argc, argv, "l:s:t:h")) != -1) {
switch (c) {
case 'l':
library = optarg;
break;
case 's':
symbol = optarg;
break;
case 't':
type = optarg;
break;
case '?':
cerr << "Unrecognized option: -" << static_cast<char> (optopt)
<< endl;
case 'h':
usage = true;
break;
}
}

if (usage) {
//TODO
return 2;
}

DynamicLib lib(library.c_str());

if(type == "int") {
cerr << "Symbol value (as int) is: " << lib.get_symbol<int> (symbol.c_str()) << endl;
}
if(type == "function") {
cerr << "Symbol value (as return of function [int ()] ) is: " << lib.get_symbol<int (*)()> (symbol.c_str())() << endl;
}
return 0;
}

The hardest one to understand here is possibly the function pointer. Basically I'm asking for a pointer to a no-argument function that returns an integer [int (*)()]. I then immediately call this function for a value to place in cerr.

Ok, now we need a shared-library to test out. Here's my dummy .cpp that I compiled into a .so (I leave that as an excersie to the user).

extern "C" {
int MY_VAL = 5;
int MY_FUNC() { return MY_VAL; }
}

Onto our outputs:

>  Debug/cpp-types -l ../dummy-shared-lib/Debug/libdummy-shared-lib.so -s MY_VAL -t int
Symbol value (as int) is: 5
>  Debug/cpp-types -l ../dummy-shared-lib/Debug/libdummy-shared-lib.so -s MY_FUNC -t function
Symbol value (as return of function [int ()] ) is: 5

Ok, so this is a little bit limited, but we have it working! YAY!!!?

In conclusion, I really want to get back into Scala again. The type system seems a little bit less verbose, and the language itself is less clunky.

Note: I later updated this to pull string-literals out of libraries. The fun part there was realizing that a symbol pointer to a string literal is *really* a [char**] that needs to be dereferenced (carefully). Basically I wrote a specialized std::string specialization of get_symbol that would perform the checks/casts and return a string. The key to testing this was ensuring that the char* literal would wind up in the symbol table of the output .so. This means you should make a dummy function that returns it, otherwise it may be optimized away. Enjoy!

Reader Response: proxy_ajp is better than mod_jk

2008-09-23T17:58:00.000-04:00

Wow, so not only do I have readers, but I have readers willing to contribute their time to improving the information available on this blog! Special thanks to Emerson for this post.

This is a complementary post to Ubuntu Dev Server - Hudson[1] with a simpler way (IMO) to integrate Tomcat with Apache.

This can be achieved by enabling mod_proxy and mod_proxy_ajp in Apache. If you also use ubuntu-server, like I do, just link them symbolically:

$ sudo ln -s /etc/apache2/mods-available/mod_proxy /etc/apache2/mods-enabled/mod_proxy
$ sudo ln -s /etc/apache2/mods-available/mod_proxy_ajp /etc/apache2/mods-enabled/mod_proxy_ajp

Now edit apache2.conf like this:

<Location /your_tomcat_app>
ProxyPass ajp://your_host:8009/your_tomcat_app
Order allow,deny
allow from all
</Location>

Then restart Apache and it's done. To add another application, just repeat the same setup above mutatis mutandis[2].

One additional setup for my application was to enable access to URLs, like <http://my-server/my-tomcat-app/servlet/my-servlet-name>. To do this, edit /etc/tomcat5.5/web.xml and find the string "invoker". You will find the following commented servlet element:

<!--
   <servlet>
        <servlet-name>invoker</servlet-name>
        <servlet-class>
          org.apache.catalina.servlets.InvokerServlet
        </servlet-class>
        <init-param>
            <param-name>debug</param-name>
            <param-value>0</param-value>
        </init-param>
        <load-on-startup>2</load-on-startup>
    </servlet>
-->

Just strip the comment tags and restart Tomcat. Integrating Tomcat with Apache should be that easy since the beginning.

[1] http://suereth.blogspot.com/2008/08/ubuntu-dev-server-hudson.html
[2] http://en.wikipedia.org/wiki/Mutatis_mutandis

Hudson on Winstone + Apache

2008-09-17T14:05:00.000-04:00

I decided to make a quick post about hudson. I've been using it for work for a few weeks now and after the initial setup it's required almost no maintenance. I also like how easier it is to extend and setup.

Anyway, just thought I'd include some screenshots of installing plugins for hudson. It's as simple as clicking through the web interface and then restarting the server.

And here's downloading a plugin:

War Games?

Hudson has the neat ability to run only using the war. It makes use of a servlet container called "winstone". This was my first look into winstone, but one of the nice features (that tomcat also has) is AJP support. Let's look at how we can install hudson as a service (I'm using CentOS 5.2 for this).

First, create a hudson user and group. Next create a /var/hudson directory. Make sure hudson owns this directory. Place the hudson.war into it.

Here's an init script I modified from the comments on Hudson's install instructions:


# chkconfig: 2345 84 16
# description: Hudson CI server

# Source function library.
. /etc/rc.d/init.d/functions
[ -z "$JAVA_HOME" -a -x /etc/profile.d/java.sh ] && . /etc/profile.d/java.sh

USER=hudson
HUDSON_HOME=/var/hudson
WAR="$HUDSON_HOME/hudson.war"
LOG="/var/log/hudson.log"
LOCK="/var/lock/subsys/hudson"
export HUDSON_HOME

RETVAL=0

pid_of_hudson() {
  ps auxwww | grep java | grep hudson | grep -v grep | awk '{print $2}'
}

start() {
  [ -e "$LOG" ] && cnt=`wc -l "$LOG" | awk '{ print $1 }'` || cnt=1

  echo -n $"Starting hudson: "

  echo "" > "$LOG"
  chown $USER $LOG
  cd "$HUDSON_HOME"

  CMD="nohup ${JAVA_HOME}/bin/java -DHUDSON_HOME=${HUDSON_HOME} -jar ${WAR} --httpPort=8080 --ajp13Port=8010 --prefix=/hudson >> ${LOG} 2>&1 &"
  su -m --command="${CMD}" hudson
  while { pid_of_hudson > /dev/null ; } &&
        ! { cat "$LOG" | grep -q 'Winstone Servlet Engine .* running' ; } ; do
      sleep 1
  done

  pid_of_hudson > /dev/null
  RETVAL=$?
  [ $RETVAL = 0 ] && success $"$STRING" || failure $"$STRING"
  echo

  [ $RETVAL = 0 ] && touch "$LOCK"
}

stop() {
  echo -n "Stopping hudson: "

  pid=`pid_of_hudson`
  [ -n "$pid" ] && kill $pid
  RETVAL=$?
  cnt=10    while [ $RETVAL = 0 -a $cnt -gt 0 ] &&
        { pid_of_hudson > /dev/null ; } ; do
      sleep 1
      ((cnt--))
  done

  [ $RETVAL = 0 ] && rm -f "$LOCK"
  [ $RETVAL = 0 ] && success $"$STRING" || failure $"$STRING"
  echo
}

status() {
  pid=`pid_of_hudson`
  if [ -n "$pid" ]; then
      echo "hudson (pid $pid) is running..."
      return 0
  fi
  if [ -f "$LOCK" ]; then
      echo $"${base} dead but subsys locked"
      return 2
  fi
  echo "hudson is stopped"
  return 3
}

# See how we were called.
case "$1" in
start)
  start
  ;;
stop)
  stop
  ;;
restart)
  stop
  start
  ;;
*)
  echo $"Usage: $0 {start|stop|restart|status}"
  exit 1
esac

exit $RETVAL

After creating the file, make sure you install it with

chkconfig --add hudson

Finally, if we only want to open port 80, it's easy to configure apache to forward to winstone using mod_ajp. Here's my apache configuration:

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
ProxyPass /hudson/ ajp://localhost:8010/hudson/

Now just head to http://localhost/hudson and you should see hudson.

Entirely Too Motivated

2008-08-26T22:06:00.000-04:00

So this week I've been very motivated to work on my more nerdy hobbies. Here's an update on my latest progress

Made use of Hudson to set up a continuous build. WOW is this software great. I'm amazed at how good free software can get!

Attempted to get a Scala job up on hudson, so I'll know early on when scala fails on windows. :Sigh: I'm not sure what's going on here, but it looks like no one at EPFL really wants to maintain Scala for windows. While I understand their sentiments, I now have a mini-project at work that I cannot develop for because they've broken the eclipse plugin. I've managed to compile with an older scala, but this means I've lost my dev environment. I've heard someone say, if you stay on the bleeding-edge, expect to bleed. I guess I was hoping scala was further along in maturity then this. Oh well, back to contributing bug-fixes/enhancements in my spare time and secretly hoping we can use this for future development.
I managed to get Archiva up and running on ubuntu. For some reason (probably my stupidity), I had a difficult time figuring out where the common/lib file was. (Archiva requires derby, mail and activation in common/lib). For those of you like myself, common/lib is under /usr/share/tomcat5.5 NOT /var/lib/tomcat5.5. Anyway, now that I have archiva + hudson + trac up and running, it's time for a really sweet development environment. I'm working on trying to wire at least hudson into our project at work. I'll probably be posting what's useful and what to avoid later.

Also I'd like to take this time for a mini-rant.
<rant>
I'm really confused here. I got on the #eclipse-dev list serve to ask some questions about the eclipse PDE project. I received answers on where to look to solve my issues. I got on the #scala list serve to ask some general architecture questions, and was told to look at some haskell apps because Scala is impure (Note: this is my wording). Ok, the haskel App engine thing looked pretty cool, but I couldn't get past the syntax...(I'd rather read LOLCode all day). How many key contributors to Scala feel it's a "less-good" Haskell? I know a lot of libraries are inspired by Haskell, but not a lot of libraries appear to be inspired from other JVM languages. This confuses me, as I thought Scala was targeted towards Java developers. There aren't a lot of Helper routines to make using Scala with existing java code/libraries easier.

Example: I'm integrating with a bunch of Java-Threaded code. It uses Runnable and callable. I created a JavaThreadConversion object that looks like this:

object JavaThreadConversion
{
  def functionToRunnable(f : => Unit) = new Runnable() {
     override def run() = f
  }
  import java.util.conncurrent.Callable
  def functionToCallable[A](f : => A) = new Callable() {
     override def call() = f
  }
}

That little bit of code save me a lot of typing when interfacing with my old java libraries.

Why don't you rewrite them in scala Not enough time/money/priority. Not gonna happen any time soon.

Oh well. Hopefully as more Java developers see the power of scala, they'll begin to contribute more utilities that can make transitioning off java easier. Until then, I'll see what of mine really is useful and can pass the "functional purity" filter.

Amusing note: At point I wondered why there weren't any Scala + OSGi helpers as I hear a lot of people asking how to do XYZ scala task using OSGi. I'd like to see how much that make a FP purist cringe... OSGi seems to be the bane of FP. Even worse would be the mixing of a Scala core with a Groovy plugin system...
</rant>

Anyway, don't mean to offend anyone. Just frustrated with certain things, including my slow-learning-process for various FP concepts (like monads)

Statically Duck-Typed (Take 2)

2008-08-25T20:24:00.001-04:00

So I've been reading the "Programming in Scala" book in my spare time. There's a lot of neat patterns to learn from the Scala language. Anyway, I was reading about curried functions and the loan pattern, and decided to write an updated version of my "doThenClose" helper method.

To accomplish this, we want to change the "doThenClose" method (which was on a "RichCloseable" object). Here's what we need to do:

Rename to "withResource"

Make doThenClose global

Make it take in a "resource" or "closeable"

Modify the type parameters so we don't lose the original type of the closeable. (This way we can "loan" it to the function that uses it

Modify the type of functions we take to those that operate on the type based in and return something

Here's our original "doThenClose" method

/**
   * Defines new scala-awesome features we want.
   */
  class PimpedCloseable(closeable: MyCloseable) {
     /**
      * Executes a function, then closes the resource.
      */
     def doThenClose[A](f : => A) : Option[A]= {
      try {
        return Some(f)
      } catch {
      case _ => System.err.println("Trouble executing", f);
      } finally {
        CloseUtils.close(closeable);
      }
      return None;
    }
  }

And here's the new version:

  def withResource[A <: Closeable, B](closeable : A)( f : A => B) = {
    try {
      f(closeable)
    } finally {
      CloseUtils close closeable
    }
  }

First of, let's describe the type parameters. "A <: Closeable" means "anything meeting the closeable interface" and "B" is the return type of the withResource method (which we infer from the argument that will use our resource). Also remember that Closeable is a TYPE not a class (i.e. type Closeable = { def close() : Unit } ). This is where the "static duck-typing" comes in.

Now, the method (withResource) defines what appears to be two argument lists. What this is really doing is creating two functions, the first generates the second. This is the function currying. The result of this, is that when calling the function, you use two argument lists as well. Because scala performs some syntactic sugar for 1-argument functions, we can use {} instead of () to surround either of the arguments. What this means is we can write the following test application:

object BlogTest {
  def main(args : Array[String]) : Unit = {
    import CloseUtils._
    import java.io._
    withResource(new FileInputStream("src/com/suereth/CloseUtils.scala")) {
      file =>
       val reader = new BufferedReader(new InputStreamReader(file))
       var line = reader.readLine()
       while ( line != null ) {
         System.out.println("" + line.trim.size + ":" + line)
         line = reader.readLine
       }
    }
  }
}

Notice the second argument list to "withResource" is { file => ...}. This is a function that is "loaned" the resource created. The function can do whatever it wants to the resource, however when it exists (via return or throw), the resource *will* be closed. Although I'm not a fan of the "resource =>" syntax, it's still a pretty amazing feature *and* totally user-defined.