Untupled

Unfiltered validation has been on my to-do list for as long as there’s been an Unfiltered. I tried a few different approaches, none of which were good enough to move it to-done.

First I tried doing it with extractors, the golden hammer of Unfiltered.

Requests are commonly accompanied by named parameters, either in the query string of the URL or in the body of a POST. Unfiltered supports access to these parameters with the Params extractor.

– Extracting Params

The results were unsatisfying. If you want typed results, and you probably do, the only thing you can do with unacceptable input is refuse to match it, typically responding with a 404.

I never expected the extractor approach to work out, but the extractors are easy to build and easy to use – for mediocre results. My grander scheme was to support parameter validation such that it would be easy to these things:

1. build your own reusable validators.
2. build your own validators inline.
3. respond to multiple unacceptable parameters with multiple error messages.

And I tried for a while to do this by hacking for-expressions. It was difficult to build, and difficult to use. I didn’t use it much myself and periodically forgot how it worked. Something kept me from ever documenting it. Common sense?

Later, Jan-Anders Teigen contributed directives to Unfiltered. Directives use for-expressions to validate requests in a straightforward way. Missing a header we require? Respond with the appropriate status code. You can also use them for routing, by orElse-ing through non-fatal failures. What you couldn’t do was accumulate errors, my requirement #3.

I had a week with no internet in the Adirondacks, and it seemed as good a time as any to face my old nemesis, parameter validation for the people.

The first thing I did was to build into directives a syntax for interpreting parameters into any type and producing errors when interpretation fails.

This took some time to get right, mostly choosing what to call things and crafting an API for both explicit and implicit use. Define your own implicit interpreters to produce your own types and error responses, then you can collect data like a lovesick NSA officer.

val result = for {
  device <- data.as.Required[Device] named "udid"
} yield device.location

This could report failure in different ways according to your own interpreter; the udid parameter is missing, not in the right format, isn’t in the database, the database is down, and so on.

After this I decided to tackle my dreaded requirement #3. This time I wouldn’t abuse for-expressions or change the way directives fundamentally work. When directives are flat-mapped together, the result is a mapping of all their successes or the first failure.

So how do we produce multiple responses for multiple failures? I settled on the idea of combining multiple directives into one, which would itself produce a mapping of all their successes or a combination of all their failures. This combined directive would then, typically, be flat-mapped to other directives in a for-expression.

The combination step is easy enough to express, even if it was a little tricky to implement.

scala> (data.as.Required[String] named "a") &
  (data.as.Required[Int] named "b") 
res1: unfiltered.directives.Directive[
  Any,
  unfiltered.directives.JoiningResponseFunction[String,Any],
  (String, Int)] = <function1>

The first type parameter of Directive has to do with the underlying request, the second is the joined error response type, and the third is the success type – a tuple of the two directives’ success types.

The joining method & produces a tuple so that successes preserve all their type information. We might use it in a for expression like so:

(a, b) <- (data.as.Required[String] named "a") &
  (data.as.Required[Int] named "b") 

But what happens if there’s more than one independent parameter?

scala> (data.as.Required[String] named "a") &
  (data.as.Required[Int] named "b") &
  (data.as.Required[BigInt] named "c")
res2: unfiltered.directives.Directive[
  Any,
  unfiltered.directives.JoiningResponseFunction[String,Any],
  ((String, Int), BigInt)] = <function1>

Oh dear — our tuples are nested. To assign the values now, we would need to nest tuples exactly the same.

((a, b), c) <- (data.as.Required[String] named "a") &
  (data.as.Required[Int] named "b") &
  (data.as.Required[BigInt] named "c")

This could get rather confusing, especially when we want to add or remove a parameter later.

To understand why the tuples are nested, think about what the & method does. For d1 & d2, it produces a new directive where the success values are a tupled pair. It does this always, according to its return type.

Now consider the case of three directives: d1 & d2 & d3. We could write it without infix notation: d1.&(d2).&(d3). It’s clearer still with parenthesis grouping the infix operations in their normal order of evaluation: ((d1 & d2) & d3). With repeated applications of & we’ll necessarily produce typed, nested pairs. You can see a simpler example with the standard library’s tuple constructor:

scala> 1 -> 2 -> 3
res3: ((Int, Int), Int) = ((1,2),3)

So this makes sense, even if we don’t like it. We’d rather access the results as if the structure were flat. A Seq would allow that, but we’d lose the component types. Another approach, would be to apply a single joining function across all the directives we want to combine:

a, b, c <- &(data.as.Required[String] named "a"),
  (data.as.Required[Int] named "b"),
  (data.as.Required[BigInt] named "c")

This looks pretty nice, but it comes at a high cost: the function & would have to be defined specifically for 2 arguments, 3 arguments, and so on up to 22. And if somebody wanted to use it for 23 independent parameters, too bad. You’ll see that kind of code in some libraries including the standard library; I think it’s usually generated. But I really didn’t want to add it to Unfiltered if I didn’t have to.

And I didn’t have to.

In Scala, common data types are built into the standard library instead of the language. So we can do things like this without List being special, or known to the compiler at all.

scala> val a :: b :: c :: Nil = 1 :: 2 :: 3 :: Nil
a: Int = 1
b: Int = 2
c: Int = 3

In order to support rich user-defined interfaces comparable to language-level features in other languages, the Scala language itself has features like infix notation that are surprising to the newcomer. This is all Scala 101, but on occasion I’m still impressed by the possibilities that basic design decision gives to the programmer.

Let’s try our list example again, with grouping parenthesis.

scala> val (a :: (b :: (c :: Nil))) = (1 :: (2 :: (3 :: Nil)))
a: Int = 1
b: Int = 2
c: Int = 3

Because of the right-associativity of methods ending with a colon, the nesting is reversed, but you can probably see that a solution to flattening our nested tuples is getting closer.

The standard library provides a :: case class as a helper to the :: method of List, and like all case classes it has a companion extractor object. The above relies on infix notation for the extractor; the constructor style makes it a little more plain.

scala> val ::(a, ::(b, ::(c, Nil))) = (1 :: (2 :: (3 :: Nil)))
a: Int = 1
b: Int = 2
c: Int = 3

That’s just how you expect case class extraction to work. On the right-hand side, :: is a method call on a list but its definition constructs the same case class. See for yourself. So actually, it’s more like the standard library provides the method as a helper to the case class.

That’s great for lists and we’d like to do the same for a pair, but we can’t use the same case class technique since Tuple2 is itself a case class. Not to worry, even though case classes are a language feature, the extractor functionality they use is available to any object. (It was a short holiday for this golden hammer.) We’ll call ours & since we want it to partner with the & method of Directive.

scala> object & {
     |   def unapply[A,B](tup: Tuple2[A,B]) = Some(tup)
     | }
defined module $amp

Let’s try it out with modest constructor notation first.

scala> val &(a,b) = (1,2)
a: Int = 1
b: Int = 2

Infix?

scala> val a & b = (1,2)
a: Int = 1
b: Int = 2

Nesting???

scala> val a & b & c = ((1,2),3)
a: Int = 1
b: Int = 2
c: Int = 3

Sweet!!!

Now we can assign arbitrarily many nested success values from combined directives in simple flat statement.

a & b & c <- (data.as.Required[String] named "a") &
  (data.as.Required[Int] named "b") &
  (data.as.Required[BigInt] named "c")

And that is basically how it works. One caveat is that Unfiltered already had a & extractor object for pattern matching on requests, but I was able to overload the unapply method without issue — once again, I’m rescued by the soundness of Scala’s core features.

You might wonder why I didn’t name it ->. And indeed, I could have.

scala> object -> {
     | def unapply[A,B](tup: Tuple2[A,B]) = Some(tup)
     | }
defined module $minus$greater

scala> val a -> b -> c = 1 -> 2.0 -> "3"
a: Int = 1
b: Double = 2.0
c: String = 3

Isn’t that pretty? I don’t know why this isn’t defined in the standard library already, perhaps it’s been discussed before. I think it would be useful, and I would have used that here instead of defining my own extractor. It would promote the pattern of nesting tuples, rather than defining 22 methods while taking a shot of vodka after each one.

(I am also aware, vaguely, that I have wondered onto ground inhabited by HLists. Don’t shoot! I come in peace.)

In any case, an object -> is not in the standard library. I don’t want to invite the possibility of a collision, should it be added later. And also, if nested tuples and -> extractors were a common pattern, it would be one thing. Since they are not yet, I don’t want to have to explain to people why & is used to join directives while a -> extractor is used to split them.

All told, I’m very pleased with the resulting API for parameter directives, and hope people will get good use out of it. Directives are now fully documented and Unfiltered parameter validation is finally done.