As programmers, we are very incautious with our use of the word “type”. The concept of “type” is sufficiently abstract and specific that we are tempted to understand it by analogy, so much that we begin to confuse analogy with sameness.

The colloquial “runtime type”, a fair approximation of “class”, makes it tempting to equate types with “classes, interfaces, traits, that sort of thing”, which I will name classes for the rest of this article. But they aren’t the same. The type system is much richer and more interesting than the class system, even in Java.

To appreciate this richness, we must stop thinking of types as classes and stop drawing conclusions from that weak analogy. Luckily, the compiler will readily reveal how unlike classes types are, if we ask it some simple questions.

One value with class, many variables with type

val greeting: String = "hi there!"

Here I have constructed a String and assigned it to a variable. (I have also constructed the char array in the String and various other details, but immediately handed those off to the String and forgotten about them.) This value has class String. It has several classes, really.

1. String
2. java.io.Serializable
3. CharSequence
4. Comparable[String]
5. Object/AnyRef

That seems like a lot of classes for one value. And they are genuine classes of greeting, though 2-5 are all implied by #1.

greeting also has all five of these types. We can ask the compiler to verify that this type truth holds, entirely separately from the class truth.

scala> (greeting: String, greeting: java.io.Serializable,
        greeting: CharSequence, greeting: Comparable[String],
        greeting: AnyRef)
res3: (String, java.io.Serializable, CharSequence,
       Comparable[String], AnyRef) =
  (hi there!,hi there!,hi there!,hi there!,hi there!)

So we have exhausted the classes, but aren’t quite done with types.

scala> greeting: greeting.type
res0: greeting.type = hi there!

greeting.type is not like the other five types we just tested. It is a strict subtype of String, and has no class with the same name.

// If and only if call compiles, A is a subtype of B.
def conformance[A, B >: A]: Unit = ()

scala> conformance[greeting.type, String]

scala> conformance[String, greeting.type]
<console>:14: error: type arguments [String,greeting.type] do not conform
              to method conformance's type parameter bounds [A,B >: A]

Fine, we can accept that object identity is represented at the type level without our universe imploding, by inventing the theory that this is about object identity; hold on, though:

scala> val salutation = greeting
salutation: String = hi there!

Fine, salutation is just another name for greeting, right?

scala> conformance[salutation.type, String]

scala> implicitly[greeting.type =:= salutation.type]
<console>:14: error: Cannot prove that greeting.type =:= salutation.type.

Now we have seven. I’ll spare you spelling out the induction: each new variable defined like salutation will yield a new alias with a distinct type. This is not about objects; this is about variables!

// find a type for the literal "hi there!"
scala> val literalHiThere = shapeless.Witness("hi there!")
literalHiThere: shapeless.Witness.Aux[String("hi there!")] = shapeless.Witness$$anon$1@1d1537bb

scala> conformance[greeting.type, literalHiThere.T]
<console>:15: error: type arguments [greeting.type,literalHiThere.T] do not conform
              to method conformance's type parameter bounds [A,B >: A]

scala> conformance[literalHiThere.T, greeting.type]
<console>:15: error: type arguments [literalHiThere.T,greeting.type] do not conform
              to method conformance's type parameter bounds [A,B >: A]

As local variables are a strictly compile-time abstraction, and we have anyway seen that the numbers don’t match up, that should be the end of the “types are classes” confusion for you. But maybe this is just some Scala oddity! And anyhow I haven’t even begun to demonstrate the overwhelming richness of the type model as it blindingly outshines the paucity of the class model. Let’s go further.

No values, infinite types: method type parameters

To our small program of a greeting, we can add a small method.

def pickGreeting[G](grt: G, rand: Int) = grt

scala> pickGreeting(greeting, 42)
res9: String = hi there!

It seems like G must be String, because the argument passed to pickGreeting is a string, and in that case so must its return value be, according to the implementation. And from the perspective of this call, outside pickGreeting’s implementation, it is String indeed.

But that implementation’s perspective matters, too; it is also part of our program. And it sees things quite differently. We can ask its thoughts on the matter by adding to its body

def pickGreeting[G](grt: G, rand: Int) = {
  implicitly[G =:= String]
  grt
}

<console>:12: error: Cannot prove that G =:= String.
         implicitly[G =:= String]
                   ^

In fact, G bears no direct relationship to String at all.

// replace implicitly with
conformance[G, String]

<console>:13: error: type arguments [G,String] do not conform
              to method conformance's type parameter bounds [A,B >: A]
         conformance[G, String]
                    ^

// or with
conformance[String, G]

<console>:13: error: type arguments [String,G] do not conform
              to method conformance's type parameter bounds [A,B >: A]
         conformance[String, G]
                    ^

Let’s apply the pigeonhole principle. Imagine that you had a list of every class that ever was or ever will be. Imagine that, somehow, all of these classes, from String to AbstractFactoryMethodProxyBuilder, were on your classpath, available to your program.

Next, imagine that you had the time and inclination to try the =:= test with every last one of these classes.

implicitly[G =:= javax.swing.JFrame]
implicitly[G =:= AbstractFactoryMethodProxyBuilder]
// ad [in]finitum

Your search will be futile; every class on your list-of-every-class will give the same compiler error we got with String.

So, since G is not equal to anything on this list, it must be something else that doesn’t appear on the list. Because this list contains all classes, G must be something other than a class.

It must not necessarily be anything

It seems like it might be convenient to say “well, in this program G is only ever String by substitution, so therefore it is, even if the compiler doesn’t see that.” However, thinking like this misses out on the second key advantage of type parameterization, the one not based on multiplicity of substitution, or the type-safety of callers: blindness.

The implementation of type-parameterized classes and methods are required to treat each type parameter uniquely, uniformly, and without prejudice. The compiler enforces this by making the implementation blind to what that parameter, like G, could be. It can only use what the caller, the “outside”, has told it about G—arguments whose types contain G, like List[G], (G, G) => G, or G itself, like the argument to pickGreeting. This is information-hiding at the type level; if you find information-hiding a useful tool for implementing correct programs, you will find the same of the fresh, unique, and mysterious types induced by each introduction of a type parameter.

How many calls are there?

Put another way, when implementing the code in the scope of a type parameter, your implementation must be equally valid for all possible G substitutions, including the ones that haven’t been invented yet. This is why we call it universal quantification.

But it is not merely each declaration of a type parameter that yields a distinct type—each call does! Consider two consecutive calls to pickGreeting.

pickGreeting(greeting, 42)
pickGreeting(33, 84)

Externally, there are two G types. However, the possibility of writing this demands another level of uniqueness treatment when typechecking pickGreeting’s definition: whatever G is now, like String, it might be something else in the next call, like Int in the above example. With recursion, it might even be two different things at the same time. There’s nothing to hold this at two, either: there may be an unbounded number of substitutions for a given type parameter within a single program, at a single point in time.

While G may be the same between two invocations of pickGreeting, it might not. So we have no choice but to treat the G types of each call as separate types. There may be infinitely many calls, so there are so many types.

Incidentally, the same happens for singleton types. Each time val greeting comes into scope, it induces a separate singleton type. It is easy enough to arrange for an unbounded number of scope entries in a particular program. This isn’t so practical as the type parameter phenomenon, though.

More types from variable copies

Suppose we’d like to wait a while to compute our greeting. We can define a type-and-class to represent that conveniently.

// like Coyoneda Id, if that helps
sealed abstract class Later[A] {
  type I
  val i: I
  val f: I => A
}

def later[In, A](now: In)(later: In => A)
  : Later[A] = new Later[A] {
    type I = In
    val i = now
    val f = later
}

val greeting3 = later(3){
  n => List.fill(n)("hi").mkString(" ")
}

How many classes are involved here, in the type of greeting3?

1. Later, obviously;
2. Function1, the greeting3.f overall class;
3. String, the output type of greeting3.f;
4. Int, the I type.

How many types?

The first difference is that greeting3.I is not Int.

scala> implicitly[greeting3.I =:= Int]
<console>:14: error: Cannot prove that greeting3.I =:= Int.
       implicitly[greeting3.I =:= Int]
                 ^

They are unrelated for much the same reason as G was unrelated to String in the previous example: the only things code following val greeting3 may know are those embodied in the greeting3.i and greeting3.f members. You can almost think of them as “arguments”.

But that’s not all.

val salut3 = greeting3

scala> greeting3.f(greeting3.i)
res11: String = hi hi hi

scala> salut3.f(salut3.i)
res12: String = hi hi hi

scala> greeting3.f(salut3.i)
<console>:14: error: type mismatch;
 found   : salut3.i.type (with underlying type salut3.I)
 required: greeting3.I
       greeting3.f(salut3.i)
                          ^

scala> implicitly[greeting3.I =:= salut3.I]
<console>:14: error: Cannot prove that greeting3.I =:= salut3.I.

Just like every call to pickGreeting induces a new G type, each simple val copy of greeting3 will induce a new, unique I type. It doesn’t matter that they’re all the same value; this is a matter of variables, not values, just as with singleton types.

But that’s still not all.

One value with class, many variable references with types

The preceding is more delicate than it seems.

var allo = greeting3

scala> allo.f(allo.i)
<console>:13: error: type mismatch;
 found   : Later[String]#I
 required: _1.I where val _1: Later[String]
       allo.f(allo.i)
                   ^

All we have done differently is use a mutable var instead of an immutable val. Why is this enough to throw a wrench in the works?

Suppose you had another value of the Later[String] type.

val bhello = later("olleh")(_.reverse)

The I substitution here is String. So the f takes a String argument, and the I is a String.

bhello is of a compatible type with the allo var. So this assignment will work.

allo = bhello

In a sense, when this mutation occurs, the I type also mutates, from Int to String. But that isn’t quite right; types cannot mutate.

Suppose that this assignment happened in the middle of that line of code that could not compile. We could imagine the sequence of events, were it permitted.

1. allo.f (which is greeting3.f) evaluates. It is the function (n: Int) => List.fill(n)("hi").mkString(" ").
2. The allo = bhello assignment occurs.
3. allo.i (which is bhello.i) evaluates. It is the string "olleh".
4. We attempt to pass "olleh" as the (n: Int) argument to complete the evaluation, and get stuck.

Just as it makes no difference what concrete substitutions you make for G, it makes no difference whether such an assignment could ever happen in your specific program; the compiler takes it as a possibility because you declared a var. (def allo = greeting3 gets the same treatment, lest you think non-functional programs get to have all the fun here.) Each reference to allo gets a new I type member. That failing line of code had two allo references, so was working with two incompatible I types.

Since the number of references to a variable in a program is also unbounded…you get the picture.

How do we tell the two apart?

All of this is simply to say that we must be working with two separate concepts here.

1. The runtime shape and properties of the values that end up flying around when a program actually runs. This we call class.
2. The compile-time, statically-discoverable shape and properties of the expressions that fly around when a program is written. This we call type.

The case with var is revealing. Maybe the I type will always be the same for a given mutable variable. But demonstrating that this holds true for one run of the program (#1, class) isn’t nearly good enough to prove that it will be true for all runs of the program (#2, type).

We refuse to apply the term “type” to the #1, ‘class’ concept because it does not live up to the name. The statement “these two types are the same” is another level of power entirely; “these two values have the same class” is extraordinarily weak by comparison.

It is tempting to use the term “runtime type” to refer to classes. However, in the case of Scala, as with all type systems featuring parametric polymorphism, classes are so dissimilar to types that the similar-sounding term leads to false intuition, not helpful analogy. It is a detriment to learning, not an aid.

Types are compile-time, and classes are runtime.

When are types real?

The phase separation—compile-time versus runtime—is the key to the strength of types in Scala and similar type systems. The static nature of types means that the truths they represent must be universally quantified—true in all possible cases, not just some test cases.

We need this strength because the phase separation forbids us from taking into account anything that cannot be known about the program without running it. We need to think in terms of “could happen”, not “pretty sure it doesn’t”.

How do classes give rise to types?

There appears to be some overlap between the classes of greeting and its types. While greeting has the class String, it also has the type String.

We want types to represent static truths about the expressions in a program. That’s why it makes sense to include a “model of the classes” in the type system. When we define a class, we also define an associated type or family of types.

When we use a class to construct a value, as in new Blob, we would like to assign as much specific meaning to that expression as we can at compile time. So, because we know right now that this expression will make a value of class Blob, we assign it the type Blob too.

How do the types disappear?

There’s a common way to throw away type information in Scala, especially popular in object-oriented style.

val absGreeting: CharSequence = greeting

absGreeting has the same value as greeting, so it has the same five classes. However, it only has two of those five types, because we threw away the other three statically. It has lost some other types, too, namely greeting.type, and acquired some new ones, namely absGreeting.type.

Once a value is constructed, the expression will naturally cast off the types specifying its precise identity, as it moves into more abstract contexts. Ironically, the best way to preserve that information as it passes through abstract contexts is to take advantage of purely abstract types—type parameters and type members.

scala> pickGreeting[greeting.type](greeting, 100)
res16: greeting.type = hi there!

While the implementation must treat its argument as being of the abstract type G, the caller knows that the more specific greeting.type must come out of that process.

How do the types come back?

There is a feature in Scala that lets you use class to get back some type information via a dynamic, runtime test.

absGreeting match {
  case hiAgain: String =>
    conformance[hiAgain.type, String] // will compile
}

The name “type test” for this feature is poorly chosen. The conclusion affects the type level—hiAgain is, indeed, proven statically to be of type String—but the test occurs only at the class level.

The compiler will tell you about this limitation sometimes.

def pickGreeting2[G](grt: G, rand: Int): G =
  ("magic": Any) match {
    case ok: G => ok
    case _ => sys error "failed!"
  }

<console>:13: warning: abstract type pattern G is unchecked
              since it is eliminated by erasure
           case ok: G => ok
                    ^

But reflecting the runtime classes back to compile-time types is a subtle art, and the compiler often can’t explain exactly what you got wrong.

def pickGreeting3[G](grt: G, rand: Int): G =
  grt match {
    case _: String =>
      "Surely type G is String, right?"
    case _ => grt
  }

<console>:14: error: type mismatch;
 found   : String("Surely type G is String, right?")
 required: G
             "Surely type G is String, right?"
             ^

I’ve touched upon this mistake in previous articles, but it’s worth taking at least one more look. Let’s examine how tempting this mistake is.

String is a final class. So it is true that G can contain no more specific class than String, if the first case matches. For example, given trait MyAwesomeMixin, G cannot be String with MyAwesomeMixin if this case succeeds, because that can’t be instantiated; you would need to create a subclass of String that implemented MyAwesomeMixin.

This pattern match isn’t enough evidence to say that G is exactly String. There are still other class-based types it could be, like Serializable.

pickGreeting3[java.io.Serializable](greeting, 4055)

Instead, it feels like this pattern match confirms Serializable as a possibility, instead of denying it.

But we don’t need G = String for this code to compile; we only need G >: String. If that was true, then "Surely type G is String, right?", a String, could simply upcast to G.

However, even G >: String is unproven. There are no subclasses of String, but there are infinitely many subtypes of String. Including the G created by each entry into pickGreeting3, every abstract and existential type bounded by String, and every singleton type of String variable definitions.

This mistake is, once again, confusing a demonstration of one case with a proof. Pattern matching tells us a great deal about one value, the grt argument, but very little about the type G. All we know for sure is that “grt is of type G, and also of type String, so these types overlap by at least one value.” In the type system, if you don’t know something for sure, you don’t know it at all.

Classes are a concrete source of values

In the parlance of functional Scala, concrete classes are often called “data constructors”.

When you are creating a value, you must ultimately be concrete about its class, at the bottom of all the abstractions and indirections used to hide this potentially messy detail.

scala> def pickGreeting4[G]: G = new G
<console>:12: error: class type required but G found
       def pickGreeting4[G]: G = new G
                                     ^

You’ll have to do something else here, like take an argument () => G, to let pickGreeting4 construct Gs.

The truly essential role that classes play is that they encapsulate instructions for constructing concrete values of various types. In a safe program, this is the only feature of classes you’ll use.

In Scala, classes leave fingerprints on the values that they construct, without fail. This is merely an auxiliary part of their primary function as value factories, like a “Made in class Blah” sticker on the back. Pattern matching’s “type tests” work by checking this fingerprint of construction.

Most runtime “type test” mechanisms do not work for types

These fingerprints only come from classes, not types. So “type tests” only work for “classy” types, like String and MyAwesomeMixin. They also work for specific singleton types because construction also leaves an “object identity” fingerprint that the test can use.

The ClassTag typeclass does not change this restriction. When you add a ClassTag or TypeTag context bound, you also prevent that type parameter from working with most types.

scala> implicitly[reflect.ClassTag[greeting3.I]]
<console>:14: error: No ClassTag available for greeting3.I
       implicitly[reflect.ClassTag[greeting3.I]]
                 ^

As such, judicious use of ClassTag is not a great solution to excessive use of type tests in abstract contexts. There are so many more types than classes that this is to confine the expressivity of your types to a very small, class-reflective box. Set them free!

“But doesn’t Python/JavaScript/&c have both types and classes at runtime?”

In JavaScript, there’s a very general runtime classification of values called “type”, meant to classify built-in categories like string, number, and the like.

>> typeof "hi"
"string"
>> typeof 42
"number"
>>> typeof [1, 'a']
"object"

Defining a class with the new class keyword doesn’t extend this partition with new “types”; instead, it further subdivides one of those with a separate classification.

>> class Foo() {}
>> class Bar() {}
>> typeof (new Foo)
"object"
>> typeof (new Bar)
"object"
>> new Foo().constructor
function Foo()
>> new Bar().constructor
function Bar()

So, if you treat JavaScript’s definition of the word “type” as analogous to the usage in this article, then yes, JavaScript has “runtime types”.

But JavaScript can only conveniently get away with this because its static types are uninteresting. It has one type—the type of all values—and no opportunities to do interesting type-level modeling, at least not as part of the standard language.

Hence, JavaScript is free to repurpose the word “type” for a flavor of its classes, because our “types” aren’t a tool you make much use of in JavaScript. But when you come back to Scala, Haskell, the ML family, et al, you need a word for the static concept once again.

Thinking about types as just classes leads to incorrect conclusions

Setting aside the goal of principled definition of terms, this separation is the one that makes the most sense for a practitioner of Scala. Consider the practicalities:

Types and classes have different behavior, are equal and unequal according to different rules, and there are a lot more types than classes. So we need different words to distinguish them.

Saying “compile-time type” or “runtime type” is not a practical solution—no one wants to speak such an unwieldy qualifier every time they refer to such a commonly-used concept.

While I’ve given a sampling of the richness of the type system in this article, it’s not necessary to know that full richness to appreciate or remember the difference between the two: types are static and compile-time; classes are dynamic and runtime.

This article was tested with Scala 2.12.1, Shapeless 2.3.2, and Firefox 53.0a2.

Licensing

Unless otherwise noted, all content is licensed under a Creative Commons Attribution 3.0 Unported License.

The mixin style of importing in which classes and traits are defined within traits, as seen in scala.reflect.Universe, ScalaTest, and other Scala styles, seems to be infectious. By that, I mean once you define something in a trait to be mixed in, to produce another reusable module that calls that thing, you must define another trait, and so must anyone using your module, and so on and so forth. You effectively become “trapped in the cake”.

However, we can use type parameters that represent singleton types to write functions that are polymorphic over these “cakes”, without being defined as part of them or mixed in themselves. For example, you can use this to write functions that operate on elements of a reflection universe, without necessarily passing that universe around all over the place.

Well, for the most part. Let’s see how far this goes.

Our little universe

Let’s set aside the heavyweight real-world examples I mentioned above in favor of a small example. Then, we should be able to explore the possibilities in this simpler space.

final case class LittleUniverse() {
  val haystack: Haystack = Haystack()

  final case class Haystack() {
    def init: Needle = Needle()
    def iter(n: Needle): Needle = n
  }

  final case class Needle()
}

For brevity, I’ve defined member classes, but this article equally applies if you are using abstract types instead, as any Functional programmer of pure, virtuous heart ought to!

Suppose we have a universe.

scala> val lu: LittleUniverse = LittleUniverse()
lu: LittleUniverse = LittleUniverse()

The thing that Scala does for us is not let Haystacks and Needle s from one universe be confused with those from another.

val anotherU = LittleUniverse()

scala> lu.haystack.iter(anotherU.haystack.init)
<console>:14: error: type mismatch;
 found   : anotherU.Needle
 required: lu.Needle
       lu.haystack.iter(anotherU.haystack.init)
                                          ^

The meaning of this error is “you can’t use one universe’s Haystack to iter a Needle from another universe”.

This doesn’t look very important given the above code, but it’s a real boon to more complex scenarios. You can set up a lot of interdependent abstract invariants, verify them all, and have the whole set represented with the “index” formed by the singleton type, here lu.type or anotherU.type.

Working with a universe on hand

Refactoring in macro-writing style seems to be based upon passing the universe around everywhere. We can do that.

def twoInits(u: LittleUniverse): (u.Needle, u.Needle) =
  (u.haystack.init, u.haystack.init)

def stepTwice(u: LittleUniverse)(n: u.Needle): u.Needle =
  u.haystack.iter(u.haystack.iter(n))

The most important feature we’re reaching for with these fancy dependent method types, and the one that we have to keep reaching for if we want to write sane functions outside the cake, is preserving the singleton type index.

scala> twoInits(lu)
res3: (lu.Needle, lu.Needle) = (Needle(),Needle())

scala> stepTwice(anotherU)(anotherU.haystack.init)
res4: anotherU.Needle = Needle()

These values are ready for continued itering, or whatever else you’ve come up with, in the confines of their respective universes. That’s because they’ve “remembered” where they came from.

By contrast, consider a simple replacement of the path-dependencies with a type projection.

def brokenTwoInits(u: LittleUniverse)
    : (LittleUniverse#Needle, LittleUniverse#Needle) =
  (u.haystack.init, u.haystack.init)

scala> val bti = brokenTwoInits(lu)
bti: (LittleUniverse#Needle, LittleUniverse#Needle) = (Needle(),Needle())

That seems to be okay, until it’s time to actually use the result.

scala> lu.haystack.iter(bti._1)
<console>:14: error: type mismatch;
 found   : LittleUniverse#Needle
 required: lu.Needle
       lu.haystack.iter(bti._1)
                            ^

The return type of brokenTwoInits “forgot” the index, lu.type.

Getting two needles without a universe

When we pass a LittleUniverse to the above functions, we’re also kind of passing in a constraint on the singleton type created by the argument variable. That’s how we know that the returned u.Needle is a perfectly acceptable lu.Needle in the caller scope, when we pass lu as the universe.

However, as the contents of a universe become more complex, there are many more interactions that need not involve a universe at all, at least not directly.

def twoInitsFromAHaystack[U <: LittleUniverse](
    h: U#Haystack): (U#Needle, U#Needle) =
  (h.init, h.init)

scala> val tifah = twoInitsFromAHaystack[lu.type](lu.haystack)
tifah: (lu.Needle, lu.Needle) = (Needle(),Needle())

Since we didn’t pass in lu, how did it know that the returned Needles were lu.Needles?

1. The type of lu.haystack is lu.Haystack.
2. That type is shorthand for lu.type#Haystack.
3. We passed in U = lu.type, and our argument meets the resulting requirement for a lu.type#Haystack (after expanding U).
4. The type of the expression h.init is u.Needle forSome {val u: U}. We use an existential because the relevant variable (and its singleton type) is not in scope.
5. This type widens to U#Needle, satisfying the expected return type.

This seems like a more complicated way of doing things, but it’s very freeing: by not being forced to necessarily pass the universe around everywhere, you’ve managed to escape the cake’s clutches much more thoroughly. You can also write syntax enrichments on various members of the universe that don’t need to talk about the universe’s value, just its singleton type.

Unless, you know, the index appears in contravariant position.

Syntactic stepTwice

One test of how well we’ve managed to escape the cake is to be able to write enrichments that deal with the universe. This is a little tricky, but quite doable if you have the universe’s value.

With the advent of implicit class, this became a little easier to do wrongly, but it’s a good start.

implicit class NonWorkingStepTwice(val u: LittleUniverse) {
  def stepTwiceOops(n: u.Needle): u.Needle =
    u.haystack.iter(u.haystack.iter(n))
}

That compiles okay, but seemingly can’t actually be used!

scala> lu stepTwiceOops lu.haystack.init
<console>:15: error: type mismatch;
 found   : lu.Needle
 required: _1.u.Needle where val _1: NonWorkingStepTwice
       lu stepTwiceOops lu.haystack.init
                                    ^

There’s a hint in that we had to write val u, not u, nor private val u, in order for the implicit class itself to compile. This signature tells us that there’s an argument of type LittleUniverse, and a member u: LittleUniverse. However, whereas with the function examples above, we [and the compiler] could trust that they’re one and the same, we have no such guarantee here. So we don’t know that an lu.Needle is a u.Needle. We didn’t get far enough, but we don’t know that a u.Needle is an lu.Needle, either.

Relatable variables

Instead, we have to expand a little bit, and take advantage of a very interesting, if obscure, element of the type equivalence rules in the Scala language.

class WorkingStepTwice[U <: LittleUniverse](val u: U) {
  def stepTwice(n: u.Needle): u.Needle =
    u.haystack.iter(u.haystack.iter(n))
}

implicit def WorkingStepTwice[U <: LittleUniverse](u: U)
    : WorkingStepTwice[u.type] =
  new WorkingStepTwice(u)

Unfortunately, the ritual of expanding the implicit class shorthand is absolutely necessary; the implicit class won’t generate the dependent-method-typed implicit conversion we need.

Now we can get the proof we need.

scala> lu stepTwice lu.haystack.init
res7: _1.u.Needle forSome { val _1: WorkingStepTwice[lu.type] } = Needle()

// that's a little weird, but reduces to what we need
scala> res7: lu.Needle
res8: lu.Needle = Needle()

How does this work?

1. Implicitly convert lu, giving us a conv: WorkingStepTwice[lu.type].
2. This means that conv.u: lu.type, by expansion of U.
3. This in turn means that conv.u.type <: lu.type.

The next part is worth taking in two parts. It may be worth having §3.5.2 “Conformance” of the language spec open for reference. First, let’s consider the return type (a covariant position), which is simpler.

1. The return type expands to conv.u.type#Needle.
2. The ninth conformance bullet point tells us that the left side of a # projection is covariant, so because conv.u.type <: lu.type (see above), the return type widens to lu.type#Needle.
3. For this, lu.Needle is a shorthand.

It was far longer until I realized how the argument type works. You’ll want to scroll up on the SLS a bit, to the “Equivalence” section. Keep in mind that we are trying to widen lu.Needle to conv.u.Needle, which is the reverse of what we did for the return type.

1. Our argument’s type expands to lu.type#Needle.
2. The second bullet point under “Equivalence” says that “If a path p has a singleton type q.type, then p.type ≡ q.type.” From this, we can derive that conv.u.type = lu.type. This is a stronger conclusion than we reached above!
3. We substitute the left side of the # using the equivalence, giving us conv.u.type#Needle.

I cannot characterize this feature of the type system as anything other than “really freaky” when you first encounter it. It seems like an odd corner case. Normally, when you write val x: T, then x.type is a strict subtype of T, and you can count on that, but this carves out an exception to that rule. It is sound, though, and an absolutely essential feature!

val sameLu: lu.type = lu

scala> sameLu.haystack.iter(lu.haystack.init)
res9: sameLu.Needle = Needle()

Without this rule, even though we have given it the most specific type possible, sameLu couldn’t be a true substitute for lu in all scenarios. That means that in order to make use of singleton type indices, we would be forever beholden to the variable we initially stored the value in. I think this would be extremely inconvenient, structurally, in almost all useful programs.

With the rule in place, we can fully relate the lu and conv.u variables, to let us reorganize how we talk about universes and values indexed by their singleton types in many ways.

A pointless argument

Let’s try to hide the universe. We don’t need it, after all. We can’t refer to u in the method signature anymore, so let’s try the same conversion we used with twoInitsFromAHaystack. We already have the U type parameter, after all.

class CleanerStepTwice[U <: LittleUniverse](private val u: U) {
  def stepTwiceLively(n: U#Needle): U#Needle =
    ???
}

implicit def CleanerStepTwice[U <: LittleUniverse](u: U)
    : CleanerStepTwice[u.type] =
  new CleanerStepTwice(u)

This has the proper signature, and it’s cleaner, since we don’t expose the unused-at-runtime u variable anymore. We could refine a little further, and replace it with a U#Haystack, just as with twoInitsFromAHaystack.

This gives us the same interface, with all the index preservation we need. Even better, it infers a nicer return type.

scala> def trial = lu stepTwiceLively lu.haystack.init
trial: lu.Needle

Now, let’s turn to implementation.

class OnceMoreStepTwice[U <: LittleUniverse](u: U) {
  def stepTwiceFinally(n: U#Needle): U#Needle =
    u.haystack.iter(u.haystack.iter(n))
}

<console>:18: error: type mismatch;
 found   : U#Needle
 required: OnceMoreStepTwice.this.u.Needle
           u.haystack.iter(u.haystack.iter(n))
                                           ^

This is the last part of the escape! If this worked, we could fully erase the LittleUniverse from most code, relying on the pure type-level index to prove enough of its existence! So it’s a little frustrating that it doesn’t quite work.

Let’s break it down. First, the return type is fine.

1. Since u: U, u.type <: U. (This is true, and useful, in the scope of u, which is now invisible to the caller.)
2. iter returns a u.type#Needle.
   • Note: since u is not in scope for the caller, if we returned this as is, it would effectively widen to the existentially bound u.type#Needle forSome {val u: U}. But the same logic in the next step would apply to that type.
3. By the # left side covariance, u.type#Needle widens to U#Needle.

Pretty simple, by the standards of what we’ve seen so far.

Contravariance is the root of all…

But things break down when we try to call iter(n). Keep in mind that n: U#Needle and the expected type is u.Needle. Specifically: since we don’t know in the implementation that U is a singleton type, we can’t use the “singleton type equivalence” rule on it! But suppose that we could; that is, suppose that we could constrain U to be a singleton type.

1. The argument type is U#Needle.
2. By singleton equivalence, since u: U and u is stable, so u.type = U.
3. By substituting the left-hand side of the #, we get u.type#Needle.
4. This shortens to u.Needle.

If we are unable to constrain U in this way, though, we are restricted to places where U occurs in covariant position when using cake-extracted APIs. We can invoke functions like init, because they only have the singleton index occurring in covariant position.

Invoking functions like iter, where the index occurs in contravariant or invariant position, requires being able to add this constraint, so that we can use singleton equivalence directly on the type variable U. This is quite a bit trickier.

Extracting more types

We have the same problem with the function version.

def stepTwiceHaystack[U <: LittleUniverse](
    h: U#Haystack, n: U#Needle): U#Needle =
  h.iter(h.iter(n))

<console>:18: error: type mismatch;
 found   : U#Needle
 required: _1.Needle where val _1: U
         h.iter(h.iter(n))
                       ^

Let’s walk through it one more time.

1. n: U#Needle.
2. h.iter expects a u.type#Needle for all val u: U.
3. Suppose that we constrain U to be a singleton type:
   1. [The existential] u.type = U, by singleton equivalence.
   2. By # left side equivalence, h.iter expects a U#Needle.

The existential variable complicates things, but the rule is sound.

As a workaround, it is commonly suggested to extract the member types in question into separate type variables. This works in some cases, but let’s see how it goes in this one.

def stepTwiceExUnim[N, U <: LittleUniverse{type Needle = N}](
    h: U#Haystack, n: N): N = ???

This looks a lot weirder, but should be able to return the right type.

scala> def trial2 = stepTwiceExUnim[lu.Needle, lu.type](lu.haystack, lu.haystack.init)
trial2: lu.Needle

But this situation is complex enough for the technique to not work.

def stepTwiceEx[N, U <: LittleUniverse{type Needle = N}](
    h: U#Haystack, n: N): N =
  h.iter(h.iter(n))

<console>:18: error: type mismatch;
 found   : N
 required: _1.Needle where val _1: U
         h.iter(h.iter(n))
                       ^

Instead, we need to index Haystack directly with the Needle type, that is, add a type parameter to Haystack so that its Needle arguments can be talked about completely independently of the LittleUniverse, and then to write h: U#Haystack[N] above. Essentially, this means that any time a type talks about another type in a Universe, you need another type parameter to redeclare a little bit of the relationships between types in the universe.

The problem with this is that we already declared those relationships by declaring the universe! All of the non-redundant information is represented in the singleton type index. So even where the above type-refinement technique works (and it does in many cases), it’s still redeclaring things that ought to be derivable from the “mere” fact that U is a singleton type.

The fact that it’s a singleton type

(The following is based on enlightening commentary by Daniel Urban on an earlier draft.)

Let’s examine the underlying error in stepTwiceEx more directly.

scala> def fetchIter[U <: LittleUniverse](
    h: U#Haystack): U#Needle => U#Needle = h.iter
<console>:14: error: type mismatch;
 found   : _1.type(in method fetchIter)#Needle
             where type _1.type(in method fetchIter) <: U with Singleton
 required: _1.type(in value $anonfun)#Needle
             where type _1.type(in value $anonfun) <: U with Singleton
           h: U#Haystack): U#Needle => U#Needle = h.iter
                                                    ^

It’s a good thing that this doesn’t compile. If it did, we could do

fetchIter[LittleUniverse](lu.haystack)(anotherU.haystack.init)

Which is unsound.

§3.2.1 “Singleton Types” of the specification mentions this Singleton, which is in a way related to singleton types.

A stable type is either a singleton type or a type which is declared to be a subtype of trait scala.Singleton.

Adding with Singleton to the upper bound on U causes fetchIter to compile! This is sound, because we are protected from the above problem with the original fetchIter.

def fetchIter[U <: LittleUniverse with Singleton](
    h: U#Haystack): U#Needle => U#Needle = h.iter

scala> fetchIter[lu.type](lu.haystack)
res3: lu.Needle => lu.Needle = $$Lambda$1397/1159581520@683e7892

scala> fetchIter[LittleUniverse](lu.haystack)
<console>:16: error: type arguments [LittleUniverse] do not conform
                     to method fetchIter's type parameter bounds
                     [U <: LittleUniverse with Singleton]
       fetchIter[LittleUniverse](lu.haystack)
                ^

Let’s walk through the logic for fetchIter. The expression h.iter has type u.Needle => u.Needle for some val u: U, and our goal type is U#Needle => U#Needle. So we have two subgoals: prove u.Needle <: U#Needle for the covariant position (after =>), and U#Needle <: u.Needle for the contravariant position (before =>).

First, covariant:

1. Since u: U, u.type <: U.
2. Since the left side of # is covariant, #1 implies u.type#Needle <: U#Needle.
3. This re-sugars to u.Needle <: U#Needle, which is the goal.

Secondly, contravariant. We’re going to have to make a best guess here, because it’s not entirely clear to me what’s going on.

1. Since [existential] path u has a singleton type U (if we define “has a singleton type” as “having a type X such that X <: Singleton”), so u.type = U by the singleton equivalence.
2. Since equivalence implies conformance, according to the first bullet under “Conformance”, #1 implies U <: u.type.
3. Since the left side of # is covariant, #2 implies that U#Needle <: u.type#Needle.
4. This resugars to U#Needle <: u.Needle, which is the goal.

I don’t quite understand this, because U doesn’t seem to meet the requirements for “singleton type”, according to the definition of singleton types. However, I’m fairly sure it’s sound, since type stability seems to be the property that lets us avoid the universe-mixing unsoundness. Unfortunately, it only seems to work with existential vals; we seem to be out of luck with vals that the compiler can still see.

// works fine!
def stepTwiceSingly[U <: LittleUniverse with Singleton](
    h: U#Haystack, n: U#Needle): U#Needle = {
  h.iter(h.iter(n))
}

// but alas, this form doesn't
class StepTwiceSingly[U <: LittleUniverse with Singleton](u: U) {
  def stepTwiceSingly(n: U#Needle): U#Needle =
    u.haystack.iter(u.haystack.iter(n))
}

<console>:15: error: type mismatch;
 found   : U#Needle
 required: StepTwiceSingly.this.u.Needle
           u.haystack.iter(u.haystack.iter(n))
                                            ^

We can work around this by having the second form invoke the first with the Haystack, thus “existentializing” the universe. I imagine that most, albeit not all, cakes can successfully follow this strategy.

So, finally, we’re almost out of the cake.

1. Escape covariant positions with universe variable: complete.
2. Escape contravariant/invariant positions with universe variable: complete.
3. Escape covariant positions with universe singleton type: complete!
4. Escape contravariant/invariant positions with universe singleton type: 90% there!

This article was tested with Scala 2.12.1.

Licensing

Unless otherwise noted, all content is licensed under a Creative Commons Attribution 3.0 Unported License.

This is a guest post by Tomas Mikula. It was initially published as a document in the hasheq. It has been slightly edited and is being republished here with the permission of the original author.

This article describes what we mean when we say that the data structures in this library are equivalence-aware in a type-safe fashion.

Equivalence

Set is a data structure that doesn’t contain duplicate elements. An implementation of Set must therefore have a way to compare elements for “sameness”. A useful notion of sameness is equivalence, i.e. a binary relation that is reflexive, symmetric and transitive. Any reasonable implementation of Set is equipped with some equivalence relation on its element type.

Here’s the catch: For any type with more than one inhabitant there are multiple valid equivalence relations. We cannot (in general) pick one that is suitable in all contexts. For example, are these two binary trees same?

  +            +
 / \          / \
1   +        +   3
   / \      / \
  2   3    1   2

It depends on the context. They clearly have different structure, but they are both binary search trees containing the same elements. For a balancing algorithm, they are different trees, but as an implementation of Set, they represent the same set of integers.

Equality

Despite the non-uniqueness, there is one equivalence relation that stands out: equality. Two objects are considered equal when they are indistinguishable to an observer. Formally, equality is required to have the substitution property:

\[ \forall a,b \in A, \forall f \in (A \to B): a=_A b \implies f(a)=_B f(b) \]

(Here, $=_A$ denotes equality on $A$, $=_B$ denotes equality on $B$.)

Equality is the finest equivalence: whenever two elements are equal, they are necessarily equivalent with respect to every equivalence.

Choices in libraries

Popular Scala libraries take one of these two approaches when dealing with comparing elements for “sameness”.

The current approach of cats is equality. Instances of the cats.Eq[A] typeclass are required to have all the properties of equality, including the substitution property above. The problem with this approach is that for some types, such as Set[Int], equality is too strict to be useful: Are values Set(1, 2) and Set(2, 1) equal? For that to be true, they have to be indistinguishable by any function. Let’s try (_.toList):

scala> Set(1, 2).toList == Set(2, 1).toList
res0: Boolean = false

So, Set(1, 2) and Set(2, 1) are clearly not equal. As a result, we cannot use Set[Int] in a context where equality is required (without cheating).

On the other hand, scalaz uses unspecified equivalence. Although the name scalaz.Equal[A] might suggest equality, instances of this typeclass are only tested for properties of equivalence. As mentioned above, there are multiple valid equivalence relations for virtually any type. When there are also multiple useful equivalences for a type, we are at risk of mixing them up (and the fact that they are usually resolved as implicit arguments only makes things worse).

Equivalence-aware sets (a.k.a. setoids)

Let’s look at how we deal with this issue. We define typeclass Equiv with an extra type parameter that serves as a “tag” identifying the meaning of the equivalence.

trait Equiv[A, Eq] {
  def equiv(a: A, b: A): Boolean
}
// defined trait Equiv

For the compiler, the “tag” is an opaque type. It only has specific meaning for humans. The only meaning it has for the compiler is that different tags represent (intensionally) different equivalence relations.

An equivalence-aware data structure then carries in its type the tag of the equivalence it uses.

import hasheq._
// import hasheq._

import hasheq.immutable._
// import hasheq.immutable._

import hasheq.std.int._
// import hasheq.std.int._

scala> HashSet(1, 2, 3, 4, 5)
res0: hasheq.immutable.HashSet[Int] = HashSetoid(5, 1, 2, 3, 4)

What on earth is HashSetoid? A setoid is an equivalence-aware set. HashSetoid is then just a setoid implementated using hash-table. Let’s look at the definition of HashSet:

type HashSet[A] = HashSetoid[A, Equality.type]

So HashSet is just a HashSetoid whose equivalence is equality. To create an instance of HashSet[Int] above, we needed to have an implicit instance of Equiv[Int, Equality.type] in scope.

implicitly[Equiv[Int, Equality.type]]

For the compiler, Equality is just a rather arbitrary singleton object. It only has the meaning of mathematical equality for us, humans.

There is a convenient type alias provided for equality relation:

type Equal[A] = Equiv[A, Equality.type]

implicitly[Equal[Int]]

So how do we deal with the problem of set equality mentioned above, i.e. that HashSet(1, 2) and HashSet(2, 1) are not truly equal? We just don’t provide a definition of equality for HashSet[Int].

scala> implicitly[Equal[HashSet[Int]]]
<console>:22: error: could not find implicit value for parameter e: hasheq.Equal[hasheq.immutable.HashSet[Int]]
       implicitly[Equal[HashSet[Int]]]
                 ^

But that means we cannot have a HashSet[HashSet[Int]]! (Remember, for a HashSet[A], we need an instance of Equal[A], and we just showed we don’t have an instance of Equal[HashSet[Int]].)

scala> HashSet(HashSet(1, 2, 3, 4, 5))
<console>:22: error: could not find implicit value for parameter A: hasheq.Hash[hasheq.immutable.HashSet[Int]]
       HashSet(HashSet(1, 2, 3, 4, 5))
              ^

But we can have a HashSetoid[HashSet[Int], E], where E is some equivalence on HashSet[Int].

scala> HashSet.of(HashSet(1, 2, 3, 4, 5))
res5: hasheq.immutable.HashSetoid[hasheq.immutable.HashSet[Int],hasheq.immutable.Setoid.ContentEquiv[Int,hasheq.Equality.type]] = HashSetoid(HashSetoid(5, 1, 2, 3, 4))

HashSet.of(elems) is like HashSet(elems), except it tries to infer the equivalence on the element type, instead of requiring it to be equality.

Notice the equivalence tag: Setoid.ContentEquiv[Int, Equality.type]. Its meaning is (again, for humans only) that two setoids are equivalent when they contain the same elements (here, of type Int), as compared by the given equivalence of elements (here, Equality).

The remaining question is: How does this work in the presence of multiple useful equivalences?

Let’s define another equivalence on Int (in addition to the provided equality).

// Our "tag" for equivalence modulo 10.
// This trait will never be instantiated.
sealed trait Mod10

// Provide equivalence tagged by Mod10.
implicit object EqMod10 extends Equiv[Int, Mod10] {
  def mod10(i: Int): Int = {
    val r = i % 10
    if (r < 0) r + 10
    else r
  }
  def equiv(a: Int, b: Int): Boolean = mod10(a) == mod10(b)
}

// Provide hash function compatible with equivalence modulo 10.
// Note that the HashEq typeclass is also tagged by Mod10.
implicit object HashMod10 extends HashEq[Int, Mod10] {
  def hash(a: Int): Int = EqMod10.mod10(a)
}

Now let’s create a “setoid of sets of integers”, as before.

scala> HashSet.of(HashSet(1, 2, 3, 4, 5))
res13: hasheq.immutable.HashSetoid[hasheq.immutable.HashSet[Int],hasheq.immutable.Setoid.ContentEquiv[Int,hasheq.Equality.type]] = HashSetoid(HashSetoid(5, 1, 2, 3, 4))

This still works, because HashSet requires an equality on Int, and there is only one in the implicit scope (the newly defined equivalence EqMod10 is not equality). Let’s try to create a “setoid of setoids of integers”:

scala> HashSet.of(HashSet.of(1, 2, 3, 4, 5))
<console>:24: error: ambiguous implicit values:
 both method hashInstance in object int of type => hasheq.Hash[Int]
 and object HashMod10 of type HashMod10.type
 match expected type hasheq.HashEq[Int,Eq]
       HashSet.of(HashSet.of(1, 2, 3, 4, 5))
                            ^

This fails, because there are now more equivalences on Int in scope. (There are now also multiple hash functions, which is what the error message actually says.) We need to be more specific:

scala> HashSet.of(HashSet.of[Int, Mod10](1, 2, 3, 4, 5))
res15: hasheq.immutable.HashSetoid[hasheq.immutable.HashSetoid[Int,Mod10],hasheq.immutable.Setoid.ContentEquiv[Int,Mod10]] = HashSetoid(HashSetoid(5, 1, 2, 3, 4))

Finally, does it prevent mixing up equivalences? Let’s see:

scala> val s1 = HashSet(1,  2,  3,         11, 12, 13    )
s1: hasheq.immutable.HashSet[Int] = HashSetoid(1, 13, 2, 12, 3, 11)

scala> val s2 = HashSet(    2,  3,  4,  5,         13, 14)
s2: hasheq.immutable.HashSet[Int] = HashSetoid(5, 14, 13, 2, 3, 4)

scala> val t1 = HashSet.of[Int, Mod10](1,  2,  3,         11, 12, 13    )
t1: hasheq.immutable.HashSetoid[Int,Mod10] = HashSetoid(1, 2, 3)

scala> val t2 = HashSet.of[Int, Mod10](    2,  3,  4,  5,         13, 14)
t2: hasheq.immutable.HashSetoid[Int,Mod10] = HashSetoid(5, 2, 3, 4)

Combining compatible setoids:

scala> s1 union s2
res16: hasheq.immutable.HashSetoid[Int,hasheq.Equality.type] = HashSetoid(5, 14, 1, 13, 2, 12, 3, 11, 4)

scala> t1 union t2
res17: hasheq.immutable.HashSetoid[Int,Mod10] = HashSetoid(5, 1, 2, 3, 4)

Combining incompatible setoids:

scala> s1 union t2
<console>:26: error: type mismatch;
 found   : hasheq.immutable.HashSetoid[Int,Mod10]
 required: hasheq.immutable.HashSetoid[Int,hasheq.Equality.type]
       s1 union t2
                ^

scala> t1 union s2
<console>:26: error: type mismatch;
 found   : hasheq.immutable.HashSet[Int]
    (which expands to)  hasheq.immutable.HashSetoid[Int,hasheq.Equality.type]
 required: hasheq.immutable.HashSetoid[Int,Mod10]
       t1 union s2
                ^

Conclusion

We went one step further in the direction of type-safe equivalence in Scala compared to what is typically seen out in the wild today. There is nothing very sophisticated about this encoding. I think the major win is that we can design APIs so that the extra type parameter (the “equivalence tag”) stays unnoticed by the user of the API as long as they only deal with equalities. As soon as the equivalence tag starts requesting our attention (via an ambiguous implicit or a type error), it is likely that the attention is justified.

This article was tested with Scala 2.11.8 and hasheq version 0.3.

Licensing

Unless otherwise noted, all content is licensed under a Creative Commons Attribution 3.0 Unported License.

Haskell is a pure language. Every Haskell expression is referentially transparent, meaning that you can substitute that expression with its evaluated result without changing the program. Or, put into code:

-- this program
f expr expr -- apply function f to arguments expr, expr

-- is equivalent to this one, which factors out `expr`
let
  x = expr -- introduce a new variable `x` with the value of `expr`
in
  f x x

And this is true for all expressions e, and all functions f. These could be complex expressions which describe ways of manipulating network channels or window buffers, or something trivial like a numeric literal. You can always substitute the expression with its value.

This is not true in Scala, simply because Scala allows unrestricted side-effects. Unlike Haskell, Scala puts no limitations on where and when we can use things like mutable state (vars) or evaluated external effects like println or launchTheMissiles. Since there are no restrictions on where and when we can do evil, the Scala equivalent to the above just doesn’t work:

f(e, e)
// isn't really equivalent to!
val x = e
f(x, x)

The reason it isn’t equivalent comes from the different sorts of expressions that we could find in e. For example, what if e is println("hi!"). If we make that substitution, our snippet looks like the following:

f(println("hi"), println("hi"))
// isn't really equivalent to!
val x = println("hi")
f(x, x)

Clearly these are not the same two programs. The first prints "hi" twice, while the second only prints it once. This is a violation of referential transparency, and it’s why we sometimes say that Scala is an impure language. Any expression which is not referentially transparent must contain side-effects, by definition.

Now of course, we found this problem by using a side-effecting function: namely, println. Haskell clearly has the ability to print to standard output, so how does it avoid this issue? If we build the same program in Haskell, can we violate referential transparency?

f (putStrLn "hi") (putStrLn "hi")
-- is equivalent to
let x = putStrLn "hi" in f x x

As it turns out, this is still referentially transparent! These two programs still have the same meaning. This is possible only because neither program actually prints anything!

In Haskell, effects are treated as first-class values. The putStrLn function doesn’t print to standard out, it returns a value (of type IO ()) which describes how to print to standard out, but stops short of actually doing it. These sorts of values can be composed using the monadic operators (in Scala, flatMap and pure), allowing Haskell programmers to build up expressions composed of sequences of dependent effects, all of which are merely descriptions of the side-effects which will eventually be performed by the runtime. Ultimately, the description which comprises your whole program is the return result from the main function. The Haskell runtime runs the main function to get this description of all your effects, and then runs the effects per your instructions.

This is kind of a clever trick. It allows Haskell to simultaneously be pure and still have excellent support for manipulating effects and interacting with the “real world”. But why is it relevant to Scala? After all, Scala is an impure language. We don’t need to go through this complex rigmarole of describing our effects and composing those descriptions; the language lets us just do it! So why wouldn’t we just, you know, evaluate the effects that we need evaluated?

The answer is that we want to reason about where and when our effects are evaluated. And of course, we want to be able to leverage laws and abstractions which assume equational semantics for expressions (i.e. referential transparency). Cats is full of these sorts of abstractions, and cats-laws provides a vast set of laws which describe them. But all of these abstractions and all of these laws break down the moment you introduce some sort of side-effecting expression. Because, much like our referential transparency example from earlier, these abstractions assume that you can substitute expressions with their evaluated results, and that’s just not true in the presence of side-effects.

What we need is a data type which allows us to encapsulate Scala-style side-effects in the form of a pure value, on which referential transparency holds and which we can compose using other well-defined abstractions, such as Monad. Scalaz defines two such data types which meet these criteria: scalaz.effect.IO and scalaz.concurrent.Task. But in practice, nearly everyone uses Task instead of IO because of its support for asynchronous effects.

Cats does not define any such abstraction, and what’s worse is the cats ecosystem also doesn’t really provide any such abstraction. There are two Task implementations that are relatively commonly used with cats – namely, monix.eval.Task and fs2.Task – but these are not part of cats per se, nor are they deeply integrated into its abstraction hierarchy. Additionally, the proliferation of broadly equivalent options has led to confusion in the ecosystem, with middleware authors often forced to choose a solution for their end-users, and end-users uncertain as to which choice is “right”.

Introducing cats-effect

The cats-effect project aims to change all of that. The goal of cats-effect is to provide an “easy default” IO type for the cats ecosystem, deeply integrated with cats-core, with all of the features and performance that are required for real world production use. Additionally, cats-effect defines a set of abstractions in the form of several typeclasses which describe what it means to be a pure effect type. These abstractions are extremely useful both in enabling MTL-style program composition and to ensure that other pre-existing Task implementations remain first-class citizens of the ecosystem. IO does not overshadow monix.eval.Task or fs2.Task; it complements them by providing a set of abstractions and laws which allow users to write safe, parametric code which supports each of them equally.

One important sidebar here: cats-effect does not provide any concurrency primitives. scalaz.concurrent.Task and monix.eval.Task are both notable for providing functions such as both, which takes two Tasks and runs them in parallel, returning a Task of a tuple of the results. The cats.effect.IO type does not provide any such function, and while it would be possible to define such a function (and others like it!), we strongly encourage users to instead consider full-on streaming frameworks such as fs2 or Monix for their concurrency needs, as these frameworks are able to provide a much sounder foundation for such functions. See here for a rough outline of why this is. Also note that some Task implementations, such as Monix’s, can and do provide parallelism on a sound foundation by enriching their internal algebraic structures. Thus, monix.eval.Task is actually quite different from cats.effect.IO, despite having a similar core set of operations.

Enough Talk…

What does this look like in practice? Well, ideally, as convenient as possible! Let’s look at our println example:

def putStrLn(line: String): IO[Unit] =
  IO { println(line) }

f(putStrLn("hi!"), putStrLn("hi!"))

// is equivalent to

val x = putStrLn("hi!")
f(x, x)

Great! We can write Haskell fanfic in Scala. 😛

The notable element here is the use of the IO.apply constructor to wrap the println effect in a pure IO value. This pattern can be applied to any side-effect. You can think of this sort of like an FFI that converts impure code (like println) into pure code (like putStrLn). The goal of this API was to be as simple and straightforward as possible. If you have a curly brace block of impure side-effecting code, you can wrap it in a composable and pure abstraction by just adding two characters: IO. You can wrap arbitrarily large or small blocks of code, potentially involving complex allocations, JNI calls, resource semantics, etc; but it is generally considered a best practice to wrap side-effects into the smallest composable units that make sense and do all of your sequentialization using flatMap and for-comprehensions.

For example, here’s a program that performs some simple user interaction in the shell:

import cats.effect.IO

val program = for {
  _ <- IO { println("Welcome to Scala!  What's your name?") }
  name <- IO { Console.readLine }
  _ <- IO { println(s"Well hello, $name!") }
} yield ()

We could have just as easily written this program in the following way:

val program = IO {
  println("Welcome to Scala!  What's your name?")
  val name = Console.readLine
  println(s"Well hello, $name!")
}

But this gives us less flexibility for composition. Remember that even though program is a pure and referentially transparent value, its definition is not, which is to say that IO { expr } is not the same as val x = expr; IO { x }. Anything inside the IO { … } block is not referentially transparent, and so should be treated with extreme care and suspicion. The less of our program we have inside these blocks, the better!

As a sidebar that is actually kinda cool, we can implement a readString IO action that wraps Console.readLine as a val!

val readString = IO { Console.readLine }

This is totally valid! We don’t need to worry about the difference between def and val anymore, because IO is referentially transparent. So you use def when you need parameters, and you use val when you don’t, and you don’t have to think about evaluation semantics. No more subtle bugs caused by accidentally memoizing your effects!

Of course, if program is referentially transparent, then clearly repeated values of program cannot possibly run the effects it represents multiple times. For example:

program
program
program

// must be the same as!

program

If this weren’t the case, then we would be in trouble when trying to construct examples like the Haskell one from earlier. But there is an implication here that is quite profound: IO cannot eagerly evaluate its effects, and similarly cannot memoize its results! If IO were to eagerly evaluate or to memoize, then we could no longer replace references to the expression with the expression itself, since that would result in a different IO instance to be evaluated separately.

This is precisely why scala.concurrent.Future is not a suitable type for encapsulating effects in this way: constructing a Future that will eventually side-effect is itself a side-effect! Future evaluates eagerly (sort of, see below) and memoizes its results, meaning that a println inside of a given Future will only evaluate once, even if the Future is sequenced multiple times. This in turn means that val x = Future(...); f(x, x) is not the same program as f(Future(...), Future(...)), which is the very definition of a violation of referential transparency.

Coming back to IO… If program does not evaluate eagerly, then clearly there must be some mechanism for asking it to evaluate. After all, Scala is not like Haskell: we don’t return a value of type IO[Unit] from our main function. IO provides an FFI of sorts for wrapping side-effecting code into pure IO values, so it must also provide an FFI for going in the opposite direction: taking a pure IO value and evaluating its constituent actions as side-effects.

program.unsafeRunSync()    // uh oh!

This function is called unsafeRunSync(). Given an IO[A], the unsafeRunSync() function will give you a value of type A. You should only call this function once, ideally at the very end of your program! (i.e. in your main function) Just as with IO.apply, any expression involving unsafeRunSync() is not referentially transparent. For example:

program.unsafeRunSync()
program.unsafeRunSync()

The above will run program twice. So clearly, referential transparency is out the window whenever we do this, and we cannot expect the normal laws and abstractions to remain sound in the presence of this function.

A sidebar on Future’s eager evaluation

As Viktor Klang is fond of pointing out, Future doesn’t need to evaluate eagerly. It is possible to define an ExecutionContext in which Future defers its evaluation until some indefinitely later point. However, this is not the default mode of operation for 99% of all Futures ever constructed; most people just use ExecutionContext.global and leave it at that. Additionally, if someone hands me an arbitrary Future, perhaps as a return value from a function, I really have no idea whether or not that Future is secretly running without my consent. In other words, the referential transparency (or lack thereof) of functions that I write using Future is dependent on the runtime configuration of some other function which is hidden from me. That’s not referential transparency anymore. Because we cannot be certain that Future is deferring its evaluation, we must defensively assume that it is not.

This, in a nutshell, is precisely why Future is not appropriate for functional programming. IO provides a pair of functions (fromFuture and unsafeToFuture) for interacting with Future-using APIs, but in general, you should try to stick with IO as much as possible when manipulating effects.

Asynchrony and the JVM

Scala runs on three platforms: the JVM, JavaScript and LLVM. For the moment, we’ll just focus on the first two. The JVM has support for multiple threads, but those threads are native (i.e. kernel) threads, meaning that they are relatively expensive to create and maintain in the runtime. They are a very limited resource, sort of like file handles or heap space, and you can’t just write programs which require an unbounded number of them. The exact upper bound on the JVM varies from platform to platform, and varies considerably depending on your GC configuration, but a general rule of thumb is “a few thousand”, where “few” is a small number. In practice, you’re going to want far less threads than that if you want to avoid thrashing your GC, and most applications will divide themselves into a bounded “main” thread pool (usually bounded to exactly the number of CPUs) on which all CPU-bound tasks are performed and most of the program runs, as well as a set of unbounded “blocking” thread pools on which blocking IO actions (such as anything in java.io) are run. When you add NIO worker pools into the mix, the final number of threads in a practical production service is usually around 30-40 on an 8 CPU machine, growing roughly linearly as you add CPUs. Clearly, this is not a very large number.

On JavaScript runtimes (such as node or in the browser), the situation is even worse: you have exactly one thread! JavaScript simply doesn’t have multi-threading in any (real) form, and so it’s like the JVM situation, but 30-40x more constraining.

For this reason, we need to be very careful when writing Scala to treat threads as an extremely scarce resource. Blocking threads (using mechanisms such as wait, join or CountDownLatch) should be considered absolutely anathema, since it selfishly wastes a very finite and very critical resource, leading to thread starvation and deadlocks.

This is very different from how things are in Haskell though! The Haskell runtime is implemented around the concept of green threads, which is to say, emulated concurrency by means of a runtime dispatch lock. Haskell basically creates a global bounded thread pool in the runtime with the same number of threads as your machine has CPUs. On top of that pool, it runs dispatch trampolines that schedule and evict expression evaluation, effectively emulating an arbitrarily large number of “fake” threads atop a small fixed set of “real” threads. So when you write code in Haskell, you generally just assume that threads are extremely cheap and you can have as many of them as you want. Under these circumstances, blocking a thread is not really a big deal (as long as you don’t do it in FFI native code), so there’s no reason to go out of your way to avoid it in abstractions like IO.

This presents a bit of a dilemma for cats-effect: we want to provide a practical pure abstraction for encapsulating effects, but we need to run on the JVM and on JavaScript which means we need to provide a way to avoid thread blocking. So, the IO implementation in cats-effect is going to necessarily end up looking very, very different from the one in Haskell, providing a very different set of operations.

Specifically, cats.effect.IO provides an additional constructor, async, which allows the construction of IO instances from callback-driven APIs. This is generally referred to as “asynchronous” control flow, as opposed to “synchronous” control flow (represented by the apply constructor). To see how this works, we’re going to need a bit of setup.

Consider the following somewhat-realistic NIO API (translated to Scala):

trait Response[T] {
  def onError(t: Throwable): Unit
  def onSuccess(t: T): Unit
}
// defined trait Response

trait Channel {
  def sendBytes(chunk: Array[Byte], handler: Response[Unit]): Unit
  def receiveBytes(handler: Response[Array[Byte]]): Unit
}
// defined trait Channel

This is an asynchronous API. Neither of the functions sendBytes or receiveBytes attempt to block on completion. Instead, they schedule their operations via some underlying mechanism. This interface could be implemented on top of java.io (which is a synchronous API) through the use of an internal thread pool, but most NIO implementations are actually going to delegate their scheduling all the way down to the kernel layer, avoiding the consumption of a precious thread while waiting for the underlying IO – which, in the case of network sockets, may be a very long wait indeed!

Wrapping this sort of API in a referentially transparent and uniform fashion is a very important feature of IO, precisely because of Scala’s underlying platform constraints. Clearly, sendBytes and receiveBytes both represent side-effects, but they’re different than println and readLine in that they don’t produce their results in a sequentially returned value. Instead, they take a callback, Response, which will eventually be notified (likely on some other thread!) when the result is available. The IO.async constructor is designed for precisely these situations:

def send(c: Channel, chunk: Array[Byte]): IO[Unit] = {
  IO async { cb =>
    c.sendBytes(chunk, new Response[Unit] {
      def onError(t: Throwable) = cb(Left(t))
      def onSuccess(v: Unit) = cb(Right(()))
    })
  }
}
// send: (c: Channel, chunk: Array[Byte])cats.effect.IO[Unit]

def receive(c: Channel): IO[Array[Byte]] = {
  IO async { cb =>
    c.receiveBytes(new Response[Array[Byte]] {
      def onError(t: Throwable) = cb(Left(t))
      def onSuccess(chunk: Array[Byte]) = cb(Right(chunk))
    })
  }
}
// receive: (c: Channel)cats.effect.IO[Array[Byte]]

Obviously, this is a little more daunting than the println examples from earlier, but that’s mostly the fault of the anonymous inner class syntactic ceremony. The IO interaction is actually quite simple!

The async constructor takes a function which is handed a callback (represented above by cb in both cases). This callback is itself a function of type Either[Throwable, A] => Unit, where A is the type produced by the IO. So when our Response comes back as onSuccess in the send example, we invoke the callback with a Right(()) since we’re trying to produce an IO[Unit]. When the Response comes back as onSuccess in the receive example, we invoke the callback with Right(chunk), since the IO produces an Array[Byte].

Now remember, IO is still a monad, and IO values constructed with async are perfectly capable of all of the things that “normal”, synchronous IO values are, which means that you can use these values inside for-comprehensions and other conventional composition! This is incredibly, unbelievably nice in practice, because it takes your complex, nested, callback-driven code and flattens it into simple, easy-to-read sequential composition. For example:

val c: Channel = null // pretend this is an actual channel

for {
  _ <- send(c, "SYN".getBytes)
  response <- receive(c)

  _ <- if (response == "ACK".getBytes)   // pretend == works on Array[Byte]
    IO { println("found the guy!") }
  else
    IO { println("no idea what happened, but it wasn't good") }
} yield ()

This is kind of amazing. There’s no thread blocking at all in the above (other than the println blocking on standard output). The receive could take quite a long time to come back to us, and our thread is free to do other things in the interim. Everything is driven by callbacks under the surface, and asynchronous actions can be manipulated just as easily as synchronous ones.

Of course, this is an even bigger win on JavaScript, where nearly everything is callback-based, and gigantic, deeply nested chunks of code are not unusual. IO allows you to flatten those deeply nested chunks of code into a nice, clean, linear and sequential formulation.

Thread Shifting

Now there is a caveat here. When our Response handler is invoked by Channel, it is very likely that the callback will be run on a thread which is part of a different thread pool than our main program. Remember from earlier where I described how most well-designed Java services are organized:

• A bounded thread pool set to num CPUs in size for any non-IO actions
• A set of unbounded thread pools for blocking IO
• Some bounded internal thread worker pools for NIO polling

We definitely want to run nearly everything on that first pool (which is probably ExecutionContext.global), but we’re probably going to receive the Response callback on one of the third pools. So how can we force the rest of our program (including those printlns) back onto the main pool?

The answer is the shift function.

import scala.concurrent._
implicit val ec = ExecutionContext.global

for {
  _ <- send(c, "SYN".getBytes)
  response <- receive(c).shift    // there's no place like home!

  _ <- if (response == "ACK".getBytes)   // pretend == works on Array[Byte]
    IO { println("found the guy!") }
  else
    IO { println("no idea what happened, but it wasn't good") }
} yield ()

shift’s functionality is a little complicated, but generally speaking, you should think of it as a “force this IO onto this other thread pool” function. Of course, when receive executes, most of its work isn’t done on any thread at all (since it is simply registering a hook with the kernel), and so that work isn’t thread shifted to any pool, main or otherwise. But when receive gets back to us with the network response, the callback will be handled and then immediately thread-shifted back onto the main pool, which is passed implicitly as a parameter to shift (you can also pass this explicitly if you like). This thread-shifting means that all of the subsequent actions within the for-comprehension – which is to say, the continuation of receive(c) – will be run on the ec thread pool, rather than whatever worker pool is used internally by Channel. This is an extremely common use-case in practice, and IO attempts to make it as straightforward as possible.

Another possible application of thread shifting is ensuring that a blocking IO action is relocated from the main, CPU-bound thread pool onto one of the pools designated for blocking IO. An example of this would be any interaction with java.io:

import java.io.{BufferedReader, FileReader}
// import java.io.{BufferedReader, FileReader}

def readLines(name: String): IO[Vector[String]] = IO {
  val reader = new BufferedReader(new FileReader(name))
  var back: Vector[String] = Vector.empty

  try {
    var line: String = null
    do {
      line = reader.readLine()
      back :+ line
    } while (line != null)
  } finally {
    reader.close()
  }

  back
}
// readLines: (name: String)cats.effect.IO[Vector[String]]

for {
  _ <- IO { println("Name, pls.") }
  name <- IO { Console.readLine }
  lines <- readLines("names.txt")

  _ <- if (lines.contains(name))
    IO { println("You're on the list, boss.") }
  else
    IO { println("Get outa here!") }
} yield ()

Clearly, readLines is blocking the underlying thread while it waits for the disk to return the file contents to us, and for a large file, we might be blocking the thread for quite a long time! Now if we’re treating our thread pools with respect (as described above), then we probably have a pair of ExecutionContext(s) sitting around in our code somewhere:

import java.util.concurrent.Executors

implicit val Main = ExecutionContext.global
val BlockingFileIO = ExecutionContext.fromExecutor(Executors.newCachedThreadPool())

We want to ensure that readLines runs on the BlockingFileIO pool, while everything else in the for-comprehension runs on Main. How can we achieve this?

With shift!

for {
  _ <- IO { println("Name, pls.") }
  name <- IO { Console.readLine }
  lines <- readLines("names.txt").shift(BlockingFileIO).shift(Main)

  _ <- if (lines.contains(name))
    IO { println("You're on the list, boss.") }
  else
    IO { println("Get outa here!") }
} yield ()

Now we’re definitely in bizarro land. Two calls to shift, one after the other? Let’s break this apart:

readLines("names.txt").shift(BlockingFileIO)

One of the functions of shift is to take the IO action it is given and relocate that action onto the given thread pool. In the case of receive, this component of shift was meaningless since receive didn’t use a thread under the surface (it was asynchronous!). However, readLines does use a thread under the surface (hint: it was constructed with IO.apply rather than IO.async), and so that work will be relocated onto the BlockingFileIO pool by the above expression.

Additionally, the continuation of this work will also be relocated onto the BlockingFileIO pool, and that’s definitely not what we want. The evaluation of the contains function is definitely CPU-bound, and should be run on the Main pool. So we need to shift a second time, but only the continuation of the readLines action, not readLines itself. As it turns out, we can achieve this just by adding the second shift call:

readLines("names.txt").shift(BlockingFileIO).shift(Main)

Now, readLines will be run on the BlockingFileIO pool, but the continuation of readLines (namely, everything that follows it in the for-comprehension) will be run on Main. This works because shift creates an asynchronous IO that schedules the target action on the given thread pool and invokes its continuation from a callback. The ExecutionContext#execute function should give you an idea of how this works. This means that the result of the first shift is an IO constructed with async, and cannot itself be thread-shifted (unlike an IO constructed with apply), but its continuation can be thread-shifted, which is exactly what happens.

This sort of double-shift idiom is very common in production service code that makes use of legacy blocking IO libraries such as java.io.

Synchronous vs Asynchronous Execution

Speaking of asynchrony, readers who have been looking ahead in the class syllabus probably realized that the type signature of unsafeRunSync() is more than a little suspicious. Specifically, it promises to give us an A immediately given an IO[A]; but if that IO[A] is an asynchronous action invoked with a callback, how can it achieve this promise?

The answer is that it blocks a thread. (gasp!!!) Under the surface, a CountDownLatch is used to block the calling thread whenever an IO is encountered that was constructed with IO.async. Functionally, this is very similar to the Await.result function in scala.concurrent, and it is just as dangerous. Additionally, it clearly cannot possibly work on JavaScript, since you only have one thread to block! If you try to call unsafeRunSync() on JavaScript with an underlying IO.async, it will just throw an exception rather than deadlock your application.

This is not such a great state of affairs. I mean, it works if unsafeRunSync() is being run in test code, or as the last line of your main function, but sometimes we need to interact with legacy code or with Java APIs that weren’t designed for purity. Sometimes, we just have to evaluate our IO actions before “the end of the world”, and when we do that, we don’t want to block any of our precious threads.

So IO provides an additional function: unsafeRunAsync. This function takes a callback (of type Either[Throwable, A] => Unit) which it will run when (and if) the IO[A] completes its execution. As the name implies, this function is also not referentially transparent, but unlike unsafeRunSync(), it will not block a thread.

As a sidebar that will be important in a few paragraphs, IO also defines a safe function called runAsync which has a very similar signature to unsafeRunAsync, except it returns an IO[Unit]. The IO[Unit] which is returned from this function will not block if you call unsafeRunAsync(). In other words, it is always safe to call unsafeRunSync() on the results of runAsync, even on JavaScript.

Another way to look at this is in terms of unsafeRunAsync. You can define unsafeRunAsync in terms of runAsync and unsafeRunSync():

def unsafeRunAsync[A](ioa: IO[A])(cb: Either[Throwable, A] => Unit): Unit =
  ioa.runAsync(e => IO { cb(e) }).unsafeRunSync()
// unsafeRunAsync: [A](ioa: cats.effect.IO[A])(cb: Either[Throwable,A] => Unit)Unit

This isn’t the actual definition, but it would be a valid one, and it would run correctly on every platform.

Abstraction and Lawfulness

As mentioned earlier (about 10000 words ago…), the cats-effect project not only provides a concrete IO type with a lot of nice features, it also provides a set of abstractions characterized by typeclasses and associated laws. These abstractions collectively define what it means to be a type which encapsulates side-effects in a pure fashion, and they are implemented by IO as well as several other types (including fs2.Task and monix.eval.Task). The hierarchy looks like this:

Monad and MonadError are of course a part of cats-core, while everything else is in cats-effect. MonadError is functionally equivalent to the familiar scalaz.Catchable typeclass, which was commonly used in conjunction with scalaz.concurrent.Task. It literally means “a monad with error-handling capabilities”. IO certainly fits that description, as any exceptions thrown within its apply method (or within async) will be caught and may be handled in pure code by means of the attempt function. Sync, Async, LiftIO and Effect are the new typeclasses.

Sync simply describes the IO.apply function (in the typeclasses, this function is called delay). Which is to say, any type constructor F[_] which has a Sync[F] has the capability to suspend synchronous side-effecting code. Async is very similar to this in that it describes the async function. So any type constructor F[_] which has an Async[F] can suspend asynchronous side-effecting code. LiftIO should be familiar to Haskell veterans, and is broadly useful for defining parametric signatures and composing monad transformer stacks.

Effect is where everything is brought together. In addition to being able to suspend synchronous and asynchronous side-effecting code, anything that has an Effect instance may also be asynchronously interpreted into an IO. The way this is specified is using the runAsync function:

import cats.effect.{Async, LiftIO, Sync}

trait Effect[F[_]] extends Sync[F] with Async[F] with LiftIO[F] {
  def runAsync[A](fa: F[A])(cb: Either[Throwable, A] => IO[Unit]): IO[Unit]
}

What this is saying is that any Effect must define the ability to evaluate as a side-effect, but of course, we don’t want to have side-effects in our pure and reasonable code. So how are side-effects purely represented? With IO!

From a parametric reasoning standpoint, IO means “here be effects”, and so any type signature which involves IO thus also involves side-effects (well, effects anyway), and any type signature which requires side-effects must also involve IO. This bit of trickery allows us to reason about Effect in a way that would have been much harder if we had defined unsafeRunAsync as a member, and it ensures that downstream projects which write code abstracting over Effect types can do so without using any unsafe functions if they so choose (especially when taken together with the liftIO function).

Conclusion

The lack of a production-ready Task-like type fully integrated into the cats ecosystem has been a sticking point for a lot of people considering adopting cats. With the introduction of cats-effect, this should no longer be a problem! As of right now, the only releases are snapshots with hash-based versions, the latest of which can be found in the maven badge at the top of the readme. These snapshots are stable versions (in the repeatable-build sense), but they should not be considered stable, production-ready, future-proof software. We are quickly moving towards a final 0.1 release, which will depend on cats-core and will represent the stable, finalized API.

Once cats releases a final 1.0 version, cats-effect will also release version 1.0 which will depend on the corresponding version of cats-core. Changes to cats-effect are expected to be extremely rare, and thus the dependency should be considered quite stable for the purposes of upstream compatibility. Nevertheless, the release and versioning cycle is decoupled from cats-core to account for the possibility that breaking changes may need to be made independent of the cats-core release cycle.

Check out the sources! Check out the documentation. Play around with the snapshots, and let us know what you think! Now is the time to make your opinion heard. If IO in its current form doesn’t meet your needs, we want to hear about it!

Licensing

Unless otherwise noted, all content is licensed under a Creative Commons Attribution 3.0 Unported License.

Dimensional analysis

When we code, we code in numbers - doubles, floats and ints. Those numbers always represent real world quantities.

For example, the number of people in a room can be represented as an integer, as can the number of chairs. Adding people and chairs together gives a nonsensical result, but dividing the number of people by the number of chairs gives a useful indicator of how full up the room is.

val numberOfPeople = 9
val numberOfChairs = 10
numberOfPeople + numberOfChairs // this is a bug
numberOfPeople.toDouble / numberOfChairs.toDouble // this is useful

This is actually a form of dimensional analysis. We’re mentally assigning the dimension Person to the quantity of people, and Chair to the quantity of chairs. Dimensional analysis can be summarized in two laws.

1. Quantities can only be added or subtracted to quantities of the same dimension
2. Quantities of different dimensions can be multiplied or divided

Why is it important?

Ignoring the laws can result in serious problems. Take the Mars Climate Orbiter, a $200 million space probe which successfully reached Mars after a year long voyage, but suddenly crashed into the Martian atmosphere on arrival. Most components on the orbiter were using metric units, however a single component was sending instructions in Imperial units. The other components did not detect this, and instead began a sudden descent causing the orbiter to burn up. This was a simple unit conversion error! It was a basic mistake that could have been easily avoided. It should have been picked up during testing, or in the runtime validation layer.

In fact, it could even have been caught at compile time.

Compile time dimensional analysis

We’re going to use a similar problem to demonstrate compile time dimensional analysis. To fit with the theme of rocket physics, we will tackle a rocket launch towards the distant constellation of Libra. We’ll begin by working through our calculation in doubles before adding compile time safety with dependent types and finally supporting compile time dimensional analysis with typeclass induction.

Destination: Alpha Librae

The star that we’re aiming for is Alpha Librae. This is pretty far, so we can only send one very small person. We have been given the following quantities to work with:

• rocket mass of a small person - 40kg
• fuel mass of a lot of fuel - 10⁴kg
• exhaust speed of a decent fuel - 10⁶ms^-1
• distance to Alpha Librae - 77 ly

We want to calculate when the rocket will arrive.

To do so, we’re going to make use of a formula known as the Ideal Rocket Equation. This calculates the speed of a rocket in ideal conditions.

val rocketSpeed = exhaustSpeed * log((rocketMass + fuelMass) / rocketMass)

Once we have the speed, we can work out the travel time.

val time = distance / rocketSpeed

Plugging the numbers in

Let’s do what we’re used to doing and use doubles:

val rocketSpeed = 1000000.0 * log((40.0 + 10000.0) / 40.0)
// rocketSpeed: Double = 5525452.939131783

val time = 77.0 / rocketSpeed
// time: Double = 1.39355091515989E-5

Fantastic! We can get to Libra in less than a day!

Unfortunately, this time estimate is too far off to be valid. We can’t get to Libra that quickly at light speed, let alone rocket speed. We’ve clearly made a mistake somewhere. Instead of pouring over our code to find out where that is, let’s try and use the compiler.

Using types

We can add some type safety to this problem by using a case class to represent each quantity.

case class Quantity[A](value: Double)

A represents the quantity dimension. So given the following dimensions:

type Kilogram
type Metre
type Second
type MetresPerSecond
type C
type LightYear
type Year

We can create quantities:

val rocketMass = Quantity[Kilogram](40.0)
val fuelMass = Quantity[Kilogram](10000.0)
val exhaust = Quantity[MetresPerSecond](1000000.0)
val distance = Quantity[LightYear](77.0)

It’s important to note that these are types, not classes. We never instantiate a MetresPerSecond - we’re just using it to differentiate between Quantity[MetresPerSecond] and Quantity[Year] at the type level.

So how does this change the code?

val rocketSpeed = Quantity[MetresPerSecond](exhaust.value * log((rocketMass.value + fuelMass.value) / rocketMass.value))
// rocketSpeed: Quantity[Types.MetresPerSecond] = Quantity(5525452.939131783)

val time = Quantity[Year](distance.value / rocketSpeed.value)
// time: Quantity[Types.Year] = Quantity(1.39355091515989E-5)

In short, it doesn’t. The code might be clearer, but we don’t know what the bug is. This is because the compiler isn’t doing anything with the types we’ve added.

Operating on quantities

We can encode our first law of addition at compile time by creating a function to add quantities:

def add[A](q0: Quantity[A], q1: Quantity[A]): Quantity[A] = Quantity(q0.value + q1.value)

This ensures that quantities can only be added to other quantities of the same type. Trying to add quantities of different types will result in a compilation error.

A quantity can also be multiplied by a dimensionless scalar value to give a quantity of the same dimension.

def times[A](q: Quantity[A], v: Double): Quantity[A] = Quantity(q.value * v)

It would be great if we could divide quantities too. Writing a divide function is more difficult:

def divide[A, B, Magic](q0: Quantity[A], q1: Quantity[B]): Quantity[Magic] =
  Quantity[Magic](q0.value / q1.value)

There’s a clear problem with trying to do this. When we divide a quantity by another, we don’t know what the Magic output type should be. The output type is dependent on what the input types are (for example, dividing Metre by Second should give MetresPerSecond). The compiler needs a way of working out what the output is, provided that it knows the input types.

Dependent types

What we actually want is a dependent type. A division operation should occur at the type level, taking two input types and supplying a dependent output type. We can create the trait Divide with a dependent output type:

trait Divide[A, B] {
  type Out
}

We also need to define an Aux type alias. This is known as the Aux pattern and makes it easier to refer to all three types at once.

object Divide {
  type Aux[A, B, Out0] = Divide[A, B] { type Out = Out0 }
}

We can create instances of this divide typeclass with different output types, so the output type is dependent on the value of the divide typeclass instance.

When dividing, the compiler looks for this implicit typeclass instance and returns a quantity corresponding to the output type.

def divide[A, B](q0: Quantity[A], q1: Quantity[B])(implicit d: Divide[A, B]): Quantity[d.Out] =
  Quantity[d.Out](q0.value / q1.value)

So given that we want to divide A by B, the compiler will look for a value of Divide[A, B] and find the Out type of it. If no instance exists, the code doesn’t compile.

We’ll need some more types to represent the result of a division:

type LightYearSecondsPerMetre
type MetresPerSecondPerC
type Dimensionless

And we’ll need to write instances for all combinations of dimensions.

implicit val kgDivideKg: Divide.Aux[Kilogram, Kilogram, Dimensionless] =
  new Divide[Kilogram, Kilogram] { type Out = Dimensionless }

implicit val lyDivideC: Divide.Aux[LightYear, C, Year] =
  new Divide[LightYear, C] { type Out = Year }

implicit val lyDivideMps: Divide.Aux[LightYear, MetresPerSecond, LightYearSecondsPerMetre] =
  new Divide[LightYear, MetresPerSecond] { type Out = LightYearSecondsPerMetre }

implicit val mpsDivideC: Divide.Aux[MetresPerSecond, C, MetresPerSecondPerC] =
  new Divide[MetresPerSecond, C] { type Out = MetresPerSecondPerC }

implicit val mpsDivideMpsPerC: Divide.Aux[MetresPerSecond, MetresPerSecondPerC, C] =
  new Divide[MetresPerSecond, MetresPerSecondPerC] { type Out = C }

And so on.

Unfortunately, there are an infinite number of combinations, so there are an infinite number of instances. Nevertheless, let’s plough on with the ones we’ve written. We can modify our rocket equation to use add, times and divide:

val rocketSpeed = times(exhaust, log(divide(add(rocketMass, fuelMass), rocketMass).value))
// rocketSpeed: Quantity[Types.MetresPerSecond] = Quantity(5525452.939131783)

val time: Quantity[Year] = divide(distance, rocketSpeed)
// <console>:31: error: type mismatch;
//  found   : Quantity[lyDivideMps.Out]
//     (which expands to)  Quantity[MoreTypes.LightYearSecondsPerMetre]
//  required: Quantity[Types.Year]
//        val time: Quantity[Year] = divide(distance, rocketSpeed)
//                                         ^

Great! We’ve caught our bug! The result was in LightYearSecondsPerMetre, not Year. We made a unit conversion error, just like the Mars orbiter.

We can now fix this by adding a conversion:

val metresPerSecondPerC: Quantity[MetresPerSecondPerC] = divide(Quantity[MetresPerSecond](300000000.0), Quantity[C](1.0))
// metresPerSecondPerC: Quantity[MoreTypes.MetresPerSecondPerC] = Quantity(3.0E8)

val speedInC = divide(rocketSpeed, metresPerSecondPerC)
// speedInC: Quantity[mpsDivideMpsPerC.Out] = Quantity(0.018418176463772612)

val time: Quantity[Year] = divide(distance, speedInC)
// time: Quantity[Types.Year] = Quantity(4180.65274547967)

It seems like it’s going to take a lot longer than we hoped to get to Libra. Perhaps it’s unwise to send a person.

Automatic derivation

We found the bug, but we needed to explicitly write out typeclass instances for every combination of dimensions. This might have worked for our small problem, but it just doesn’t scale in the long run. We need to figure out a way of deriving the typeclass instances automatically. To attempt this, we first need to generalize what a combination of dimensions actually is.

Representing dimensions

We can represent a combination of dimensions as a heterogeneous list (HList) of base dimensions. HLists are defined in shapeless, a cornerstone of most functional libraries, and can be thought of as a type level list.

type LightYearSeconds = LightYear :: Second :: HNil

This is good for multiples of dimensions, such as LightYearSeconds, but doesn’t represent combinations created from division, such as MetresPerSecond. To do this, we need some way of representing integer exponents as types. We can represent integers as types using Singleton types. We actually need these singleton types in type position. This is supported by a new feature present in Typelevel Scala, called literal types:

scalaOrganization := "org.typelevel"
scalacOptions += "-Yliteral-types"

We need to represent a key value pair of dimension and integer exponent. We could use a Tuple for this, but will use a shapeless FieldType instead. This is similar to a Tuple, but is more compatible with some of shapeless’s typeclasses.

type MetresPerSecond = FieldType[Metre, 1] :: FieldType[Second, -1] :: HNil

It’s important to note that the number 1 above is a type, not a value. Because it’s a type, the compiler can work with it.

Operations on Singleton types

When we multiply and divide dimensions, we want to add or subtract from these exponents.

We can use a library called singleton ops to do this. This provides us with type level integer operations using the OpInt typeclass:

OpInt.Aux[1 + 2, 3]
OpInt.Aux[3 * 2, 6]

It also provides a convenient alias for integer singleton types

type XInt = Singleton with Int

The type 1, for example, is a subtype of XInt.

Deriving typeclass instances

We now need to automatically derive typeclass instances of Divide. To do this, we’re going to derive instances for Invert and Multiply operations first. Deriving Divide then becomes much simpler.

The technique we’re going to use to automatically derive instances is known as typeclass induction.

Typeclass Induction

Aaron Levin gave a great introduction to induction in his talk earlier at the Typelevel Summit. In summary, you can derive an implicit typeclass instance for all cases by:

1. Providing it for the base case
2. Providing it for the n + 1 case, given that the n case is provided

This is similar to the mathematical method of proof by induction.

Invert

We’re first going to derive inductive typeclass instances for the Invert operation. Inverting a quantity raises it to the exponent of -1. This means that the exponents of all dimensions must be negated.

For example, the inverse of FieldType[Metre, 1] :: HNil is FieldType[Metre, -1] :: HNil. Invert takes one input type and returns one output type:

trait Invert[A] {
	type Out
}
object Invert {
	type Aux[A, Out0] = Invert[A] { type Out = Out0 }
}

To inductively derive typeclass instances for inverting, we need to prove that:

1. We can derive an instance for HNil (the base case)
2. We can derive an instance for a non-empty HList (the n + 1 case), provided there is an existing instance for its tail (the n case)

The base case operates on HNil

implicit def baseCase: Invert.Aux[HNil, HNil] = new Invert[HNil] { type Out = HNil }

The inductive case assumes that the tail has an instance, and derives an instance for the head by negating the exponent:

implicit def inductiveCase[D, Exp <: XInt, NExp <: XInt, Tail <: HList,
  OutTail <: HList](
    implicit negateEv: OpInt.Aux[Negate[Exp], NExp],
    tailEv: Invert.Aux[Tail, OutTail]
): Invert.Aux[FieldType[D, Exp] :: Tail, FieldType[D, NExp] :: OutTail] =
  new Invert[FieldType[D, Exp] :: Tail] {
    type Out = FieldType[D, NExp] :: OutTail
}

When the compiler looks for the implicit instance for FieldType[Metre, 1] :: HNil:

• It finds that the inductiveCase method has a return type which fits the signature
• It can find the required evidence negateEv for negating 1 from singleton ops
• It requires evidence of an implicit instance for the tail HNil
• It finds that baseCase provides this evidence

So in hunting for implicit typeclass instance for the whole list, the compiler goes and finds instances for the tail (the n case), right up until the base. If we provide an inductive proof with a baseCase and an inductiveCase, we fit the bill for what the compiler needs.

Multiply

Now that we’ve tested a basic example of induction, we can go on to a more complex one.

We want to multiply two HLists of dimensions together. This means that the exponents should be added.

trait Multiply[A, B] {
  type Out
}
object Multiply {
  type Aux[A, B, Out0] = Multiply[A, B] { type Out = Out0 }
}

This is harder to make inductive because there are two input lists involved. Luckily, we only need to recurse over one of them, as we can pick dimensions from the other using shapeless’s Selector. We will recurse over the left list and can pick elements from the right list.

Our base case can be the same:

implicit def baseCase: Multiply.Aux[HNil, HNil, HNil] = new Multiply[HNil, HNil] {
  type Out = HNil
}

We can define the inductive case using the following logic:

1. Pick the exponent in the right list corresponding to the head dimension in the left list
2. Add the left and right exponents together
3. Filter the term from the right list to get the remaining elements
4. Look for a typeclass instance for the left list tail and the remaining elements in the right list

implicit def inductiveCase[D, R <: HList, LExp <: XInt , RExp <: XInt,
  OutExp <: XInt, RTail <: HList, LTail <: HList, OutTail <: HList](
  implicit pickEv: Selector.Aux[R, D, RExp],
  addEv: OpInt.Aux[LExp + RExp, OutExp],
  filterEv: FilterNot.Aux[R, FieldType[D, RExp], RTail],
  tailEv: Multiply.Aux[LTail, RTail, OutTail]
): Multiply.Aux[FieldType[D, LExp] :: LTail, R, FieldType[D, OutExp] :: OutTail] =
  new Multiply[FieldType[D, LExp] :: LTail, R] {
    type Out = FieldType[D, OutExp] :: OutTail
}

When the compiler looks for an implicit instance of multiply for FieldType[Metre, -1] :: HNil and FieldType[Metre, 3] :: HNil:

• It finds that the inductiveCase has a return type which fits the signature
• Given that the head of the left list is Metre, it selects the exponent for Metre from the right list
• It can find the evidence addEv to add the exponents -1 and 3
• It filters Metre from the right list to get HNil
• It requires evidence of an instance for HNil and HNil
• This is provided by the base case

The compiler can now find instances of Multiply, as long as a dimension appears in both the left and right lists. This can be extended to when a dimension doesn’t appear by writing a few more inductive cases.

Divide

The reason we went to the effort of writing Invert and Multiply was to divide. Dividing a numerator by a denominator is as simple as inverting the denominator and multiplying it by the numerator. We can write this in a single non-inductive instance:

implicit def divide[L <: HList, R <: HList, RInverted <: HList,
  Divided <: HList](
  implicit invertEv: Invert.Aux[R, RInverted],
  multiplyEv: Multiply.Aux[L, RInverted, Divided]
): Divide.Aux[L, R, Divided] = new Divide[L, R] {
  type Out = Divided
}

That’s far simpler than the work we’ve done before - we’re just building on the typeclasses we wrote to do this.

Automatically derived instances

We can now have compile time dimensional analysis without writing out divide instances for every combination of dimensions:

val rocketMass = Quantity[FieldType[Kilogram, 1] :: HNil](40.0)
val fuelMass = Quantity[FieldType[Kilogram, 1] :: HNil](10000.0)
val exhaust = Quantity[FieldType[Metre, 1] :: FieldType[Second, -1] :: HNil](1000000.0)
val distance = Quantity[FieldType[LightYear, 1] :: HNil](77.0)

val rocketSpeed = times(exhaust, log(divide(add(rocketMass, fuelMass), rocketMass).value))
val time: Quantity[FieldType[Year, 1] :: HNil] = divide(distance, rocketSpeed)
// error: type mismatch; found: FieldType[LightYear, 1] :: FieldType[Metre, -1] :: FieldType[Second, 1] :: HNil; required: FieldType[Year, 1] :: HNil

Yay! We’ve solved the problem! It looks a lot more verbose than what we started with, but we can tidy this up by using extension methods:

implicit final class DoubleOps(val d: Double) {
  def ly: Quantity[FieldType[LightYear,1] :: HNil] = Quantity(d)
  def kg: Quantity[FieldType[Kilogram, 1] :: HNil] = Quantity(d)
  def yr: Quantity[FieldType[Year, 1] :: HNil]     = Quantity(d)
  def mps: Quantity[FieldType[Metre, 1] :: FieldType[Second, -1] :: HNil] = Quantity(d)
  def c: Quantity[FieldType[LightYear, 1] :: FieldType[Year, -1] :: HNil] = Quantity(d)
}

We can also add symbolic infix operators for add, times and divide:

case class Quantity[A](value: Double) {
  def +(q: Quantity[A]): Quantity[A] = Quantity(value + q.value)
  def *(v: Double): Quantity[A] = Quantity(value + v)
  def /[B](q: Quantity[B])(implicit d: Divide[A, B]): Quantity[d.Out] = Quantity(value / q.value)
}

We’ve reached Libra!

We started with doubles:

val rocketSpeed = 1000000.0 * log((40.0 + 10000.0) / 40.0)
val time = 77.0 / rocketSpeed

And we finished with compile time dimensional analysis:

val rocketSpeed = 1000000.0.mps * log(((40.0.kg +10000.0.kg) /40.0.kg).value)
val speedConversion = 300000000.0.mps / 1.c
val speedInC = rocketSpeed / speedConversion
val time = 77.0.ly / speedInC
//time: Quantity[FieldType[Year, 1] :: HNil] = Quantity(4180.65274634)

The code isn’t more verbose - if anything, it’s more explanatory and just as easy to work with.

Rolling this out to more problems

All we need to provide for the business logic of our rocket launch problem are the dimensions and DoubleOps. We could roll this out to any other problem. Let’s say we wanted to do a currency conversion between GBP and DKK:

val exchangeRate: Quantity[FieldType[DKK, 1] :: FieldType[GBP, -1] :: HNil] =
   currentExchangeRate()
Val krone: Quantity[FieldType[DKK, 1] :: HNil] = 10.gbp * exchangeRate

We get dimensional analysis for any problem domain out of the box!

Most of the code we’ve written is library code. In fact, it’s Libra code! Libra is a dimensional analysis library based on typelevel induction. It performs compile time dimensional analysis to any problem domain. It also uses spire for its numeric typeclasses, so can be used for far more than just doubles.

Conclusion

It’s been a long way from the humble Double. We started with basic types, explored dependent types, took a look at Typelevel Scala along the way, before finally ending up performing typelevel induction. As a result, we’ve managed to achieve compile time dimensional analysis for any problem. If you’re curious about typelevel induction take a look at the Libra codebase for more examples. Enjoy!

Licensing

Unless otherwise noted, all content is licensed under a Creative Commons Attribution 3.0 Unported License.

The need for configuration arises in almost every application, as we want to be able to run in different environments – for example, local, testing, and production environments. Configurations are also used as a way to keep secrets, like passwords and keys, out of source code and version control. By having configurations as untyped structured data in files, we can change and override settings without having to recompile our software.

In this blog post, we’ll take a look at configurations with configuration files, to see how we can make the loading process less error-prone, while overcoming obstacles with boilerplate, testing, and validation. We’ll also identify when it’s suitable to use Scala as a configuration language for improved compile-time safety, convenience, and flexibility; and more specifically, how Ciris helps out.

Configuration Files

Traditionally, configuration files, and libraries like Typesafe Config, have been used to load configurations. This involves writing your configuration file, declaring values and how they’re loaded, and then writing very similar Scala code for loading that configuration. That kind of boilerplate code typically looks something along the lines of the following example.

import com.typesafe.config.{Config, ConfigFactory}

// The settings class, wrapping Typesafe Config
final case class Settings(config: Config) {
  object http {
    def apiKey = config.getString("http.api-key")
    def timeoutSeconds = config.getInt("http.timeout-seconds")
    def port = config.getInt("http.port")
  }
}

// The configuration file, here represented in code
val config =
  ConfigFactory.parseString(
    """
      |http {
      |  api-key = ${?API_KEY}
      |  timeout-seconds = 10
      |  port = 989
      |}
    """.stripMargin
  ).resolve()

val settings = Settings(config)

show(settings)
// Settings(Config(SimpleConfigObject({"http":{"port":989,"timeout-seconds":10}})))

This is a tedious, error-prone process that rarely sees any testing efforts. PureConfig (and other libraries, like Case Classy) were created to remove that boilerplate. Using macros and conventions, they inspect your configuration model (nested case classes) and generate the necessary configuration loading code. This eliminates a lot of errors typically associated with configuration loading. Following is an example of how you can load that very same configuration with PureConfig.

final case class HttpSettings(
  apiKey: String,
  timeoutSeconds: Int,
  port: Int
)

final case class Settings(http: HttpSettings)

val settings = pureconfig.loadConfig[Settings](config)

show(settings)
// Left(ConfigReaderFailures(KeyNotFound("http.api-key", None, Set()), List()))

Encoding Validation

In both previous examples, we do not check whether our configurations are valid to use with our application. In the case of Typesafe Config, we hit a runtime exception if the key is missing or if the type conversion fails, and in PureConfig’s case, we will instead get a ConfigReaderFailures. But in neither case do we care what values are being loaded, as long as they can be converted to the appropriate types. For example, we might require a key of certain length and that it only contains certain characters, the timeout needs to be positive, and the port must be a non-system port number (value in the inclusive range between 1024 and 65535).

You could write an additional validation step to ensure the configuration is valid after it has been loaded – which can be tedious to write and requires testing. One could also argue that the types of the configuration values are too permissive: why use String for the key if you do not accept all String values? And why use an Int for timeout and port, if you only allow a limited subset of values?

We could write these custom types ourselves, including the validation logic, and tell PureConfig how to load them – which would be tedious to write for many types and would require testing. Another alternative is to use refined, which allows you to do type-level refinements (apply predicates) to types. I found this approach so useful that I wrote a small integration between PureConfig and refined at the end of last year (see blog post), so that PureConfig can now load refined’s types.

import eu.timepit.refined.api.Refined
import eu.timepit.refined.numeric.Interval
import eu.timepit.refined.pureconfig._
import eu.timepit.refined.string.MatchesRegex
import eu.timepit.refined.types.numeric.PosInt
import eu.timepit.refined.W

type ApiKey = String Refined MatchesRegex[W.`"[a-zA-Z0-9]{25,40}"`.T]

type NonSystemPort = Int Refined Interval.Closed[W.`1024`.T, W.`65535`.T]

final case class HttpSettings(
  apiKey: ApiKey,
  timeoutSeconds: PosInt,
  port: NonSystemPort
)

final case class Settings(http: HttpSettings)

val settings = pureconfig.loadConfig[Settings](config)

show(settings)
// Left(
//   ConfigReaderFailures(
//     KeyNotFound("http.api-key", None, Set()),
//     List(
//       CannotConvert(
//         "989",
//         "eu.timepit.refined.api.Refined[Int,eu.timepit.refined.boolean.And[eu.timepit.refined.boolean.Not[eu.timepit.refined.numeric.Less[Int(1024)]],eu.timepit.refined.boolean.Not[eu.timepit.refined.numeric.Greater[Int(65535)]]]]",
//         "Left predicate of (!(989 < 1024) && !(989 > 65535)) failed: Predicate (989 < 1024) did not fail.",
//         None,
//         "http.port"
//       )
//     )
//   )
// )

As you can see in the example above, refined already contains type aliases for many common refinement types, like PosInt (for Int values greater than zero). You can also easily define your own predicates, like the one for the key and port. The W here is a shorthand for shapeless’ Witness: a way to encode literal-based singleton types (essentially, values on the type-level). If you’re using Typelevel Scala with the -Yliteral-types flag, you can write values directly in the type declaration, without having to use Witness.

If you’re not convinced configurations need to be validated, I can recommend reading the paper Early Detection of Configuration Errors to Reduce Failure Damage, and to read through the slides of Leif Wickland’s (one of the authors behind PureConfig) recent presentation Defusing the Configuration Time Bomb on the subject.

In many ways, think of configurations as user input – would you happily accept any values provided to your application from its users? Probably not: you would validate the input, and sanitize it if possible. Think about configurations in the same way, except that the user here might happen to be a developer of the application. The key here, as discussed in the paper linked above, is to check that your configuration is valid as soon as possible, ideally at compile-time, or as soon as the application starts. We want to avoid situations where we’re running the application and suddenly discover that configuration values are invalid or cannot be loaded – or worse, continue running with an invalid configuration, not to discover issues until much later on.

Improving Compile-time Safety

We’ve now got a way to encode validation in the types of our configurations, and a boilerplate-free way of loading values of those types from configuration files – is there still room for improvement? To answer that question, we first need to ask why we are using configuration files in the first place.

Whether you thought about it or not, the main reason for using configuration files is so that we can change settings without having to recompile the software. In my experience, most developers default to using configuration files, and almost always change values by pushing commits to a version control repository. This is followed by a new release of the software, either manually or via a continuous integration system. In scenarios like this, and in general when it’s easy to change and release software (particularly when employing continuous deployment practices), configuration files are not used for the benefit of being able to change values without recompile.

In such cases, why are we not writing the configurations directly in source code? Christopher Vogt has written an excellent blog post (and given a presentation) on the subject. The tricky part here is managing values which need to be dynamic in the environment (like the port to bind) and are secret (like passwords and keys). Depending on your requirements and preferences, you more or less have two alternatives.

• If you know which environments your application will run in, and what the configuration values will be in those environments, you can just include the configurations in your application code (if it has no secrets), or store, compile, and bundle the configuration separately. If you have a requirement that secrets shouldn’t touch persistent storage, this might not be a feasible alternative. You might also appreciate the fact that all code relating to your application is in the same version control repository and gets compiled together, in which case this approach might not be suitable.

• Alternatively, you can include the configuration in your application, but load secrets and values which need to be dynamic from the environment during runtime. This is necessary when configuration values cannot be determined beforehand – because you do not know what environment your application will run in, or if you use a vault (like credstash, for example) or a configuration service (like ZooKeeper, for example) – or if you prefer having your configuration together with your application code and in the same version control repository.

In this post, we’ll only focus on the latter case. While it’s possible to not use any libraries in the latter case, loading values from the environment typically means dealing with: different environments and configuration sources, type conversions, error handling, and validation. This is where Ciris comes in: a small library, dependency-free at its core, helping you to deal with all of that more easily.

Introducing Ciris

Imagine for the moment that no part of your configuration is secret and that your application only ever runs in one environment. You can then just write your configuration in code.

import eu.timepit.refined.auto._

final case class Config(
  apiKey: ApiKey,
  timeoutSeconds: PosInt,
  port: NonSystemPort
)

val config =
  Config(
    apiKey = "RacrqvWjuu4KVmnTG9b6xyZMTP7jnX",
    timeoutSeconds = 10,
    port = 4000
  )

You then realize that it’s a bad idea to put the key in the source code, because source code can easily get into the wrong hands. You decide that you’ll instead read an environment variable for the key. Since you want to make sure that your configuration is valid, you have used refinement types, so you’ll have to make sure to check that the key conforms to the predicate. You would also welcome a helpful error message if the key is missing or invalid. This sounds like more work than it should be, so let’s see how Ciris can help us.

Ciris method for loading configurations is loadConfig and it works in two steps: first define what to load, and then how to load the configuration. For reading a key from an environment variable, you can use env[ApiKey]("API_KEY") which reads the environment variable API_KEY as an ApiKey. Ciris has a refined integration in a separate module, so you just need to add an appropriate import. Loading the configuration is then just a function accepting the loaded values as arguments.

import ciris._
import ciris.refined._

val config =
  loadConfig(
    env[ApiKey]("API_KEY")
  ) { apiKey =>
    Config(
      apiKey = apiKey,
      timeoutSeconds = 10,
      port = 4000
    )
  }

show(config)
// Left(ConfigErrors(MissingKey(API_KEY, Environment)))

Ciris deals with type conversions, error handling, and error accumulation, so you can focus on your configuration. The loadConfig method returns an Either[ConfigErrors, T] instance back, where T is the result of your configuration loading function. You can retrieve the accumulated error messages by using messages on ConfigErrors.

show { config.left.map(_.messages) }
// Left(Vector("Missing environment variable [API_KEY]"))