Haoyi's Programming Blog

Twitter
Resume
Github
Learn Scala
Subscribe
Talks
Table of Contents

12 years of the com.lihaoyi Scala Platform

Posted 2024-06-09
ScalaSql: a New SQL Database Query Library for the com-lihaoyi Scala Ecosystem

The com.lihaoyi platform is an ecosystem of Scala tools and libraries that allow you to write Scala in a way that is easy, rather than powerful. This post discusses the history of the com.lihaoyi platform, how it came about and evolved over the past decade, and what role I think it may play in the future of the Scala language.

About the Author: Haoyi is a software engineer, and the author of many open-source Scala tools such as the Ammonite REPL and the Mill Build Tool. If you enjoyed the contents on this blog, you may also enjoy Haoyi's book Hands-on Scala Programming

This is the companion blog post to the keynote talk I gave at Scala Matsuri, 8 June Tokyo 2024.

What is the com.lihaoyi Platform?

The most notable projects in the com.lihaoyi platform are listed below, along with the date they were first published:

The goal of these projects is to make Scala easy. These projects pivot away from the more dogmatic approaches to writing Scala - whether Reactive-Async actors with Akka or strict pure functional programming with Cats or ZIO - towards a more "quick and easy" style of writing Scala that aims to maximize up-front productivity so you can hit the ground running.

The com.lihaoyi platform does make tradeoffs. These libraries may not support the ultra-high-concurrency use cases that Akka does, nor do they support the strict purity that the pure functional programming approaches do. But they are very much battle-tested, letting you be productive from your first line of code in the REPL and robust enough as your codebase grows into a large scale, high performance, complex system. I myself have deployed systems running on thousands of machines, many tens of thousands of cores, over multiple clouds using the com.lihaoyi libraries.

Before we go too deeply into what the com.lihaoyi ecosystem is today, let's backtrack a bit to discuss how it all got started...

Origins of the com.lihaoyi platform

The com.lihaoyi platform started together with my interest in the Scala language. The first project, Scalatags, arose out of my experience with the XHP project I used as a web developer during my 2012 college internship at Facebook:

XHP was one of the first projects allowing you to mix HTML markup with your server-side application code. And not just as a templating language (as PHP already was), but treating the HTML snippets as values you could manipulate and pass around your application.

if ($_POST['name']) {
  echo <span>Hello, {$_POST['name']}</span>;
} else {
  echo <form method="post">
    What is your name?<br />
    <input type="text" name="name" />
    <input type="submit" />
  </form>;
}

XHP at the time was an amazing piece of technology:

Since then, XHP has spawned a zoo of downstream projects, including a Python equivalent Pyxl that I used in my time at Dropbox, and the now-ubiquitous Javascript framework React.js and JSX. Now tools like XHP are table stakes, but back in 2012 it was cutting edge.

After my internship, I wanted to continue using XHP for my own small web development projects. But I couldn't! XHP's native environment was the Facebook Linux development and production systems, and as a college-student I did not have the skills necessary to port it to run on my Windows laptop. But as a college student, what I did have was a lot of energy and a lot of free time, so the natural thing to do was to re-implement the whole thing myself. So that's what I did, multiple times, in multiple languages!

Touring Languages

Most of these implementations pre-date my knowledge of version control, and have been lost to the mists of time. But they include implementations in a variety of languages:

  1. PHP, again! Maybe I couldn't figure out how to install it, but I could figure out how to re-implement it. I had PHP installed on my windows laptop (though not without difficulty). Within a few weeks I had my own clean-room re-implementation of the XHP PHP pre-processor: parser, code generator, runtime library, the whole works. It worked, more or less, but the PHP interpreter kept giving me segfaults on valid generated code. Debugging them was beyond my capabilities, which led me to abandon this implementation

  2. Python, lingua-franca of my university computer science department. This worked, but was very slow to run. Rendering typical HTML pages took hundreds of milliseconds even without counting any business logic. There are always things you can do to try and speed things up, but the fundamental slowness of Python seemed unavoidable.

  3. Java, from my middle-school programming class. This was also when I decided to abandon the XML-based <body><p>text</p></body> XML-ish syntax preprocessor in favor of a more concise body(p("hello")) syntax. Java didn't segfault, and performance was great. But the Java language proved to be very verbose and inflexible, so I moved on.

  4. C#, which I learned on my own in high-school. C# is a more flexible language than Java, including features like Implicit Conversions. C# is great in many ways, and I like the language. But it was still pretty verbose and I still had trouble making it look good

  5. F#! This relatively obscure language is a hybrid between C# and Ocaml. Its functional programming constructs like map/filter/reduce were an epiphany for me. But it had a lot of odd warts, e.g. the syntax was weird (more on that below), the fact that IDE support in Visual Studio sucked despite being a first-party Microsoft language, or the way you had to keep repeating the type of the collection you were working with e.g. let newList = oldList |> List.map(fun x -> x + 1) |> List.filter(fun x -> x > 10). A good language, but flawed in many ways.

  6. Scala! That's what eventually grew to become Scalatags:

html(
  head(
    script(src:="..."),
    script(
      "alert('Hello World')"
    )
  ),
  body(
    div(
      h1(id:="title", "This is a title"),
      p("This is a big paragraph of text")
    )
  )
)

In many ways, the issues I had trying to write my XHP-esque HTML templating library were a microcosm of the programming language ecosystem as a whole: PHP's unreliability, Python's poor performance, Java and C#'s verbosity, F#'s weirdness and poor tooling. While this specific exercise was trying to implement a HTML templating library, the same complaints could be expected from someone using these languages for any other purpose.

After the exercise of implementing and re-implementing the same HTML templating library in 6 languages, spanning many months of work, I ended up sticking with Scala.

Good Parts: The Scala Language

Below are some of the nice properties of the Scala language that made me stick around:

Operation Fsharp Scala
Array Access a.[i] a(i)
Dictionary Access a.["foo"] a("foo")
Copy Constructor {p with Y = 0.0} p.copy(y = 0.0)
List Constructor [a; b; c] List(a, b, c)
Array Constructor [| a; b; c |] Array(a, b, c)
Collection Take values[..5] values.take(5)
Collection Drop values[5..] values.drop(5)
Collection Slice values[1..5] values.slice(1, 5)

This was in 2012, with Scala 2.9, but already Scala just seemed to hit all the right notes for me. Sure there were issues, but my college-student web-developer self loved the language. I ended up building some websites for various school or freelance projects and fun side projects like a Metacircular JVM or various Scala.js browser games.

However, not everything was as rosy...

Bad Parts: The Scala Ecosystem

Although the Scala language felt great to me, even in 2012, the ecosystem was a mess. Some fun memories of the time:

Enumeratees.png

JSON parser Score (higher is better)
fastparse 159.5
circe 332.4
argonaut 149.1
uJson 266.6
json4s 100.9
play-json 226.6
scala-parser-combinators 0.9
Operator Meaning
<++=
FishBones.png
?
|@|
TIEfighter2.png
??
<**>
TIEBomber.png
???

While I felt the Scala language was nice when I was writing standalone code living in its own little bubble, the Scala ecosystem was a complete mess when I actually wanted to do anything. Recursing over linked lists or transforming case class data structures was great, but god forbid you actually wanted to serialize your data structure to a file or return it from a JSON web endpoint. In my efforts to use Scala for projects, both personal or professional, this was a constant drag.

I even managed to write some "production" Scala code at my first job at Dropbox, which is a Python shop. I was using Scala to statically analyze multi-million-line Python and Javascript codebases, and finding >10 bugs every day that were not caught by tests. But I could not grow usage of the Scala language at Dropbox: how do you convince your colleagues to invest in a new language when the first time they try to serialize a dictionary the compiler hangs?

An Ecosystem Of My Own

Given I liked the Scala language, and did not like the Scala ecosystem, what else was there to do but to write my own ecosystem? How hard could it be to write a JSON parser or HTTP client anyway?

Building out the com.lihaoyi platform took the better part of a decade, but the direction was always clear: I wanted to be able to write Scala in a way that was easy, not powerful. Scala-the-language already provided all the language features and syntax necessary to make nice-to-use APIs, Scala-the-ecosystem just needed to start making use of them.

Easy, not Powerful

For example rather than using the Akka-HTTP client to write:

import akka.actor.typed.ActorSystem
import akka.actor.typed.scaladsl.Behaviors
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import scala.util.{ Failure, Success }

implicit val system = ActorSystem(Behaviors.empty, "SingleRequest")

implicit val executionContext = system.executionContext

val responseFuture = Http()
  .singleRequest(HttpRequest(uri = "http://akka.io"))

responseFuture.onComplete {
  case Success(res) => println(res)
  case Failure(_) => sys.error("something wrong")
}

I wanted to be able to use Requests-Scala to write

val res = requests.get("https://akka.io")

println(res)

Rather than using Scopt to parse command-line arguments via:

case class Config(foo: String = null,
                  num: Int = 2,
                  bool: Boolean = false)

val builder = OParser.builder[Config]
val parser1 = {
  import builder._
  OParser.sequence(
    programName("run"),
    opt[String]("foo")
      .required()
      .action((x, c) => c.copy(foo = x)),
    opt[Int]("num")
      .action((x, c) => c.copy(num = x)),
    opt[Unit]("bool")
      .action((_, c) => c.copy(bool = true))
  )
}

val parsed = OParser.parse(parser1, args, Config())
for(Config(foo, num, bool) <- parsed){
  println(foo * num + " " + bool.value)
}

I wanted to be able to use MainArgs to write

@main
def run(foo: String, num: Int = 2, bool: Flag) = {
  println(foo * num + " " + bool.value)
}

Rather than using verbose Java APIs via:

import java.nio.file.{Paths, Files}
import java.nio.charset.StandardCharsets

Files.write(Paths.get("filename.txt"), "hello world".getBytes(StandardCharsets.UTF_8))

I wanted to be able to use OS-Lib to write

os.write(os.pwd / "filename.txt", "hello world")

The goal of this is not just to make the code shorter, but also to make the code easier. Both easier to write and also easier to read. Someone who doesn't know the com.lihaoyi libraries, even someone who doesn't know Scala, should be able to glance at requests.get("https://akka.io") or os.write(os.pwd / "filename.txt", "hello world"), guess in a second what the code does, and guess correctly.

Compounding Simplicity

The com.lihaoyi platform is not just about cute one-liners; as your requirements and workflows become more complex, the cute one-liners can be combined together into concise 2-liners or 5-liners that pack in a lot of meaning into a short and easy to read snippet.

For example, given how you fetch data via HTTP:

requests.get("https://akka.io")

And how you write data to a file:

os.write(os.pwd / "filename.txt", "hello world")

Fetching data and then writing it to a file is both trivial and obvious:

os.write(os.pwd / "filename.txt", requests.get("https://akka.io"))

If you want to do so in a streaming fashion, it's just adding a single .stream:

os.write(os.pwd / "filename.txt", requests.get.stream("https://akka.io"))

Even if you want to stream the downloaded data through an external subprocess, and take the output of the subprocess and stream that to a file, the code remains short and easy to understand:

os.write(
   os.pwd / "filename.txt",
   os.proc("gzip")
     .spawn(stdin = requests.get.stream("https://akka.io"))
     .stdout
)

Consider what we are doing above: making a HTTP call, streaming the result through a gzip subprocess, and then streaming the output of the subprocess to a file on disk. This isn't rocket science, but could easily be a bunch of messy, finnicky code. But with the com.lihaoyi libraries like OS-Lib providing os.write and os.proc, and Requests-Scala providing requests.get, it becomes a short snippet of extremely simple code that is easy to write and easy to read!

Contrast this with similar code written without the com-lihaoyi libraries: using Akka-HTTP client, java.lang.ProcessBuilder, and java.nio.file. Even without streaming, it's already quite a huge mess of code:

import akka.actor.typed.ActorSystem
import akka.actor.typed.scaladsl.Behaviors
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import scala.util.{ Failure, Success }
import java.nio.file.{Paths, Files}
import java.nio.charset.StandardCharsets

implicit val system = ActorSystem(
  Behaviors.empty,
  "SingleRequest"
)

implicit val executionContext = system.executionContext

val responseFuture = Http()
  .singleRequest(HttpRequest(uri = "http://akka.io"))

responseFuture
  .flatMap { resp => resp.entity.toStrict(timeout)}
  .map { strictEntity => strictEntity.data.toArrayUnsafe }
  .onComplete {
    case Success(res) =>
      val proc = new java.lang.ProcessBuilder().command("gzip").start()
      val out = proc.getOutputStream
      out.write(bytes)
      out.close()
      proc.waitFor()

      Files.write(
        Paths.get("filename.txt"),
        proc.getInputStream.readAllBytes
      )

    case Failure(_) => sys.error("something wrong")
  }

This code isn't rocket science, but it is pretty verbose with lots of details you don't probably don't care about, and lots of places where you can possibly get things subtly wrong. And although requests.get, os.write, os.proc are "just" helper methods, it should be clear that at some point the convenience of helper methods results in a qualitative change in how you work with your code: how fast you can try something, how fast you can throw together a prototype to show your colleague/boss/customer, or how fast you can read/understand/modify to fix bugs or adapt to changing requirements. This lets you focus on the specific business logic unique to your specific problem, rather than spending your time juggling actors, futures, monads or InputStreams and OutputStreams.

And that what the com.lihaoyi ecosystem is all about: making using Scala easy, not just when working with your own code, but also when interacting with the external APIs necessary to do real work.

Where We Stand Today

The Scala ecosystem in 2024 is very different from the Scala ecosystem in 2012. People joke about having 5 good Scala JSON libraries to choose from in 2024, but that's a far better situation than having zero good JSON libraries back in 2012! SBT is much easier to use. Scala-CLI, Scalafmt, and other tools greatly increase quality of life. The entire ecosystem has evolved greatly from when I started using Scala in 2012, as has the com.lihaoyi platform itself. But let's look at some high-level points:

The Reactive-Async mania has burned out

People have realized that perhaps Akka actors aren't, in fact, the future of everything. Akka has gone closed source, many of the supporting libraries like SLICK or Play have been largely abandoned. Many of the folks standing on stage 12 years ago evangelizing Akka, now work elsewhere doing other things. While the exact details of this shift can be debated, fundamentally the Akka folks ran out of money because they were not demonstrating enough value to their users and customers to sustainably fund the project.

There are certainly niches where concurrency based on distributed-actors is incredibly powerful, and I expect Akka and it's fork Pekko to continue being used indefinitely. But it has proven to be a niche tool: extremely valuable in certain scenarios, and extremely inappropriate to use in others. Wrapping everything under the sun in Futures/Actors/Reactive-Streams was not the future of the software industry. Even burning through 10 years and 80 million US$ of VC funding was not enough to change this fundamental fact.

Pure Functional programming has found a strong niche

The Pure functional side of the Scala community is definitely healthy. The old FP framework Scalaz may be gone, but Cats and ZIO have risen from the ashes, both groups having built out a strong ecosystem of libraries and tools and knowledge. Many large companies use these frameworks and many enthusiastic developers contribute to them.

But pure-functional programming and IO-monad techniques has always been a niche approach in the broader programming community, and I don't believe the Scala language is special enough to change that. There will always be a community of people for which pure functional programming resonates, and there will always be use cases where IO-monad-style concurrency becomes very valuable. This sub-community forms a core part of the Scala community today, and I hope it remains healthy for the foreseeable future. But I do not think it is the engine that will drive the growth of the Scala community for the next 12 years.

The com.lihaoyi platform has built out a third way

What started out as a few scattered libraries here and there has been fleshed out into a rich ecosystem, providing everything you need to build a CLI tool or website or distributed system. Even today, the bulk of Scala users are still on Reactive or Pure Functional Programming stacks, but folks using com.lihaoyi libraries form a significant minority.

The com.lihaoyi platform may still be small, but it is relatively complete. Just like you can assemble one-liners into useful code snippets, you can also assembly entire libraries into useful applications:

  1. Mill + OS-Lib = Build System
  2. uJson + Requests-Scala = API Client
  3. Cask + Scalatags + ScalaSql = Website
  4. Cask + uPickle + ScalaSql = API Server
  5. Mill + OS-Lib + Scalatags = Static Site Generator
  6. MainArgs + OS-Lib = CLI tool

The individual libraries let you parse command-line arguments, work with files, serialize data, run HTTP clients and servers, render HTML websites, query SQL databases. Combined, these building blocks let you write websites, API servers, API clients, CLI tools. All of them concrete use cases that you may do as part of your day-to-day job!

Again, the com.lihaoyi platform is still only used by a minority of Scala programmers. But if I meet a Python programmer looking for a platform that's more performant, or a Java developer looking for a platform that's more expressive and concise, com.lihaoyi Scala is where I would send them.

Looking Into The Future

So given where Scala and the com.lihaoyi platform are today, where are they going in the future?

Scala Needs to Evolve

Scala needs to continue evolving. While I think Scala in 2012 was significantly better than most other languages in 2012, and Scala in 2024 is significantly better than Scala in 2012, other languages in 2024 are making great strides as well:

In 2012, XHP and Scalatags were cutting edge, but in 2024 every language has some sort of in-line HTML template builder. In 2012, Scala compiling to Javascript via Scala.js and sharing code between the browser and server was amazing. In 2024, every backend language compiles to Javascript in some way, and Javascript itself has grown a strong backend runtime Node.js, a strong static-type-checker in Typescript, and a massive ecosystem of tools and libraries.

While I may not agree with every change that went into Scala 3, at a high level Scala needs to continue evolving to remain relevant. The rollout of indentation-syntax, enums, and other Scala 3 improvements are table stakes. More needs to be done. This evolution needs to be managed carefully, and with the support of the stakeholders and community, but fundamentally the language needs to keep moving forward.

Scala Needs to be more than Reactive-Async or Pure-FP

Reactive-Async-Scala and Pure-FP-Scala will continue to be major players in the Scala ecosystem, but I hope that com.lihaoyi Scala can provide a viable third way. In a recent thread on Reddit Why have you stopped using Scala, it is notable how many people move away from Scala got burned by using advanced Reactive or FP techniques when perhaps they were not a good fit for the people and problem at hand:

We need to stop using very complicated concepts where not needed, that would help Scala adoption. In many applications plain Scala would be enough, but many of us threw Akka, or ZIO, etc. at the problem, even if it is not necessary, making it harder to onboard new team members.

I get that a lot of the core community in Scala really loves the deep FP stuff, but it’s a bit like switching from infix notation to reverse Polish notation on a calculator: neat and clever, but too clever so it becomes this write once edifice that’s hard for others to contribute to.

My company (a bank) is trying to port all scala code back to java code, mostly because there are less and less scala developers to recruit. These ports require major efforts because of the non-trivial scala code such as implicits, early usage of scalaz, ZIO or Akka and... there are less and less scala developers that want to do those ports. It is a mess.

Reactive-Async or Pure-FP-IO-Monad programming patterns are relatively specialized tools outside the Scala community. In Scala, we have seen people picking them "by default" and getting burned when they are not a good fit for the people or problem. As you can see from the above quotes, many of these folks get burned and end up moving off of Scala entirely. I have been burned myself, as have many people I know personally.

But this presents an opportunity: if we can retain the folks who would have written off Scala as "that weird actor/monad language", by instead selling it as "that concise, safe, performant language with good tooling and ecosystem", that would be a huge boon. It would allow us to evolve the Scala ecosystem, not by reducing the size of the Reactive/FP niche, but by growing the size of the rest of the Scala community! And it seems like a viable path forward: what programmer doesn't want a concise, safe, and performant language with good tooling and ecosystem?

The Scala Community Needs to Grow

Reflecting on the past 12 years, there are two things that are true:

  1. The Scala ecosystem has accomplished amazing things with very few people. For example, Scala.js is largely a 1-2 person project, com.lihaoyi has been largely a 1-2 person project, etc.. The fact that these efforts have done so much with so little is notable

  2. Other ecosystems have been less efficient, but have nonetheless managed to accomplish as much or more. Dart may have required an order of magnitude more people to accomplish what Scala.js did, but they put in the work and did it. Many more people use Dart than Scala.js today in 2024

In the end, it doesn't matter if the Scala language is 10x more productive than its competitors, if its competitors have more than 10x as large a community and firepower behind their efforts: their IDEs will be better, their tooling will be better, their libraries and frameworks will be better. Simply by brute force effort.

Conversely, although Scala may be a very productive language, we have seen many projects languish or be entirely abandoned just because the single maintainer graduated from their PhD, found a new job doing something else, or just got busy with life in general. Scala being 10x more productive means nothing when the number of people on a project drops to zero.

The Scala community is smaller in 2024 than it was around 2015-2020. Much of the hype has passed, the evangelism funded by Lightbend's VC funded marketing budget has ended, the novelty-seekers who adopted Scala when it was new and exciting have since moved on to newer and more exciting things. You still have big companies like Databricks/Asana/Apple/Disney heavily invested into Scala, but there is no question at all that the community interest is smaller than it used to be. The Peak of Inflated Expectations has passed. The open question now is whether will be able to rise from the Trough of Disillusionment to some Plateau of Productivity, or whether it will continue to decline into obscurity and oblivion.

HypeCycle.jpg

To avoid irrelevance, the Scala community needs to grow: that would mean more contributors, more ideas, more bus factor, more firepower behind important efforts. But growing the Scala community means getting people without prior Scala experience to have a good time using the Scala language. And that's exactly what the com.lihaoyi platform is designed for!

The com.lihaoyi Platform Will Endure

The com.lihaoyi platform has always been newbie friendly. In a way, it is the ideal way to learn Scala: by focusing on Scala itself, being productive doing useful things with it, and leaving the complexity of Reactive Actors or IO Monads for later (if ever). But there has always been the question of whether com.lihaoyi as a small part-time volunteer effort is sustainable, compared to other much larger - and sometimes much more well-funded - efforts in the Scala community. But I think by now, the answer is clear.

Over the past 12 years, we have seen the well-funded Reactive-Async effort run out of money and sputter out, today still alive but running on fumes. We’ve seen the death of ScalaZ and it’s rebirth as Cats and ZIO. We’ve seen countless projects be announced, launch with great fanfare, and fizzle out in obscurity.

Throughout all this, the com.lihaoyi platform trudges on, slowly and steadily building out a platform of carefully chosen building blocks: simple, solid, and stable. Think bricks, not cathedrals. Bricks that require minimal maintenance to serve their purpose over time. Bricks that are not themselves interesting, but are meant for you to build something interesting using them!

Perhaps uniquely among the programming languages of the world, Scala is meant to be a Scalable Language: usable for anything from small-scale scripts to large-scale enterprise deployments. Scala has always supported scripts and a REPL. But it is com.lihaoyi projects like the Ammonite REPL, OS-lib, and Requests-Scala that make small-scale Scala useful for actual work. Apart from providing a third way to write Scala apart from Reactive-Async and Pure-FP, the com.lihaoyi platform making small-scale Scala useful is key to the long term value proposition of Scala as a Scalable Language.

Get Involved In com.lihaoyi Scala!

We've now discussed the origins of the com.lihaoyi platform, how it has evolved over the last 12 years, and where I think it will go in future. So here's the call to action: if you are a fan of the Scala language, but aren't fully onboard with the Reactive-Async-Actors or Pure-Functional-IO-Monad styles of writing Scala code, get involved in the com.lihaoyi ecosystem!

The com.lihaoyi ecosystem is by no means perfect. We need help by interested people to maintain it and move it forward. We don't have 80 million US$ of VC funding for development and evangelism, nor do we have binders full of online influencers and personalities to market for us on twitter. Just a handful of like-minded volunteers who love the Scala language and want to make their own experience writing Scala better.

The core thesis of the com.lihaoyi ecosystem is that the Scala language can be more than a host for Reactive-Async-Actors or Pure-Functional-IO-Monad frameworks. That Scala's conciseness, safety, performance, and ecosystem is valuable in its own right. Not everyone agrees with that thesis. But if that's something you agree on, then the com.lihaoyi ecosystem needs your help!

About the Author: Haoyi is a software engineer, and the author of many open-source Scala tools such as the Ammonite REPL and the Mill Build Tool. If you enjoyed the contents on this blog, you may also enjoy Haoyi's book Hands-on Scala Programming

ScalaSql: a New SQL Database Query Library for the com-lihaoyi Scala Ecosystem
Updated 2024-06-09 2024-06-09 2024-06-09 2024-06-09 2024-06-09 2024-06-09 2024-06-09 2024-06-09 2024-06-09 2024-06-09