Fork me on GitHub

µPickle 0.7.1


uPickle (pronounced micro-pickle) is a lightweight JSON and binary (MessagePack) serialization library for Scala. It's key features are:

Getting Started


Add the following to your SBT config:

libraryDependencies += "com.lihaoyi" %% "upickle" % "0.7.1"

And then you can immediately start writing and reading common Scala objects to strings:

import upickle.default._

write(1)                          ==> "1"

write(Seq(1, 2, 3))               ==> "[1,2,3]"

read[Seq[Int]]("[1,2,3]")         ==> List(1, 2, 3)

write((1, "omg", true))           ==> """[1,"omg",true]"""

read[(Int, String, Boolean)]("""[1,"omg",true]""") ==> (1, "omg", true)

Or to compact byte arrays, using the MessagePack format:

import upickle.default._

writeBinary(1)                          ==> Array(1)

writeBinary(Seq(1, 2, 3))               ==> Array(0x93.toByte, 1, 2, 3)

readBinary[Seq[Int]](Array[Byte](0x93.toByte, 1, 2, 3))  ==> List(1, 2, 3)

val serializedTuple = Array[Byte](0x93.toByte, 1, 0xa3.toByte, 111, 109, 103, 0xc3.toByte)

writeBinary((1, "omg", true))           ==> serializedTuple

readBinary[(Int, String, Boolean)](serializedTuple) ==> (1, "omg", true)

ScalaJS

For ScalaJS applications, use this dependencies instead:

libraryDependencies += "com.lihaoyi" %%% "upickle" % "0.7.1"

Other than that, everything is used the same way. upickle-0.7.1 is only compatible with ScalaJS 0.6.x.

Scala 2.10

uPickle does not support Scala 2.10; only 2.11 and 2.12 are supported

Basics


Builtins

This is a non-comprehensive list of what the most commonly-used types pickle to using uPickle. To begin, let's import upickle

import upickle.default._

Booleans are serialized as JSON booleans

write(true: Boolean)              ==> "true"
write(false: Boolean)             ==> "false"

Numbers are serialized as JSON numbers

write(12: Int)                    ==> "12"
write(12: Short)                  ==> "12"
write(12: Byte)                   ==> "12"
write(Int.MaxValue)               ==> "2147483647"
write(Int.MinValue)               ==> "-2147483648"
write(12.5f: Float)               ==> "12.5"
write(12.5: Double)               ==> "12.5"

Except for Longs, which too large for Javascript. These are serialized as JSON Strings, keeping the interchange format compatible with the browser's own JSON parser, which provides the best performance in Scala.js

write(12: Long)                   ==> "12"
write(4000000000000L: Long)       ==> "4000000000000"
// large longs are written as strings, to avoid floating point rounding
write(9223372036854775807L: Long) ==> "\"9223372036854775807\""

Special values of Doubles and Floats are also serialized as Strings

write(1.0/0: Double)              ==> "\"Infinity\""
write(Float.PositiveInfinity)     ==> "\"Infinity\""
write(Float.NegativeInfinity)     ==> "\"-Infinity\""

Both Chars and Strings are serialized as Strings

write('o')                        ==> "\"o\""
write("omg")                      ==> "\"omg\""

Arrays and most immutable collections are serialized as JSON lists

write(Array(1, 2, 3))             ==> "[1,2,3]"

// You can pass in an `indent` parameter to format it nicely
write(Array(1, 2, 3), indent = 4)  ==>
  """[
    |    1,
    |    2,
    |    3
    |]""".stripMargin

write(Seq(1, 2, 3))               ==> "[1,2,3]"
write(Vector(1, 2, 3))            ==> "[1,2,3]"
write(List(1, 2, 3))              ==> "[1,2,3]"
import collection.immutable.SortedSet
write(SortedSet(1, 2, 3))         ==> "[1,2,3]"

Options are serialized as JSON lists with 0 or 1 element

write(Some(1))                    ==> "[1]"
write(None)                       ==> "[]"

Tuples of all sizes (1-22) are serialized as heterogenous JSON lists

write((1, "omg"))                 ==> """[1,"omg"]"""
write((1, "omg", true))           ==> """[1,"omg",true]"""

Case classes of sizes 1-22 are serialized as JSON dictionaries with the keys being the names of each field. To begin with, you need to define a serializer in the Case Class's companion object:

import upickle.default.{ReadWriter => RW, macroRW}

After that, you can begin serializing that case class.

case class Thing(myFieldA: Int, myFieldB: String)
object Thing{
  implicit val rw: RW[Thing] = macroRW
}
case class Big(i: Int, b: Boolean, str: String, c: Char, t: Thing)
object Big{
  implicit val rw: RW[Big] = macroRW
}
import upickle._
write(Thing(1, "gg"))             ==> """{"myFieldA":1,"myFieldB":"gg"}"""
read[Thing]("""{"myFieldA":1,"myFieldB":"gg"}""") ==> Thing(1, "gg")
write(Big(1, true, "lol", 'Z', Thing(7, ""))) ==>
  """{"i":1,"b":true,"str":"lol","c":"Z","t":{"myFieldA":7,"myFieldB":""}}"""

write(Big(1, true, "lol", 'Z', Thing(7, "")), indent = 4) ==>
  """{
    |    "i": 1,
    |    "b": true,
    |    "str": "lol",
    |    "c": "Z",
    |    "t": {
    |        "myFieldA": 7,
    |        "myFieldB": ""
    |    }
    |}""".stripMargin

Sealed hierarchies are serialized as tagged values, the serialized object tagged with the full name of the instance's class:

sealed trait IntOrTuple
object IntOrTuple{
  implicit val rw: RW[IntOrTuple] = RW.merge(IntThing.rw, TupleThing.rw)
}
case class IntThing(i: Int) extends IntOrTuple
object IntThing{
  implicit val rw: RW[IntThing] = macroRW
}
case class TupleThing(name: String, t: (Int, Int)) extends IntOrTuple
object TupleThing{
  implicit val rw: RW[TupleThing] = macroRW
}
write(IntThing(1)) ==> """{"$type":"upickle.example.Sealed.IntThing","i":1}"""

write(TupleThing("naeem", (1, 2))) ==>
  """{"$type":"upickle.example.Sealed.TupleThing","name":"naeem","t":[1,2]}"""

// You can read tagged value without knowing its
// type in advance, just use type of the sealed trait
read[IntOrTuple]("""{"$type":"upickle.example.Sealed.IntThing","i":1}""") ==> IntThing(1)

Serializability is recursive; you can serialize a type only if all its members are serializable. That means that collections, tuples and case-classes made only of serializable members are themselves serializable

case class Foo(i: Int)
object Foo{
  implicit val rw: RW[Foo] = macroRW
}
case class Bar(name: String, foos: Seq[Foo])
object Bar{
  implicit val rw: RW[Bar] = macroRW
}
write((((1, 2), (3, 4)), ((5, 6), (7, 8)))) ==>
  """[[[1,2],[3,4]],[[5,6],[7,8]]]"""

write(Seq(Thing(1, "g"), Thing(2, "k"))) ==>
  """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]"""

write(Bar("bearrr", Seq(Foo(1), Foo(2), Foo(3)))) ==>
  """{"name":"bearrr","foos":[{"i":1},{"i":2},{"i":3}]}"""

Nulls serialize into JSON nulls, as you would expect

write(Bar(null, Seq(Foo(1), null, Foo(3)))) ==>
  """{"name":null,"foos":[{"i":1},null,{"i":3}]}"""

uPickle only throws exceptions on unpickling; if a pickler is properly defined, serializing a data structure to a String should never throw an exception.

All these examples can be similarly serialized to MessagePack-formatted binaries, in the same way: JSON booleans become MessagePack booleans, lists become MessagePack lists, and so on. Reading and writing MessagePack binary data is typically significantly faster than reading and writing JSON, and the serialized data is also significantly smaller.

Read/Writing Other Things

Apart from reading & writing java.lang.Strings, allows you to easily read from alternate sources such as CharSequences, Array[Byte]s, java.io.Files and java.nio.file.Paths:

import upickle.default._
val original = """{"myFieldA":1,"myFieldB":"gg"}"""
read[Thing](original) ==> Thing(1, "gg")
read[Thing](original: CharSequence) ==> Thing(1, "gg")
read[Thing](original.getBytes) ==> Thing(1, "gg")
import upickle.default._
val original = """{"myFieldA":1,"myFieldB":"gg"}"""

import java.nio.file.Files
val f = Files.createTempFile("", "")
Files.write(f, original.getBytes)

read[Thing](f) ==> Thing(1, "gg")
read[Thing](f.toFile) ==> Thing(1, "gg")
read[Thing](Files.newByteChannel(f)) ==> Thing(1, "gg")

Reading from large files is automatically streamed so you do not read the entire file into memory. You can use writeTo to serialize your data to an arbitrary java.io.Writer/java.io.OutputStream: this can be streamed directly to files or over the network without having to accumulate the serialized JSON in memory.

Defaults

If a field is missing upon deserialization, uPickle uses the default value if one exists

read[FooDefault]("{}")                ==> FooDefault(10, "lol")
read[FooDefault]("""{"i": 123}""")    ==> FooDefault(123,"lol")

If a field at serialization time has the same value as the default, uPickle leaves it out of the serialized blob

write(FooDefault(i = 11, s = "lol"))  ==> """{"i":11}"""
write(FooDefault(i = 10, s = "lol"))  ==> """{}"""
write(FooDefault())                   ==> """{}"""

This allows you to make schema changes gradually, assuming you have already pickled some data and want to add new fields to the case classes you pickled. Simply give the new fields a default value (e.g. "" for Strings, or wrap it in an Option[T] and make the default None) and uPickle will happily read the old data, filling in the missing field using the default value.

Supported Types

Out of the box, uPickle supports writing and reading the following types:

Readability/writability is recursive: a container such as a Tuple or case class is only readable if all its contents are readable, and only writable if all its contents are writable. That means that you cannot serialize a List[Any], since uPickle doesn't provide a generic way of serializing Any. Case classes are only serializable up to 22 fields.

Case classes are serialized using the apply and unapply methods on their companion objects. This means that you can make your own classes serializable by giving them companions apply and unapply. sealed hierarchies are serialized as tagged unions: whatever the serialization of the actual object, together with the fully-qualified name of its class, so the correct class in the sealed hierarchy can be reconstituted later.

That concludes the list of supported types. Anything else is not supported by default, but you can add support using Custom Picklers

Common Operations

The following common operations are available on any uPickle module, e.g. upickle.default or upickle.legacy:

extends upickle.core.Types
with implicits.Readers
with implicits.Writers
with WebJson
with Api.NoOpMappers
with JsReadWriters
with MsgReadWriters{
*
* Reads the given MessagePack input into a Scala value
*/
f readBinary[T: Reader](s: upack.Readable): T = s.transform(reader[T])

*
* Reads the given JSON input into a Scala value
*/
f read[T: Reader](s: ujson.Readable): T = s.transform(reader[T])

f reader[T: Reader] = implicitly[Reader[T]]

*
* Write the given Scala value as a JSON string
*/
f write[T: Writer](t: T,
                   indent: Int = -1,
                   escapeUnicode: Boolean = false): String = {
transform(t).to(ujson.StringRenderer(indent, escapeUnicode)).toString

*
* Write the given Scala value as a MessagePack binary
*/
f writeBinary[T: Writer](t: T): Array[Byte] = {
transform(t).to(new upack.MsgPackWriter(new ByteArrayOutputStream())).toByteArray


*
* Write the given Scala value as a JSON struct
*/
f writeJs[T: Writer](t: T): ujson.Value = transform(t).to[ujson.Value]

*
* Write the given Scala value as a MessagePack struct
*/
f writeMsg[T: Writer](t: T): upack.Msg = transform(t).to[upack.Msg]

*
* Write the given Scala value as a JSON string to the given Writer
*/
f writeTo[T: Writer](t: T,
                     out: java.io.Writer,
                     indent: Int = -1,
                     escapeUnicode: Boolean = false): Unit = {
transform(t).to(new ujson.Renderer(out, indent = indent, escapeUnicode))

*
* Write the given Scala value as a MessagePack binary to the given OutputStream
*/
f writeBinaryTo[T: Writer](t: T, out: java.io.OutputStream): Unit = {
transform(t).to(new upack.MsgPackWriter(out))


f writer[T: Writer] = implicitly[Writer[T]]

f readwriter[T: ReadWriter] = implicitly[ReadWriter[T]]

se class transform[T: Writer](t: T) extends upack.Readable with ujson.Readable {
def transform[V](f: Visitor[_, V]): V = writer[T].transform(t, f)
def to[V](f: Visitor[_, V]): V = transform(f)
def to[V](implicit f: Reader[V]): V = transform(f)

Customization


Custom Picklers

import upickle.default._
case class Wrap(i: Int)
implicit val fooReadWrite: ReadWriter[Wrap] =
  readwriter[Int].bimap[Wrap](_.i, Wrap(_))

write(Seq(Wrap(1), Wrap(10), Wrap(100))) ==> "[1,10,100]"
read[Seq[Wrap]]("[1,10,100]") ==> Seq(Wrap(1), Wrap(10), Wrap(100))

You can use the readwriter[T].bimap[V] function to create a pickler that reads/writes a type V, using the pickler for type T, by providing a conversion function between them.

The type you are .bimaping to doesn't need to be a case class, or be pickleable in any way, as long as the type you are .bimaping from is pickleable. The following example demonstrates using .bimap to define a serializer for non-case Scala class

class CustomThing2(val i: Int, val s: String)
object CustomThing2 {
  implicit val rw = upickle.default.readwriter[String].bimap[CustomThing2](
    x => x.i + " " + x.s,
    str => {
      val Array(i, s) = str.split(" ", 2)
      new CustomThing2(i.toInt, s)
    }
  )
}

Note that when writing custom picklers, it is entirely up to you to get it right, e.g. making sure that an object that gets round-trip pickled/unpickled comes out the same as when it started.

Lastly, if you want more control over exactly how something is serialized, you can use readwriter[Js.Value].bimap to give yourself access to the raw JSON AST:

import upickle.default._
case class Bar(i: Int, s: String)
implicit val fooReadWrite: ReadWriter[Bar] =
  readwriter[ujson.Value].bimap[Bar](
    x => ujson.Arr(x.s, x.i),
    json => new Bar(json(1).num.toInt, json(0).str)
  )

write(Bar(123, "abc")) ==> """["abc",123]"""
read[Bar]("""["abc",123]""") ==> Bar(123, "abc")

Custom Keys

uPickle allows you to specify the key that a field is serialized with via a @key annotation

case class KeyBar(@upickle.implicits.key("hehehe") kekeke: Int)
object KeyBar{
  implicit val rw: RW[KeyBar] = macroRW
}
write(KeyBar(10))                     ==> """{"hehehe":10}"""
read[KeyBar]("""{"hehehe": 10}""")    ==> KeyBar(10)

Practically, this is useful if you want to rename the field within your Scala code while still maintaining backwards compatibility with previously-pickled objects. Simple rename the field and add a @key("...") with the old name so uPickle can continue to work with the old objects correctly.

You can also use @key to change the name used when pickling the case class itself. Normally case classes are pickled without their name, but an exception is made for members of sealed hierarchies which are tagged with their fully-qualified name. uPickle allows you to use @key to override what the class is tagged with:

sealed trait A
object A{
  implicit val rw: RW[A] = RW.merge(B.rw, macroRW[C.type])
}
@upickle.implicits.key("Bee") case class B(i: Int) extends A
object B{
  implicit val rw: RW[B] = macroRW
}
case object C extends A
write(B(10))                          ==> """{"$type":"Bee","i":10}"""
read[B]("""{"$type":"Bee","i":10}""") ==> B(10)

This is useful in cases where:

Custom Configuration

Often, there will be times that you want to customize something on a project-wide level. uPickle provides hooks in letting you subclass the upickle.Api trait to create your own bundles apart from the in-built upickle.default and upickle.legacy. The following example demonstrates how to customize a bundle to automatically snake_case all dictionary keys.

object SnakePickle extends upickle.AttributeTagged{
  def camelToSnake(s: String) = {
    s.split("(?=[A-Z])", -1).map(_.toLowerCase).mkString("_")
  }
  def snakeToCamel(s: String) = {
    val res = s.split("_", -1).map(x => x(0).toUpper + x.drop(1)).mkString
    s(0).toLower + res.drop(1)
  }

  override def objectAttributeKeyReadMap(s: CharSequence) =
    snakeToCamel(s.toString)
  override def objectAttributeKeyWriteMap(s: CharSequence) =
    camelToSnake(s.toString)

  override def objectTypeKeyReadMap(s: CharSequence) =
    snakeToCamel(s.toString)
  override def objectTypeKeyWriteMap(s: CharSequence) =
    camelToSnake(s.toString)
}

// Default read-writing
upickle.default.write(Thing(1, "gg")) ==>
  """{"myFieldA":1,"myFieldB":"gg"}"""

upickle.default.read[Thing]("""{"myFieldA":1,"myFieldB":"gg"}""") ==>
  Thing(1, "gg")

implicit def thingRW: SnakePickle.ReadWriter[Thing] = SnakePickle.macroRW

// snake_case_keys read-writing
SnakePickle.write(Thing(1, "gg")) ==>
  """{"my_field_a":1,"my_field_b":"gg"}"""

SnakePickle.read[Thing]("""{"my_field_a":1,"my_field_b":"gg"}""") ==>
  Thing(1, "gg")

If you are using uPickle to convert JSON from another source into Scala data structures, you can also configure it to map Option[T]s to nulls when the option is None:

object OptionPickler extends upickle.AttributeTagged {
  override implicit def OptionWriter[T: Writer]: Writer[Option[T]] =
    implicitly[Writer[T]].comap[Option[T]] {
      case None => null.asInstanceOf[T]
      case Some(x) => x
    }

  override implicit def OptionReader[T: Reader]: Reader[Option[T]] =
    implicitly[Reader[T]].mapNulls{
      case null => None
      case x => Some(x)
    }
}

This custom configuration allows you to treat nulls as Nones and anything else as Some(...)s. Simply import OptionPickler._ instead of the normal uPickle import throughout your project and you'll have the customized reading/writing available to you.

You can also use a custom configuration to change how 64-bit Longs are handled. By default, small longs that can be represented exactly in 64-bit Doubles are written as raw numbers, while larger values (n > 2^53) are written as strings. This is to ensure the values are not truncated when the serialized JSON is then manipulated, e.g. by Javascript which truncates all large numbers to Doubles. If you wish to always write Longs as Strings, or always write them as numbers (at risk of truncation), you can do so as follows:

upickle.default.write(123: Long) ==> "123"
upickle.default.write(Long.MaxValue) ==> "\"9223372036854775807\""

object StringLongs extends upickle.AttributeTagged{
  override implicit val LongWriter = new Writer[Long] {
    def write0[V](out: Visitor[_, V], v: Long) = out.visitString(v.toString, -1)
  }
}

StringLongs.write(123: Long) ==> "\"123\""
StringLongs.write(Long.MaxValue) ==> "\"9223372036854775807\""

object NumLongs extends upickle.AttributeTagged{
  override implicit val LongWriter = new Writer[Long] {
    def write0[V](out: Visitor[_, V], v: Long) = out.visitFloat64String(v.toString, -1)
  }
}

NumLongs.write(123: Long) ==> "123"
NumLongs.write(Long.MaxValue) ==> "9223372036854775807"

Limitations


uPickle doesn't currently support:

Most of these limitations are inherent in the fact that ScalaJS does not support reflection, and are unlikely to ever go away. In general, uPickle by default can serialize statically-typed, tree-shaped, immutable data structures. Anything more complex requires Custom Picklers

Manual Sealed Trait Picklers

Due to a bug in the Scala compiler SI-7046, automatic sealed trait pickling can fail unpredictably. This can be worked around by instead using the macroRW and merge methods to manually specify which sub-types of a sealed trait to consider when pickling:

sealed trait TypedFoo
object TypedFoo{
  import upickle.default._
  implicit val readWriter: ReadWriter[TypedFoo] = ReadWriter.merge(
    macroRW[Bar], macroRW[Baz], macroRW[Quz]
  )

  case class Bar(i: Int) extends TypedFoo
  case class Baz(s: String) extends TypedFoo
  case class Quz(b: Boolean) extends TypedFoo
}

uJson


uJson is uPickle's JSON library, which can be used to easily manipulate JSON source and data structures without converting them into Scala case-classes. This all lives in the ujson package. Unlike many other Scala JSON libraries that come with their own zoo of new concepts, abstractions, and techniques, uJson has a simple & predictable JSON API that should be instantly familiar to anyone coming from scripting languages like Ruby, Python or Javascript.

uJson comes bundled with uPickle, or can be used stand-alone via the following package coordinates:

libraryDependencies += "com.lihaoyi" %% "ujson" % "0.7.1"

Construction

You can use ujson to conveniently construct JSON blobs, either programmatically:

import ujson.Js

val json0 = ujson.Arr(
  ujson.Obj("myFieldA" -> ujson.Num(1), "myFieldB" -> ujson.Str("g")),
  ujson.Obj("myFieldA" -> ujson.Num(2), "myFieldB" -> ujson.Str("k"))
)

val json = ujson.Arr( // The `ujson.Num` and `ujson.Str` calls are optional
  ujson.Obj("myFieldA" -> 1, "myFieldB" -> "g"),
  ujson.Obj("myFieldA" -> 2, "myFieldB" -> "k")
)

json0 ==> json

Or parsing them from strings, byte arrays or files:

val str = """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]"""
val json = ujson.read(str)
json(0)("myFieldA").num   ==> 1
json(0)("myFieldB").str   ==> "g"
json(1)("myFieldA").num   ==> 2
json(1)("myFieldB").str   ==> "k"

ujson.write(json)         ==> str

ujson.Js ASTs are mutable, and can be modified before being re-serialized to strings:

val str = """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]"""
val json: ujson.Js = ujson.read(str)

json.arr.remove(1)
json(0)("myFieldA") = 1337
json(0)("myFieldB") = json(0)("myFieldB").str + "lols"

You can also use the `_` shorthand syntax to update a JSON value in place, without having to duplicate the whole path:

val str = """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]"""
val json: ujson.Js = ujson.read(str)

json(0)("myFieldA") = _.num + 100
json(1)("myFieldB") = _.str + "lol"

case classes or other Scala data structures can be converted to ujson.Js ASTs using upickle.default.writeJs, and the ujson.Js ASTs can be converted back using upickle.default.readJs or plain upickle.default.read:

val data = Seq(Thing(1, "g"), Thing(2, "k"))
val json = upickle.default.writeJs(data)

json.arr.remove(1)
json(0)("myFieldA") = 1337

upickle.default.read[Seq[Thing]](json)   ==> Seq(Thing(1337, "g"))

JSON Utilities

uJson comes with some convenient utilities on the `ujson` package:

def transform[T](t: Readable, v: upickle.core.Visitor[_, T]) = t.transform(v)

/**
  * Read the given JSON input as a JSON struct
  */
def read(s: Readable): Value.Value = transform(s, Value)

def copy(t: Value.Value): Value.Value = transform(t, Value)

/**
  * Write the given JSON struct as a JSON String
  */
def write(t: Value.Value,
          indent: Int = -1,
          escapeUnicode: Boolean = false): String = {
  transform(t, StringRenderer(indent, escapeUnicode)).toString
}

/**
  * Write the given JSON struct as a JSON String to the given Writer
  */
def writeTo(t: Value.Value,
            out: java.io.Writer,
            indent: Int = -1,
            escapeUnicode: Boolean = false): Unit = {
  transform(t, Renderer(out, indent, escapeUnicode))
}

/**
  * Parse the given JSON input, failing if it is invalid
  */
def validate(s: Readable): Unit = transform(s, NoOpVisitor)
/**
  * Parse the given JSON input and write it to a string with
  * the configured formatting
  */
def reformat(s: Readable, indent: Int = -1, escapeUnicode: Boolean = false): String = {
  transform(s, StringRenderer(indent, escapeUnicode)).toString
}
/**
  * Parse the given JSON input and write it to a string with
  * the configured formatting to the given Writer
  */
def reformatTo(s: Readable, out: java.io.Writer, indent: Int = -1, escapeUnicode: Boolean = false): Unit = {
  transform(s, Renderer(out, indent, escapeUnicode)).toString
}

Transformations

uJson allows you seamlessly convert between any of the following forms you may find your JSON in:

This is done using the ujson.transform(source, dest) function:

// It can be used for parsing JSON into an AST
val exampleAst = ujson.Arr(1, 2, 3)

ujson.transform("[1, 2, 3]", Js) ==> exampleAst

// Rendering the AST to a string
ujson.transform(exampleAst, StringRenderer()).toString ==> "[1,2,3]"

// Or to a byte array
ujson.transform(exampleAst, BytesRenderer()).toBytes ==> "[1,2,3]".getBytes

// Re-formatting JSON, either compacting it
ujson.transform("[1, 2, 3]", StringRenderer()).toString ==> "[1,2,3]"

// or indenting it
ujson.transform("[1, 2, 3]", StringRenderer(indent = 4)).toString ==>
  """[
    |    1,
    |    2,
    |    3
    |]""".stripMargin

// `transform` takes any `Transformable`, including byte arrays and files
ujson.transform("[1, 2, 3]".getBytes, StringRenderer()).toString ==> "[1,2,3]"

All transformations from A to B using ujson.transform happen in a direct fashion: there are no intermediate JSON ASTs being generated, and performance is generally very good.

You can use ujson.transform to validate JSON in a streaming fashion:

ujson.transform("[1, 2, 3]", NoOpVisitor)

intercept[IncompleteParseException](
  ujson.transform("[1, 2, 3", NoOpVisitor)
)
intercept[ParseException](
  ujson.transform("[1, 2, 3]]", NoOpVisitor)
)

The normal upickle.default.read/write methods to serialize Scala data-types is just shorthand for ujson.transform, using a upickle.default.transform(foo) as the source or a upickle.default.reader[Foo] as the destination:

ujson.transform("[1, 2, 3]", upickle.default.reader[Seq[Int]]) ==>
  Seq(1, 2, 3)

ujson.transform(upickle.default.transform(Seq(1, 2, 3)), StringRenderer()).toString ==>
  "[1,2,3]"

Other ASTs

uJson does not provide any other utilities are JSON that other libraries do: zippers, lenses, combinators, etc.. However, uJson can be used to seamlessly convert between the JSON AST of other libraries! This means if some other library provides a more convenient API for some kind of processing you need to do, you can easily parse to that library's AST, do whatever you need, and convert back after.

As mentioned earlier, conversions are fast and direct, and happen without creating intermediate JSON structures in the process. The following examples demonstrate how to use the conversion modules for Argonaut, Circe, Json4s, and Play Json.

Each example parses JSON from a string into that particular library's JSON AST, manipulates the AST using that library, un-pickles it into Scala data types, then serializes those data types first into that library's AST then back to a st.ring

Argonaut

Maven Coordinates
libraryDependencies += "com.lihaoyi" %% "ujson-argonaut" % "0.7.1"
Usage
import ujson.argonaut.ArgonautJson
val argJson: argonaut.Json = ArgonautJson(
  """["hello", "world"]"""
)

val updatedArgJson = argJson.withArray(_.map(_.withString(_.toUpperCase)))

val items: Seq[String] = ArgonautJson.transform(
  updatedArgJson,
  upickle.default.reader[Seq[String]]
)

items ==> Seq("HELLO", "WORLD")

val rewritten = upickle.default.transform(items).to(ArgonautJson)

val stringified = ArgonautJson.transform(rewritten, StringRenderer()).toString

stringified ==> """["HELLO","WORLD"]"""

Circe

Maven Coordinates
libraryDependencies += "com.lihaoyi" %% "ujson-circe" % "0.7.1"
Usage
import ujson.circe.CirceJson
val circeJson: io.circe.Json = CirceJson(
  """["hello", "world"]"""
)

val updatedCirceJson =
  circeJson.mapArray(_.map(x => x.mapString(_.toUpperCase)))

val items: Seq[String] = CirceJson.transform(
  updatedCirceJson,
  upickle.default.reader[Seq[String]]
)

items ==> Seq("HELLO", "WORLD")

val rewritten = upickle.default.transform(items).to(CirceJson)

val stringified = CirceJson.transform(rewritten, StringRenderer()).toString

stringified ==> """["HELLO","WORLD"]"""

Play-Json

Maven Coordinates
libraryDependencies += "com.lihaoyi" %% "ujson-play" % "0.7.1"
Usage
import ujson.play.PlayJson
import play.api.libs.json._
val playJson: play.api.libs.json.JsValue = PlayJson(
  """["hello", "world"]"""
)

val updatedPlayJson = JsArray(
  for(v <- playJson.as[JsArray].value)
    yield JsString(v.as[String].toUpperCase())
)

val items: Seq[String] = PlayJson.transform(
  updatedPlayJson,
  upickle.default.reader[Seq[String]]
)

items ==> Seq("HELLO", "WORLD")

val rewritten = upickle.default.transform(items).to(PlayJson)

val stringified = PlayJson.transform(rewritten, StringRenderer()).toString

stringified ==> """["HELLO","WORLD"]"""

Json4s

Maven Coordinates
libraryDependencies += "com.lihaoyi" %% "ujson-json4s" % "0.7.1"
Usage
import org.json4s.JsonAST
val json4sJson: JsonAST.JValue = Json4sJson(
  """["hello", "world"]"""
)

val updatedJson4sJson = JsonAST.JArray(
  for(v <- json4sJson.children)
    yield JsonAST.JString(v.values.toString.toUpperCase())
)

val items: Seq[String] = Json4sJson.transform(
  updatedJson4sJson,
  upickle.default.reader[Seq[String]]
)

items ==> Seq("HELLO", "WORLD")

val rewritten = upickle.default.transform(items).to(Json4sJson)

val stringified = Json4sJson.transform(rewritten, StringRenderer()).toString

stringified ==> """["HELLO","WORLD"]"""

Cross-Library Conversions

uJson lets you convert between third-party ASTs efficiently and with minimal overhead: uJson converts one AST to the other directly and without any temporary compatibility data structures. The following example demonstrates how this is done: we parse a JSON string using Circe, perform some transformation, convert it to a Play-Json AST, perform more transformations, and finally serialize it back to a String and check that both transformations were applied:

import ujson.circe.CirceJson
val circeJson: io.circe.Json = CirceJson(
  """["hello", "world"]"""
)

val updatedCirceJson =
  circeJson.mapArray(_.map(x => x.mapString(_.toUpperCase)))

import ujson.play.PlayJson
import play.api.libs.json._

val playJson: play.api.libs.json.JsValue = CirceJson.transform(
  updatedCirceJson,
  PlayJson
)

val updatedPlayJson = JsArray(
  for(v <- playJson.as[JsArray].value)
    yield JsString(v.as[String].reverse)
)

val stringified = PlayJson.transform(updatedPlayJson, StringRenderer()).toString

stringified ==> """["OLLEH","DLROW"]"""

uPack


uPack is uPickle's MessagePack library, which can be used to easily manipulate MessagePack source and data structures without converting them into Scala case-classes. This all lives in the upack package.

uJson comes bundled with uPickle, or can be used stand-alone via the following package coordinates:

libraryDependencies += "com.lihaoyi" %% "upack" % "0.7.1"

The following basic functions are provided in the upack package to let you read and write MessagePack structs:

def transform[T](t: Readable, v: upickle.core.Visitor[_, T]) = t.transform(v)

/**
  * Read the given MessagePack input into a MessagePack struct
  */
def read(s: Readable): Msg = transform(s, Msg)

def copy(t: Msg): Msg = transform(t, Msg)
/**
  * Write the given MessagePack struct as a binary
  */
def write(t: Msg): Array[Byte] = {
  transform(t, new MsgPackWriter()).toByteArray
}
/**
  * Write the given MessagePack struct as a binary to the given OutputStream
  */
def writeTo(t: Msg, out: java.io.OutputStream): Unit = {
  transform(t, new MsgPackWriter(out))
}
/**
  * Parse the given MessagePack input, failing if it is invalid
  */
def validate(s: Readable): Unit = transform(s, NoOpVisitor)

MessagePack structs are represented using the upack.Msg type. You can construct ad-hoc MessagePack structs using upack.Msg, and can similarly parse binary data into upack.Msg for ad-hoc querying and manipulation, without needing to bind it to Scala case classes or data types:

val msg = upack.Arr(
  upack.Obj(upack.Str("myFieldA") -> upack.Int32(1), upack.Str("myFieldB") -> upack.Str("g")),
  upack.Obj(upack.Str("myFieldA") -> upack.Int32(2), upack.Str("myFieldB") -> upack.Str("k"))
)

val binary: Array[Byte] = upack.write(msg)

val read = upack.read(binary)
assert(msg == read)

You can read/write Scala values to upack.Msgs using readBinary/writeMsg:

val big = Big(1, true, "lol", 'Z', Thing(7, ""))
val msg: upack.Msg = upickle.default.writeMsg(big)
upickle.default.readBinary[Big](msg) ==> big

Or include upack.Msgs inside Seqs, case-classes and other data structures when you read/write them:

val msgSeq = Seq[upack.Msg](
  upack.Str("hello world"),
  upack.Arr(upack.Int32(1), upack.Int32(2))
)

val binary: Array[Byte] = upickle.default.writeBinary(msgSeq)

upickle.default.readBinary[Seq[upack.Msg]](binary) ==> msgSeq

You can also convert the uPack messages or binaries to ujson.Values via upack.transform. This can be handy to help debug what's going on in your binary message data:

val msg = upack.Arr(
  upack.Obj(upack.Str("myFieldA") -> upack.Int32(1), upack.Str("myFieldB") -> upack.Str("g")),
  upack.Obj(upack.Str("myFieldA") -> upack.Int32(2), upack.Str("myFieldB") -> upack.Str("k"))
)

val binary: Array[Byte] = upack.write(msg)

// Can pretty-print starting from either the upack.Msg structs,
// or the raw binary data
upack.transform(msg, new ujson.StringRenderer()).toString ==>
  """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]"""

upack.transform(binary, new ujson.StringRenderer()).toString ==>
  """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]"""

// Some messagepack structs cannot be converted to valid JSON, e.g.
// they may have maps with non-string keys. These can still be pretty-printed:
val msg2 = upack.Obj(upack.Arr(upack.Int32(1), upack.Int32(2)) -> upack.Int32(1))
upack.transform(msg2, new ujson.StringRenderer()).toString ==> """{[1,2]:1}"""

Note that such a conversion between MessagePack structs and JSON data is lossy: some MessagePack constructs, such as binary data, cannot be exactly represented in JSON and have to be converted to strings. Thus you should not rely on being able to round-trip data between JSON <-> MessagePack and getting the same thing back, although round tripping data between Scala-data-types <-> JSON and Scala-data-types <-> MessagePack should always work.

Some of the differences between the ways things are serialized in MessagePack and JSON include:

If you need to construct Scala case classes or other data types from your MessagePack binary data, you should directly use upickle.default.readBinary and upickle.default.writeBinary: these bypass the upack.Msg struct entirely for the optimal performance.

Performance


The uPickle has a small set of benchmarks in bench/ that tests reading and writing performance of a few common JSON libraries on a small, somewhat arbitrary workload. The numbers below show how many times each library could read/write a small data structure in 25 seconds (bigger numbers better). In some libraries, caching the serializers rather than re-generating them each read/write also improves performance: that effect can be seen in the (Cached) columns.

JVM Case Class Serialization Performance

uPickle runs 30-50% faster than Circe for reads/writes, and ~200% faster than play-json.

Library Reads Writes Reads (Cached) Write (Cached)
Jackson-Scala 2,080,682 8,905,996
Play Json 1,123,923 1,518,832 1,360,201 1,763,854
Circe 2,172,638 2,057,883 2,519,163 2,416,923
upickle.default 3,078,442 4,018,176 3,564,215 4,749,887
upickle.default binary 4,907,232 6,812,322 6,332,698 9,367,948

As you can see, uPickle's JSON serialization is pretty consistently ~30% faster than Circe for reads and ~50% faster for writes. It is also faster than Jackson-Scala on reads by a similar margin, although somehow Jackson-Scala has write performance that far outstrips everyone else. uPickle JSON is 100-200% faster than Play-Json, depending on workload.

uPickle's binary MessagePack backend is then another 50-100% than uPickle JSON.

uPickle achieves this speed by avoiding the construction of an intermediate JSON AST: while most libraries parse from String -> AST -> CaseClass, uPickle parses input directly from String -> CaseClass. uPickle also provides a ujson.Js AST that you can use to manipulate arbitrary JSON, but ujson.Js plays no part in parsing things to case-classes and is purely for users who want to manipulate JSON.

JVM AST Serialization Performance

If you benchmark the various libraries parsing from String -> AST, instead of String -> CaseClass, uPickle's numbers are in line with everyone else's:

Library AST Reads AST Writes
uJson 2,809,313 4,846,925
uPack (binary) 5,927,786 13,218,990
Play Json 3,372,847 3,291,835
Circe 5,744,314 5,669,830
Argonaut 2,189,890 2,699,175
Json4s Native 3,459,126 1,302,335

One thing to note is that uPickle's AST-serialization performance isn't any better than uPickle's CaseClass-serialization performance. This may seem surprising, but it makes sense when you realize that when serializing case classes, uPickle goes straight from the case class to the output string without converting to the AST in between

JS Case Class Serialization Performance

While all libraries are much slower on Scala.js/Node.js than on the JVM, uPickle runs 4-5x as fast as Circe or Play-Json for reads and writes.

Library Reads Writes Reads (Cached) Write (Cached)
Play Json 117,808 201,701 148,964 268,816
Circe 136,181 461,873 160,860 678,861
upickle.default 666,626 970,840 931,945 1,462,505
upickle.default.web 581,024 970,332 931,505 1,734,548
upickle.default binary 601,333 497,954 931,505 1,734,548

Apart from the huge difference between uPickle and it's alternatives, this Scala.js/Node.js benchmark also shows a much larger effect for caching the picklers: 80-100% speedup. It seems likely that Scala.js/Node.js just isn't as good at optimizing away the overhead of temporary/throwaway data-structures, whether they're intermediate-ASTs or temporary-pickler-instances.

Another thing of note is the .web entries; these are benchmarking uPickle using the Node.js JSON.parse and JSON.stringify builtin functions, rather than our own parser. It turns out that these builtins appear to be slower than our parser for this benchmark, but they're available as the upickle.default.web.read/write functions or the upickle.WebJson object if you for some reason want to use them.

In Scala.js, uPickle's binary MessagePack backend isn't any faster than upickle JSON. This makes sense when you consider that Scala.js isn't optimized for handling binary data.

uJson is a fork of Erik Osheim's excellent [Jawn](https://github.com/non/jawn) JSON library, and inherits a lot of it's performance from Erik's work.

Version History


0.7.1

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.0

0.5.1

0.4.3

0.4.1

0.4.1

0.4.0

0.3.9

0.3.8

0.3.7

0.3.6

0.3.5

0.3.4

0.3.3

0.3.2

0.3.1

0.3.0

0.2.8

0.2.7

0.2.6

0.2.5

0.2.4

0.2.3

0.2.2

0.2.1

0.2.0

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3