Have you ever got bored writing Kotlin classes which serialise a JSON payload?

Me too. Fortunately, we’re going to write a Kotlin library that automates the whole process, using Square’s awesome KotlinPoet.

We’re also going to create a command-line tool and Spring Boot application, then deploy both to production.

Creating the core Kotlin Library

We’ll start by generating an empty Kotlin project with a module named core.

This module will contain all the conversion code, so we should define an API that encapsulates most of the gory details. Firstly, we’ll need a method that converts a JSON InputStream to an OutputStream:

fun convert(input: InputStream, output: OutputStream, args: ConversionArgs)

Several JSON serialisation libraries require annotations like @SerializedName. We’ll provide callbacks via a delegate for each time a class or property is added, which will allow us to tack on annotations after the source has been generated:

class Kotlin2JsonConverter(
    private val buildDelegate: SourceBuildDelegate = GsonBuildDelegate()
)

For the Spring Boot and command-line applications, we’ll create separate modules which both depend on the core module. More on that later.

JSON Conversion Algorithms

JSON is a tree, where each array and object node could contain child nodes.

Our algorithm will start at the bottom of the tree, and work its way up to the top, one level at a time. For each level, we should group similar objects together, and build a Kotlin type representation for each grouping.

A conversion example

That all sounds very abstract. Consider the following JSON:

{
  "obj": {
    "foo": "a string value"
  },
  "another": {
    "foo": "another string value"
  },
  "primitive": true
}

We can clearly see that both objects near the bottom of the tree have a key of foo, and a type of String:

{
  "foo": "a string value"
},
{
  "foo": "another string value"
}

Therefore we’ll group them together, and represent them with a single Kotlin type:

data class Obj(val foo: String)

When we go up a level, we can immediately see that there are two fields of type Obj, and one primitive field of type Boolean:

{
  "obj": Obj,
  "another": Obj,
  "primitive": Boolean
}

And as we’ve reached the root node, we’ve finished converting our JSON to Kotlin, and can write the results to our OutputStream:

data class Example(val obj: Obj, val another: Obj, val primitive: Boolean)
data class Obj(val foo: String)

JSON is simple, right?

Of course, it’s not always as easy as our example. JSON is deceptively complex, and there are a surprising number of scenarios which we’ll need to address. For example, here are just a few to think about:

Sounds hard! We’d better write some tests.

Parameterised Unit Tests for Source Code generation

We want to verify that given a JSON input, the correct Kotlin output is generated. As we’re going to check many different JSON structures, this is a prime candidate for a JUnit parameterised test.

Here’s a simplified version of our test, which parameterises two filenames:

@RunWith(Parameterized::class)
class JsonConverterTest(val expectedFilename: String,
                        val jsonFilename: String) {

    companion object {
        @JvmStatic
        @Parameterized.Parameters
        fun filenamePairs(): Collection<Array<String>> {
            return listOf(arrayOf("HelloWorld.kt", "hello_world.json"))
        }
    }
}

JUnit will run the test once for each parameter pair. Therefore, all we need to do is compare the generated source code against the expected source code, and supply filenames:

@Test
fun testJsonToKotlinConversion() {
    val outputStream = ByteArrayOutputStream()
    jsonConverter.convert(json, outputStream, ConversionArgs())
    val generatedSource = String(outputStream.toByteArray())
    val expectedContents = fileReader.readContents(expectedFilename)
    Assert.assertEquals(msg, expectedContents)
}

And then our test case will verify that a JSON input matches an output similar to this:

{"foo":"Hello World!"} // "Example.json"
data class Example(val foo: String) "Example.kt"

This assumes that a JSON and Kotlin file are present in src/test/resources, and that the file contents can be read via the ClassLoader:

val classLoader = ResourceFileReader::class.java.classLoader
val inputStream = classLoader.getResourceAsStream("HelloWorld.kt")

The full test takes this even further, by recursively detecting JSON/Kotlin files within the resources directory. This means we can just add a pair of JSON and Kotlin files, and we’ll automatically have a test case.

The latest test suite is available on the Github repository.

Implementing the Json2Kotlin converter

Before we look at our approach in more detail, let’s summarise the steps:

  1. Sanitise input and generate a JSON tree using GSON.
  2. Use breadth-first search to push each JSON node onto a Stack, along with some additional metadata.
  3. Traverse in reverse level order and generate type information, group common objects, then reduce common objects to a single type types.
  4. Pop the stack of generated type information and write it to an OutputStream.

JSON Tree traversal

We’ll start by implementing a breadth-first search that pushes non-primitive JSON nodes to a stack.

private fun buildStack(bfsStack: Stack<TypedJsonElement>,
                       parent: JsonElement,
                       key: String?) {
    val queue = LinkedList<TypedJsonElement>()
    queue.add(TypedJsonElement(parent, key!!, 0))

    while (queue.isNotEmpty()) {
        val element = queue.poll()
        bfsStack.push(element)

        val complexChildren = with(element) {
            when {
                isJsonObject -> convertParent(asJsonObject, level + 1)
                isJsonArray -> convertParent(asJsonArray, jsonKey, level + 1)
                else -> Collections.emptyList()
            }
        }
        queue.addAll(complexChildren)
    }
}

We’ll then pop from the stack and process JSON nodes. For each level, we’ll generate a TypeSpec for each distinct object grouping.

fun generateTypeSpecs(bfsStack: Stack<TypedJsonElement>): Stack<TypeSpec> {
    val typeSpecs = Stack<TypeSpec>()
    var level = -1
    val levelQueue = LinkedList<TypedJsonElement>()

    while (bfsStack.isNotEmpty()) {
        val pop = bfsStack.pop()

        if (level != -1 && pop.level != level) {
            processTreeLevel(levelQueue, typeSpecs)
        }
        levelQueue.add(pop)
        level = pop.level
    }
    processTreeLevel(levelQueue, typeSpecs)
    return typeSpecs
}

Grouping common objects

Our grouping strategy will be very simple. For objects to belong to the same type, they must share 15 or more of the same keys. The only exception to this will be an empty class, which will be considered to share 1 key automatically.

// builds a list of common objects, implementation omitted
fun groupCommonJsonObjects
(jsonElements: MutableList<TypedJsonElement>): List<List<TypedJsonElement>>

internal fun defaultGroupingStrategy(lhs: TypedJsonElement,
                                     rhs: TypedJsonElement): Boolean {
    val lhsKeys = lhs.asJsonObject.keySet()
    val rhsKeys = rhs.asJsonObject.keySet()
    val lhsSize = lhsKeys.size
    val rhsSize = rhsKeys.size
    val emptyClasses = (lhsKeys.isEmpty() || rhsKeys.isEmpty())

    val maxKeySize = if (lhsSize > rhsSize) lhsSize else rhsSize
    val commonKeyCount = if (emptyClasses) {
        1
    } else  {
        lhsKeys.intersect(rhsKeys).size
    }

    // at least a fifth of keys must match
    return (commonKeyCount * 5) >= maxKeySize
}

We’ll also make it easy to alter the grouping strategy at a later date, by accepting a function reference as a constructor parameter:

// typealias used to make method signature more human-readable
typealias GroupingStrategy =
 (lhs: TypedJsonElement, rhs: TypedJsonElement) -> Boolean
class JsonFieldGrouper(private val strategy: GroupingStrategy = ::myStrategy)

Generating KotlinPoet TypeSpecs

Now that our objects are grouped, we can generate type representations using KotlinPoet. We’ll start by building a TypeSpec:

private fun buildClass(commonElements: List<TypedJsonElement>,
                       fields: Collection<String>): TypeSpec.Builder {

    val identifier = commonElements.last().kotlinIdentifier
    val classBuilder = TypeSpec.classBuilder(identifier.capitalize())
    val constructor = FunSpec.constructorBuilder()

    if (fields.isNotEmpty()) {
        val fieldTypeMap = typeReducer.findDistinctTypes(fields,
                                                         commonElements,
                                                         jsonElementMap)
        fields.forEach {
            buildProperty(it, fieldTypeMap,
                          commonElements, classBuilder, constructor)
        }
        classBuilder.addModifiers(KModifier.DATA)
        classBuilder.primaryConstructor(constructor.build())
    }

    delegate.prepareClass(classBuilder, commonElements.last())
    return classBuilder
}

There’s quite a lot going on here. First, we create a TypeSpec.Builder, which holds the information used to build a class.

We then add the data modifier, build a constructor, and for each JSON field, call the buildProperty method:

private fun buildProperty(fieldKey: String,
                          fieldTypeMap: Map<String, TypeName>,
                          commonElements: List<TypedJsonElement>,
                          classBuilder: TypeSpec.Builder,
                          constructor: FunSpec.Builder) {

    val kotlinIdentifier = fieldKey.toKotlinIdentifier()
    val typeName = fieldTypeMap[fieldKey]

    val initializer = PropertySpec.builder(kotlinIdentifier, typeName!!)
                                  .initializer(kotlinIdentifier)
    delegate.prepareProperty(initializer, kotlinIdentifier,
                            fieldKey, commonElements)

    classBuilder.addProperty(initializer.build())
    constructor.addParameter(kotlinIdentifier, typeName)
  }

We add a property for each JSON field by generating a PropertySpec.

For our case, all we need to include is the Kotlin identifier, and the property’s Kotlin type, which is represented by TypeName. The typename may be a Standard Library type such as String?, but could also be a type generated earlier on, such as Foo.

Each generated TypeSpec will then be pushed to a Stack, where it will eventually be written to an OutputStream as a Kotlin source file.

Writing a Kotlin command-line application

Our conversion tool is working pretty nicely by this point, so we’ll start writing a command-line app. This module will depend on the core module and the Apache Commons CLI, which does all the hard work of parsing arguments for us.

compile project(":core")
compile "commons-cli:commons-cli:1.4"

Handling command line arguments

We’ll start off by supporting help and input arguments:

private fun prepareOptions(): Options {
    return with(Options()) {
        addOption(Option.builder("input")
                .desc("The JSON file input")
                .numberOfArgs(1)
                .build())
        addOption(Option.builder("help")
                .desc("Displays help on available commands")
                .build())
    }
}

Now we need to handle each argument in our main method. If the arguments were invalid or not present, then we’ll print a message indicating that was the case, otherwise we’ll attempt to convert the JSON to Kotlin:

try {
    val cmd = parser.parse(prepareOptions(), args)

    if (cmd.hasOption("help") || !cmd.hasOption("input")) {
        printHelp(options)
    } else {
        val parsedOptionValue = cmd.getParsedOptionValue("input") as String
        val inputFile = Paths.get(parsedOptionValue).toFile()

        if (inputFile.exists()) {
            val outputFile = findOutputFile(inputFile)
            val input = inputFile.inputStream()
            val output = outputFile.outputStream()
            Kotlin2JsonConverter().convert(input, output, ConversionArgs())
            println("Generated source available at '$outputFile'")
        } else {
            println("Failed to find file '$inputFile'")
        }
    }
} catch (e: ParseException) {
    println("Failed to parse arguments: ${e.message}")
}

Distributing a Kotlin command-line application

JVM languages run pretty much anywhere in a JAR, and Gradle has a task that simplifies the distribution process. Our build file needs a few modifications, such as specifying the location of our main class:

apply plugin: 'application'

mainClassName = "com.fractalwrench.json2kotlin.AppKt"

jar {
    manifest {
        attributes "Main-Class": "$mainClassName"
    }
    from {
        configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
    }
}
distributions {
    main {
        baseName = 'json2kotlin'
    }
}

We will then run the following commands to distribute and test our application:

./gradlew assemble
unzip cmdline/build/distributions/json2kotlin.zip -d json2kotlin
./cmdline -input /c/Users/<User>/json2kotlin/bin/test.json

Our final step is to put the archive somewhere that people can download it, which in this case is GitHub.

Onto Spring Boot.

Writing a Spring Boot app in Kotlin

Spring Boot is a Java Framework that can be used to create web applications, and has recently announced first-class Kotlin support. It comes with sensible defaults, so we should be able to write a useful app in very few lines of code.

Adding a controller for GET requests

We’ll start off by creating an empty Spring Boot project by following Pivotal’s very handy guide. We’ll create a class annotated with Controller, and setup a RequestMapping to the root endpoint:

@Controller
class ConversionController {
    @GetMapping("/")
    fun displayConversionForm(model: Model): String {
        model.addAttribute("conversionForm", ConversionForm())
        model.addAttribute("kotlin", "class Example")
        return "conversion"
    }
}

There’s a lot of magic going on here.

Behind the scenes, Spring will detect the @Controller annotation and route HTTP requests to our displayConversionForm method.

This method then adds the generated Kotlin source as an attribute of a Model. Finally it returns a view, which corresponds to a HTML template stored under src/main/resources/static/templates. This template may look familiar to anyone who has used the Android Data Binding Library before, and the principle is the same:

<html>
  <head></head>
  <body>
    <textarea th:text="${kotlin}"></textarea>
  </body>
</html>

Thymeleaf binds model attributes to the view by evaluating any expressions in the th namespace. The generated HTML is then returned as an HTTP response to the user:

<html>
  <head></head>
  <body>
    <textarea text="class Example"></textarea>
  </body>
</html>

Adding a POST request mapping

We’re getting slightly ahead of ourselves here, as the generated Kotlin can’t be displayed until a user submits their JSON input. Therefore we need to display a form to the user, which will POST the JSON to our / endpoint:

<form id="jsonForm"
      th:action="@{/}"
      th:object="${conversionForm}"
      method="post"
      onsubmit="return validateForm()">
  <textarea maxlength="10000"
            placeholder="Paste JSON here..."
            th:field="*{json}"></textarea>
    <input type="reset" value="Reset"/>
    <input type="submit" value="Convert"/>
</form>

You may have noticed that the HTML form binds the conversionForm attribute in our previous method, as a form object.

When a POST request is submitted, the conversionForm method parameter will contain the text entered into our <textarea> element. We can then pass the user’s input into the Kotlin2JsonConverter, and our HTML response will contain dynamically generated Kotlin:

@PostMapping("/")
fun convertToKotlin(model: Model,
                    @ModelAttribute conversionForm: ConversionForm): String {
    val os = ByteArrayOutputStream()
    Kotlin2JsonConverter().convert(conversionForm.json, os, ConversionArgs())
    model.addAttribute("kotlin", String(os.first.toByteArray()))
    return displayConversionForm(model)
}

Of course, this isn’t quite production ready - we’d certainly want to sanitise user input before attempting to convert it.

We’ll skip a few steps here such as adding validation, and making the HTML prettier with CSS. If you’re interested in how this functionality works, I’d encourage you to browse through the Spring module of the project.

Deploying a Kotlin Spring Boot app to AWS Elastic Beanstalk

Now that we’ve completed an MVP web app, we’re going to deploy to AWS using the free tier, which meets the needs of most hobby projects. Some level of familiarity with AWS is assumed from here on out, but here’s a quick refresher on the services we’ll use:

Building a deployable JAR

The first step we’ll take is to update our JAR metadata, as we did for the command-line application:

jar {
    baseName = 'json2kotlin'
    version = '0.1.0'
    manifest {
        attributes 'Main-Class': 'AppKt'
    }
    from { configurations.compile.collect {
      it.isDirectory() ? it : zipTree(it) }
    }
}

We’ll also want to add Spring Actuator as a dependency, as it exposes several endpoints that provide useful information for devops.

compile("org.springframework.boot:spring-boot-starter-actuator")

Finally, we’ll need to update our server port in application.properties so that it can communicate with the load balancer:

server.port=8888

Registering a domain

The next step is to register a domain name, in this case json2kotlin.co.uk, by following the AWS guide.

After the rest of setup is completed, we’ll configure a hosted zone, which routes requests towards our Elastic Beanstalk environment.

Setting up an Elastic Beanstalk environment

We now need to setup an Elastic Beanstalk environment, which can be achieved by following this very helpful blog from AWS. ./gradlew bootRepackage will generate a JAR of our Spring Boot application.

Our application will use a load balancer, which will automatically scale up EC2 instances in the face of heavy traffic. Depending on your anticipated traffic, it’s possible that you could skip this step.

If you do use a load balancer, it is vital that a health check is setup, as otherwise the load balancer will assume that all the instances are unhealthy, and all the requests will timeout. Fortunately, Spring Actuator contains a ready-made /health endpoint, so we’ll configure our environment to use this.

Setup crash reporting with Bugsnag

There are probably a few bugs lurking in our application, so our next step is to add an error-detection SDK that reports any uncaught exceptions that occur in the wild. I chose Bugsnag for this task, because:

(Full disclaimer, I work for Bugsnag, so you can blame me if anything goes wrong)

The end result

The final web application is available here, or alternatively you can grab the command-line tool from Github.

You can also browse the complete source of this project on GitHub. At less than 1000 lines of code, this would be a nice project to wrap your head around if you’re looking to submit pull requests, issues, or just general feedback!

Thank You

I hope you’ve enjoyed learning about Kotlin source generation, and will sleep easy at night in the knowledge that you’ll never have to write data classes by hand again.

If you have any questions, feedback, or would like to suggest a topic for me to write about, please get in touch via Twitter!