Processing Big Data in Kotlin: An Unexpected and Powerful Tandem

Why Kotlin is the New "Swiss Army Knife" for Big Data?

When it comes to processing big data, Python, Java, or Scala usually come to mind. But there is one language that has quietly but confidently burst into this world — Kotlin. And if you think Kotlin is only about Android, you are seriously behind the times.

Kotlin for Big Data is like finding a sports car in an old barn: unexpected, powerful, and damn pleasant. It combines the conciseness of Python, the type safety of Java, and the functional features of Scala. And most importantly, it is fully compatible with the JVM ecosystem. This means you can run big data processing in Kotlin using all the same Hadoop, Spark, or Kafka, but write 2 times less code.

In this article, we will break down how to use Kotlin for real Big Data tasks: from writing ETL pipelines to working with Apache Spark and coroutines for asynchronous processing. No fluff, only code and practice.

Why is Kotlin Better than Java and Scala for a Data Engineer?

Let's be honest: Java is too verbose. One data class in Java takes 30 lines with getters, setters, equals, and hashCode. In Kotlin — one line:

data class User(val id: Long, val name: String, val email: String)

Scala, in turn, has powerful functional capabilities, but its syntax often scares beginners with its complexity. Kotlin, however, offers a "golden mean" — functional programming that doesn't break your brain.

Key advantages of Kotlin for data processing:

• Null-safety — you will forget about NullPointerException at runtime.
• Extension functions — you can extend Spark or Kafka classes without inheritance.
• Coroutines — lightweight asynchrony for stream processing.
• Type inference — code is written faster, read easier.
• Java Interop — you can use any Java library as if it were native.

For a data engineer, Kotlin is an opportunity to write production code that runs fast, doesn't glitch, and is easy to maintain.

Toolkit: What We Work With

Before writing code, let's decide which libraries and frameworks we will use for data analysis in Kotlin:

• Apache Spark for Kotlin — via the Java API. Everything works "out of the box".
• Kotlin DataFrame — a library from JetBrains for working with tabular data. An analog of Pandas, but in Kotlin.
• Kotlinx.serialization — for working with JSON/Parquet.
• Kotlin Coroutines — for asynchronous loading and stream processing.
• Exposed — an ORM for working with databases.

Now let's move on to the most interesting part — the code.

Example 1: ETL Pipeline in Kotlin and Spark

Let's say we have website visit logs in JSON format (10 GB). We need to load them, filter only paying users, and save them in Parquet. A classic Kotlin data processing task.

import org.apache.spark.sql.SparkSessionimport org.apache.spark.sql.functions._

fun main() {    val spark = SparkSession.builder()        .appName("Kotlin Big Data ETL")        .master("local[*]")        .getOrCreate()

    // Read JSON    val logsDF = spark.read()        .option("multiline", "true")        .json("hdfs://data/logs/*.json")

    // Filter paying users    val payingUsers = logsDF        .filter(col("is_paid") === true)        .select("user_id", "event_time", "revenue")

    // Aggregate: total revenue by user    val revenueByUser = payingUsers        .groupBy("user_id")        .agg(sum("revenue").alias("total_revenue"))\