HTML Parsing in Kotlin: jsoup, ktor, khtml and Other Tools

HTML Parsing in Kotlin: A Complete Guide to Tools and Libraries

HTML parsing (extracting data from web pages) is one of the key tasks when creating parsers, content analyzers, or integrating external data. In the Kotlin ecosystem, there are several powerful libraries that allow you to work with HTML efficiently. In this article, we will take a detailed look at the best tools, their features, and provide practical code examples.

1. jsoup — The De Facto Standard for HTML Parsing

jsoup is the most popular Java library for working with HTML, which works perfectly in Kotlin as well. It provides a convenient API for loading, parsing, and extracting data from HTML documents.

Key Features:

• Loading HTML from a URL, file, or string
• Support for CSS selectors to find elements
• Cleaning HTML from unwanted tags
• Modifying the DOM tree

Example of using jsoup in Kotlin:

import org.jsoup.Jsoupimport org.jsoup.nodes.Document

fun main() {    // Load the HTML document    val doc: Document = Jsoup.connect("https://example.com").get()        // Extract the page title    val title: String = doc.title()    println("Title: $title")        // Find all links on the page    val links = doc.select("a[href]")    links.forEach { link ->        val href = link.attr("href")        val text = link.text()        println("Link: $text -> $href")    }        // Extract text from a specific block    val content = doc.select("div.content").text()    println("Content: $content")        // Working with forms    val form = doc.select("form").first()    form?.select("input")?.forEach { input ->        println("Field: ${input.attr("name")} = ${input.attr("value")}")    }}

2. Ktor Client + khtml — A Modern Kotlin Stack

Ktor Client is an asynchronous HTTP client for Kotlin that pairs perfectly with the khtml library (Kotlin HTML parser). Together they form a powerful tool for web scraping.

Advantages of this approach:

• Native support for coroutines (suspend functions)
• Asynchronous page loading
• Flexible configuration of HTTP requests
• Support for cookies, sessions, and proxies

Example with Ktor Client and khtml:

import io.ktor.client.*import io.ktor.client.request.*import io.ktor.client.statement.*import kotlinx.coroutines.runBlockingimport org.jsoup.Jsoup

suspend fun fetchAndParse(url: String) {    val client = HttpClient()        try {        // Asynchronous HTML loading        val response: HttpResponse = client.get(url)        val html: String = response.bodyAsText()                // Parsing with jsoup        val doc = Jsoup.parse(html)        val headlines = doc.select("h2, h3").map { it.text() }                println("Headlines on the page:")        headlines.forEach { println("- $it") }            } catch (e: Exception) {        println("Error during loading: ${e.message}")    } finally {        client.close()    }}

fun main() = runBlocking {    fetchAndParse("https://kotlinlang.org")}

3. Selenium WebDriver — For Dynamic Pages

If a website heavily uses JavaScript for rendering content, you will need Selenium WebDriver. It launches a real browser and allows you to interact with dynamic elements.

Example with Selenium in Kotlin:

import org.openqa.selenium.Byimport org.openqa.selenium.WebDriverimport org.openqa.selenium.chrome.ChromeDriverimport org.openqa.selenium.chrome.ChromeOptions

fun scrapeDynamicPage() {    val options = ChromeOptions().apply {        addArguments("--headless") // Headless mode        addArguments("