Parsing HTML in Java: Jsoup, HtmlUnit, and Other Tools

Parsing HTML in Java: Overview of Libraries and Tools

Parsing HTML is the process of extracting structured data from HTML documents. In Java, there are several powerful libraries for this task, allowing you not only to parse static pages but also to handle dynamic content. In this article, we will look at three of the most popular tools: Jsoup, HtmlUnit, and Jericho HTML Parser. Each has its own features and areas of application.

1. Jsoup — Universal HTML Parser

Jsoup is the most popular Java library for working with HTML. It provides a convenient API for loading, parsing, and extracting data, and also supports CSS selectors and DOM tree manipulation. Jsoup is ideal for static pages and simple web scraping tasks.

Key Features:

• Loading HTML from a URL, file, or string
• Extracting data using CSS selectors
• Cleaning HTML from unwanted tags
• DOM modification (adding/removing elements)
• Automatic correction of malformed HTML

Code Example: Extracting News Headlines

import org.jsoup.Jsoup;import org.jsoup.nodes.Document;import org.jsoup.select.Elements;

public class JsoupExample {    public static void main(String[] args) throws Exception {        // Load the HTML page        Document doc = Jsoup.connect("https://example.com/news")                            .userAgent("Mozilla/5.0")                            .timeout(5000)                            .get();                // Extract all h2 headings with class "news-title"        Elements titles = doc.select("h2.news-title");                // Print the text of each heading        titles.forEach(title -> System.out.println(title.text()));    }}

2. HtmlUnit — Parsing with JavaScript Support

HtmlUnit is a headless browser in Java that emulates the operation of a web browser. Unlike Jsoup, it executes JavaScript code, handles AJAX requests, and supports cookies. This is an indispensable tool for parsing modern SPA applications and sites with dynamic content.

Key Features:

• Full browser emulation (Chrome, Firefox, IE)
• JavaScript execution (including jQuery and React)
• Support for forms, clicks, and page navigation
• Working with cookies, sessions, and redirects
• Built-in support for XPath and CSS selectors

Code Example: Waiting for Dynamic Content to Load

import com.gargoylesoftware.htmlunit.WebClient;import com.gargoylesoftware.htmlunit.html.HtmlPage;import com.gargoylesoftware.htmlunit.html.HtmlElement;

public class HtmlUnitExample {    public static void main(String[] args) throws Exception {        try (WebClient webClient = new WebClient()) {            // Disable logging and timeouts            webClient.getOptions().setThrowExceptionOnScriptError(false);            webClient.getOptions().setJavaScriptEnabled(true);                        // Load the page            HtmlPage page = webClient.getPage("https://example.com/dynamic");                        // Wait 2 seconds for JS to execute            webClient.waitForBackgroundJavaScript(2000);                        // Extract data using XPath            HtmlElement element = page.getFirstByXPath("//div[@class='result']/p");            System.out.println("Result: " + element.getTextContent());        }    }}

3. Jericho HTML Parser — Lightweight and Fast

Jericho HTML Parser is a lightweight library with minimal dependencies. It handles parsing invalid HTML very well and provides low-level access to the document structure.