HTML Parsing in Java: Jsoup and HtmlUnit Libraries

HTML Parsing in Java: Overview of Jsoup and HtmlUnit Libraries

HTML parsing (or web scraping) is the process of extracting data from HTML documents. In Java, there are several powerful libraries for this task, among which Jsoup (for static HTML) and HtmlUnit (for dynamic pages with JavaScript) stand out. In this article, we will thoroughly examine both libraries, show code examples, and help you choose the right tool for your project on pythonlib.ru.

What is Jsoup?

Jsoup is a lightweight library for parsing, extracting, and manipulating HTML documents. It supports CSS selectors, making it intuitive for web developers. Jsoup works with any HTML, even if it is poorly formatted.

Installing Jsoup

Add the dependency to your pom.xml (Maven):

<dependency>    <groupId>org.jsoup</groupId>    <artifactId>jsoup</artifactId>    <version>1.17.2</version></dependency>

Parsing Example with Jsoup

Let's look at a basic example — getting the page title and all links:

import org.jsoup.Jsoup;import org.jsoup.nodes.Document;import org.jsoup.nodes.Element;import org.jsoup.select.Elements;

public class JsoupExample {    public static void main(String[] args) throws Exception {        // 1. Load the HTML document        Document doc = Jsoup.connect("https://example.com").get();

        // 2. Extract the page title        String title = doc.title();        System.out.println("Title: " + title);

        // 3. Find all links (<a> tags)        Elements links = doc.select("a[href]");        for (Element link : links) {            String href = link.attr("href");            String text = link.text();            System.out.println("Link: " + href + " — text: " + text);        }

        // 4. Extract text from a specific block (e.g., <div class="content">)        Element content = doc.selectFirst("div.content");        if (content != null) {            System.out.println("Block content: " + content.text());        }    }}

In this example, Jsoup.connect() makes an HTTP request, and select() uses CSS selectors to find elements. Jsoup can also work with local files and HTML strings.

What is HtmlUnit?

HtmlUnit is a "headless" browser in Java. It not only parses HTML but also executes JavaScript, handles AJAX requests, and emulates user actions. This is an ideal choice for sites where content is loaded dynamically.

Installing HtmlUnit

<dependency>    <groupId>net.sourceforge.htmlunit</groupId>    <artifactId>htmlunit</artifactId>    <version>2.70.0</version></dependency>

Parsing Example with HtmlUnit

Parsing a page that loads data via JavaScript:

import com.gargoylesoftware.htmlunit.WebClient;import com.gargoylesoftware.htmlunit.html.HtmlPage;import com.gargoylesoftware.htmlunit.html.HtmlElement;

public class HtmlUnitExample {    public static void main(String[] args) throws Exception {        // 1. Create a WebClient (browser analog)        try (WebClient webClient = new WebClient()) {            // Disable CSS and logging for speed            webClient.getOptions().setCssEnabled(false);            webClient.getOptions().setJavaScriptEnabled(true);            webClient.getOptions().setThrowExceptionOnScriptError(false);

            // 2. Load the page (JavaScript will execute automatically)            HtmlPage page = webClient.getPage("https://example.com/dynamic");

            // 3. Wait, p