Removing Duplicates from an Array in JavaScript: Under the Hood Analysis

Where Do Duplicates Come From and Why Are They a Problem?

Imagine: you are collecting data from an API, parsing a CSV file, or processing user input. In 90% of cases, you encounter arrays where the same element appears multiple times. This is not just an aesthetic problem — duplicates can skew statistics, break list rendering, or lead to infinite loops in algorithms.

The question "Remove duplicate values from a JavaScript array" on Stack Overflow has gathered thousands of views for a reason. This is a basic but critically important operation that every developer should know. Today, we will look under the hood and break down how different approaches work — from naive to the most performant.

Method #1: Set — The Most Elegant and Fastest

If you ask a modern JavaScript developer: "How do you remove duplicates?", the first thing they will say is Set. And rightly so. But how does it work?

const numbers = [1, 2, 2, 3, 4, 4, 5];\nconst uniqueNumbers = [...new Set(numbers)];\nconsole.log(uniqueNumbers); // [1, 2, 3, 4, 5]

What Happens Under the Hood of Set?

The Set object stores unique values of any type. When you pass an array to the constructor new Set(numbers), the JavaScript engine (V8, SpiderMonkey, etc.) performs the following steps:

\n
• Iteration over the array: the engine goes through each element of the original array.
\n
• Uniqueness check: an internal hash function is called for each element. If the element already exists in a "bucket", it is ignored. If not, it is added.
\n
• Result assembly: the spread operator [...set] expands the iterable Set object back into an array.
\n

An important nuance: Set uses the SameValueZero comparison algorithm. This means that NaN is considered equal to NaN (unlike ===), and +0 and -0 are considered different values. In practice, this rarely matters, but it is good to know.

Set Performance

The complexity of the operation is O(n). Each element is checked exactly once. This is the best performance among all methods. For arrays with 10,000 elements, the difference between Set and other methods can be tens of milliseconds — critical for high-load applications.

Method #2: filter + indexOf — A Classic Approach

Before the advent of ES6 (2015), developers used the combination of filter and indexOf. This method is still found in legacy code:

const numbers = [1, 2, 2, 3, 4, 4, 5];\nconst uniqueNumbers = numbers.filter((value, index, self) => {\n    return self.indexOf(value) === index;\n});\nconsole.log(uniqueNumbers); // [1, 2, 3, 4, 5]

How Does It Work?

\n
• filter iterates over each element of the array.
\n
• For each element, indexOf(value) is called, which returns the first index where this value is found.
\n
• If the current index of the element matches the first found index — it means this is the first occurrence, and the element is kept. If it does not match — it is a duplicate and is discarded.
\n

Pitfalls

The main problem is performance. indexOf internally uses linear search (O(n) for each call). As a result, for an array of N elements, the complexity becomes O(n²). With 1000 elements, this is still tolerable, but with 100,000, the slowdown becomes noticeable.

Another downside: indexOf does not work correctly with NaN. If the array contains NaN, indexOf(NaN) will always return -1, and the element will be added again. To check for NaN, you need to use