Working with Task Queues in C#: A Practical Guide

Introduction: Why Do You Need Task Queues in C#?

In modern C# (.NET) development, there is often a need to organize asynchronous data processing. A classic example is a web server that accepts requests and must process them in the background without blocking the main thread. Or an application that downloads files from the internet and processes already downloaded ones in parallel. For such scenarios, the "Producer-Consumer" pattern is ideal, and its foundation is a task queue.

A task queue allows you to safely transfer work from one thread (or asynchronous operation) to another, synchronizing access to shared data. In this article, we will look at three main ways to implement queues in C#: ConcurrentQueue, BlockingCollection, and the modern System.Threading.Channels. Each is suitable for different scenarios, and we will break them down with practical examples.

1. ConcurrentQueue — Basic Thread-Safe Queue

ConcurrentQueue<T> is the simplest thread-safe queue from the System.Collections.Concurrent namespace. It uses lock-free algorithms (compare-and-swap, CAS) to ensure high performance. However, it does not block the thread when the queue is empty — you will have to implement waiting yourself.

Example: Loading Data from Multiple Sources

Suppose we have a task: download 10 web pages and process them. One thread (Producer) adds URLs to the queue, and several threads (Consumers) take them and download them.

using System;\nusing System.Collections.Concurrent;\nusing System.Threading.Tasks;\n\npublic class ConcurrentQueueExample\n{\n    public static async Task RunAsync()\n    {\n        var queue = new ConcurrentQueue<string>();\n        var urls = new[] { "https://example.com", "https://dotnet.microsoft.com" };\n\n        // Producer: adding tasks\n        foreach (var url in urls)\n        {\n            queue.Enqueue(url);\n        }\n\n        // Consumers: processing tasks\n        var tasks = new Task[3];\n        for (int i = 0; i < tasks.Length; i++)\n        {\n            tasks[i] = Task.Run(() =>\n            {\n                while (queue.TryDequeue(out var url))\n                {\n                    Console.WriteLine($"Processing: {url}");\n                    // Simulating work\n                    Task.Delay(500).Wait();\n                }\n            });\n        }\n\n        await Task.WhenAll(tasks);\n        Console.WriteLine("All tasks completed.");\n    }\n}

Pros: simplicity, high performance with a large number of operations.
\nCons: no built-in waiting mechanism (requires a polling loop), difficult to limit queue size.

2. BlockingCollection — Queue with Blocking and Bounding

BlockingCollection<T> is a wrapper over a thread-safe collection that adds the ability to block the consumer thread when the queue is empty and the producer thread when the queue is full. By default, it uses ConcurrentQueue, but you can substitute any collection implementing IProducerConsumerCollection<T>.

Example: Order Processing with Buffer Limit

Imagine an order processing system. A producer generates orders, and three consumers process them. The buffer is limited to 10 items to avoid consuming too much memory.

using System;\nusing System.Collections.Concurrent;\nusing System.Threading.Tasks;\n\npublic class BlockingCollectionExample\n{\n    public static async Task RunAsync()\n    {\n        using var orders = new BlockingCollection<string>(boundedCapacity: 10);\n\n        // Producer\n        var producer = Task.Run(() =>\n        {\n            for (int i = 1; i <= 20; i++)\n