Go and Files: From Opening to Streaming Data Processing

Why working with files in Go is not scary, but interesting?

When I first opened the documentation for the os package in Go, it seemed like a boring list of system calls awaited me. But after digging deeper, I realized: working with files and the file system in Go is not just about reading bytes. It's a powerful tool that gives you full control over data, whether it's configuration files, logs, binaries, or entire databases in CSV.

Go was created for system programming, so its authors (Ken Thompson, Rob Pike, and others) built file handling directly into the core of the language. No external dependencies, just a standard library that works fast and predictably. In this article, we'll cover all scenarios: from simple file reading to recursive directory traversal and streaming processing of large data.

If you write in Go and want to confidently work with the file system — you're in the right place. Let's go!

Basics: open, read, close

The very first step is opening a file. In Go, the os.Open() function handles this. It returns a file pointer and an error. Errors in Go are not exceptions, but values. You must always check them.

package main\n\nimport (\n    "fmt"\n    "os"\n)\n\nfunc main() {\n    file, err := os.Open("example.txt")\n    if err != nil {\n        fmt.Println("Error opening file:", err)\n        return\n    }\n    defer file.Close() // обязательно закрываем файл\n\n    // Read the contents\n    buf := make([]byte, 1024)\n    n, err := file.Read(buf)\n    if err != nil {\n        fmt.Println("Error reading:", err)\n        return\n    }\n    fmt.Printf("Read %d bytes: %s\\n", n, string(buf[:n]))\n}\n

Note the defer file.Close(). This is a Go idiom: a deferred call ensures the file closes even in case of a panic or early return. Forgetting to close a file is bad practice, leading to descriptor leaks.

Reading an entire file in one line

If the file is small (up to a few megabytes), you can use os.ReadFile() (replaced the deprecated ioutil.ReadFile). This is convenient for configs or text data.

data, err := os.ReadFile("config.yaml")\nif err != nil {\n    log.Fatal(err)\n}\nfmt.Println(string(data))\n

The function opens, reads, and closes the file itself. But remember: for large files, this approach is dangerous — you'll load the entire file into memory.

Writing to a file: create and append

For writing, use os.Create() (creates a new file or truncates an existing one) and os.OpenFile() for more fine-grained control (e.g., appending to the end).

// Create a file\nfile, err := os.Create("output.txt")\nif err != nil {\n    panic(err)\n}\ndefer file.Close()\n\n// Write a string\n_, err = file.WriteString("Hello, Go files!\\n")\nif err != nil {\n    panic(err)\n}\n\n// Write bytes\nbytes := []byte("Writing a byte slice\\n")\n_, err = file.Write(bytes)\nif err != nil {\n    panic(err)\n}\n

To append data to the end of an existing file, use flags:

file, err := os.OpenFile("log.txt", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)\nif err != nil {\n    log.Fatal(err)\n}\ndefer file.Close()\n\nfile.WriteString("New log entry\\n")\n

The os.O_APPEND flag ensures data is appended to the end, not overwriting the beginning.

Working with directories: create, delete, traverse

The file system is not just files, but also folders. Go provides convenient functions for working with directories.

Creating and deleting folders

// Create a single folder\nerr := os.Mkdir("newdir", 0755)\nif err != nil {\n    log.Fatal(err)\n}\n\n// Create nested folders (like mkdir -p)\nerr = os.MkdirAll("path/to/nested/dir", 0755)\nif err != nil {\n    log.Fatal(err)\