Working with S3 and Cloud Storage in Go (Golang): Libraries and Examples

Working with S3 and Cloud Storage in Go (Golang)

Cloud object storage compatible with the Amazon S3 protocol has become the de facto standard for storing unstructured data: images, videos, logs, backups. In the Go (Golang) ecosystem, there are several powerful libraries for interacting with S3 that allow you to easily integrate cloud storage into your applications. In this article, we will look at two main libraries: minio-go and aws-sdk-go-v2, and provide practical examples of their use.

1. The minio-go Library

MinIO is a high-performance open-source object storage that is fully compatible with the Amazon S3 API. Its client library minio-go is one of the most popular for working with S3 in Go. It supports all basic operations: creating and deleting buckets, uploading, downloading, and deleting objects, as well as working with access policies.

Installing the library:

go get github.com/minio/minio-go/v7

Example: Connecting and Uploading a File

package main

import (    "context"    "log"

    "github.com/minio/minio-go/v7"    "github.com/minio/minio-go/v7/pkg/credentials")

func main() {    ctx := context.Background()

    // Specify the endpoint, access key, and secret key of your S3-compatible storage    endpoint := "play.min.io"    accessKeyID := "Q3AM3UQ867SPQQA43P2F"    secretAccessKey := "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"    useSSL := true

    // Initialize the client    minioClient, err := minio.New(endpoint, &minio.Options{        Creds:  credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),        Secure: useSSL,    })    if err != nil {        log.Fatalln(err)    }

    // Create a bucket (if it doesn't exist)    bucketName := "my-bucket"    err = minioClient.MakeBucket(ctx, bucketName, minio.MakeBucketOptions{})    if err != nil {        // Check if the bucket already exists        exists, errBucketExists := minioClient.BucketExists(ctx, bucketName)        if errBucketExists == nil && exists {            log.Printf("Bucket %s already exists", bucketName)        } else {            log.Fatalln(err)        }    }

    // Upload a file    objectName := "myfile.txt"    filePath := "/tmp/myfile.txt"    contentType := "text/plain"

    n, err := minioClient.FPutObject(ctx, bucketName, objectName, filePath, minio.PutObjectOptions{        ContentType: contentType,    })    if err != nil {        log.Fatalln(err)    }

    log.Printf("Successfully uploaded %s of size %d", objectName, n)}

2. The AWS SDK for Go v2 Library

The official SDK from Amazon (aws-sdk-go-v2) provides a complete set of tools for working with AWS services, including S3. It supports all S3 features, including multipart upload, encryption, and version management. The SDK uses model-based code generation, which ensures high performance and strong typing.

Installation:

go get github.com/aws/aws-sdk-go-v2go get github.com/aws/aws-sdk-go-v2/configgo get github.com/aws/aws-sdk-go-v2/service/s3

Example: Uploading a File Using AWS SDK v2

package main

import (    "context"    "log"    "os"

    "github.com/aws/aws-sdk-go-v2/aws"    "github.com/aws/aws-sdk-go-v2/config"    "github.com/aws/aws-sdk-go-v2/service/s3")

func main() {    ctx := context.Background()

    // Load configuration from environment variables or ~/.aws/credentials file    cfg, err := config.LoadDefaultConfig(ctx)    if err != nil {        log.Fatalf("unable to load SDK config, %v", err)    }

    // Create an S3 client    client := s3.NewFromConfig(cfg)

    // Open the file for uploading    file, err := os.Ope