File Handling in Go

File handling is one of the most common operations in Go—reading configuration, writing logs, and data persistence all depend on it. Go's standard library provides a concise yet powerful set of file I/O interfaces, covering scenarios from one-shot reads/writes to streaming processing.

There are 5 core packages related to file handling, each with its own role:

Go File Handling Core Packages os Core: create, read/write, delete, permissions io Generic Reader/Writer Interface bufio Buffered read/write, performance optimization filepath Cross-platform path handling ioutil ⚠ Deprecated in Go 1.16 → os / io Typical usage flow os.Open() bufio.Scanner Process data defer Close()

Version note:ioutilThe package was deprecated in Go 1.16, and its functionality has been moved toos(such asos.ReadFile、os.WriteFile) andio(such asio.ReadAll). New code should directly useosandiopackage.

1. File creation

os.CreateCreate a new file. If the file already exists,will be truncated (contents cleared). The returned file object must be closed to release system resources:

Example

package main

import (
        "log"
        "os"
)

func main() {
        // os.Create creates a file (truncates if it already exists)
        file, err := os.Create("test.txt")
        if err != nil {
                log.Fatal(err)
        }
        defer file.Close() // defer ensures the file is closed when the function ends

        // Write content to verify creation succeeded
        file.WriteString("File created successfully\n")
        log.Println("File created successfully")
}

defer file.Close()is a best practice for Go file operations.deferensures that even if subsequent code panics, the file will be correctly closed, avoiding file descriptor leaks.

2. Opening and Closing Files

osThe package provides three ways to open files, suitable for different scenarios:

Function Mode Description
os.Open(name) Read-only The simplest way is to open read-only
os.Create(name) Read/Write + Create/Truncate Create a new file or clear an existing file
os.OpenFile(name, flag, perm) Custom You can specify flags such as read/write, append, create, etc.

Example

package main

import (
        "fmt"
        "os"
)

func main() {
        // Method 1: open read-only
        file, err := os.Open("example.txt")
        if err != nil {
                fmt.Println("Open failed:, err)
                return
        }
        defer file.Close()
        fmt.Println("File opened successfully")

        // Method 2: OpenFile custom mode
        // os.O_WRONLY write-only
        // os.O_CREATE create if it does not exist
        // os.O_APPEND append mode
        // os.O_TRUNC truncate if it exists
        f, err := os.OpenFile("log.txt",
                os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
        if err != nil {
                fmt.Println("Open failed:, err)
                return
        }
        defer f.Close()
        f.WriteString("Append a line of log\n")
}

OpenFile Flag Descriptions

Flags Description
os.O_RDONLY Read-only (default)
os.O_WRONLY Write-only
os.O_RDWR Read-Write
os.O_APPEND Append mode, writes content appended to the end of the file
os.O_CREATE Create if the file does not exist
os.O_TRUNC Clear the file content when opening
os.O_EXCL Used with CREATE, errors if the file already exists

3. File reading

Go provides three main ways to read files; choose based on file size and processing requirements:

Three ways to read files Read all at once os.ReadFile("file.txt") Suitable for small files (config, JSON, etc.) Automatically opens/closes, simplest Line-by-line reading bufio.NewScanner(file) Suitable for large files (logs, CSV, etc.) Memory-friendly, processes line by line Streaming read io.ReadAll(reader) Suitable for network responses and already-open files Generic Reader interface

3.1 One-time Read (Recommended for Small Files)

os.ReadFileis the simplest way—automatically open, read, and close the file in one step:

Example

package main

import (
        "fmt"
        "log"
        "os"
)

func main() {
        // Read entire file at once (Go 1.16+)
        data, err := os.ReadFile("config.json")
        if err != nil {
                log.Fatal(err)
        }

        fmt.Println(string(data))
}

3.2 Reading line by line (recommended for large files)

For large files, usebufio.ScannerProcess line by line to avoid loading everything into memory at once:

Example

package main

import (
        "bufio"
        "fmt"
        "log"
        "os"
)

func main() {
        file, err := os.Open("large-file.log")
        if err != nil {
                log.Fatal(err)
        }
        defer file.Close()

        // Scan line by line
        scanner := bufio.NewScanner(file)
        lineNum := 0
        for scanner.Scan() {
                lineNum++
                line := scanner.Text() // Get current line content
                fmt.Printf("Line %d: %s\n", lineNum, line)
        }

        // Check if an error occurred during scanning
        if err := scanner.Err(); err != nil {
                log.Fatal("Read error:", err)
        }
}

Note:bufio.ScannerThe default maximum line length is 64KB. If the file contains overly long lines, you need to, when callingScan()Presetscanner.Buffer(make([]byte, 0), maxSize)to increase the buffer.

3.3 Reading with io.ReadAll

When you already have an openio.Reader(such as network responses, opened files), you can useio.ReadAll：

Example

package main

import (
        "fmt"
        "io"
        "log"
        "os"
)

func main() {
        file, err := os.Open("example.txt")
        if err != nil {
                log.Fatal(err)
        }
        defer file.Close()

        // Read all data from Reader
        data, err := io.ReadAll(file)
        if err != nil {
                log.Fatal(err)
        }

        fmt.Println(string(data))
}

4. File writing

Go provides multiple ways to write, from simple one-shot writes to high-performance buffered writes:

4.1 One-time Write

os.WriteFileWrite data to the file in one shot (overwriting existing content):

Example

package main

import (
        "log"
        "os"
)

func main() {
        content := []byte("Hello, Go!\nThis is the second line\n")

        // 0644: owner read/write, others read-only
        err := os.WriteFile("output.txt", content, 0644)
        if err != nil {
                log.Fatal(err)
        }
        log.Println("Write successful")
}

4.2 Writing using the File object

Writing through the file object allows you to write content in multiple passes:

Example

package main

import (
        "fmt"
        "log"
        "os"
)

func main() {
        file, err := os.Create("output.txt")
        if err != nil {
                log.Fatal(err)
        }
        defer file.Close()

        // Method 1: Write string
        file.WriteString("Write string directly\n")

        // Method 2: Write byte slice
        data := []byte("Write byte slice\n")
        file.Write(data)

        // Method 3: Formatted write
        fmt.Fprintf(file, "Formatted write: %d + %d = %d\n", 3, 4, 7)
}

4.3 Buffered Writing (Recommended for Large Amounts of Data)

bufio.WriterIt first writes data to an in-memory buffer, then writes to disk in batches once enough has accumulated, significantly reducing the number of I/O operations.Make sure to call before finishingFlush()：

Example

package main

import (
        "bufio"
        "log"
        "os"
)

func main() {
        file, err := os.Create("buffered-output.txt")
        if err != nil {
                log.Fatal(err)
        }
        defer file.Close()

        // Create buffered writer
        writer := bufio.NewWriter(file)

        // Write multiple lines (go to buffer first, not immediately to disk)
        for i := 0; i < 1000; i++ {
                writer.WriteString("This is line " + itoa(i) + " line\n")
        }

        // Flush writes remaining buffered data to the file
        if err := writer.Flush(); err != nil {
                log.Fatal("Failed to flush buffer:", err)
        }
        log.Println("Buffered write complete")
}

// Simple helper function to convert int to string
func itoa(n int) string {
        return fmt.Sprintf("%d", n)
}

5. Appending to Files

Usageos.O_APPENDThe flag appends content at the end of the file without overwriting existing data:

Example

package main

import (
        "log"
        "os"
        "time"
)

func main() {
        // O_APPEND: append mode
        // O_CREATE: create if it doesn't exist
        // O_WRONLY: write-only
        file, err := os.OpenFile("app.log",
                os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
        if err != nil {
                log.Fatal(err)
        }
        defer file.Close()

        // Appends each run, does not overwrite
        timestamp := time.Now().Format("2006-01-02 15:04:05")
        file.WriteString("[" + timestamp + "] Application started\n")
}

Comparison:os.CreateIt will clear the existing content;os.OpenFileCoordinationO_APPENDIt will append the content. In logging scenarios, be sure to use append mode.

6. Deleting and Renaming Files

Example

package main

import (
        "log"
        "os"
)

func main() {
        // Rename / Move file
        err := os.Rename("old.txt", "new.txt")
        if err != nil {
                log.Fatal("Rename failed:", err)
        }
        log.Println("Rename successful")

        // Delete file
        err = os.Remove("temp.txt")
        if err != nil {
                log.Fatal("Delete failed:", err)
        }
        log.Println("Delete successful")
}

7. File Information and Inspection

7.1 Getting File Information

Example

package main

import (
        "fmt"
        "log"
        "os"
)

func main() {
        info, err := os.Stat("test.txt")
        if err != nil {
                log.Fatal(err)
        }

        fmt.Println("File name: ", info.Name())
        fmt.Println("Size: ", info.Size(), "bytes")
        fmt.Println("Permissions: ", info.Mode())
        fmt.Println("Modified: ", info.ModTime())
        fmt.Println("Is directory: ", info.IsDir())
}

7.2 Checking Whether a File Exists

Go does not have a dedicated "file exists" function; byos.Stat + os.IsNotExistDetermine:

Example

package main

import (
        "fmt"
        "os"
)

// fileExists checks whether the file exists
func fileExists(path string) bool {
        _, err := os.Stat(path)
        return !os.IsNotExist(err)
}

func main() {
        if fileExists("test.txt") {
                fmt.Println("File exists")
        } else {
                fmt.Println("File does not exist")
        }
}

8. Directory operations

Example

package main

import (
        "fmt"
        "log"
        "os"
)

func main() {
        // Create a single directory
        err := os.Mkdir("newdir", 0755)
        if err != nil {
                log.Fatal(err)
        }

        // Recursively create multi-level directories (automatically creates parent directories even if they don't exist)
        err = os.MkdirAll("path/to/deep/dir", 0755)
        if err != nil {
                log.Fatal(err)
        }

        // Read directory contents
        entries, err := os.ReadDir(".")
        if err != nil {
                log.Fatal(err)
        }
        for _, entry := range entries {
                mark := "&#x1f4c4;"
                if entry.IsDir() {
                        mark = "&#x1f4c1;"
                }
                fmt.Printf("%s %s\n", mark, entry.Name())
        }

        // Delete empty directory
        os.Remove("newdir")

        // Recursively delete the directory and all its contents (&#x26a0; use with caution)
        os.RemoveAll("path")
}

9. Advanced operations

9.1 File copying

Usageio.CopyEfficiently copy data between two files:

Example

package main

import (
        "io"
        "log"
        "os"
)

// copyFile is a generic function for copying files
func copyFile(src, dst string) (int64, error) {
        srcFile, err := os.Open(src)
        if err != nil {
                return 0, err
        }
        defer srcFile.Close()

        dstFile, err := os.Create(dst)
        if err != nil {
                return 0, err
        }
        defer dstFile.Close()

        // io.Copy automatically manages the buffer for efficient copying
        return io.Copy(dstFile, srcFile)
}

func main() {
        n, err := copyFile("source.txt", "destination.txt")
        if err != nil {
                log.Fatal(err)
        }
        log.Printf("Copy completed, %d bytes total", n)
}

9.2 Temporary files and directories

Example

package main

import (
        "fmt"
        "log"
        "os"
)

func main() {
        // Create a temporary file (automatically named, prefix is "app-")
        tmpFile, err := os.CreateTemp("", "app-*.txt")
        if err != nil {
                log.Fatal(err)
        }
        defer os.Remove(tmpFile.Name()) // Clean up after use
        fmt.Println("Temporary file:", tmpFile.Name())

        // Create a temporary directory
        tmpDir, err := os.MkdirTemp("", "app-*")
        if err != nil {
                log.Fatal(err)
        }
        defer os.RemoveAll(tmpDir) // Clean up after use
        fmt.Println("Temporary directory:", tmpDir)
}

9.3 Recursively traversing directories

filepath.WalkYou can recursively traverse all files and subdirectories in the directory tree:

Example

package main

import (
        "fmt"
        "log"
        "os"
        "path/filepath"
)

func main() {
        root := "." // Start traversing from the current directory

        err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
                if err != nil {
                        return err // When an error occurs, you can choose to skip or return
                }

                // Print path and size
                size := ""
                if !info.IsDir() {
                        size = fmt.Sprintf(" (%d bytes)", info.Size())
                }
                fmt.Println(path + size)
                return nil
        })

        if err != nil {
                log.Fatal(err)
        }
}

9.4 Cross-platform Path Joining

Never use string concatenation to build paths(such asdir + "/" + file), usefilepath.JoinTo ensure cross-platform compatibility:

// 错误：Windows 上路径分隔符是 \
path := "dir" + "/" + "file.txt"

// 正确：自动适配当前操作系统
path := filepath.Join("dir", "file.txt")

Scenario quick reference

Scenarios Recommended approach Key code
Read small file os.ReadFile data, err := os.ReadFile("file.txt")
Read large files line by line bufio.Scanner scanner := bufio.NewScanner(file)
Write small file os.WriteFile os.WriteFile("f.txt", data, 0644)
Writing large amounts of data bufio.Writer writer := bufio.NewWriter(file)
Append log os.OpenFile + O_APPEND os.OpenFile("log", os.O_APPEND|os.O_WRONLY, 0644)
File copy io.Copy io.Copy(dst, src)
Traverse directory filepath.Walk filepath.Walk(".", callback)
Path join filepath.Join filepath.Join("dir", "file.txt")
Check file existence os.Stat + os.IsNotExist _, err := os.Stat(path)
Create temporary files os.CreateTemp os.CreateTemp("", "prefix-*.txt")

File permission description:0644Indicates the owner can read and write (6=4+2), while group and other users are read-only (4). Directories typically use0755(adds execute permission, otherwise you cannot enter the directory)

other extensions