dailyrotate - a Go library for rotating files daily

Krzysztof Kowalczyk
Sep 20 '19 · 2 min read · 648 views

What is dailyrotate?

dailyrotate is a Go library for rotating files daily.
References:

Imagine you're writing a backend for a web app in Go.
Logging is important for observability and debugability.
Web backends are long-running and you don't want your logs to grow without bounds.
Log rotation solves the issue of infinite growth: when the log file grows beyond a certain size or at certain time, a new log file is created.
To implement log rotation you can use external programs like logrotate but I prefer simpler solution and thanks to io.Writer interface, Go makes implementing log rotation simple.
dailyrotate is a Go library that makes it easy to implement log rotation.
By defaultd logs rotate daily, at midnight UTC time.
I use dailyrotate in production (for example in presstige.io).

How to use dailyrotate

Open log file

Logging happens everywhere in the code so typically we would have a global variable for the log file and open the log at program start
var (
	logFile *dailyrotate.File
)

func openLogFile(pathFormat string, onClose func(string, bool)) error {
	w, err := dailyrotate.NewFile(pathFormat, onLogClose)
	if err != nil {
		return err
	}
	logFile = w
	return nil
}

func main() {
	logDir := "logs"

	// we have to ensure the directory we want to write to
	// already exists
	err := os.MkdirAll(logDir, 0755)
	if err != nil {
		log.Fatalf("os.MkdirAll()(")
	}

	pathFormat := filepath.Join(logDir, "2006-01-02.txt")
	err = openLogFile(pathFormat, onLogClose)
	if err != nil {
		log.Fatalf("openLogFile failed with '%s'\n", err)
	}

  // ... the rest of your program
}
Just like for regular os.Create() we have to ensure that the directory for log files exists with os.MkdirAll(dir, 0755).
Rotating files implies that the name of the file changes.
I like the convention of using date in the name of the file so dailyrotate uses time.Format formatting layout for file paths.
I use 2006-01-02.txt for the file format (which is YYYY-MM-DD + .txt).
It's easy to locate a log for a given day and the names sort by day.
Log file is opened in append mode.

Write to log file

dailyrotate.File implements io.Writer so to write we uses the standard f.Write(d []byte) (int, error).
func writeToLog(msg string) error {
	_, err := logFile.Write([]byte(msg))
	return err
}

Close log file

dailyrotate.File implements io.Close so to close we use Close() error.
It's safe to call Close multiple times.
func closeLogFile() error {
	return logFile.Close()
}

Execute code on log rotation

Imagine that when a log rotates you want to immediately backup the file to online storage like S3.
dailyrotate.NewFile takes an optional function that will be called when the log file is closed. A skeleton of a callback:
func onLogClose(path string, didRotate bool) {
	fmt.Printf("we just closed a file '%s', didRotate: %v\n", path, didRotate)
	if !didRotate {
		return
	}
	// process just closed file e.g. upload to S3 for backup
	go func() {
		// if processing takes a long time, do it in background
	}()
}
The file can be closed either because you called Close explicitly (e.g. because the program is exiting) or implicitly due to rotation.
You can distinguish the 2 cases with didRotate argument.

Rotate at a different time

By default logs rotate at midnight UTC time. We use UTC because "midnight" means different time in different timezones. In California, it's 5 PM and in Paris it's 2 AM.
You can change UTC to a different time zone by setting File.Location to a timezone Location (which you can load with time.LoadLocation):
loc, errr := time.LoadLocation("America/Los_Angeles")
if err != nil {
	log.Fatalf("time.LoadLocation() failed with '%s'\n", err)
}

f, err := dailyrotate.NewFile("2006-01-02.txt", nil)
if err != nil {
	log.Fatalf("dailyrotate.NewFile() failed with '%s'\n", err)
}

f.Location = loc
Updating...

Share on