Logging is a nearly indispensable part of any networked service, but especiallywith high traffic installations care needs to be taken to ensure that log filesdon’t exhaust storage space and cause maintenance or outage nightmares.
Traffic Server provides a two-pronged solution: log rotation (also called log rolling) tokeep individual logs as manageable in size as possible for easier ingestion andanalysis by humans and other programs, and log retention to keep logs fromusing more space than available and necessary.
This section covers both features.
Rotation Options¶
Traffic Server provides automatic log file rolling. At specific intervals during the dayor when log files reach a certain size, Traffic Server closes its current set of logfiles and opens new log files. Depending on the amount of traffic your serversare exposed to, you may find that increasing the frequency of log rolling isbeneficial, or even necessary, to maintain manageable log file sets. Traffic Server nodesprocessing moderately high levels of traffic may want to start by rolling logsevery six hours, and adjusting from there.
Log file rolling offers the following benefits:
It defines a consistent interval over which log analysis can be performed.
It keeps any single log file from becoming too large and helps tokeep the logging system within the specified space limits.
It provides an easy way to identify files that are no longer beingused so that an automated script can clean the logging directory andrun log analysis programs.
Rolled Log Filename Format¶
Traffic Server provides a consistent naming scheme for rolled log files that enables youto easily identify log files. When Traffic Server rolls a log file, it saves and closesthe old file before it starts a new file. Traffic Server renames the old file to includethe following information:
The original log file’s name (such as
access.log).The hostname of the Traffic Server node that generated the log file.
Two timestamps separated by a hyphen (
-). The first timestamp isa lower bound for the timestamp of the first record in the logfile. The lower bound is the time when the new buffer for log recordsis created. Under low load, the first timestamp in the filename canbe different from the timestamp of the first entry. Under normalload, the first timestamp in the filename and the timestamp of thefirst entry are similar. The second timestamp is an upper boundfor the timestamp of the last record in the log file (this isnormally the rolling time).The suffix
.old, which makes it easy for automated scripts tofind rolled log files.
Timestamps have the following format:
%Y%M%D.%Hh%Mm%Ss-%Y%M%D.%Hh%Mm%Ss
The following table describes the format:
Format | Description | Sample |
|---|---|---|
| The year in four-digit format. | 2000 |
| The month in two-digit format, from 01-12. | 07 |
| The day in two-digit format, from 01-31. | 19 |
| The hour in two-digit format, from 00-23. | 21 |
| The minute in two-digit format, from 00-59. | 52 |
| The second in two-digit format, from 00-59. | 36 |
The following is an example of a rolled log filename:
access.log.mymachine.20110912.12h00m00s-20000913.12h00m00s.old
The logging system buffers log records before writing them to disk. Whena log file is rolled, the log buffer might be partially full. If it is,then the first entry in the new log file will have a timestamp earlierthan the time of rolling. When the new log file is rolled, its firsttimestamp will be a lower bound for the timestamp of the first entry.
For example, suppose logs are rolled every three hours, and the firstrolled log file is:
access.log.mymachine.20110912.12h00m00s-19980912.03h00m00s.old
If the lower bound for the first entry in the log buffer at 3:00:00 is2:59:47, then the next log file will have the following timestamp whenrolled:
access.log.mymachine.20110912.02h59m47s-19980912.06h00m00s.old
The contents of a log file are always between the two timestamps. Logfiles do not contain overlapping entries, even if successive timestampsappear to overlap.
Rolling Intervals¶
Log files are rolled at specific intervals relative to a given hour of the day.Three options may be used to control when log files are rolled:
A file size threshold, which will prevent any individual log from growingtoo large.
The offset hour, which is an hour between
0(midnight) and23.The rolling interval.
Both the offset hour and the rolling interval determine when log file rollingstarts. Rolling occurs every rolling interval and at the offset hour. Forexample, if the rolling interval is six hours and the offset hour is 0(midnight), then the logs will roll at midnight (00:00), 06:00, 12:00, and18:00 each day. If the rolling interval is 12 hours and the offset hour is3, then logs will roll at 03:00 and 15:00 each day.
To set log file rolling options and/or configure Traffic Server to roll log files whenthey reach a certain size, adjust the following settings inrecords.config:
Enable log rolling with proxy.config.log.rolling_enabled.
CONFIG proxy.config.log.rolling_enabled INT 1
Configure the upper limit on log file size withproxy.config.log.rolling_size_mb.
CONFIG proxy.config.log.rolling_size_mb INT 1024
Set the offset hour with proxy.config.log.rolling_offset_hr.
CONFIG proxy.config.log.rolling_offset_hr INT 0
Set the interval (in seconds) withproxy.config.log.rolling_interval_sec.
CONFIG proxy.config.log.rolling_interval_sec INT 21600
Set the minimum number of rolled files withproxy.config.log.rolling_min_count.
CONFIG proxy.config.log.rolling_min_count INT 0
Run the command traffic_ctl config reload to apply the configurationchanges.
You can fine-tune log file rolling settings for individual log files in thelog.* specification in logging.yaml. The custom log file uses therolling settings provided in the relevant log function call, which overridethe default settings you specify in Traffic Manager or records.configdescribed above.
Retention Options¶
Traffic Server enables you to control the amount of disk space that the logging directorycan consume. This allows the system to operate smoothly within a specifiedspace window for a long period of time. After you establish a space limit,Traffic Server continues to monitor the space in the logging directory. When the freespace dwindles to the headroom limit, it enters a low space state and takes thefollowing actions:
If the autodelete option is enabled, then Traffic Server identifies previously-rolledlog files (log files with the
.oldextension). It starts deleting filesone by one, beginning with the oldest file with largest ratio between currentnumber of files and the minimum rolling count, until it emerges from the lowstate. The default minimum rolling count of 0 is treated as INT_MAX duringratio calculation. Hence the rolling_min_count is not guaranteed to bereserved; instead, it is used as a reference to decide the priority of logfiles to delete. In low space state, even when all log files are below minimumcount, Traffic Server still tries to delete files until it emerges from the low state.Traffic Server logs a record of all deleted files in the system error log.If the autodelete option is disabled or there are not enough old log filesto delete for the system to emerge from its low space state, then Traffic Serverissues a warning and continues logging until the allocated log space isexhausted (which, if configured appropriately, will be well before youractual filesystem space is fully consumed and causes additional problems).At this point, event logging stops even though proxy traffic is still servedwithout client-visible interruption. Traffic Server resumes event logging when enoughspace becomes available for it to exit the low space state. To make spaceavailable, either explicitly increase the logging space limit or removefiles from the logging directory manually.
You can run a cron(8) script in conjunction with Traffic Server toautomatically remove old log files from the logging directory before Traffic Serverenters the low space state. Relocate the old log files to a temporarypartition, where you can run a variety of log analysis scripts. Followinganalysis, either compress the logs and move to an archive location, or simplydelete them.
Traffic Server periodically checks the amount of log space used against both the logspace allocation configured by proxy.config.log.max_space_mb_for_logsand the actual amount of space available on the disk partition. The used logspace is calculated by summing the size of all files present in the loggingdirectory and is published in theproxy.process.log.log_files_space_used metric. Theproxy.config.log.max_space_mb_headroom configuration variablespecifies an amount of headroom that is subtracted from the log spaceallocation. This can be tuned to reduce the risk of completely filling the diskpartition.
Setting Log File Management Options¶
To set log management options, follow the steps below:
In the records.config file, edit the following variables
proxy.config.log.max_space_mb_for_logs
proxy.config.log.max_space_mb_headroom
Run the command traffic_ctl config reload to apply the configurationchanges.
Retaining Logs For No More Than a Specified Period¶
If for security reasons logs need to be purged to make sure no log entry remains on the boxfor more then a specified period of time, we could achieve this by setting the rolling interval,the maximum number of rolled log files, and forcing Traffic Server to roll even when there is no traffic.
Let us say we wanted the oldest log entry to be kept on the box to be no older than 2-hour old.
Set proxy.config.log.rolling_interval_sec (yaml: rolling_interval_sec) to 3600 (1h)which will lead to rolling every 1h.
Set proxy.config.log.rolling_max_count (yaml: rolling_max_count) to 1which will lead to keeping only one rolled log file at any moment (rolled will be trimmed on every roll).
Set proxy.config.log.rolling_allow_empty (yaml: rolling_allow_empty) to 1 (default: 0)which will allow logs to be open and rolled even if there was nothing to be logged during the previous period(i.e. no requests to Traffic Server).
The above will ensure logs are rolled every 1h hour, only 1 rolled log file to be kept(rest will be trimmed/removed) and logs will be rolling (“moving”) even if nothing is logged(i.e. no traffic to Traffic Server).
