sar (System Activity Reporter) is an oldy but a goody that should be in your bag of tricks and installed on every *nix system you administer. It collects and logs system data at regular intervals, allowing analysis of system actity historically, trending, baselining and much much more. It comes as part of the systat package, along with the indispensible iostat. After installing the package, run the rc script to start the collection daemon for the first time. After that, it will be called by cron in crontabs installed by the package. The Red Hat and SuSE RPMs install the crontabs in /etc/cron.d/sysstat, so you can adjust the sampling rate there. You may want to increase the granularity for busy systems with performance issues. Reduce the frequency of sampling for systems that are cruising along.
How it Works
The sysstat crontab invokes a shell script, sa1, which in turn calls a program named sadc (System Activity Data Collector) which gathers data samples on CPU, memory, io, and network devices, creating a snapshot of system performance for that moment in time. The data is written out, in a binary format, to /var/log/sa/saXX where the XX is the two digit day of the month. This scheme allows for the files to be overwritten as the month rolls over, so if they need to be archived, set up a shell script to rotate them elsewhere. Also, be aware that, on Red Hat at least, the /usr/lib/sa/sa2 script which creates the daily summaries each night, also removes any sar data files older than 7 days. This is configurable in /etc/sysconfig/sysstat.
The sar program is invoked by the user. When called without the -f (file) option, it reads the file in /var/log/sa for today. When called with no options or arguments, sar gives us a nice little summary of CPU activity.
Linux 2.6.9-34.EL (sv-1.example.com) 02/10/2007 12:00:01 AM CPU %user %nice %system %iowait %idle 12:10:01 AM all 0.01 0.00 0.05 0.00 99.93 12:20:01 AM all 0.01 0.00 0.05 0.00 99.94 12:30:01 AM all 0.01 0.00 0.05 0.00 99.94
Use -f (file) to look at data from another day,
ot@sv-1 sa]# sar -f /var/log/sa/sa08 Linux 2.6.9-34.EL (sv-1.example.com) 02/08/2007 05:23:13 PM LINUX RESTART 05:30:01 PM CPU %user %nice %system %iowait %idle 05:40:01 PM all 0.05 0.00 0.38 0.08 99.49 05:50:01 PM all 0.07 0.00 0.65 0.01 99.27 06:00:01 PM all 0.07 0.00 0.71 0.02 99.20
sar can also be used to collect real time data at intervals. In this example, we sample at five second intervals three times.
[root@sv-1 sa]# sar 5 3 Linux 2.6.9-34.EL (sv-1.example.com) 02/10/2007 03:59:24 PM CPU %user %nice %system %iowait %idle 03:59:29 PM all 0.00 0.00 0.00 0.00 100.00 03:59:34 PM all 0.00 0.00 0.20 0.00 99.80 03:59:39 PM all 0.00 0.00 0.00 0.00 100.00 Average: all 0.00 0.00 0.07 0.00 99.93
With the -o (out) option, these stats can be written out to a file.
Sometimes, if someone’s looking over my shoulder and I blank out on which comes first, interval or count, I’ll take ten ten second intervals. But that won’t happen to you, because I’m about to offer you the first of Urbana’s supersar mnemonics. When you run sar to collect real time data, you can remember that interval comes first by repeating the phrase, “Sar, I See.” Interval count. I See.
The most common options you’ll want to use with sar are probably the options for disk activity, network activity, CPU, and memory. We’ll look at how to use sar for all these and more, in Part 2 of this article.