Using PerfPMR for Performance Diagnosis

The PerfPMR package was developed to ensure that reports of suspected performance problems in AIXwere accompanied by enough data to permit problem diagnosis by IBM. This makes the shell scripts in PerfPMR useful to other performance analysts as well. PerfPMR is an optionally installable part of the AIX Version 4 Base Operating System. It is located in /usr/sbin/perf/pmr. See the discussion in Installing AIX Version 4 PerfPMR. A version of PerfPMR is also available for AIX Version 3.2.5. See Obtaining and Installing AIX Version 3.2.5 PerfPMR.

The script perfpmr is the highest-level script of the package, but it collects data, such as configuration information, that a local performance analyst probably knows already. The lower-level script monitor collects a coordinated set of performance information for a specified number of seconds and summarizes the data. The syntax of monitor is:

monitor seconds [-n] [-p]

The seconds parameter must be at least 60. If seconds is 600 or less, the interval for the periodic reports is 10 seconds; otherwise, the interval is 60 seconds. The -n flag suppresses collection of netstat and nfsstat data. The -p flag suppresses collection of process-profile data (see below). The monitor script should not be run at the same time as any other operation that uses the system trace facility.

A single monitor request creates:

• A monitor.int file containing:
  • Combined output of ps -elk and ps gv commands run at the beginning and end of the monitoring period.
  • Output of a sar -A command with the appropriate interval.
  • Output of an iostat command with the appropriate interval. The initial, cumulative report is omitted.
  • Output of a vmstat command with the appropriate interval. The initial, cumulative report is omitted.
• A monitor.sum file containing:
  • For those processes that were active at both the beginning and end of the monitor run, the differences between end and start values of various resource-use statistics.
  • The "Average" lines from the sar -A command output.
  • Averages of the iostat interval statistics.
  • Averages of the vmstat interval statistics.
  • The "after - before" differences of the statistics produced by the vmstat -s command.
• If the -n option was not specified, a netstat.int file containing:
  • The output at the beginning of the monitor run of the following commands:

    netstat -v

    netstat -m

    netstat -rs

    netstat -s

  • The output of a netstat command with the appropriate interval.
  • The output at the end of the monitor run of the following commands:

    netstat -v

    netstat -m

    netstat -rs

    netstat -s

• If the -n option was not specified, an nfsstat.int file containing:
  • The output at the beginning and end of the monitor run of a nfsstat -csnr command.
• If the -p option was not specified, a pair of files named Pprof.stt and Pprof.flow. Pprof.stt contains the starting and ending times of the run. Pprof.flow contains process-profile data. The columns in the Pprof.flow file are:
  1. Process name
  2. Process ID
  3. Time of first occurrence of the process within the measurement period
  4. Time of last occurrence of the process
  5. Total process execution time
  6. Begin/end flag (sum of Begin + End, below). Describes the beginning and ending state of the process.

     Begin:
       execed:          0
       forked:          1
       Alive at Start:  2
     End:
       Alive at end:    0
       execed away:     4
       Exited:          8

  7. Parent process ID

Check before You Change

One particularly important use of the PerfPMR package is the creation of a configuration and performance baseline prior to a significant change in system hardware or software. Just as you probably back up critical files before such a change, you should make a record of the configurations and the performance they were providing. If a performance degradation should occur after the change, you will have detailed data that will let you perform a rigorous before-and-after analysis of the system.

To get the most complete data possible, you should run:

$ perfpmr 3600

during the busiest hour of the day. The output files from this measurement run will appear in directory /var/perf/tmp. (If you are running on a pre-Version 4 system, the output files will appear in the current working directory.) Be sure to move these files to a safe haven before beginning the configuration change.