Chapter 13. Handling a Possible AIX Performance Bug

If you believe that you have found a possible performance problem in AIX, there are tools and procedures for reporting the problem and supplying problem-analysis data. They are intended to ensure that you get a prompt and accurate response with a minimum of effort and time on your part.

The main sections in this topic are:

• Measuring the Baseline
• Reporting the Problem
• Obtaining and Installing AIX Version 3.2.5 PerfPMR
• Installing PerfPMR from the Web (includes Version 4)
• Problem-Analysis Data

Measuring the Baseline

Performance problems are often reported right after some change to the system's hardware or software. Unless there is a pre-change baseline measurement with which to compare post-change performance, quantification of the problem is impossible. Still better would be collection of a full set of performance and configuration information using the PerfPMR package, as recommended in Check Before You Change.

Having the Performance Diagnostic Tool (PDT) installed and operational also provides a baseline of overall system performance.

Reporting the Problem

You should report suspected AIX performance problems to the IBM Software Service organization. Use your normal software problem-reporting channel. If you are not familiar with the correct problem-reporting channel for your organization, check with your IBM representative.

When you report the problem, you should supply the following basic information:

• A description of the problem that can be used to search the problem-history database to see if a similar problem has already been reported.
• What aspect of your analysis led you to conclude that the problem is due to a defect in AIX?
• What is the hardware/software configuration in which the problem is occurring?
  • Is the problem confined to a single system, or does it affect multiple systems?
  • What are the models, memory sizes, and number and size of disks on the affected system(s)?
  • What kinds of LAN and other communications media are connected to the system(s)?
  • Does the overall configuration include non-AIX systems? Non-UNIX systems?
• What are the characteristics of the program or workload that is experiencing the problem?
  • Does an analysis with time, iostat, and vmstat indicate that it CPU-limited or I/O-limited?
  • Are the workloads being run on the affected system(s): workstation, server, multiuser, or a mixture?
• What are the performance objectives that are not being met?
  • Is the primary objective in terms of console or terminal response time, throughput, or real-time responsiveness?
  • Were the objectives derived from measurements on another AIX system? If so, what was its configuration?

If this is the first report of the problem, you will receive a PMR number for use in identifying any additional data you supply and for future reference.

You will probably be asked to provide data to help IBM analyze the problem. An IBM-provided tools package called PerfPMR can collect the necessary data. On AIX Version 3.2.5, PerfPMR is an informal tool available from your IBM representative. On AIX Version 4, PerfPMR is an optionally installable package on the AIX Base Operating System distribution medium.

Obtaining and Installing AIX Version 3.2.5 PerfPMR

Your IBM representative can obtain a copy of AIX Version 3.2.5 PerfPMR on suitable media.

To install PerfPMR, you:

1. Log in as root or use the su command to obtain root authority.
2. Create the perfpmr directory and move to that directory (this example assumes the directory built is under /tmp).

   # cd /tmp
   # mkdir perfpmr
   # cd perfpmr

3. Copy the compressed tar file from diskette (this example assumes the diskette drive used is fd0):

   # tar -xvf/dev/fd0 perfpmr.tarbinz

4. Rename the compressed tar file:

   # mv perfpmr.tarbinz perfpmr.tarbin.Z

5. Uncompress the tar file with:

   # uncompress perfpmr.tarbin.Z

6. Extract the shell scripts from the tar file with:

   # tar -xvf perfpmr.tarbin

7. Install the shell scripts with:

   # ./Install

Installing PerfPMR from the Web (includes Version 4)

To acquire the correct version of perfpmr, visit IBM's web site: www.ibm.com. Enter perfpmr in the search box, and search All IBM. The result of the query is a window entitled "AIX Performance PMR Data Collection Scripts - perfpmr." Click the download button to transfer to the perfpmr ftp site. Several different versions of perfpmr are listed (for example, perf32, perf41, perf42, and perf43).

Each folder contains the following:

To install PerfPMR from the Web, you:

1. Download the correct version of perfpmr into /tmp:
2. Log in as root or use the su command to obtain root authority
3. Create the perf41 directory (for example) and move to that directory

   # mkdir /tmp/perf41
   
   # cd /tmp/perf41

4. Extract the shell scripts out of the compressed tar file:

   # zcat /tmp/perf41.tar.Z | tar -xvf -

5. Install the shell scripts

   # sh ./Install

The installation process places the PerfPMR package in a directory called /usr/sbin/perf/pmr. The package takes approximately 200KB of disk space.

Problem-Analysis Data

All of the following items should be included when the supporting information for the PMR is first gathered:

• A means of reproducing the problem
  • If possible, a program or shell script that demonstrates the problem should be included.
  • At a minimum, a detailed description of the conditions under which the problem occurs is needed.
• Data collected by the PerfPMR tools
  • On each system involved
  • At the same time
  • While the performance problem is occurring
• The application experiencing the problem
  • If the application is, or depends on, a software product, the exact version and release of that product should be identified, even if the software is not an IBM product.
  • If the source code of a user-written application cannot be released, the exact set of compiler parameters used to create the executable should be documented.

Capturing the Data

To capture and package the data in usable form, perform the following steps on each of the systems involved with the problem. If possible, step 6 should be performed on all of the systems at (approximately) the same time.

1. Log in as root or use the su command to obtain root authority.
2. PerfPMR captures more information if the tprof, filemon, and netpmon performance tools are available. In AIX Version 4, these tools are packaged as part of the Performance Toolbox for AIX. To determine whether the performance tools have been installed on the system, check with:

   $ lslpp -lI perfagent.tools

   If this package has been installed, the tools are available.

3. Make sure that your PATH variable includes the directory that contains the PerfPMR executables.

   In AIX Version 4, add /usr/sbin/perf/pmr to the PATH. For example:

   # echo $PATH
   /usr/bin:/etc:/usr/local/bin:/usr/ucb:.:
   # PATH=$PATH:/usr/sbin/perf/pmr:
   # export PATH

   In Version 3, add to the PATH the directory in which you installed PerfPMR (in place of /usr/sbin/perf/pmr) and the directory for the performance tools, /usr/lpp/bosperf.

4. In Version 4, the output of perfpmr will be written to /var/perf/tmp. In Version 3, you should:
   1. cd to a suitable directory, such as /tmp, in a file system that has at least 5MB of free space.
   2. Create a subdirectory to hold the data and switch to it, with:

      # mkdir perfdata
      # cd perfdata

5. Track system activity for 1 hour with:

   # perfpmr 3600

   (in Version 3, perfpmr is named perfpmr.sh.)

6. Combine the files into one compressed tar file with:

   # cd ..
   # tar -cvf pmrnumber.tarbin perfdata
   # compress pmrnumber.tarbin

   Where pmrnumber is the number assigned to the PMR by Software Service.

7. Put the file on a diskette (or other portable volume) with, for example:

   # tar -cvf /dev/fd0 pmrnumber.tarbin.Z

8. Label the portable volume with:
   • PMR number
   • Date the information was gathered
   • Command and flags that should be used to remove the data from the portable volume, for example:

     # tar -xvf /dev/fd0

9. Within the United States, send the data to:

   IBM Corporation, Mail Drop 2900
   11400 Burnet Road
   Austin, TX 78758
   Attn: AIX Program Services: V4DEFECT (or V3DEFECT, as appropriate)

   Outside the United States, send the data to your IBM Software Service organization.