Perfmon For The Pentium^® Pro User's Guide

Introduction

Perfmon for the Pentium Pro is a tool for monitoring the performance of the Intel Pentium Pro processor and the Intel P6 family of processors. It is a device driver that runs with the Sun Solaris^® operating system kernel. It allows user level or non-privileged code to read and write to the performance-monitoring counters on the processor. This tool can be used in refining hardware designs, optimizing code, or diagnosing some system failures. It can also be used as a research tool for cross-platform or cross-operating-system benchmarking and comparisons. For an in-depth look at perfmon design please refer to the Perfmon for the Pentium Pro Design document.

Supported Processors and Operating Systems

The following processors are supported.

• Pentium^® Pro processor.
• Pentium^® II processor.

Currently only the Solaris^® operating system is supported.

There is also Perfmon support for the UltraSPARC-I and UltraSPARC-II CPU's running the Solaris operating system. This perfmon can be obtained from the following link: http://www.cse.msu.edu/~enbody/perfmon/perfmon.html.

Installation

The driver source code, binary and documentation, including this document, are "tarred" in the file perfmon.tar. After downloading the tar file, uncompressing it and untarring it in a directory, the following directories should be created:

• docs - Contains this document and the perfmon design document.
• examples - Contains source code example programs showing how to use Perfmon for the Pentium Pro.
• include - Contains the perfmon.h file that has all the commands and functions declarations.
• lib - Contains the library of X86 functions that the user code has to link with.
• src - Has the X86 assembly code for the kernel driver as well as the kernel-driver interface code, plus the final binary driver.

An installation script, perfinstall, has been provided to install the perfmon device driver. This script has to be run with an administrator account. The script does the following:

• copy the driver files to the appropriate system directories.
• If there is an existing perfmon device driver, it removes it from the kernel and the /etc/devlink.tab file.
• adds the perfmon device driver to the kernel.
• adds an entry in the /etc/devlink.tab file
• does synching of the disk

The perfinstall script should be run from the directory where perfmon.tar has been expanded:

%<become root>
%cd <perfmon directory>
%perfinstall

Note that Perfmon will only work on the Intel Pentium Pro or the Intel P6 based machines.

Usage

Requirements

In order to write programs using Perfmon, you will need to have access to perfmon.h (found in the include directory), and libperfmon.a found in the lib directory. Perfmon.h contains all the commands needed to control the performance counters and read the time-stamp-counter (TSC). libperfmon.a contains the actual library routines that can be run in user level mode.

Note that code written using Perfmon will only run on machines where the Perfmon driver is installed and loaded. Attempting to run programs using Perfmon on other machines may result in strange behavior, core dumps, illegal instruction errors, etc.

Writing code

There are two types of Perfmon routines:

• driver routines - These are routines that live inside of the device driver because they need perform operations that are only allowed by code running in privileged mode.
• library routines - These routines are part of a library you link with your code.

The library routines can be run inside of a user application as you would expect. The driver routines need to be accessed via ioctl(). To use ioctl(), you must first open the Perfmon device (accessible through /dev/perfmon). After the device is open, you simply use ioctl() to communicate to the driver what routine you wish to run (passing arguments as necessary). Here is an example code segment which opens the device and issues a write back and invalidate external caches request on the current CPU:

	#include <stdio.h>
	#include <fcntl.h>
	#include "perfmon.h"

	main()
	{
	    int fd;
	    int rc;

	    fd = open("/dev/perfmon", O_RDONLY);
	    if (fd == -1) {
		perror("open(/dev/perfmon)");
		exit(1);
	    }
	    /* Enable reading of the TSC register and PerfCtrs */
	    rc = ioctl(fd, TSC_PERFCTRS_EN);
	    if (rc < 0) {
        	perror("ioctl(TSC_PERFCTRS_EN)");
                 exit(1);
            } 

	    /* Write back and invalidate all external caches of the current CPU */
	    rc = ioctl(fd, WBINVD_CACHES);
	    if (rc < 0) {
		perror("ioctl(PERFMON_FLUSH_CACHE)");
		exit(1);
	    }
	}

Pentium Pro Performance Monitoring Counters

The Pentium Pro processor has two 40-bit performance counters, allowing two types of events to be monitored simultaneously. These counters can either count events or measure duration. When counting events, a counter is incremented each time a specified event takes place or a specified number of events takes place. When measuring duration, a counter counts the number of processor clocks that occur while a specified condition is true. The counters can count events or measure duration that occur at any privilege level.

The performance monitoring counters are supported by four Model Specific Registers (MSR's): the performance event select registers (PerfEvtSel0 and PerfEvtSel1) , and the performance counter MSR's (PerfCtr0 and PerfCtr1). These registers reflect events that happen on a per-processor basis. For best results, it is recommended that you run your program on an MP machine and bind your process to a specific CPU to prevent process migration to another CPU and the loss of performance data that has been collected.

Access to the PerfEvtSel0/1 registers is privileged. They can only be accessed by using the PERFEVTSEL0/1_W and PERFEVTSEL0/1_R ioctl() routines in the Perfmon driver (see the next section for more details). Each of the PerfEvtSel0 and PerfEvtSel1 has the following bitfields (taken from section 10.6.1 Vol 3 of the Pentium Pro Family Developer's Manual):

All other bitfields are reserved and should be set to 0.

The performance-counter MSR's (PerfCtr0 and PerfCtr1) contain the event or duration count for the selected events being counted. Writing to these counters can be only done at privilege level 0 and is accomplished via ioctl() calls to the perfmon device driver. The perfmon also provides ioctl() call to make reading of the PerfCtr0/1 and the TSC register non-privileged. Reading the PerfCtr0/1 and the TSC register is done via perfmon library calls.

The include file perfmon.h has encodings for the various fields in the PerfCtr0/1 registers. These encodings are pre-shifted so that when they are inclusive-ORed together, they produce a value suitable for writing directly to the PerfCtr0/1 register. The defined values are:

For example, if you wanted to accumulate the number of instructions executed in PerfCtr0 in user-mode only, you would use: EVT_INST_RETIRED | USER_ MODE. This value would then be passed to the PERFEVTSEL0_W ioctl(). For more description of the above events please refer to Appendix A of the Intel Architecture Software Developer's Manual, Volume 3: System Programming Guide . This is a pdf file that is included with the perfmon documentation.

ioctl() Routines

Library Routines

These library routines are prototyped in perfmon.h and can be included in your code by adding the compile time options: -L$PERFMON_HOME/lib -lperfmon. These routines are in the table below:

Here is an example to compile a user source code, tick.c, and how to link it with the library routines and using the GNU gcc compiler:.

# Compile using the library routines:
gcc -I$(PERFMONHOME)/include -o tick -O tick.c -L$(PERFMON_HOME)/lib -lperfmon

Examples

In the examples directory, there are several example programs that show how to use Perfmon:

• tick - This is a simple program that shows how to open the Perfmon driver and do some simple reads of the TSC register and computing CPI and IPC. Try this with and without compiler optimization (-O option) to see how the results vary.
• matrix - This program computes CPI and IPC and the number of cycles while a miss to the Data Cache Unit (L1) is outstanding. It does the multiply using both a cold (empty) and warm (full) cache. This isn't the greatest example for showing the warm/cold differences, but with a really small matrix (50x50), you can see the difference. However, this is a good program for seeing the effects of compiler optimization on the number of instructions executed and CPI.
• tick_interesting - Same as tick above, but there is an #ifdef in the source. Read the source for more details.
• timeit - This program attempts to calculate the number of machine cycles required to call the library function gethrtime(). The function gethrtime() makes a trap into the kernel to get the high-res timer information. You can use this info to estimate how long a context switch takes on an P6 family machine runnin Solaris.