Perfmon For The Pentim^® Pro Design

Perfmon

Perfmon for the Pentium Pro is a tool that allows user-level code to access the performance counters present in the Intel^® P6 family of microprocessors running the Solaris^® operating system by Sun^® Microsystems. This is accomplished by a loadable driver that re-programs devices with performance counters so that user-level code can access these counters (normally, access to these counters is restricted to code running in privileged mode). Accessing the performance counters requires special machine instructions. There is a user library component of Perfmon that provides access to these instructions via C function calls. The library also includes a function for accessing the Time-Stamp-Counter (TSC) that is incremented every processor clock tick. Currently, the only devices supported are the Pentium Pro and the Pentium II processors. See the section on Future Work for devices that may be supported in future versions of Perfmon.

Background

There are two parts to collecting performance data on the P6 CPU. The first is to program one or both of the Performance-Event-Select registers (PerfEvtSel0/1) indicating the type of events that you wish to count. Access to the PerfEvtSel0/1 is always privileged and requires a call into the Perfmon device driver. The second part is to read one or both of the Performance Counter (PerfCtr0/1) registers to get the current count of the watched events. Access to the PerfCtr0/1 is normally privileged, but can be made non-privileged by setting the PCE bit (bit 8) of the CR4 register. (Control Register 4). In addition , user access to the TSC register (which is incremented once per machine clock cycle at all times) is also enabled. This is done by turning off the TSD bit of the CR4 register. See the Perfmon For the Pentium Pro User's Guide and the the Intel Architecture Software Developer's Manual, Volume 3: System Programming Guide for more information.

Design

There were two basic requirements in designing Perfmon. The first was to allow user programs to access the performance counters, which is a privileged operation. The second was to have lightweight access to the accumulated data to minimize the amount of error introduced by the act of reading the performance registers as well as the TSC register.

Kernel Component

Since access to the PerfEvtSel0/1 is always privileged, and access to the PerfCtr0/1 and TSC is by default privileged, it was necessary to write some code that runs in the kernel context. For maximum flexibility and ease of installation, it was decided to write a loadable device driver rather than have a specially modified kernel.

The loadable driver is a standard, autoconfiguring SVR4 character device driver. In addition to the static structures and functions needed to support a device driver (see Writing Device Drivers in AnswerBook for more details), there are two functions in the device driver that needed Perfmon-specific code to be written.

When the device driver is initially loaded, there is a certain sequence that of events that happen:

• The kernel loads the device driver into kernel space and calls the driver's _init() routine (which must exist). The _init() function creates a mutex lock (for later use) and then calls the kernel routine mod_install() which does some housekeeping, allocation of kernel data structures, and does run-time linking to resolve any entries from the driver symbol table against the kernel symbol table.
• The driver then stays in this state until it is probed by the autoconfiguration code in the kernel (which is usually initiated with the drvconfig command). The kernel looks up the driver-specific attach function through the dev_ops structure (which must exist) and winds up calling perfmon_attach().
• The perfmon_attach() function calls ddi_create_minor_node() which creates the device node (using the major device number found for the driver in /etc/name_to_major) under the /devices directory. Assuming that the driver attach succeeded, perfmon_attach() calls perfmon_init() which finishes up driver initialization.

At this point, the driver is loaded and ready to accept requests from the user. The communication channel that is used between a user program and the device driver is the ioctl() system call. The sequence of events from the driver's point of view is:

• The user program performs and open() on the device node, /dev/perfmon. This ends up calling perfmon_open(), which the kernel locates through the static cb_ops structure. In the case of the Perfmon device driver, there is no special permission checking or state information that needs to be taken care of upon an open(), so perfmon_open() always returns successfully.
• The user then calls ioctl() at various points through the program, which in turn calls perfmon_ioctl(), which the kernel finds through the cb_ops structure, to handle the request. For each of the supported ioctl()s, here is what happens:
  • TSC_PERFCTRS_EN: an ioctl() call with this command invokes the pm_tsc_perfctrs() call which sets the appropriate bits in the CR4 register to enable non-privileged access to the TSC and the PerfCtr0/1 registers.
  • PERFEVTSEL0_W: The user passes in a pointer to a 32-bit value that is to be stored in the PerfEvtSel0 register. But since the pointer is to a user address, and is not valid in kernel space, we need to call copyin() to map the user's buffer into the kernel so that we can get at the value. After we have the value, we simply call pm_set_perfevtsel0() which sets the value of the PerfEvtSel0 register.
  • PERFEVTSEL1_W: This is same as for the above but for is done for PerfEvtSel1 register.
  • PERFEVTSEL0_R: The user passes in a pointer to a 32-bit buffer in which the current value of the PerEvtSel0 register is to be stored. We have the same memory mapping problem as before, so we call pm_get_perfevtsel0() to get the current PerfEvtSel0 value and then store it in the user's buffer using copyout().
  • PERFEVTSEL1_R: This is the same as above except it is done for the PerfEvtSel1 register.
  • STARTPERFCTRS: This command invokes the pm_startperfctrs() function that sets bit 22 of the PerfEvtSel0 register to enable both PerCtr0/1 registers.
  • STOPPERFCTRS: This command does the opposite of the above command and clears bit 22 of the PerfEvtSel0 register.
  • STOPERFCTR0: This command write zero value to the PerfEvtSel0 register disabling PerfCtr0 but it leave bit 22 unchanged.
  • STOPERFCTR1: This is similar to the above except it writes zero value into the PerfEvtSel1 register disabling PerfCtr1.
  • PERFCTR0_W: The user passes in a pointer to 32-bit buffer that is to stored in the lower 32-bit of the PerfCtr0 register the upper 8 bits are sign-extended from bit 31.
  • PERFCTR1_W: The same as above except it is done for the PerfCtr1 register.
  • WBINVD_CACHES: This command invokes the function pm_wbinvd() which simply issues the wbind instruction. This instruction can only executed in privileged code and causes all external caches to be written back to memory then invalidated.
• The user program then performs a close() on the device, which calls perfmon_close(), which the kernel located through the cb_ops structure. Since we needed no special processing on a device open, we also need nothing special on close, so perfmon_close() always returns successfully.

If the driver is ever unloaded, there is a sequence of events that take place:

• The kernel calls the driver detach routine, pointed to by the dev_ops structure, perfmon_detach(). This function removes the device node, which was created during attach, by calling the function ddi_remove_minor_node().
• The kernel then calls the driver's _fini() function which does some housekeeping. It first calls mod_remove(), which takes the driver out of the kernels data structures. It then calls perfmon_cleanup() to do any driver-specific cleanup, which is nothing in our case. Lastly, the mutex lock we created earlier is destroyed and its memory freed.

User Component

The library functions are all written in assembly since they need to use special machine instructions to do their work. Both of the read_perfctr0 () and read_perfctr1() calls as well as the read_tsc() call return a 64-bit value. Calling these functions can only be done after calling the ioctl() call with the TSC_PERFCTRS_EN command. The read_perfctr0/1() calls simply issue a RDPMC instruction to read the appropriate counter. The read_tsc() function issues a RDTSC instruction.

Also a cpu_serialize() call has been implemented. This call issues a serializing instruction, specifically CPUID, so that all modification to flags, registers, and memory by previous instructions are completed before the next instruction is fetched and executed and all buffered writes have been drained to memory. This call, however is not necessary since serializing is implemented in the read_perfctr0/1 () calls by using a CPUID instruction. All the other calls via ioctl() to the driver are serialized by nature since using the privileged instructions (RDMSR, WRMSR) are serializing instructions, and therefore, the user needs not to use the cpu_serialize() call except for possible special cases.

Another issue for writing the user-level code was the fact that the performance counters are kept on a per-processor basis rather than a per-process basis. This means that if you run your program on an MP machine, and it migrates between CPUs during its run-time (which is pretty likely given Solaris' work-grabbing scheduler), the data read from the CPU performance counters is useless. Fortunately, there is a non-privileged system call named processor_bind() that will let you bind your process (or a single LWP) to a particular CPU.

Also, since there is some setup required by most programs using Perfmon, a skeleton program was provided to minimize development and testing of programs. The basic outline of the skeleton program is:

• Parse command-line arguments. Since you usually want to bind your process to a CPU to prevent migration, it would be handy to be able to specify which CPU you wish to be bound to. By default, the process is bound to CPU 0, but by specifying the "-r " option, you can dictate which CPU on which you wish the program to be bound.
• Bind to the specified processor (CPU 0 by default).
• Get a handle to the Perfmon device driver by opening /dev/perfmon.
• Enable reading of the PerfCtr0/1 and the TSC register by calling ioctl().
• Set the PerfCtr0 register to accumulate CPU cycles in PerfCtr0 and instructions retired in PerfCtr1 while in user mode. This can be easily changed.
• Run an example function. This includes the following steps to get accurate data:
  • Call gethrtime() to get the current value of the hi-res clock.
  • Call pm_set_perfctr0/1() to clear the values of both PerfCtr0/1 registers (this is to help prevent overflow from occurring).
  • Call pm_start_perfctrs()
  • Call the function that does the computations in which you're interested.
  • Call pm_stop_perfctrs()
  • Get the current value of both PerfCtr0/1 registers.
  • Call gethrtime().
  • Lastly, the elapsed wall-clock time, instructions executed, CPU cycles taken, and CPI are calculated and printed.

Interaction

The interaction between user code and driver code of a typical user program using Perfmon would go something like this:

            User Code             |             Kernel Code
       C             assembly     |      C                     assembly
----------------------------------+------------------------------------------------
open("/dev/perfmon")              | perfmon_open()                   
ioctl(TSC_PERFCTRS_EN)            | perfmon_ioctl()                  
                                  |                       pm_tsc_perfctrs_enable()
ioctl(PERFEVTSEL0_W)              | perfmon_ioctl()                   
                                  | copyin()                         
                                  |                       pm_set_perfevtsel0()
                                  |                                  
ioctl(WBINVD_CACHES)              | perfmon_ioctl()                  
                                  |                       pm_wbinvd()               
ioctl(PERFCTR0_W)                 | perfmon_ioctl()                                 
gethrtime()                       |                       pm_set_perfctr0()         
ioctl(STARTPERFCTRS)              | perfmon_ioctl()                     
                                  |                       pm_start_perfctrs()      
                                  |                                  
Run code to be analyzed           |                                  
                                  |                                  
                                  |                                  
ioctl(STOPPERFCTRS)               | perfmon_ioctl()                  
                                  |                       pm_stop_perfctrs()
                  read_perfctr0() |                                  
                                  |                                  
gethrtime()                       |                                  
Analyze results                   |                                  
close("/dev/perfmon")             | perfmon_close()                  
                                  |                                  

Installation

Since Perfmon requires the installation of a device driver on every machine that is to run it, the installation procedure was scripted so it can be used by the system administrator. Running the script does the following:

• All necessary files are installed into the proper places on the system. In our case, there are just two files, which make up the device driver:
  • /platform/i86pc/kernel/drv/perfmon
  • /platform/i86pc/kernel/drv/perfmon.conf
  • The file /etc/devlink.tab is examined and any existing entries relating to Perfmon are deleted.
  • If the Perfmon driver is loaded, it is unloaded and its device node is removed.
  • The Perfmon driver is added to the system via the add_drv command. This causes a major device number to be picked by Solaris and registered in the file /etc/name_to_major. The file /etc/minor_perm is updated to reflect the desired permission on the Perfmon device node when it gets created.
  • The file /etc/devlink.tab gets the entry for the Perfmon device driver added to it. This will cause a symbolic link to be created from /dev/perfmon to the actual device node, which usually resides at /devices/pseudo/perfmon@0:perfmon. This is just for convenience to the user.
  • The perfmon driver is loaded and its device node created. This is forced by calling drvconfig -i perfmon.
  • The symbolic link is created according to the rules in /etc/devlink.tab. This is forced by running the program devlinks

At this point, the Perfmon driver is installed, loaded, and ready for use.

Future work:

• Support for other Processors: Currently there is Perfmon for the Intel P6 family running Solaris and Perfmon for the UltraSPARC-I and UltraSPARC-II CPU's running Solaris. Possible other processors are the MIPS^® family of processors for the Silicon Graphics workstations or the DEC Alpha family of processors. Other operating systems such Windows NT^® are possible candidate for future support.
  
• Support for other I/O devices: Non-CPU devices typically have memory-mapped registers that allow access to the performance counters. These registers are accessible only to code running in privileged mode. One possible way to allow user code access to these registers is to extend the Perfmon driver to support the mmap() routine. User programs would use the mmap() system call on the /dev/perfmon file descriptor. The device driver could then use the passed-in memory address to decode which ASIC registers the user wanted access to and provide a mapping for those memory addresses to the user's context.

• Support for per-context counters - This would mainly be for CPU performance counters, but might be extended for other devices as well. In order to provide per-context information, the context switching routines in the kernel would have to be modified to save the desired performance counters. There are a couple of downsides to this, though. First, the core kernel would have to be modified, which would require installing a special kernel/driver set for each machine on which you wish to collect performance data. Second, there should be a way of turning per-context collecting on or off, otherwise you would need to keep rebooting and switching kernels. Lastly, per-context data may not make sense for all devices.