1The following are additional notes on the cputimes command,
2
3
4* How cputimes works
5
6cputimes accurately measures time consumed by the kernel, idle therads
7and processes, by tracking the activity of the schedular. In particular we
8track on-cpu and off-cpu events for kernel therads, measuring the timestamps
9at each event.
10
11
12* Why cputimes?
13
14If you are interested in how much time processes are consuming, the data 
15given by "prstat" or "prstat -m" is fine. However there is no easy way to 
16see kernel consumed time, which is the idea behind cputimes.
17
18
19* What does it mean?
20
21The output shows categories of threads by the sum of time, in nanoseconds.
22
23A nanosecond is 10^-9, or 0.000000001 of a second. This program uses
24nanoseconds as units, but does not have nanosecond accuracy. It would be 
25reasonable to assume that this has microsecond accuracy (10^-6), so in 
26practise ignore the last three digits of the times.
27
28The sections reported are,
29
30PROCESSES - the sum of all the process time on the CPU. 
31KERNEL - the sum of the time spent in the kernel.
32IDLE - the time the kernel spent in the idle thread, waiting for some work.
33
34If your system isn't doing much, then the idle time will be quite large. If
35your system is running many applications, then there may be no idle time
36at all - instead most of the time appearing under processes.
37
38
39* When is there a problem?
40
41Expect to see most of the time in processes or idle, depending on how busy 
42your server is. Seeing a considerable amout of time in kernel would 
43definately be interesting.
44
45The kernel generally doesn't use much CPU time, usually less than 5%. 
46If it were using more, that may indicate heavy activity from an interrupt 
47thread, or activity caused by DTrace. 
48
49For example,
50
51   # cputimes 1
52   2005 Apr 27 23:49:32,
53            THREADS        TIME (ns)
54               IDLE         28351679
55             KERNEL        436022725
56            PROCESS        451304688
57
58In this sample the kernel is using a massive amount of the CPUs, around 47%.
59This sample was taken during heavy network utilisation, the time consumed 
60by the TCP/IP and network driver threads (and DTrace). The "intrstat" command
61could be used for further analysis of the interrupt threads responsible
62for servicing the network interface.
63
64
65* Problems with cputimes
66
67The way cputimes measures schedular activity turns out to be a lot of work. 
68There are many scheduling events per second where one thread steps onto a 
69CPU and another leaves. It turns out that cputimes itself causes some degree 
70of kernel load.
71
72Here we run 1 cputimes,
73
74   # cputimes 1
75   2005 May 15 12:00:41,
76            THREADS        TIME (ns)
77             KERNEL         12621985
78            PROCESS        982751579
79   2005 May 15 12:00:42,
80            THREADS        TIME (ns)
81             KERNEL         12267577
82            PROCESS        983513765
83   [...]
84
85Now a second cputimes is run at the same time,
86
87   # cputimes 1
88   2005 May 15 12:02:06,
89            THREADS        TIME (ns)
90             KERNEL         17366426
91            PROCESS        978804165
92   2005 May 15 12:02:07,
93            THREADS        TIME (ns)
94             KERNEL         17614829
95            PROCESS        978671601
96   [...]
97
98And now a third,
99
100   # cputimes 1
101   2005 May 15 12:03:09,
102            THREADS        TIME (ns)
103             KERNEL         21303089
104            PROCESS        974925124
105   2005 May 15 12:03:10,
106            THREADS        TIME (ns)
107             KERNEL         21222992
108            PROCESS        975152727
109   [...]
110
111Each extra cputimes is consuming an extra 4 to 5 ms of the CPU as kernel time.
112Around 0.5%. This can be used as an estimate of the kernel load caused by 
113running cputimes, and a similar strategy could be used to measure the kernel 
114load of other DTrace scripts.
115
116However the following CPU characteristics must be taken into consideration,
117
118   # psrinfo -v
119   Status of virtual processor 0 as of: 05/15/2005 12:06:05
120     on-line since 04/30/2005 13:32:32.
121     The i386 processor operates at 867 MHz,
122           and has an i387 compatible floating point processor.
123
124as well as the type of activity that was also running on the system, which
125cputimes was monitoring (frequency of scheduling events).
126
127A system with a slower CPU will use a larger proportion of kernel time to
128perform the same tasks. Also, a system that is context switching more 
129(switching between different processes) is likely to consume more kernel time
130as well.
131
132
133
134