Categories
benchmarking time unix

What do ‘real’, ‘user’ and ‘sys’ mean in the output of time(1)?

2112

$ time foo
real        0m0.003s
user        0m0.000s
sys         0m0.004s
$

What do real, user and sys mean in the output of time?

Which one is meaningful when benchmarking my app?

3

  • 2

    @Casillass Real – stackoverflow.com/questions/2408981/…

    Mar 6, 2015 at 16:46

  • 13

    If your program exits that fast, none of them are meaningful, it’s all just startup overhead. If you want to measure the whole program with time, have it do something that will take at least a second.

    Nov 25, 2017 at 0:01

  • 14

    It is really important to note that time is a bash keyword. So typing man time is not giving you a man page for the bash time, rather it is giving the man page for /usr/bin/time. This has tripped me up.

    Jul 3, 2019 at 12:33

2471

Real, User and Sys process time statistics

One of these things is not like the other. Real refers to actual elapsed time; User and Sys refer to CPU time used only by the process.

  • Real is wall clock time – time from start to finish of the call. This is all elapsed time including time slices used by other processes and time the process spends blocked (for example if it is waiting for I/O to complete).

  • User is the amount of CPU time spent in user-mode code (outside the kernel) within the process. This is only actual CPU time used in executing the process. Other processes and time the process spends blocked do not count towards this figure.

  • Sys is the amount of CPU time spent in the kernel within the process. This means executing CPU time spent in system calls within the kernel, as opposed to library code, which is still running in user-space. Like ‘user’, this is only CPU time used by the process. See below for a brief description of kernel mode (also known as ‘supervisor’ mode) and the system call mechanism.

User+Sys will tell you how much actual CPU time your process used. Note that this is across all CPUs, so if the process has multiple threads (and this process is running on a computer with more than one processor) it could potentially exceed the wall clock time reported by Real (which usually occurs). Note that in the output these figures include the User and Sys time of all child processes (and their descendants) as well when they could have been collected, e.g. by wait(2) or waitpid(2), although the underlying system calls return the statistics for the process and its children separately.

Origins of the statistics reported by time (1)

The statistics reported by time are gathered from various system calls. ‘User’ and ‘Sys’ come from wait (2) (POSIX) or times (2) (POSIX), depending on the particular system. ‘Real’ is calculated from a start and end time gathered from the gettimeofday (2) call. Depending on the version of the system, various other statistics such as the number of context switches may also be gathered by time.

On a multi-processor machine, a multi-threaded process or a process forking children could have an elapsed time smaller than the total CPU time – as different threads or processes may run in parallel. Also, the time statistics reported come from different origins, so times recorded for very short running tasks may be subject to rounding errors, as the example given by the original poster shows.

A brief primer on Kernel vs. User mode

On Unix, or any protected-memory operating system, ‘Kernel’ or ‘Supervisor’ mode refers to a privileged mode that the CPU can operate in. Certain privileged actions that could affect security or stability can only be done when the CPU is operating in this mode; these actions are not available to application code. An example of such an action might be manipulation of the MMU to gain access to the address space of another process. Normally, user-mode code cannot do this (with good reason), although it can request shared memory from the kernel, which could be read or written by more than one process. In this case, the shared memory is explicitly requested from the kernel through a secure mechanism and both processes have to explicitly attach to it in order to use it.

The privileged mode is usually referred to as ‘kernel’ mode because the kernel is executed by the CPU running in this mode. In order to switch to kernel mode you have to issue a specific instruction (often called a trap) that switches the CPU to running in kernel mode and runs code from a specific location held in a jump table. For security reasons, you cannot switch to kernel mode and execute arbitrary code – the traps are managed through a table of addresses that cannot be written to unless the CPU is running in supervisor mode. You trap with an explicit trap number and the address is looked up in the jump table; the kernel has a finite number of controlled entry points.

The ‘system’ calls in the C library (particularly those described in Section 2 of the man pages) have a user-mode component, which is what you actually call from your C program. Behind the scenes, they may issue one or more system calls to the kernel to do specific services such as I/O, but they still also have code running in user-mode. It is also quite possible to directly issue a trap to kernel mode from any user space code if desired, although you may need to write a snippet of assembly language to set up the registers correctly for the call.

More about ‘sys’

There are things that your code cannot do from user mode – things like allocating memory or accessing hardware (HDD, network, etc.). These are under the supervision of the kernel, and it alone can do them. Some operations like malloc orfread/fwrite will invoke these kernel functions and that then will count as ‘sys’ time. Unfortunately it’s not as simple as “every call to malloc will be counted in ‘sys’ time”. The call to malloc will do some processing of its own (still counted in ‘user’ time) and then somewhere along the way it may call the function in kernel (counted in ‘sys’ time). After returning from the kernel call, there will be some more time in ‘user’ and then malloc will return to your code. As for when the switch happens, and how much of it is spent in kernel mode… you cannot say. It depends on the implementation of the library. Also, other seemingly innocent functions might also use malloc and the like in the background, which will again have some time in ‘sys’ then.

15

  • 19

    Does the time spent by child processes count into real/sys?

    – ron

    Aug 29, 2011 at 10:53

  • 1

    @ron – According to the Linux man page, it aggregates the ‘c’ times with the process times, so I think it does. The parent times and child times are available separately from the times(2) call, though. I guess the Solaris/SysV version of time(1) does something similar.

    Aug 30, 2011 at 10:28


  • 5

    User+Sys lets you measure CPU usage of a process. You can use it to benchmark performance. This is particularly useful for multi-threaded code where more than one CPU core might be working on a computation.

    Nov 9, 2014 at 21:35

  • 1

    Not precisely on topic, nevertheless: Running “\time <cmd>” is interesting – it provides more detail: (forgive poor formatting in the comment): $ time ps PID TTY TIME CMD 9437 pts/19 00:00:00 bash 11459 pts/19 00:00:00 ps real 0m0.025s user 0m0.004s sys 0m0.018s $ \time ps PID TTY TIME CMD 9437 pts/19 00:00:00 bash 11461 pts/19 00:00:00 time 11462 pts/19 00:00:00 ps 0.00user 0.01system 0:00.02elapsed 95%CPU (0avgtext+0avgdata 2160maxresident)k 0inputs+0outputs (0major+103minor)pagefaults 0swaps $

    – kaiwan

    May 9, 2015 at 7:41


  • 1

    (Ran out of chars in the prev comment so): More detail? Use perf [1], [2]. [1] perf.wiki.kernel.org/index.php/Main_Page [2] brendangregg.com/perf.html

    – kaiwan

    May 9, 2015 at 7:46

344

To expand on the accepted answer, I just wanted to provide another reason why realuser + sys.

Keep in mind that real represents actual elapsed time, while user and sys values represent CPU execution time. As a result, on a multicore system, the user and/or sys time (as well as their sum) can actually exceed the real time. For example, on a Java app I’m running for class I get this set of values:

real    1m47.363s
user    2m41.318s
sys     0m4.013s

10

  • 15

    I’d always wondered about this. Since I know that my programs are single threaded, the difference between user and real time must be VM overhead, correct?

    – Quantum7

    May 5, 2010 at 0:13

  • 15

    not necessarily; the Sun JVM on Solaris machines as well as Apple’s JVM on Mac OS X manages to use more than one core even in single-threaded apps. If you do a sample of a java process, you’ll see that things like garbage collection run on separate threads (and some other stuff too that I don’t remember off the top of my head). I don’t know if you really want to term that “VM overhead” though.

    – lensovet

    May 7, 2010 at 7:10

  • @Quantum7 – no, not necessarily. See my post above. Real is elapsed time, user and sys are accumulated time slice statistics from the CPU time the process actually uses.

    Apr 12, 2011 at 16:20

  • 4

    I guess the amount of up-votes gave you enough reputation now :D. So what do you think about real exceeding user and sys total ? OS overhead such as thread context switching may be ?

    Sep 26, 2012 at 11:36


  • 31

    Another potential issue could be I/O: if your application spends a good deal of time waiting to receive a file or stream, then obviously the real time would greatly exceed the user/sys time because no CPU time is used while waiting to get access to a file or something similar.

    – lensovet

    Sep 26, 2012 at 18:39

68

real: The actual time spent in running the process from start to finish, as if it was measured by a human with a stopwatch

user: The cumulative time spent by all the CPUs during the computation

sys: The cumulative time spent by all the CPUs during system-related tasks such as memory allocation.

Notice that sometimes user + sys might be greater than real, as
multiple processors may work in parallel.

2

  • 2

    real is often described as “wall-clock” time.

    Nov 24, 2017 at 23:58

  • Or in my case, sometimes real is more than the user+sys, for me it is due to running so many parallel processes at once. 87.02 real 14.12 user 5.20 sys 41.30 real 7.03 user 3.20 sys 2387.46 real 750.67 user 282.80 sys 2.60 real 7.22 user 3.35 sys

    Feb 12 at 16:01