Comment by menaerus

17 days ago

Benchmark is simple but I find it worthwhile because of the fact that (1) it is run across 15 different platforms (different CPUs, libc's) and results are pretty much reproducible, and (2) it is run through gbenchmark which has a mechanism to make the measurements statistically significant.

Interesting thing that enforces their hypothesis, and measurements, is the fact that, for example, getpid and clock_gettime_mono_raw on some platforms run much faster (vDSO) than on the rest.

Also, the variance between different CPUs is what IMO is enforcing their results and not the other way around - I don't expect the same call to have the same cost on different CPU models. Different CPUs, different cores, different clock frequencies, different tradeoffs in design, etc.

The code is here: https://github.com/gsauthof/osjitter/blob/master/bench_sysca...

syscall() row invokes a simple syscall(423) and it seems to be expensive. Other calls such as close(999), getpid(), getuid(), clock_gettime(CLOCK_MONOTONIC_RAW, &ts), and sched_yield() are also producing the similar results. All of them basically an order of magnitude larger than 50ns.

As for the register renaming, I know what this is, but I still don't get it what register renaming has to do with making the state (registers) storage a cheaper operation.

This is from Intel manual:

  Instructions following a SYSCALL may be fetched from memory before earlier instructions complete execution, but they will not execute (even speculatively) until all instructions prior to the SYSCALL have completed execution (the later instructions may execute before data stored by the earlier instructions have become globally visible).

So, I wrongly assumed that the core has to wait before the data is completely written but it seems it acts more like a memory barrier but with relaxed properties - instructions are serialized but the data written doesn't have to become globally visible.

I think the most important aspect of it is "until all instructions prior to the SYSCALL have completed". This means that the whole pipeline has to be drained. With 20+ deep instruction pipeline, and whatnot instructions in it, I can imagine that this can likely become the most expensive part of the syscall.

8 comments

menaerus

ori_b 16 days ago

I can't reproduce. When I run The code is here: https://github.com/gsauthof/osjitter/blob/master/bench_sysca..., here are the numbers on the computers I have:

    AMD Ryzen 7 9700X Desktop:
    ----------------------------------------------------------------------------
    Benchmark                                  Time             CPU   Iterations
    ----------------------------------------------------------------------------
    bench_getuid                            38.6 ns         38.5 ns     18160546
    bench_getpid                            39.9 ns         39.9 ns     17703749
    bench_close                             45.2 ns         45.1 ns     15711379
    bench_syscall                           42.2 ns         42.1 ns     16638675
    bench_sched_yield                       81.7 ns         81.6 ns      8623522
    bench_clock_gettime                     15.9 ns         15.9 ns     44010857
    bench_clock_gettime_tai                 15.9 ns         15.9 ns     43997256
    bench_clock_gettime_monotonic           15.9 ns         15.9 ns     44012908
    bench_clock_gettime_monotonic_raw       15.9 ns         15.9 ns     43982277
    bench_nanosleep0                       49961 ns          370 ns       100000
    bench_nanosleep0_slack1                10839 ns          351 ns      1000000
    bench_nanosleep1_slack1                10878 ns          358 ns      1000000
    bench_pthread_cond_signal               1.37 ns         1.37 ns    503715097
    bench_assign                           0.563 ns        0.562 ns   1000000000
    bench_sqrt                              1.63 ns         1.63 ns    430096636
    bench_sqrtrec                           5.33 ns         5.33 ns    132574542
    bench_nothing                          0.394 ns        0.394 ns   1000000000

    12th Gen Intel(R) Core(TM) i5-12600H
    ----------------------------------------------------------------------------
    Benchmark                                  Time             CPU   Iterations
    ----------------------------------------------------------------------------
    bench_getuid                            70.0 ns         70.0 ns      9985369
    bench_getpid                            71.6 ns         71.6 ns      9763016
    bench_close                             76.7 ns         76.7 ns      9131090
    bench_syscall                           66.8 ns         66.8 ns     10533946
    bench_sched_yield                        160 ns          160 ns      4377987
    bench_clock_gettime                     12.2 ns         12.2 ns     57432496
    bench_clock_gettime_tai                 12.1 ns         12.1 ns     57826299
    bench_clock_gettime_monotonic           12.2 ns         12.2 ns     57736141
    bench_clock_gettime_monotonic_raw       12.3 ns         12.3 ns     57070425
    bench_nanosleep0                       63154 ns        11834 ns        55756
    bench_nanosleep0_slack1                 2933 ns         1700 ns       348675
    bench_nanosleep1_slack1                 2654 ns         1479 ns       467420
    bench_pthread_cond_signal               1.39 ns         1.39 ns    483995101
    bench_assign                           0.868 ns        0.868 ns    821103909
    bench_sqrt                              1.69 ns         1.69 ns    422094139
    bench_sqrtrec                           4.06 ns         4.06 ns    174511095
    bench_nothing                          0.750 ns        0.750 ns    941204159

    AMD Ryzen 5 PRO 7545U Laptop:
    ----------------------------------------------------------------------------
    Benchmark                                  Time             CPU   Iterations
    ----------------------------------------------------------------------------
    bench_getuid                             106 ns          106 ns      6581746
    bench_getpid                             111 ns          111 ns      6271878
    bench_close                              116 ns          116 ns      5944154
    bench_syscall                           85.9 ns         85.9 ns      7317584
    bench_sched_yield                        315 ns          315 ns      2249333
    bench_clock_gettime                     17.6 ns         17.6 ns     39935693
    bench_clock_gettime_tai                 17.6 ns         17.6 ns     39920957
    bench_clock_gettime_monotonic           17.5 ns         17.5 ns     39962966
    bench_clock_gettime_monotonic_raw       17.5 ns         17.5 ns     39561163
    bench_nanosleep0                       52720 ns         3058 ns       100000
    bench_nanosleep0_slack1                13815 ns         2969 ns       244790
    bench_nanosleep1_slack1                13710 ns         2722 ns       254666
    bench_pthread_cond_signal               2.66 ns         2.66 ns    264735233
    bench_assign                           0.930 ns        0.930 ns    813279743
    bench_sqrt                              2.43 ns         2.43 ns    286953468
    bench_sqrtrec                           5.67 ns         5.67 ns    123889652
    bench_nothing                          0.812 ns        0.812 ns    860562208

So, I've tested multiple times in multiple ways, and the results don't seem to match.

menaerus 16 days ago

Interesting because on my machine I can reproduce the results. It's a pretty hefty 5.3GHz and recentish (Raptor Lake) Intel i7-13850HX CPU:

  ----------------------------------------------------------------------------
  Benchmark                                  Time             CPU   Iterations
  ----------------------------------------------------------------------------
  bench_getuid                             384 ns          384 ns      1822307
  bench_getpid                             382 ns          382 ns      1835289
  bench_close                              390 ns          390 ns      1796493
  bench_syscall                            374 ns          374 ns      1874165
  bench_sched_yield                        611 ns          611 ns      1143456
  bench_clock_gettime                     44.1 ns         44.1 ns     15872740
  bench_clock_gettime_tai                 44.1 ns         44.1 ns     15879915
  bench_clock_gettime_monotonic           44.1 ns         44.1 ns     15887383
  bench_clock_gettime_monotonic_raw       44.4 ns         44.4 ns     15755225
  bench_nanosleep0                       55617 ns         4647 ns       100000
  bench_nanosleep0_slack1                 7144 ns         4362 ns       160448
  bench_nanosleep1_slack1                 7159 ns         4369 ns       160645
  bench_pthread_cond_signal               7.38 ns         7.38 ns     94670062
  bench_assign                           0.523 ns        0.523 ns   1000000000
  bench_sqrt                              8.04 ns         8.04 ns     86998912
  bench_sqrtrec                           11.4 ns         11.4 ns     61428535
  bench_nothing                          0.000 ns        0.000 ns   1000000000

EDIT: also reproducible on my skylake-x (Gold 6152) machine

With turbo-boost @3.7Ghz enabled:

  ----------------------------------------------------------------------------
  Benchmark                                  Time             CPU   Iterations
  ----------------------------------------------------------------------------
  bench_getuid                             619 ns          616 ns      1153007
  bench_getpid                             632 ns          627 ns      1150829
  bench_close                              629 ns          626 ns      1110226
  bench_syscall                            617 ns          613 ns      1160239
  bench_sched_yield                        974 ns          969 ns       702773
  bench_clock_gettime                     17.9 ns         17.8 ns     39368735
  bench_clock_gettime_tai                 17.8 ns         17.7 ns     39109544
  bench_clock_gettime_monotonic           17.9 ns         17.8 ns     39591364
  bench_clock_gettime_monotonic_raw       19.0 ns         18.8 ns     38902038
  bench_nanosleep0                       63993 ns         4381 ns       100000
  bench_nanosleep0_slack1                 7445 ns         2115 ns       328474
  bench_nanosleep1_slack1                 7346 ns         2111 ns       334833
  bench_pthread_cond_signal               2.13 ns         2.12 ns    327903411
  bench_assign                           0.167 ns        0.166 ns   1000000000
  bench_sqrt                              1.87 ns         1.85 ns    374885774
  bench_sqrtrec                          0.000 ns        0.000 ns   1000000000
  bench_nothing                          0.000 ns        0.000 ns   1000000000

With turbo-boost disabled (@2.1GHz base frequency):

  ----------------------------------------------------------------------------
  Benchmark                                  Time             CPU   Iterations
  ----------------------------------------------------------------------------
  bench_getuid                            1019 ns         1012 ns       688965
  bench_getpid                            1057 ns         1048 ns       688020
  bench_close                             1039 ns         1029 ns       684537
  bench_syscall                           1010 ns         1003 ns       696919
  bench_sched_yield                       1653 ns         1642 ns       434212
  bench_clock_gettime                     30.7 ns         30.4 ns     22999055
  bench_clock_gettime_tai                 30.5 ns         30.2 ns     23716873
  bench_clock_gettime_monotonic           29.8 ns         29.6 ns     23643198
  bench_clock_gettime_monotonic_raw       30.5 ns         30.3 ns     23277717
  bench_nanosleep0                       65256 ns         5114 ns       100000
  bench_nanosleep0_slack1                11649 ns         3402 ns       197983
  bench_nanosleep1_slack1                11572 ns         3528 ns       209371
  bench_pthread_cond_signal               3.62 ns         3.60 ns    195696177
  bench_assign                           0.255 ns        0.253 ns   1000000000
  bench_sqrt                              3.13 ns         3.10 ns    225561559
  bench_sqrtrec                          0.000 ns        0.000 ns   1000000000
  bench_nothing                          0.000 ns        0.000 ns   1000000000

I wonder why your results are so much different. Mine almost linearly scale with the core frequency.

ori_b 15 days ago

Something is definitely up. Is there a VM? are you running in a container with seccomp?
Why are your calls to sqrt so slow on your newest machine? Why is sqrtrec free on the others?

5 replies →