{"id":2065,"date":"2025-07-24T07:56:20","date_gmt":"2025-07-24T07:56:20","guid":{"rendered":"https:\/\/www.nicktailor.com\/?p=2065"},"modified":"2025-07-24T09:24:41","modified_gmt":"2025-07-24T09:24:41","slug":"complete-latency-troubleshooting-command-reference","status":"publish","type":"post","link":"https:\/\/nicktailor.com\/tech-blog\/complete-latency-troubleshooting-command-reference\/","title":{"rendered":"Complete Latency Troubleshooting Command Reference"},"content":{"rendered":"
\n

How to Read This Guide:<\/strong> Each command shows the actual output you’ll see on your system. The green\/red examples below each command show real outputs – green means your system is optimized for low latency, red means there are problems that will cause latency spikes. Compare your actual output to these examples to quickly identify issues.<\/p>\n

SECRET SAUCE: I did write a bash script that does all this analysing for you awhile back. Been meaning to push to my repos.<\/p>\n

Its sitting in one my 1000’s of text files of how to do’s. \ud83d\ude01. Im sure you all have those…..more to come…<\/p>\n<\/div>\n

System Information Commands<\/h2>\n

uname -a<\/h3>\n
uname -a<\/code><\/pre>\n
Flags:<\/strong><\/p>\n
    \n
  • -a<\/code>: Print all system information<\/li>\n<\/ul>\n
    Example Output:<\/strong><\/p>\n
    Linux trading-server 5.15.0-rt64 #1 SMP PREEMPT_RT Thu Mar 21 13:30:15 UTC 2024 x86_64 x86_64 x86_64 GNU\/Linux<\/code><\/pre>\n
    What to look for:<\/strong> PREEMPT_RT<\/code> indicates real-time kernel is active<\/p>\n
    \u2713 GOOD OUTPUT (real-time kernel):<\/span>
    \nLinux server 5.15.0-rt64 #1 SMP PREEMPT_RT Thu Mar 21 13:30:15 UTC 2024<\/code>
    \nShows “PREEMPT_RT” = real-time kernel for predictable latency
    \n<\/small>\u2717 BAD OUTPUT (standard kernel):<\/span>
    \nLinux server 5.15.0-generic #1 SMP Thu Mar 21 13:30:15 UTC 2024<\/code>
    \nShows “generic” with no “PREEMPT_RT” = standard kernel with unpredictable latency<\/small><\/div>\n
    \n
    Performance Profiling Commands<\/h2>\n
    perf stat<\/h3>\n
    perf stat [options] [command]<\/code><\/pre>\n
    Key flags:<\/strong><\/p>\n
      \n
    • -e <events><\/code>: Specific events to count<\/li>\n
    • -a<\/code>: Monitor all CPUs<\/li>\n
    • -p <pid><\/code>: Monitor specific process<\/li>\n<\/ul>\n
      Example Usage & Output:<\/strong><\/p>\n
      perf stat -e cycles,instructions,cache-misses,branch-misses .\/trading_app\n\n Performance counter stats for '.\/trading_app':\n\n     4,234,567,890      cycles                    #    3.456 GHz\n     2,987,654,321      instructions              #    0.71  insn per cycle\n        45,678,901      cache-misses              #   10.789 % of all cache refs\n         5,432,109      branch-misses             #    0.234 % of all branches<\/code><\/pre>\n
      What to look for:<\/strong> Instructions per cycle (should be >1), cache miss rate (<5% is good), branch miss rate (<1% is good)<\/p>\n
      \u2713 GOOD OUTPUT:<\/span>
      \n2,987,654,321      instructions              #    2.15  insn per cycle
      \n45,678,901      cache-misses              #    3.2 % of all cache refs
      \n5,432,109      branch-misses             #    0.8 % of all branches
      \nWhy:<\/strong> Good<\/span> = >2.0 IPC (CPU efficient), <5% cache misses, <1% branch misses.<\/p>\n
      <\/small><\/code>\u2717 BAD OUTPUT:<\/span>
      \n1,234,567,890      instructions              #    0.65  insn per cycle
      \n156,789,012      cache-misses              #   15.7 % of all cache refs
      \n89,432,109      branch-misses             #    4.2 % of all branches<\/code>
      \nWhy:<\/strong><\/small><\/code>\u00a0Bad<\/span> = <1.0 IPC (CPU starved), >10% cache misses, >4% branch misses.<\/small><\/div>\n
      \n
      eBPF Tools<\/h2>\n
      \n
      Note:<\/strong> eBPF tools are part of the BCC toolkit. Install once with: sudo apt-get install bpfcc-tools linux-headers-$(uname -r)<\/code> (Ubuntu) or sudo yum install bcc-tools<\/code> (RHEL\/CentOS). After installation, these become system-wide commands.<\/p>\n<\/div>\n
      funclatency<\/h3>\n
      sudo funclatency [options] 'function_pattern'<\/code><\/pre>\n
      Key flags:<\/strong><\/p>\n