ftrace是 Linux 内核中提供的一种调试工具。使用 ftrace 可以对内核中发生的事情进行跟踪，这在调试 bug 或者分析内核时非常有用.

    什么是ftrace: 请参考

    http://lwn.net/Articles/322666/

    trace-cmd - command line reader for ftrace: 请参考 http://lwn.net/Articles/341902/

     trace-cmd配置: 请参考 http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-Latency_Tracing_Using_trace_cmd.html

    在2.6.28-rc2以后的内核都支持的, 当然包括RHEL6(2.6.32), 我粗粗的演示下ubuntu 10.10下的使用:

$ sudo apt-get -y install trace-cmd
$ man trace-cmd-record #这个是最核心的功能,收集数据的

$ sudo trace-cmd record  -e sched_switch ls > /dev/null
disable all
enable sched_switch
offset=464000
offset=466000

$ trace-cmd report
version = 6
trace-cmd: No such file or directory
  function scsi_trace_parse_cdb not defined
  failed to read event print fmt for scsi_dispatch_cmd_start
  function scsi_trace_parse_cdb not defined
  failed to read event print fmt for scsi_dispatch_cmd_error
  Error: expected \'name\' but read \'FORMAT\'
  Error: expected \'name\' but read \'FORMAT\'
  Error: expected type 5 but read 4
  failed to read event print fmt for do_sys_open
  Error: expected type 5 but read 4
  failed to read event print fmt for open_exec
cpus=2
              ls-8588  [001] 16611.191996: sched_switch:         8588:120:R ==> 8587:120: trace-cmd
       trace-cmd-8587  [001] 16611.192040: sched_switch:         8587:120:S ==> 8588:120: ls
            Xorg-1360  [000] 16611.192868: sched_switch:         1360:120:R ==> 1979:120: alsa-sink
              ls-8588  [001] 16611.192979: sched_switch:         8588:120:R ==> 1909:120: pulseaudio
       ...

    类似的功能,当然可以用systemtap来写,但是工具总是更方便.

    trace-cmd使用的模式总是先record收集数据,然后report.

    目前支持以下插件plugins:

     blk function_graph mmiotrace wakeup_rt wakeup function sched_switch nop

     对于大部分的系统调优是足够的.

    祝大家玩的开心.