Autotrace – Debug on Steroids

Autotrace is a tool that allows you to debug a process and:

View the output of multiple debug commands at once
Record this output for others to review
Replay the output

All of this is done in a single window.

You can think of it as a realtime sosreport or just a cool way to learn more about what’s going on when you run something.

Why?

Have you ever done something like this?

$ some_command
^Z
[1]+  Stopped                 some_command
imiell   17440 22741  0 10:47 pts/3    00:00:00 grep some_command
$ strace -p 17440 > strace.log 2>&1
$ fg
[some_command continues]

That is, you:

Ran a process
Realised you want to run some tracing on it
Suspended it
Found the pid
Ran your trace command(s), outputting to logfiles
Continued the process

Tedious, right?

LMATFY

autotrace can automate that. By default it finds the latest backgrounded pid, attaches to it, runs some default tracing commands on it, records the output, allows you to pause it, replay it elsewhere, and more.

Here’s what it looks like doing something similar to the above scenario:

If you remember you have autotrace before you run the command, you can specify all those commands automatically:

Pause

You can pause the session and scroll back and forth, then continue tracking. It suspends the processes while paused.

Other Features

Record and Replay

It also automatically records those sessions and can replay them – perfect for debugging or sharing debug information in real time.

Here’s the above example being tarred up and replayed somewhere else:

Zoom In and Out

You can zoom in and out by hitting the session number:

Move Windows in and Out

Got more than four commands you want to track output for? No problem, you can supply as many commands as you like.

This command runs five commands, for example:

autotrace \
  'ping google.com' \
  'iostat 1' \
  "bash -c 'while true; do cat /proc/PID/status; sleep 2; done'" \
  'vmstat 1' 
  'bash -c "while true; do echo I was in a hidden window; sleep 1; done"'

The last is hidden at the start, but by hitting ‘m‘ you can move the panes around (the ‘main’ ping one is protected). Like this:

Examples

These examples work on Linux. To work on Mac, you need to find replacements for strace/pstree/whatever.

What does nmap get up to?

sudo autotrace 'nmap meirionconsulting.com' 'strace -p PID' 'tcpdump -XXs 20000'

What about find?

sudo autotrace 'find /' 'strace -p PID'

A monster debug session on nmap involving lsof, pstree, and a tracking of /proc/interrupts (Linux only):

sudo autotrace \
    'nmap localhost' \
    'strace -p PID' \
    'tcpdump -XXs 20000' \
    'bash -c "while true;do free;sleep 5;done"' \
    'bash -c "while true;do lsof -p PID | tail -5;sleep 5;done"' \
    'bash -c "while true;do pstree -p PID | tail -5;sleep 5;done"' \
    'bash -c "while true;do cat /proc/interrupts; sleep 1;done"'