Autotrace is a tool that allows you to debug a process and:
- View the output of multiple debug commands at once
- Record this output for others to review
- Replay the output
All of this is done in a single window.
You can think of it as a realtime sosreport or just a cool way to learn more about what’s going on when you run something.
Have you ever done something like this?
$ some_command ^Z + Stopped some_command imiell 17440 22741 0 10:47 pts/3 00:00:00 grep some_command $ strace -p 17440 > strace.log 2>&1 $ fg [some_command continues]
That is, you:
- Ran a process
- Realised you want to run some tracing on it
- Suspended it
- Found the pid
- Ran your trace command(s), outputting to logfiles
- Continued the process
autotrace can automate that. By default it finds the latest backgrounded pid, attaches to it, runs some default tracing commands on it, records the output, allows you to pause it, replay it elsewhere, and more.
Here’s what it looks like doing something similar to the above scenario:
If you remember you have autotrace before you run the command, you can specify all those commands automatically:
You can pause the session and scroll back and forth, then continue tracking. It suspends the processes while paused.
Record and Replay
It also automatically records those sessions and can replay them – perfect for debugging or sharing debug information in real time.
Here’s the above example being tarred up and replayed somewhere else:
Zoom In and Out
You can zoom in and out by hitting the session number:
Move Windows in and Out
Got more than four commands you want to track output for? No problem, you can supply as many commands as you like.
This command runs five commands, for example:
autotrace \ 'ping google.com' \ 'iostat 1' \ "bash -c 'while true; do cat /proc/PID/status; sleep 2; done'" \ 'vmstat 1' 'bash -c "while true; do echo I was in a hidden window; sleep 1; done"'
The last is hidden at the start, but by hitting ‘
m‘ you can move the panes around (the ‘main’
ping one is protected). Like this:
These examples work on Linux. To work on Mac, you need to find replacements for strace/pstree/whatever.
What does nmap get up to?
sudo autotrace 'nmap meirionconsulting.com' 'strace -p PID' 'tcpdump -XXs 20000'
What about find?
sudo autotrace 'find /' 'strace -p PID'
A monster debug session on
pstree, and a tracking of
/proc/interrupts (Linux only):
sudo autotrace \ 'nmap localhost' \ 'strace -p PID' \ 'tcpdump -XXs 20000' \ 'bash -c "while true;do free;sleep 5;done"' \ 'bash -c "while true;do lsof -p PID | tail -5;sleep 5;done"' \ 'bash -c "while true;do pstree -p PID | tail -5;sleep 5;done"' \ 'bash -c "while true;do cat /proc/interrupts; sleep 1;done"'
pip install autotrace
The code is available here.
It relies heavily on Thomas Ballinger’s wonderful ‘Terminal Whispering’ talk and curtsies python library.
If you like this post, you might like Learn Git the Hard Way, Learn Bash the Hard Way or Docker in Practice