Here are some tips that might help you be more productive with bash.
1) ^x^y^
A gem I use all the time.
Ever typed anything like this?
$ grp somestring somefile
-bash: grp: command not found
Sigh. Hit ‘up’, ‘left’ until at the ‘p’ and type ‘e’ and return.
Or do this:
$ ^rp^rep^
grep 'somestring' somefile
$
One subtlety you may want to note though is:
$ grp rp somefile $ ^rp^rep^ $ grep rp somefile
If you wanted rep
to be searched for, then you’ll need to dig into the man page and use a more powerful history command:
$ grp rp somefile
$ !!:gs/rp/rep
grep rep somefile
$
2) pushd / popd vs ‘cd -‘
This one comes in very handy for scripts, especially when operating within a loop.
Let’s say you’re in a for
loop moving in and out of folders like this:
for d1 in $(ls -d */) do # Store original working directory. original_wd="$(pwd)" cd "$d1" for d2 in $(ls -d */) do pushd "$d2" # Do something popd done # Return to original working directory cd "${original_wd}" done
NOTE: I’m well aware the above code is unsafe – see here.
The code above is intended to illustrate pushd/popd without distraction
for a relative beginner.
There’s a post in the fact that people like me use $(ls -d */)
all
the time without deleterious consequences 99% of the time, but
that can wait. That said, it’s well worth knowing that this
kind of issue exists in bash as it can trip you up.
You can rewrite the above using the pushd
stack like this:
for d1 in $(ls -d *)
do
pushd "$d1"
for d2 in $(ls -d */)
do
pushd "$d2"
# Do something
popd
done
popd
done
Which tracks the folders you’ve pushed and popped as you go.
Note that if there’s an error in a pushd
you may lose track of the stack and popd
too many time. You probably want to set -e
in your script as well (see previous post)
There’s also cd -
, but that doesn’t ‘stack’ – it just returns you to the previous folder:
cd ~ cd /tmp cd blah cd - # Back to /tmp cd - # Back to 'blah' cd - # Back to /tmp cd - # Back to 'blah' ...
Material here based on material from my book
Learn Bash the Hard Way.
Free preview available here.
3) shopt
vs set
This one bothered me for a while.
What’s the difference between set
and shopt
?
set
s we saw before, but shopt
s look very similar. Just inputting shopt
shows a bunch of options:
$ shopt cdable_vars off cdspell on checkhash off checkwinsize on cmdhist on compat31 off dotglob off
I found a set of answers here.
Essentially, it looks like it’s a consequence of bash (and other shells) being built on sh, and adding shopt
as another way to set extra shell options.
But I’m still unsure… if you know the answer, let me know.
4) Here Docs and Here Strings
‘Here docs’ are files created inline in the shell.
The ‘trick’ is simple. Define a closing word, and the lines between that word and when it appears alone on a line become a file.
Type this:
$ cat > afile << SOMEENDSTRING > here is a doc > it has three lines > SOMEENDSTRING alone on a line will save the doc > SOMEENDSTRING $ cat afile here is a doc it has three lines SOMEENDSTRING alone on a line will save the doc $
Notice that:
- the string could be included in the file if it was not ‘alone’ on the line
- the string
SOMEENDSTRING
is more normallyEND
, but that is just convention
Lesser known is the ‘here string’:
$ cat > asd <<< 'This file has one line'
5) String Variable Manipulation
You may have written code like this before, where you use tools like sed
to manipulate strings:
$ VAR='HEADERMy voice is my passwordFOOTER' $ PASS="$(echo $VAR | sed 's/^HEADER(.*)FOOTER/1/')" $ echo $PASS
But you may not be aware that this is possible natively in bash.
This means that you can dispense with lots of sed and awk shenanigans.
One way to rewrite the above is:
$ VAR='HEADERMy voice is my passwordFOOTER'
$ PASS="${VAR#HEADER}"
$ PASS="${PASS%FOOTER}"
$ echo $PASS
- The
#
means ‘match and remove the following pattern from the start of the string’ - The
%
means ‘match and remove the following pattern from the end of the string
The second method is twice as fast as the first on my machine. And (to my surprise), it was roughly the same speed as a similar python script.
If you want to use glob patterns that are greedy (see globbing here) then you double up:
$ VAR='HEADERMy voice is my passwordFOOTER'
$ echo ${VAR##HEADER*}
$ echo ${VAR%%*FOOTER}
6) Variable Defaults
These are very handy when you’re knocking up scripts quickly.
If you have a variable that’s not set, you can ‘default’ them by using this. Create a file called default.sh
with these contents
#!/bin/bash FIRST_ARG="${1:-no_first_arg}" SECOND_ARG="${2:-no_second_arg}" THIRD_ARG="${3:-no_third_arg}" echo ${FIRST_ARG} echo ${SECOND_ARG} echo ${THIRD_ARG}
Now run chmod +x default.sh
and run the script with ./default.sh first second
.
Observer how the third argument’s default has been assigned, but not the first two.
You can also assign directly with ${VAR:=defaultval}
(equals sign, not dash) but note that this won’t work with positional variables in scripts or functions. Try changing the above script to see how it fails.
7) Traps
The trap
builtin can be used to ‘catch’ when a signal is sent to your script.
Here’s an example I use in my own cheapci
script:
function cleanup() { rm -rf "${BUILD_DIR}" rm -f "${LOCK_FILE}" # get rid of /tmp detritus, leaving anything accessed 2 days ago+ find "${BUILD_DIR_BASE}"/* -type d -atime +1 | rm -rf echo "cleanup done" } trap cleanup TERM INT QUIT
Any attempt to CTRL-C
, CTRL-
or terminate the program using the TERM
signal will result in cleanup being called first.
Be aware:
- Trap logic can get very tricky (eg handling signal race conditions)
- The KILL signal can’t be trapped in this way
But mostly I’ve used this for ‘cleanups’ like the above, which serve their purpose.
8) Shell Variables
It’s well worth getting to know the standard shell variables available to you. Here are some of my favourites:
RANDOM
Don’t rely on this for your cryptography stack, but you can generate random numbers eg to create temporary files in scripts:
$ echo ${RANDOM} 16313 $ # Not enough digits? $ echo ${RANDOM}${RANDOM} 113610703 $ NEWFILE=/tmp/newfile_${RANDOM} $ touch $NEWFILE
REPLY
No need to give a variable name for read
…
$ read my input $ echo ${REPLY}
LINENO and SECONDS
Handy for debugging
$ echo ${LINENO} 115 $ echo ${SECONDS}; sleep 1; echo ${SECONDS}; echo $LINENO 174380 174381 116
Note that there are two ‘lines’ above, even though you used ;
to separate the commands.
TMOUT
You can timeout reads, which can be really handy in some scripts
#!/bin/bash TMOUT=5 echo You have 5 seconds to respond... read echo ${REPLY:-noreply}
9) Extglobs
If you’re really knee-deep in bash, then you might want to power up your globbing. You can do this by setting the extglob shell option. Here’s the setup:
shopt -s extglob A="12345678901234567890" B=" ${A} "
Now see if you can figure out what each of these does:
echo "B |${B}|" echo "B#+( ) |${B#+( )}|" echo "B#?( ) |${B#?( )}|" echo "B#*( ) |${B#*( )}|" echo "B##+( )|${B##+( )}|" echo "B##*( )|${B##*( )}|" echo "B##?( )|${B##?( )}|"
Now, potentially useful as it is, it’s hard to think of a situation where you’d absolutely want to do it this way. Normally you’d use a tool better suited to the task (like sed
) or just drop bash and go to a ‘proper’ programming language like python.
10) Associative Arrays
Talking of moving to other languages, a rule of thumb I use is that if I need arrays then I drop bash to go to python (I even created a Docker container for a tool to help with this here).
What I didn’t know until I read up on it was that you can have associative arrays in bash.
Type this out for a demo:
$ declare -A MYAA=([one]=1 [two]=2 [three]=3) $ MYAA[one]="1" $ MYAA[two]="2" $ echo $MYAA $ echo ${MYAA[one]} $ MYAA[one]="1" $ WANT=two $ echo ${MYAA[$WANT]}
Note that this is only available in bashes 4.x+.
11) source vs ‘.’
This one confused me for a long time.
You can type:
$ cat > somescript.sh << END A=11 END $ source somescript.sh $ echo $A
which will run the script somescript.sh
and do so while retaining the environment changes in the script in your environment.
Try this to compare:
$ cat > somescript.sh << END A=12 END $ chmod +x somescript.sh $ ./somescript.sh $ echo $A
The dot (‘.
‘) command does something similar, but what’s the difference? Why does it exist?
The answer is simple: in bash they are exactly the same. The ‘.
‘ was the original command, and is more portable, since it works in the sh
shell as well as bash.
You may also be wondering what the difference between the dots in:
./somescript.sh
and
. ./somescript.sh
is. In the . ./somescript.sh
invocation, the first dot acts as an equivalent of the source
command, while the ./
after indicates that the script will be found in this folder, the dot there representing the local folder (try running cd .
to see what happens).
If you didn’t use the ./
, and .
wasn’t in your PATH
environment variable, then somescript.sh
might not be found. Simple, right?
If you like this, you might like one of my books:
If you liked this post, you might also like these:
Ten Things I Wish I’d Known About bash
How (and Why) I Run My Own DNS Servers
My Favourite Secret Weapon – strace
A Complete Chef Infrastructure on Your Laptop
#11 should be `. somescript.sh` not `./somescript.sh`.
The ‘.’ takes a file argument, so somescript.sh doesn’t come from the path. Do you don’t need the relative path.
You’re right, but it allows me to dispel the confusion about the two dots having different meanings.
That’s not what Jeff’s talking about. Your `.` example contains this line:
$ ./somescript.sh
That runs somescript.sh as a child process, so it can’t do stuff like change the current shell’s environment. It therefore does NOT do the same thing as:
$ source somescript.sh
I suspect you actually meant to write:
$ . ./somescript.sh
No, you’re supposed to think about the output and what it does. The . ./somescript.sh is mentioned afterwords in a ‘what’s the difference’ discussion.
Don’t parse ls, ever.
https://mywiki.wooledge.org/ParsingLs
Although I’d never put it in a script, I do use that construct *all the time*. It’s just too convenient and basically never fails when I use it.
I mean, who is realistically going to commit this to muscle memory?
find . -type f -print0 | while IFS= read -r -d ” filename; do
Worth being aware of the pitfalls though.
Or you just write a bash oneliner. You’re not to edgy for a for loop, right?
To add to what the above comment is saying, that directory walking logic is really better done as a single flat ‘find’ command with ‘-exec’ (or ‘xargs’ if you prefer that). If you do find you have to change the working directory inside your script, consider wrapping that and the command needing the changed directory within parentheses, so that it runs inside a subshell.
Oops, got ninjad by author’s response. I still stand by find, though. Also of note is that a lot of GNU-commands are 0-separation aware these days (with flags like ‘-0’ or ‘-z’, depending on the command), eliminating the need to memorize read loops, xargs or exec invocations. :)
Instead of “for d1 in $(ls -d *)”, you should be doing “for d1 in */”. You should never be parsing ls(1) output.
https://mywiki.wooledge.org/ParsingLs
You’re correct about *shopt* and *set*. Needless duplication of features and confusion. IMHO – The things that matter are: 1) Use `set` when you want to be sure your subshell gets your set option(s)–assuming your subshell’s pipeline components are more likely to have `SHELLOPTS` set than `BASHOPTS` — Ex: Use `set -x` when debugging. 2) Use `set` when you’re lazy. Which is shorter? `set -x` or `shopt -s extdebug` 3) use `shopt` if you need an option not available via `set`. Good examples here: https://bash.cyberciti.biz/guide/Setting_shell_options