Ten Things I Wish I’d Known About bash

Intro

Recently I wanted to deepen my understanding of bash by researching as much of it as possible. Because I felt bash is an often-used (and under-understood) technology, I ended up writing a book on it.

A preview is available here.

You don’t have to look hard on the internet to find plenty of useful one-liners in bash, or scripts. And there are guides to bash that seem somewhat intimidating through either their thoroughness or their focus on esoteric detail.

Here I’ve focussed on the things that either confused me or increased my power and productivity in bash significantly, and tried to communicate them (as in my book) in a way that emphasises getting the understanding right.

Enjoy!

hero

1)  `` vs $()

These two operators do the same thing. Compare these two lines:

$ echo `ls`
$ echo $(ls)

Why these two forms existed confused me for a long time.

If you don’t know, both forms substitute the output of the command contained within it into the command.

The principal difference is that nesting is simpler.

Which of these is easier to read (and write)?

    $ echo `echo \`echo \\\`echo inside\\\`\``

or:

    $ echo $(echo $(echo $(echo inside)))

If you’re interested in going deeper, see here or here.

2) globbing vs regexps

Another one that can confuse if never thought about or researched.

While globs and regexps can look similar, they are not the same.

Consider this command:

$ rename -n 's/(.*)/new$1/' *

The two asterisks are interpreted in different ways.

The first is ignored by the shell (because it is in quotes), and is interpreted as ‘0 or more characters’ by the rename application. So it’s interpreted as a regular expression.

The second is interpreted by the shell (because it is not in quotes), and gets replaced by a list of all the files in the current working folder. It is interpreted as a glob.

So by looking at man bash can you figure out why these two commands produce different output?

$ ls *
$ ls .*

The second looks even more like a regular expression. But it isn’t!

3) Exit Codes

Not everyone knows that every time you run a shell command in bash, an ‘exit code’ is returned to bash.

Generally, if a command ‘succeeds’ you get an error code of 0. If it doesn’t succeed, you get a non-zero code. 1 is a ‘general error’, and others can give you more information (eg which signal killed it, for example).

But these rules don’t always hold:

$ grep not_there /dev/null
$ echo $?

$? is a special bash variable that’s set to the exit code of each command after it runs.

Grep uses exit codes to indicate whether it matched or not. I have to look up every time which way round it goes: does finding a match or not return 0?

Grok this and a lot will click into place in what follows.

4) if statements, [ and [[

Here’s another ‘spot the difference’ similar to the backticks one above.

What will this output?

if grep not_there /dev/null
then
    echo hi
else
    echo lo
fi

grep’s return code makes code like this work more intuitively as a side effect of its use of exit codes.

Now what will this output?

a) hihi
b) lolo
c) something else

if [ $(grep not_there /dev/null) = '' ]
then
    echo -n hi
else
    echo -n lo
fi
if [[ $(grep not_there /dev/null) = '' ]]
then
    echo -n hi
else
    echo -n lo
fi

The difference between [ and [[ was another thing I never really understood. [ is the original form for tests, and then [[ was introduced, which is more flexible and intuitive. In the first if block above, the if statement barfs because the $(grep not_there /dev/null) is evaluated to nothing, resulting in this comparison:

[ = '' ]

which makes no sense. The double bracket form handles this for you.

This is why you occasionally see comparisons like this in bash scripts:

if [ x$(grep not_there /dev/null) = 'x' ]

so that if the command returns nothing it still runs. There’s no need for it, but that’s why it exists.

5) sets

Bash has configurable options which can be set on the fly. I use two of these all the time:

set -e

exits from a script if any command returned a non-zero exit code (see above).

This outputs the commands that get run as they run:

set -x

So a script might start like this:

#!/bin/bash
set -e
set -x
grep not_there /dev/null
echo $?

What would that script output?

6) ​​<()

This is my favourite. It’s so under-used, perhaps because it can be initially baffling, but I use it all the time.

It’s similar to $() in that the output of the command inside is re-used.

In this case, though, the output is treated as a file. This file can be used as an argument to commands that take files as an argument.

Confused? Here’s an example.

Have you ever done something like this?

$ grep somestring file1 > /tmp/a
$ grep somestring file2 > /tmp/b
$ diff /tmp/a /tmp/b

That works, but instead you can write:

diff <(grep somestring file1) <(grep somestring file2)

Isn’t that neater?

7) Quoting

Quoting’s a knotty subject in bash, as it is in many software contexts.

Firstly, variables in quotes:

A='123'  
echo "$A"
echo '$A'

Pretty simple – double quotes dereference variables, while single quotes go literal.

So what will this output?

mkdir -p tmp
cd tmp
touch a
echo "*"
echo '*'

Surprised? I was.

8) Top three shortcuts

There are plenty of shortcuts listed in man bash, and it’s not hard to find comprehensive lists. This list consists of the ones I use most often, in order of how often I use them.

Rather than trying to memorize them all, I recommend picking one, and trying to remember to use it until it becomes unconscious. Then take the next one. I’ll skip over the most obvious ones (eg !! – repeat last command, and ~ – your home directory).

!$

I use this dozens of times a day. It repeats the last argument of the last command. If you’re working on a file, and can’t be bothered to re-type it command after command it can save a lot of work:

grep somestring /long/path/to/some/file/or/other.txt
vi !$

 

​​!:1-$

This bit of magic takes this further. It takes all the arguments to the previous command and drops them in. So:

grep isthere /long/path/to/some/file/or/other.txt
egrep !:1-$
fgrep !:1-$

The ! means ‘look at the previous command’, the : is a separator, and the 1 means ‘take the first word’, the - means ‘until’ and the $ means ‘the last word’.

Note: you can achieve the same thing with !*. Knowing the above gives you the control to limit to a specific contiguous subset of arguments, eg with !:2-3.

:h

I use this one a lot too. If you put it after a filename, it will change that filename to remove everything up to the folder. Like this:

grep isthere /long/path/to/some/file/or/other.txt
cd !$:h

which can save a lot of work in the course of the day.

9) startup order

The order in which bash runs startup scripts can cause a lot of head-scratching. I keep this diagram handy (from this great page):

shell-startup-actual

It shows which scripts bash decides to run from the top, based on decisions made about the context bash is running in (which decides the colour to follow).

So if you are in a local (non-remote), non-login, interactive shell (eg when you run bash itself from the command line), you are on the ‘green’ line, and these are the order of files read:

/etc/bash.bashrc
~/.bashrc
[bash runs, then terminates]
~/.bash_logout

This can save you a hell of a lot of time debugging.

10) getopts (cheapci)

If you go deep with bash, you might end up writing chunky utilities in it. If you do, then getting to grips with getopts can pay large dividends.

For fun, I once wrote a script called cheapci which I used to work like a Jenkins job.

The code here implements the reading of the two required, and 14 non-required arguments. Better to learn this than to build up a bunch of bespoke code that can get very messy pretty quickly as your utility grows.


This is based on some of the contents of my book Learn Bash the Hard Way, available at $5:

hero

Preview available here.


I also wrote Docker in Practice 

Get 39% off with the code: 39miell2

 

Advertisements

47 Replies to “Ten Things I Wish I’d Known About bash”

  1. pretty good — and rename(1) was new to me…

    in 4, if [ “$(grep not_there /dev/null)” = ” ] works — the [ x”`cmd`” = x ] is from time test(1) could not handle empty strings even in “”

    in modern shells (like bash) it is better to replace (nonstandard) “echo -n” with printf …

    I use ‘ESC .’ in place of !$ (now that I tried !$ I accidentally wrote $! ;)

    In 4, I personally find ‘set -u’ more important than set -e — i actually start my scripts with ‘set -euf’

    I can understand why 7 is suprising — but pretty elementary once one thinks it a bit.

  2. Diagram has like 10 colors, it only says what one of them means. Here’s something else you should know about bash: it’s documentation is licensed under gfdl1.3, meaning anyone can contribute to it and redistribute it among other things. This is documentation is licensed under the default license of all rights reserved and does not give those rights. So for example, if the official documentation wanted to use some of this text, they couldn’t. That’s not nice. Please use a gfdl1.3 compatible license, such as gfdl1.3 or cc-by.

  3. ${PIPESTATUS[]} is better than $? because it gets you the status of the pipeline element you really want every time.

    Consider what happens to:

    $ grep foo /dev/null
    $ echo $?

    when you decide to try to pipe the output:

    $ grep foo /dev/null | some_command
    $ echo $?

    This is very easy to overlook when modifying something after the fact. Now if instead your original example was:

    $ grep foo /dev/null
    $ echo ${PIPESTATUS[0]

    adding “| some_command” to the first line does not alter the results of the second line.

  4. I was trying to follow your example in section 9 “Startup Order,” and was wondering why you say it’s following the green line, but it doesn’t go through ~/.bash_logout

    (It makes sense to me that an interactive session wouldn’t call the logout script, so maybe I misunderstood your example.)

    1. I was trying to follow your example in section 9 “Startup Order,” and was wondering why you say it’s following the green line, but the green line doesn’t go through ~/.bash_logout

      (It makes sense to me that an interactive session wouldn’t call the logout script, so maybe I misunderstood your example.)

      1. The green line is for a non-login shell (ie., you’re already logged on and open a bash session). So it doesn’t run ~/.bash_logout when it exits.

  5. 2) “The two asterisks are interpreted in different ways.”

    Yes, but in a regexp a “*” applies to what it follows. In a regexp, “*” by itself doesn’t mean “0 or more characters”, it means “0 or more occurrences of what precedes it” — so “.*” means “0 or more characters. In a glob, “*” by itself means “0 or more characters”.

    3) “I have to look up every time which way round it goes: does finding a match or not return 0?”

    If you use grep in a condition think of it as “does the pattern appear?”. Also, when used as a condition both “diff” and “cmp” mean “are the inputs identical (which is admittedly a little confusing in the case of diff).

    4) “This is why you occasionally see comparisons like this in bash scripts:

    if [ x$(grep not_there /dev/null) = ‘x’ ]”

    It’s cleaner to enclose the $(…) expression in double quotes, which forces it to be treated as a single word even if it’s empty.

    if [ “$(grep not_there /dev/null)” = “” ] ; then … ; fi

    5) sets

    Each of the “set -x” commands corresponds to a bash command-line option; executing “set -x” is like invoking “bash -x”. There are also more user-friendly equivalents for most or all of them. For example “set -o xtrace” is equivalent to “set -x”, and “set -o errexit” is equivalent to “set -e”. (The bash manual doesn’t make it particularly easy to find these.)

    6) <()

    The "<()" construct expands to a file name, so commands don't have to do anything special to refer to it. For example, on my system "echo <(echo foo) <(echo bar)" prints "/dev/fd/63 /dev/fd/62".

    7) quoting

    I don't find it surprising that
    echo "*"
    prints a literal '*' character. Single quotes inhibit both wildcard expansions and variable expansions. Double quotes inhibit wildcard expansions but do not inhibit variable expansions.

    8) Top three shortcuts

    !:1-$ can also be written as !*

    The :h expansion only works in history substitutions. Your example
    ls /long/path/to/some/file/or/other.txt:h
    attempts to list a file whose name literally ends in ":h". (csh and tcsh support ":h" et al on variable expansions; bash doesn't.)

    But bash does have a powerful set of modifiers for variable expansion (called "parameter expansion" in the bash manual), documented here:
    https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html

    You can also use command substitution for the above example:

    $ grep isthere /long/path/to/some/file/or/other.txt
    $ ls $(dirname !$)

  6. Great article. Lots of useful tidbits.

    Regarding ` !:1-$`. An easier form to use is` !*` (bang splat). It does the same thing and is easier to type.Also, you can add `:p` to any of these to print the result and not execute it. Then, after verifying it is what you want a quick `!!` will execute it.

    Keep writing.

  7. :h doesn’t work for me. I’m running bash 4.3.46, and I get “No such file or directory” as if it is literally making :h part of the file name.

  8. I once wrote a one liner to parse out of order arguments in long and short forms. A bash master like you might get a kick out of how this works.

    aes-encrypt.sh

    eval “declare -A vars=(“$(getopt -a -o i:o:s: –long in:,out:,sec: -n ‘aes-encrypt.sh’ — “$@” | sed ‘s/–$//g; s/-\+/-/g; s/-\([a-z]\)\([a-z]*\)[[:space:]]/[“\1″]=/g’)”)” && openssl enc -aes-128-cbc -in “${vars[i]}” -out “${vars[o]}” -p >> “${vars[s]}” || echo ‘ಠ_ಠ…pls lrn2computar’

    Essentially it creates an array of the getopts output that have been sanitized using sed. You can then use the dictionary array that is holding the arguments named by their argument letter. In this case, input is vars[i], output is vars[o] etc. This avoids long and ugly switch statements for argument parsing and is resilient to out of order and and long/short forms.

  9. Number 7, quoting: In the “echo” section, you should try three things. echo “*” and echo ‘*’ do the same thing, which is to display an asterisk. However, echo * (i.e. without the quotes) will display the files within the folder (in this case, the letter ‘a’), because Bash will expand the unquoted asterisk before calling ‘echo’.

  10. Very good intro. The only bash feature in your list above that I haven’t used is <(), I'll definitely use it in future.

    I'd strongly recommend:
    set -u
    Which will exit on any attempts to use uninitialised variables.

    I'm a huge fan of the shebang history. Some more that you might consider using:

    – Reference the current command with !# – for example: cp filename !#^.backup
    – Print before use !23:p – this will show command 23 without executing it but will put it in history, and if it's the one you want you can just type !!
    – Search !?string? – for example (combining with the :p) the last command that contained 'vi': !?vi?:p

    Philip

  11. Hi,

    $ if [ x$(grep not_there /dev/null) = ‘x’ ]; then true; fi

    Be careful with this, as grep may return spaces:

    $ a=’1 2′; if [ x$a = x ]; then echo true; fi

    This fails as an example. This is present on ./configure in case the shell used is an antique non-POSIX shell. What about using the safer and POSIX-compatible form:

    $ a=’1 2′; if [ “$a” = ” ]; then echo true; fi

    or even:

    $ a=’1 2′; if [ -z “$a” ]; then echo true; fi

    or also

    $ a=’1 2′; if ! [ “$a” ]; then echo true; fi

    and thank you for taking the time to share what the experience revealed to be useful, it feels that the shell syntax is either perceived as mystical or simply not transmitted at all.

    Fortunately, the POSIX manual pages or the more readable man page of each implementation preserve all of it.

  12. re: cheap ci script – it’s worth stating the maintenance of a bash script of any significant size has diminishing returns compared to a programming language like Go or Python. I’ve fallen foul of this in the past (as a bash masochist).

Leave a Reply to Tran Ngoc Tuan Anh Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s