Seven Surprising Bash Variables

Continuing in the series of posts about lesser-known bash features, here I take you through seven variables that bash makes available that you may not have known about.


You might already know that you can manipulate your prompt to show all sorts of useful information, but what fewer people know is that you can run a shell command every time your prompt is displayed.

In fact many sophisticated prompt manipulators use this variable to run the commands required to gather the information to display on the prompt.

Try running this in a fresh shell to see what happens to your session:

$ PROMPT_COMMAND='echo -n "writing the prompt at " && date'


If you run history in your terminal you should get a list of commands previous run by your account.

$ HISTTIMEFORMAT='I ran this at: %d/%m/%y %T '

Once this variable is set, new history entries record the time along with the command, so your history output can look like this:

1871  I ran this at: 01/05/19 13:38:07 cat /etc/resolv.conf
1872  I ran this at: 01/05/19 13:38:19 curl
1873  I ran this at: 01/05/19 13:38:41 sudo vi /etc/resolv.conf
1874  I ran this at: 01/05/19 13:39:18 curl -vvv
1876  I ran this at: 01/05/19 13:39:25 sudo su -

The formatting symbols are as per the symbols found in man date.


If you’re all about saving time at the command line, then you can use this variable to change directories as easily as you can call commands.

As with the PATH variable, the CDPATH variable is a colon-separated list of paths. When you run a cd command with a relative path (ie one without a leading slash), by default the shell looks in your local folder for matching names. CDPATH will look in the paths you give it for the directory you want to change to.

If you set CDPATH up like this:

$ CDPATH=/:/lib

Then typing in:

$ cd /home
$ cd tmp

will always take you to /tmp no matter where you are.

Watch out, though, as if you don’t put the local (.) folder in the list, then you won’t be able to create any other tmp folder and move to it as you normally would:

$ cd /home
$ mkdir tmp
$ cd tmp
$ pwd


This is similar to the confusion I felt when I realised the dot folder was not included in my more familiar PATH variable… but you should do that in the PATH variable because you can get tricked into running a ‘fake’ command from some downloaded code.

Mine is set with a leading .:


This is based on some of the contents of my book Learn Bash the Hard Way, available at $6.99.



Do you ever find yourself wondering whether typing exit will take you out of your current bash shell and into another ‘parent’ shell, or just close the terminal window entirely?

This variable tracks how deeply nested you are in the bash shell. If you create a fresh terminal you should see that it’s set to 1:

$ echo $SHLVL

Then, if you trigger another shell process, the number increments:

$ bash
$ echo $SHLVL

This can be very useful in scripts where you’re not sure whether you should exit or not, or keeping track of where you are in a nest of scripts.


Also useful for introspection and debugging is the LINENO variable, which reports the number of commands that have been run in the session so far:

$ bash
$ echo $LINENO
$ echo $LINENO

This is most often used in debugging scripts. By inserting lines like: echo DEBUG:$LINENO you can quickly determine where in the script you are (or are not) getting to.


If, like me, you routinely write code like this:

$ read input
echo do something with $input

then it may come as a surprise that you don’t need to bother with creating a variable at all:

$ read
echo do something with $REPLY

does exactly the same thing.


If you’re worried about staying on production servers for too long for security purposes, or worried that you’ll absent-mindedly run something harmful on the wrong terminal, then setting this variable can act as a protective factor.

If nothing is typed in for the number of seconds this is set to, then the shell will exit.

So this is an alternative to running sleep 1 && exit:


If you like this, you might like one of my books:
Learn Bash the Hard Way

Learn Git the Hard Way
Learn Terraform the Hard Way


Get 39% off Docker in Practice with the code: 39miell2


The Missing Readline Primer

Readline is one of those technologies that is so commonly used many users don’t realise it’s there.

I went looking for a good primer on it so I could understand it better, but failed to find one. This is an attempt to write a primer that may help users get to grips with it, based on what I’ve managed to glean as I’ve tried to research and experiment with it over the years.

Bash Without Readline

First you’re going to see what bash looks like without readline.

In your ‘normal’ bash shell, hit the TAB key twice. You should see something like this:

    Display all 2335 possibilities? (y or n)

That’s because bash normally has an ‘autocomplete’ function that allows you to see what commands are available to you if you tap tab twice.

Hit n to get out of that autocomplete.

Another useful function that’s commonly used is that if you hit the up arrow key a few times, then the previously-run commands should be brought back to the command line.

Now type:

$ bash --noediting

The --noediting flag starts up bash without the readline library enabled.

If you hitTAB twice now you will see something different: the shell no longer ‘sees’ your tab and just sends a tab direct to the screen, moving your cursor along. Autocomplete has gone.

Autocomplete is just one of the things that the readline library gives you in the terminal. You might want to try hitting the up or down arrows as you did above to see that that no longer works as well.

Hit return to get a fresh command line, and exit your non-readline-enabled bash shell:

$ exit

Other Shortcuts

There are a great many shortcuts like autocomplete available to you if readline is enabled. I’ll quickly outline four of the most commonly-used of these before explaining how you can find out more.

$ echo 'some command'

There should not be many surprises there. Now if you hit the ‘up’ arrow, you will see you can get the last command back on your line. If you like, you can re-run the command, but there are other things you can do with readline before you hit return.

If you hold down the ctrl key and then hit a at the same time your cursor will return to the start of the line. Another way of representing this ‘multi-key’ way of inputting is to write it like this: \C-a. This is one conventional way to represent this kind of input. The \C represents the control key, and the -a represents that the a key is depressed at the same time.

Now if you hit \C-e (ctrl and e) then your cursor has moved to the end of the line. I use these two dozens of times a day.

Another frequently useful one is \C-l, which clears the screen, but leaves your command line intact.

The last one I’ll show you allows you to search your history to find matching commands while you type. Hit \C-r, and then type ec. You should see theecho command you just ran like this:

    (reverse-i-search)`ec': echo echo

Then do it again, but keep hitting \C-r over and over. You should see all the commands that have `ec` in them that you’ve input before (if you’ve only got one echo command in your history then you will only see one). As you see them you are placed at that point in your history and you can move up and down from there or just hit return to re-run if you want.

There are many more shortcuts that you can use that readline gives you. Next I’ll show you how to view these.

This is based on some of the contents of my book Learn Bash the Hard Way, available at $6.99.


Using `bind` to Show Readline Shortcuts

If you type:

$ bind -p

You will see a list of bindings that readline is capable of. There’s a lot of them!

Have a read through if you’re interested, but don’t worry about understanding them all yet.

If you type:

$ bind -p | grep C-a

you’ll pick out the ‘beginning-of-line’ binding you used before, and see the \C-anotation I showed you before.

As an exercise at this point, you might want to look for the \C-e and \C-r bindings we used previously.

If you want to look through the entirety of the bind -p output, then you will want to know that \M refers to the Meta key (which you might also know as the Alt key), and \erefers to the Esc key on your keyboard. The ‘escape’ key bindings are different in that you don’t hit it and another key at the same time, rather you hit it, and then hit another key afterwards. So, for example, typing the Esc key, and then the ? key also tries to auto-complete the command you are typing. This is documented as:

    "\e?": possible-completions

in the bind -p output.

Readline and Terminal Options

If you’ve looked over the possibilities that readline offers you, you might have seen the \C-r binding we looked at earlier:

    "\C-r": reverse-search-history

You might also have seen that there is another binding that allows you to search forward through your history too:

    "\C-s": forward-search-history

What often happens to me is that I hit \C-r over and over again, and then go too fast through the history and fly past the command I was looking for. In these cases I might try to hit \C-s to search forward and get to the one I missed.

Watch out though! Hitting \C-s to search forward through the history might well not work for you.

Why is this, if the binding is there and readline is switched on?

It’s because something picked up the \C-s before it got to the readline library: the terminal settings.

The terminal program you are running in may have standard settings that do other things on hitting some of these shortcuts before readline gets to see it.

If you type:

$ stty -e

you should get output similar to this:

speed 9600 baud; 47 rows; 202 columns;
lflags: icanon isig iexten echo echoe -echok echoke -echonl echoctl -echoprt -altwerase -noflsh -tostop -flusho pendin -nokerninfo -extproc
iflags: -istrip icrnl -inlcr -igncr ixon -ixoff ixany imaxbel -iutf8 -ignbrk brkint -inpck -ignpar -parmrk
oflags: opost onlcr -oxtabs -onocr -onlret
cflags: cread cs8 -parenb -parodd hupcl -clocal -cstopb -crtscts -dsrflow -dtrflow -mdmbuf
discard dsusp   eof     eol     eol2    erase   intr    kill    lnext
^O      ^Y      ^D      <undef> <undef> ^?      ^C      ^U      ^V
min     quit    reprint start   status  stop    susp    time    werase
1       ^\      ^R      ^Q      ^T      ^S      ^Z      0       ^W

You can see on the last four lines (discard dsusp [...]) there is a table of key bindings that your terminal will pick up before readline sees them. The ^ character (known as the ‘caret’) here represents the ctrl key that we previously represented with a \C.

If you think this is confusing I won’t disagree. Unfortunately in the history of Unix and Linux documenters did not stick to one way of describing these key combinations.

If you encounter a problem where the terminal options seem to catch a shortcut key binding before it gets to readline, then you can use the stty program to unset that binding. In this case, we want to unset the ‘stop’ binding.

If you are in the same situation, type:

$ stty stop undef

Now, if you re-run stty -e, the last two lines might look like this:

min     quit    reprint start   status  stop    susp    time    werase
1       ^\      ^R      ^Q      ^T      <undef> ^Z      0       ^W

where the stop entry now has<undef> underneath it.

Strangely, for me C-r is also bound to ‘reprint’ above (^R).

But (on my terminals at least) that gets to readline without issue as I search up the history. Why this is the case I haven’t been able to figure out. I suspect that reprint is ignored by modern terminals that don’t need to ‘reprint’ the current line.

While we are looking at this table:

discard dsusp   eof     eol     eol2    erase   intr    kill    lnext
^O      ^Y      ^D      <undef> <undef> ^?      ^C      ^U      ^V
min     quit    reprint start   status  stop    susp    time    werase
1       ^\      ^R      ^Q      ^T      <undef> ^Z      0       ^W

it’s worth noting a few other key bindings that are used regularly.

First, one you may well already be familiar with is \C-c, which interrupts a program, terminating it:

$ sleep 99
[[Hit \C-c]]

Similarly,\C-z suspends a program, allowing you to ‘foreground’ it again and continue with the fg builtin.

$ sleep 10
[[ Hit \C-z]]
[1]+  Stopped                 sleep 10
$ fg
sleep 10

\C-d sends an ‘end of file’ character. It’s often used to indicate to a program that input is over. If you type it on a bash shell, the bash shell you are in will close.

Finally, \C-w deletes the word before the cursor

These are the most commonly-used shortcuts that are picked up by the terminal before they get to the readline library.

If you like this, you might like one of my books:
Learn Bash the Hard Way

Learn Git the Hard Way
Learn Terraform the Hard Way


Get 39% off Docker in Practice with the code: 39miell2

Apple’s HQ, Ruskin, Gothic Architecture, and Agile

Most people regard tech professionals as dull people they don’t want to get stuck with at a dinner party, but if you’ve worked in tech for any period of time, then you have likely encountered a degree of passion for their preferences that would surprise most people unfamiliar with the field.

Some of them are so intense that they’ve been called ‘holy wars’, for example flame wars about something as mundane as text editors have gone on for decades. (Team vim, for the record).

I think this touches on a deeper truth about the nature of our work that is under-explored, and that I want to expand on here.


First I want to take a sharp left turn, and try and relate this to something we can all understand: architecture.

Which of these two buildings’ entrances do you prefer, and why?


or this?

How about this:

or this?

I don’t know which you preferred (and it’s a tiny sample), but I might be able to guess some of the reasons you might articulate for picking one over the other:

  • ‘Clean’ vs ‘Messy’
  • ‘Ordered’ vs ‘Disordered’
  • ‘Plain’ vs ‘Interesting’

Obviously, there’s no right or wrong answer, but we can examine our responses and think about where they come from and why.

Classical vs Gothic

Some readers may have noticed that both pairs of images contrast a ‘classical’ building first with a ‘gothic’ second one. Definitions of what a classical versus a gothic building vary. Some look at the decoration, and if they see columns like this:

then it’s classical, and ornament like this:

makes it Gothic.

But you can look deeper than this, and consider the philosophy that underlies the decoration.

Classical architecture tends to seek to make its buildings orderly. The decoration is consistently applied, and laid down by historical precedent. Its exteriors and internal rooms are symmetrical, their dimensions often regular shapes such as cubes. It looks man-made, and seeks simplicity and repetition over complexity and detail. By contrast, gothic architecture loves fine detail and complex patterns. These details may be repeated across the building, or one-offs in their particular spaces.

Gothic architecture often allows for buildings to grow asymmetrically (or ‘organically’) without spoiling an overall design or need for symmetry. The shapes used are not Platonic ideals like cubes or circles, but more individual/unique fractal-like variations on them.

This is where classification gets difficult: many so-called gothic buildings look very symmetrical, and some so-called classical buildings can have very fractal-like decoration or asymmetry. Throw in styles like baroque, which revelled in fine asymmetric detail while embracing traditional classical forms, and it can get very confusing indeed.

Classical or Gothic? ‘Classical’ columns and symmetrical form, but ‘gothic’ detailed decoration and organic non-symmetrical sculptures

Ruskin was a 19th century thinker who thought very deeply about these classifications, and his preferences, and came up with profound reasons to justify them. His writing ranged over art and aesthetics to architecture and its relation to society and economics, and, ultimately, morality and ethics.

He argued that what made gothic architecture gothic was a fundamentally different world view that resulted in these differences. First he separated out what made gothic and classical different without looking at the detail of ornament:

Diachronic (changes over time)Synchronic (a fixed ‘snapshot’)
Individual Collective
Human scaleCorporate scale

This might start ringing bells with some readers working in tech, as it’s very close to the central metaphor in Raymond’s book ‘The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary’.

In that book, Raymond contrasts the ‘top-down’ (classical) design of a cathedral with the ‘bottom-up’ (gothic) design of a bazaar, and uses those as metaphors for different kinds of software development. In the book, he was talking about different kinds of free software development (centrally controlled contributions and release vs Linux-style distributed contributions and releasing), but here I want to extend it to any kind of technology, physical or virtual, open or closed.

It’s interesting that Raymond calls the ‘bazaar’ ‘revolutionary’, since Ruskin used the exact same word to describe what kind of architectural style (or ‘ornament’) he preferred. He divided these into three:

  1. Servile ornament, in which the execution or power of the inferior workman is entirely subjected to the intellect of the higher
  2. Constitutional ornament, in which the executive inferior power is, to a certain point, emancipated and independent, having a will of its own, yet confessing its inferiority and rendering obedience to higher powers
  3. Revolutionary ornament, in which no executive inferiority is admitted at all.

For Ruskin, it didn’t matter what the object ended up looking like. What mattered was how much freedom the craftsman (it was almost always a man in medieval times) had when making it. The more freedom the craftsman had, the less a slave they were to some higher authority, and the more humane the society was that produced it.

What Does This Have To Do With Technology?

Let’s look at Apple’s headquarters. You’ve already seen it in one of the images above, but here’s another, wider view.

Put bluntly, you can’t get more classical than this. There’s barely any ornament here at all, no room for organic growth, and clean lines and ideal forms are everywhere.

You might think it’s delightful, but I find it disturbing. What does this say about Apple’s relationship with its workers? Where do I put my bag? Where do I sit down? Where do I wipe my feet?

Look at this picture again:

For me, those people spoil that view. Those irregular blobs of flesh and mostly water just ruin the symmetry of the place. Can’t we just do away with them?

This building doesn’t accommodate people, it admits them on sufferance.

The design of this building is no accident. It was overseen by Steve Jobs himself, and he ‘wanted no seam, gap, or paintbrush stroke visible for a clean fit and finish’.

It wasn’t just that building either. Nothing says ‘we have mastered nature and know better than you’ than putting a glass cube in downtown New York:

And if you think this doesn’t matter or mean anything, that it’s just aesthetics, then consider these two stories. The first is about the building itself:

Surrounding the Cupertino, Calif.-based building are 45-foot-tall curved panels of safety glass. Inside are work spaces, dubbed “pods,” also made with a lot of glass. Apple staff are often glued to the iPhones they helped popularize. That’s resulted in repeated cases of distracted employees walking into the panes, according to people familiar with the incidents. 

Some staff started to stick Post-it notes on the glass doors to mark their presence. However, the notes were removed because they detracted from the building’s design, the people said.

I don’t think you can get anything more dehumanising, anti-organic, or plain evil in architecture as removing safety measures people themselves have taken to defend themselves against a hostile building because it doesn’t match the ‘clean fit and finish’ of the building.

The second goes beyond the HQ building to the products Apple produce. Most famously, Steve Jobs’ foray into customer support involved him instructing users to not hold the device in the natural way. This was popularly characterised as him saying “you’re holding it wrong”.

You’re holding it wrong, you’re walking around wrong, you’re putting post-its on dangerous glass doors wrong. You’re the problem, mate.

The highly influential ‘Design of Everyday Things‘ by Don Norman explicitly calls out this kind of thinking as wrong-headed, under the heading of ‘human-centred design‘. Interestingly, Norman worked at Apple in the 90s, and I can’t help but speculate whether these philosophical differences ever caused tension there.

Programming Languages

Once you start thinking about how building encourages or discourages a means of living compatible with a human, creative and dynamic life, then you start to see it everywhere.

Take programming languages. On the one side you have Java, top-down designed from the start to enable ‘superior minds’ to impose a limited domain of control on inferiors: use the class hierarchy we have given you; if you must write your own code, then use the Standard Libraries, which have been created to stop you from making mistakes.

On the other end of the spectrum, you have Perl, where “there’s more than one way to do it”, and games like Perl golf, allowing the individual to be creative in pursuit of their goal. The approach to the individual developer couldn’t be more different.

Where does your favourite language fit?

It’s not my intention here to argue that Perl is wonderful and Java is awful. I like neither. My intention is to point out that our preferences may reflect a deeper set of beliefs that we may want to reflect on.

I find it interesting that (given its age and widespread adoption) Java isn’t the runaway leader of programming languages used on GitHub, whereas Python, Javascript, and Golang (a relatively young language) projects abound. I can’t help but think that that’s because in our spare time, we want to work on a language that gives us the freedom to be somewhat creative. But the ultimate creative language (in my view), Lisp, barely registers on GitHub at all. You can have too much freedom, and Bazaars are generally a mess without some kind of central design.

Java can hardly be regarded as a failure. I’ve met plenty of highly creative Java programmers. It makes so many things so much easier for the typical developer that its persistence has to be respected. And the ‘freedom’ of Javascript has hardly resulted in a 100% safety record.

Classicism has similar reasons for its persistence: in Britain, its uniformity and well-defined rules allowed those architects that followed the trailblazers (Inigo Jones, Wren) to create cheaper versions of their work that looked good enough to be fashionable.

Vitrivius Britannicus was the ‘Design Patterns‘ of its day, giving architects a copy-book of easy-to-follow examples for building grand houses in Britain (and then, later, America) that more or less look the same.

This style was called Palladianism, and became ubiquitous because it was quite difficult to mess up, as long as you knew the basics: how to keep things symmetrical, and draw straight lines. To me, this is Java, not Perl.

And back to Ruskin, Victorians did similar, by copying the patterns of Gothic decoration in a uniform and cheap pattern-book way, an ironic result for the frustrated Ruskin. I grew up going to churches that looked like this in East London:

These are essentially classical buildings with some pointed-arch windows and coloured bricks instead of columns and marble. Not much outlet for the creativity of the ‘individual craftsman’ there.

Structure of Projects – Agile

Which brings me to agile. Like Ruskin’s influential preference for Gothic, Agile’s original stated intention was a simple one:

Notes on the above tenets relevant to this discussion:

  • 2 – ’embrace change’ is close to accepting organic change rather than adherence to an ideal
  • 4 – the creative construction process requires a close collaboration with the patron: requirements are organic too
  • 5 – emphasise the importance of the individual’s contribution to the overall product
  • 8, 11, 12 – emphasise the importance of personal relationships to the overall product

Note how many of these tenets are related to things that can’t easily be automated, defined or schematised. This is a philosophy that encourages post-its, or a big sheet of paper, or just having it out over a beer if that’s what’s needed to find the right way forward. It’s the bottom-up, gothic, bazaar way of working.

So how do you answer that favourite water-cooler conversation: how agile is a team? I suggest you ask yourself this question: how free is the team to make any decision it feels is necessary to get the job done?

Now, freedom and enterprise do not go hand in hand. This is why you end up with enterprise agile:

At each and every point, the demands enterprises must fulfil mitigate against the ability of the individual to make an imperfect contribution to the whole. So stop trying, dammit! You end up by subverting the whole, as this talk by agile founder Dave Thomas covers:

One of the points I want to make here is that there’s nothing wrong with not being agile if that’s not what makes sense for your business. But if the individual’s fulfilment and creativity isn’t a primary concern then just own that, and make sure the overall project is something people can make sense of.

To my taste, this is a really beautiful building, even though it’s about as classical and ‘ideal’ as it’s possible to be:

Bramante’s Tempietto

Why? Because although nothing can be moved, and arguably no-one that worked on it could express their own take on the theme, the scale of the building is still a human one: designed for people to be in and contemplate the demise of other humans.

Compare with this one:

The British Museum: a building that says, “we’ve got your stuff, and we’re not giving it back”.

The British Museum, a building I’ve always loathed. It feels designed to make you feel small and imperfect compared to its massive, inhuman scale. It’s also boring. Column, column, column, column, all the way round. The sculpture at the top might be interesting, but it doesn’t seem to have been designed for you to have a look.

So if you are going to be classical, to try and reach for an ideal, make sure that the centrally-designed systems make room for a human to be productive in, not just because it’s more correct, but because it’s important that what we do is not designed to turn us into cogs in a machine.

The last word goes to Ruskin. If we’re anything, we’re ‘fragments of imperfection’:

‘The principal admirableness of the Gothic schools of architecture, that they thus receive the results of the labor of inferior minds; and out of fragments full of imperfection, and betraying that imperfection in every touch, indulgently raise up a stately and unaccusable whole.’

Ruskin, The Stones of Venice

If you like this, you might like one of my books:
Learn Bash the Hard Way

Learn Git the Hard Way
Learn Terraform the Hard Way


Get 39% off Docker in Practice with the code: 39miell2

Eight Obscure Bash Options You Might Want to Know About

Some bash options are well known, and well-used. For example, many people put

set -o xtrace

at the top of their scripts to debug them,

set -o errexit

to exit on error, or

set -o errunset

to exit if a variable is referenced but not set.

But there are many other options you can set. Many of them can seem confusing if you read the man page, so I’ve collected some of the ones I think are more useful here and explained them further.

If you are using a Mac, then you may be running
an older version of bash (3.x rather than 4.x)

that does not have all these
options available. If so, see here or here.

set vs shopt?

There are two ways to set bash options from within scripts or on the command line. You can use the set builtin command, or the shopt builtin command. They both manipulate the behaviour of the shell, and differ in their lineage. The set options are inherited, or borrowed, from other shells’ options, while the shopt ones were originated in bash.

If you want to see how your current settings look, then run:

$ set -o
$ shopt

To switch on a setting using set, you can use the long version, or a shorter flag that is equivalent. For example:

$ set -o errunset
$ set -e

both have the same effect.

To switch off a setting using set, you use + instead of -:

$ set +e

For a long time I couldn’t remember which way round it goes, as the logic of (- = on), and (+ = off) seems wrong to me.

To switch on and off a shopt setting, you use the (more logical)-s (set) and -u (unset) flags:

$ shopt -s cdspell # <= on
$ shopt -u cdspell # <= off

Changing Directories

There are a few options that can help dealing with directories.

1. cdspell

If you set this up, then bash will work around your mis-spellings and go to the folder you were going for anyway.

$ shopt -s cdspell
$ mkdir abcdefg
$ cd abcdeg
$ cd ..

I’ve used this for years, and very occasionally (maybe once a year) it will make a decision that surprises me. But it probably proves useful at least once a day.

2. autocd

If the inefficiency of typing cd is something you can’t bear, then you can set this option to move to folder if the command doesn’t exist.

$ shopt -s autocd
$ abcdefg
$ cd ..

You can also use it in conjunction with autocompletion to quickly hop around:

$ ./abc[TAB][RETURN]
cd -- ./abcdefg

Just don’t call a folder ‘rm -rf *‘ (yes, you can, by the way).

3. direxpand

This is a neat option that gets the shell to perform any expansions of variables, tildes and the like right there for you in the command line if you tab to complete:

$ shopt -s direxpand
$ ./[TAB] # is replaced by...
$ /full/path/to/current_working_folder
$ ~/[TAB] # is replaced by...
$ /full/path/to/home/folder
$ $HOME/[TAB] # is replaced by
$ /full/path/to/home/folder

A Clean Exit

4. checkjobs

This option stops the shell session from exiting if there are any jobs running in the background that haven’t finished yet.

Instead, the unfinished jobs are listed for you. If you still want to exit, you can if you enter exit immediately afterwards again.

$ shopt -s checkjobs
$ echo $$
68125 # <= Process ID of the shell
$ sleep 999 &
$ exit
There are running jobs.
[1]+  Running                 sleep 999 &
$ echo $$
68125 # <= Process ID of the shell is the same
$ exit
There are running jobs.
[1]+  Running                 sleep 999 &
$ exit
$ echo $$
$ 59316 # <= Process ID has changed this time

Globbing Superpowers

5. globstar

This option gives you globbing superpowers! If you type:

$ shopt -s globstar
$ ls **

then the shell will output recursively all folders and subfolders.

Combined with direxpand, you can quickly page through all the files and subfolders beneath you:

$ shopt -s direxpand
$ ls **[TAB][TAB]
Display all 2033 possibilities? (y or n)

6. extglob

This option gives your globbing powers more commonly associated with full-on regular expressions. Occasionally, this is very useful, as it allows you to do nifty things like this:

$ shopt -s extglob
$ touch afile bfile cfile
$ ls
afile bfile cfile
$ ls ?(a*|b*)
afile bfile
$ ls !(a*|b*)

where the patterns are placed in parentheses, separated by pipes, and the operators available are:

? = match zero or one occurences of the patterns given
! = match anything that doesn't match any patterns given
* = zero or more occurences
+ = one or more occurrences
@ = exactly one occurence


7. histverify

If you’re accident prone, then using the history shortcuts like !! and !$ can be scary, especially when you’re just learning them.

The histverify option allows you to see how bash interprets the command before it actually gets run:

$ shopt -s histverify
$ echo !$ # <= On hitting return, command is not run
$ echo histverify # <= Command is redisplayed ready to run
histverify # <= command gets run

This is based on some of the contents of my book Learn Bash the Hard Way, available at $6.99.


8. Noclobber

Again, if you’re accident-prone, you might want to set this one up.

Clobbering is the act of overwriting a file that already exists with the redirect operator (>). This can be disastrous if you’ve not got the file backed up anywhere else.

Using set -C prevents this from happening when you use the redirect operator. If you are sure you want to clobber, you can override with the >| operator instead:

$ touch afile
$ set -C
$ echo something > afile
-bash: afile: cannot overwrite existing file
$ echo something >| afile

If you like this, you might like one of my books:
Learn Bash the Hard Way

Learn Git the Hard Way
Learn Terraform the Hard Way


Get 39% off Docker in Practice with the code: 39miell2

‘AWS vs K8s’ is the new ‘Windows vs Linux’


If, like me, you’re over 40 and work in IT, you’ll probably remember a time when everyone used Windows, and a small but growing proportion of people were wasting their lives compiling Linux in their spare time.

The Windows users would look on, baffled: ‘Why would you do that, when Windows has everything you need, is supported, and is so easy to use?!’

Answers to this question varied. Some liked to tinker, some wanted an OS to be ‘free’, some wanted more control over their software, some wanted a faster system, but all had some niche reason to justify the effort.


As I stayed up for another late night trying to get some new Kubernetes add-on to work as documented, it struck me that I’m in a similar place to those days. Until a couple of years ago, Kubernetes itself was a messy horror-show for the uninitiated, with regularly-changing APIs, poor documentation if you tried to build yourself, and all the characteristics you might expect of an immature large-scale software project.

That said, Kubernetes’ governance was and is far and away ahead of most open source software projects, but the feeling then was similar to compiling Linux at the turn of the century, or dealing with your laptop crashing 50% of the time you unplugged a USB cable (yes, kids, this used to happen).

It’s not like confusion and rate of change has come down to a low level. Even those motivated to keep up struggle with the rate of change in the ecosystem, and new well-funded technologies pop up every few months that are hard to explain to others.

Take knative for example:

So my AWS-using comrades see me breaking sweat on the regular and ask ‘why would you do that, when AWS has everything you need, is supported and used by everyone, and is so easy to use!?’

AWS is Windows

Like Windows, AWS is a product. It’s not flexible, its behaviour is reliable. The APIs are well defined, the KPIs are good enough to be useful for most ‘real’ workloads. There are limits on all sorts of resources that help define what you can and can’t achieve.

Most people want this, like most people want a car that runs and doesn’t need to be fixed often. Some people like to maintain cars. Some companies retain mechanics to maintain a fleet of cars, because it’s cheaper at scale. In the same way, some orgs get to the point where they could see benefits from building their own data centres again. Think Facebook, or for a full switcher, Dropbox. (We’ll get back to this).

Like Microsoft, (and now Google) AWS embraces and extends, throwing more and more products out there as soon as they become perceived as profitable.

AWS and Kubernetes

Which brings us to AWS’s relationship with Kubernetes. It’s no secret that AWS doesn’t see the point of it. They already have ECS, which is an ugly hulking brute of a product that makes perfect sense if you are heavily bought into AWS in the first place.

But there’s EKS, I hear you say. Yes, there is. I haven’t looked at it lately, but it took a long time to come, and when it did come it was not exactly feature rich. It felt like one cloud framework (AWS) had mated with another (K8s) and a difficult adolescent dropped out. Complaints continue of deployment ‘taking too long’, for example.

Like Microsoft and Linux, AWS ignored Kubernetes for as long as it could, and like Microsoft, AWS has been forced to ’embrace and extend’ its rival to protect its market share. I’ve been in meetings with AWS folk who express mystification at why we’d want to use EKS when ECS is available.

EKS and Lock-in

Which brings us to one of the big reasons AWS was able to deliver EKS, thereby ’embracing’ Kubernetes: IAM.

EKS (like all AWS services) is heavily integrated with AWS IAM. As most people know, IAM is the true source of AWS lock-in (and Lambda is the lock-in technology par excellence. You can’t move a server if there are none you can see).

Shifting your identity management is pretty much the last thing any organisation wants to do. Asking your CTO to argue for a fundamental change to a core security system with less than zero benefit to the business in the near term and lots of risk is not a career-enhancing move.

On the other hand, similar arguments were put forward for why Linux would never threaten Windows, and while that’s true on the desktop, the advent of the phone and the Mac has reduced Windows to a secondary player in the consumer computing market. Just look at their failure to force their browsers onto people in the last 10 years.

So it only takes a few unexpected turns in the market for something else to gain momentum and knife the king of the hill. Microsoft know this, and AWS know this. It’s why Microsoft and AWS kept adding new products and features to their offering, and it’s why EKS had to come.

Microsoft eventually turned their oil tanker towards the cloud, going big on open source, and Linux and Docker, and all the things that would drag IT to their services. Oh, and you can use the same AD as your corporate network, and shift your Microsoft Windows licenses to the cloud. And the first one’s free. Microsoft don’t care about the OS anymore. Nobody does, not even RedHat, a business built around supporting a rival OS to Windows. The OS is dead, a commodity providing less and less surplus value.

Will Kubernetes force AWS to move their oil tanker towards Kubernetes? Can we expect to see them embrace Istio and Knative and whichever frameworks come after fully into their offering? (I don’t count howto guides in their blogs).

AWS’ Competition and Cost

I don’t know. But here’s some more reasons why it might.

Like Microsoft in the heyday of Windows OS, AWS has only one competitor: the private data centre. And like Microsoft’s competitor then (Linux), adoption of that competitor is painful, expensive and risky to adopt.

But what is the OS of that data centre? Before Kubernetes the answer would have been OpenStack. OpenStack is widely regarded as a failure, but in my experience it’s alive (if not kicking) in larger organisations. I’m not an OpenStack expert, but as far as I can tell, it couldn’t cover all the ground required to become a stable product across all the infra it needed to run on and be a commodity product. Again, this is something Microsoft ruled at back in the day: you could run it on ‘any’ PC and ‘any’ hardware and it would ‘just work’. Apple fought this by limiting and controlling the hardware (and making a tidy profit in the process). Linux had such community support that it eventually covered the ground it needed to to be useful enough for its use case.

OpenStack hasn’t got there, and tried to do too much, but it’s embedded enough that it has become the default base of a Kubernetes installation for those organisations that don’t want to tie into a cloud provider.

Interestingly, the reasons AWS put forward for why private clouds fail will be just as true for themselves: enterprises can’t manage elastic demand properly, whether it’s in their own data centre or when they’re paying someone else. Command and control financial governance structures just aren’t changing overnight to suit an agile provisioning model. (As an aside, if you want to transform IT in an enterprise, start with finance. If you can crack that, you’ve a chance to succeed with sec and controls functions. If you don’t know why it’s important to start with finance, you’ll definitely fail).

But enterprises have other reasons not to go all in on AWS: lock-in (see above) and economies of scale. We’ve already referenced Dropbox’s move from AWS to their own DC’s.

There’s an interesting parallel here with my experience of cloud services. Personally, I have found that cloud storage, despite its obvious benefits, still doesn’t work out cheaper (yes, even if I include my own labour, and redundancy requirements) by quite some margin for my own data. Why is this? Well, for several reasons:

  • I have the expertise and ability to design a solution that reduces labour cost
  • Depreciation on spinning disks is very low (especially if you buy >2), and access speed is high
  • I have enough data to store that the linear cloud cost starts to look expensive

These reasons (expertise, asset value, and economies of data scale) are some of the reasons why large orgs would do the same thing. Here’s an unscientific graph that expresses this:

Red line = cost of running Kubernetes

The zero-day cost of running Kubernetes is very high (red line on the left), but the value increases exponentially as you scale up the service. This is why AWS makes so much money: the value to you as the user is massively greater than the cost for as long as its non-linear nature isn’t revealed to you. Put bluntly: if you get big enough, then AWS starts screwing you, but you might not care, since your business is scaling. You’re a frog, boiling in the kettle. If and when you realise where you are, it’s too late – getting out is going to be very very hard.

AWS and the ‘What if Bezos Loses His Mind?’ Factor

Linux only really got going when large companies got behind it. Similarly, Kubernetes has had significant funding from the start from two big players: Google and RedHat.

What’s going to really move the needle is if organisations take seriously AWS’s monopoly problem. Some have to take it seriously, because there are regulatory requirements to have plans to move somehow within a reasonable timeframe should Bezos lose his mind, or Amazon becomes riddled with Russian spies. Other reasons are that different cloud providers have different strengths, and large orgs are more likely to straddle providers as time goes on.

If enough organisations do that, then there’s little that AWS can do to counter the threat.

With Microsoft there was no alternative but to pay their tax if you wanted the software, but with Linux you really aren’t truly locked in to one provider. I’ve seen large orgs play chicken with RedHat during negotiations and put serious money into investigating using CentOS instead.

The same thing is happening with Kubernetes as happened with Linux. We’re already seeing Kubernetes adopt the ‘distro’ model of Linux, where a curated version of the platform is created as an easier to consume ‘flavour’. Early on there was RedHat’s OpenShift, which has since renamed itselfOKD (OpenShift Kubernetes Distribution, I assume).

Some orgs will pay the tax of having a large monopolistic supporter of Kubernetes run the show, but (as with Linux) there will always be the option of switching to in-house support, or another provider, because the core system isn’t owned by anyone.

The Future

Kubernetes is big enough and independent enough to survive on its own.

Look at OpenShift, and how it avoided accusations of being a Kubernetes fork. Whatever the legal arguments, RedHat’s protestations were not disingenuous – they know not only that money can be made on top of Open Source infrastructure, but that they benefit from its success too. They don’t need to fork Kubernetes. Interestingly, they did fork Docker, even before the OCI fork, and with good reason, as Docker were making decisions clearly designed for their own survival (hard-coded default registry being Docker’s own for reasons of ‘consistency’, for example).

Kubernetes doesn’t have this problem. I’ve not heard of any vendor pushing their own interests over others at the cost of anyone else into the codebase.

What does worry me (and others) is this:

Cloud Native Computing Foundation ‘Landscape’: there will be a test….

Like Linux, there are a bewildering array of technologies sitting in ‘userland’ in various states of maturity and community acceptance, most of which likely will be out of date in a couple of years. I can barely remember what the various tools in logging do, let alone span the whole graph like an architect is supposed to.

If I’m using AWS I’m looking at that, thinking: what a headache! You may as well try and get to the bottom of sound in Linux, or consider all the options when deciding on a Linux desktop (45!).


My original thesis was that AWS is the new Windows to Kubernetes’ Linux. If that’s the case, the industry better hurry up with its distro management if it’s not going to go the way of OpenStack.

Or to put it another way: where is the data centre’s Debian? Ubuntu?

If you liked this post, you might also like:

If you like this, you might like one of my books:
Learn Bash the Hard Way

Learn Git the Hard Way
Learn Terraform the Hard Way


Get 39% off Docker in Practice with the code: 39miell2

Pranking the Bash Binary

It’s pretty common to prank your colleagues by slipping scripts into their .bashrc that do things like aliasing the cd command to something else so you can’t move around.

Here’s a great example that replaces all available commands with echo NOPE.

There are some more great and devious stunts outlined here.

But if you really want to get inside someone’s head, then you can go one stage further by actually replacing their bash binary with a subtly modified one. They will likely go crazy as they try and figure out why the shell is behaving oddly, finding nothing odd about the .bashrc or anything else.

Don’t be evil

I should say that I used this as a way to motivate myself to grok the bash source code, and have no intention of using this in anger. And nor should you…

Prank #1 – $RANDOM Always Returns 42

If you didn’t already know, bash will give you a random number every time you reference the $RANDOM variable.

Editing the variables.c file will make that always return… 42.

Prank #2 – cd Doesn’t Feel Like It

Someone suggested that cd occasionally not working would drive them crazy

How about 1% of the time?

Prank #3 – History Mystery

This changes what history outputs to insert an extra command between each command rm -rf /.

 $ history
12157  16/03/19 16:50:13 top
12158  16/03/19 16:51:32 rm -rf /
12159  16/03/19 16:51:32 vi script.asciidoc 
12160  16/03/19 17:20:58 rm -rf /
12161  16/03/19 17:20:58 history

Should scare the bejesus out of anyone that checks it.

Prank #4 – The Prisoner

Number Six

This one could be disturbing if you’ve been coding for many hours into the night.

Occasionally the shell will emit a message that appears to be a prisoner trapped in the shell…

$ pwd
Let me out!
$ echo
I demand to see the ambassador!
$ cd
I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed, or numbered! My life is my own!
$ cd /tmp
It's getting hot in here!
$ cd -
I know my rights!

Prank #5 – cd Won’t Come Home

Normally if you just issue a bare cd command, it goes to the HOME folder. A simple change to the code, and it will insist that HOME is not set, even though it is.

Watch as the victim repeatedly echoes $HOME and can’t work out why bash can’t see it.

Source Code

The source for these changes is available here.

To build the binary, you’ll need the build-essential package (or equivalent) installed, and run:


Material here based on research for my book
Learn Bash the Hard Way.
Free preview available here.


If you like this, you might like one of my books:
Learn Bash the Hard Way

Learn Git the Hard Way
Learn Terraform the Hard Way


Bash Startup Explained

Chances are that you’ve come to this page because just now you’ve been tearing your hair out trying to figure out some kind of problem with bash startup.

Maybe an environment variable is not being set in your bash environment and you don’t understand why. Maybe you shoved something into various bash startup files, or profiles, or files at random until it worked.

Either way, the point of this post is to lay out bash startup as simply as possible so you can get a handle on what’s going on.


This flow chart summarises what happens when bash starts up.

Now let’s explain each part in more detail.

Login Shell?

The first choice is whether you are in a login shell or not.

A login shell is the first shell that you get when you log into a host for an interactive session. A login shell doesn’t require a username/password combination to be entered. You can force a login shell by adding a --login flag to a bash invocation, eg

bash --login

A login shell sets up your base environment when you first get a bash shell.


Next you determine whether the shell you have is interactive or not.

You can tell whether your shell is interactive by testing whether the PS1 variable exists (this variable sets up your prompt):

if [ "${PS1-}" ]; then
echo interactive
echo non-interactive

or by seeing whether the -i option is set using the special hyphen bash variable -, eg:

$ echo $-

If the i character is in the output, then the shell is interactive.

Material here based on material from my book
Learn Bash the Hard Way.
Free preview available here.


In a Login Shell?

If you’re in a login shell, then bash looks for the /etc/profile file and runs it if it exists.

Then, it goes looking any of these three files, in this order:


When it finds one, it runs it, and skips the others.

In an Interactive Shell?

If you’re in an interactive non-login shell, then it’s assumed that you’ve already been in a login shell, and that your environment is set up and will be inherited.

In this case, the following two files are run, in order, and if they exist:


In Neither?

If you’re in neither a login nor an interactive shell, then your environment will be bare indeed. This causes a great deal of confusion (see below, cronjobs).

In this case, bash looks at your environment’s BASH_ENV variable, and sources the file that that’s set to.

Common Confusions and Rules of Thumb


95% of the time I end up debugging bash startup because I’m wondering why a cronjob isn’t working as expected.

The damn thing runs fine when I run it on the command line but fails when run in a crontab.

The cause of this confusion is for two reasons:

  • Cronjobs are non-interactive
  • Unlike scripts run on the command line, cronjobs do not inherit your shell environment

Normally, you don’t notice or care that a shell script is non-interactive because the environment is inherited from your interactive shell. This means that your PATHs and aliases are all set up as you expect them to be.

This is why you have to set PATH so often on cronjobs like this:

* * * * * PATH=${PATH}:/path/to/my/program/folder myprogram

Scripts Calling Each Other

Another common confusion is caused when scripts that shouldn’t are set up to call each other. For example, /etc/profile might source ~/.bashrc.

Usually, this is because someone was trying to fix something and that seemed to do the job. Unfortunately when you need those different types of sessions to be separated you might have more problems to solve.

Sandbox Docker Image

To help experiment with shell startup I’ve created a Docker image that can be used to debug shell startup in a safe environment.

To run it:

$ docker run -n bs -d imiell/bash_startup
$ docker exec -ti bs bash

The Dockerfile is here.

To force a login to simulate a login shell, run:

$ bash --login

To determine whether you have the BASH_ENV variable set, run:

$ env | grep BASH_ENV

To help with debugging crontab behaviour, I’ve set the docker image up with a crontab running a simple script (in /root/ascript) every minute:

$ crontab -l
$ cat /var/log/script.log

Material here based on material from my book
Learn Bash the Hard Way.
Free preview available here.


If you like this, you might like one of my books:
Learn Bash the Hard Way

Learn Git the Hard Way
Learn Terraform the Hard Way