Idempotent Bash: Writing Scripts You Can Safely Run Twice

I’ve written about what bash setup scripts turn into once they grow up into real automation, and about a self-healing script I actually run — both assumed idempotency rather than proving it. Every homelab and every ops team accumulates a folder of setup scripts — the ones that provision a new box, register a service, add a line to a config file. The quiet danger with these isn’t that they fail; it’s that they succeed twice. I wanted a concrete example of what “succeed twice” actually does to a system, so I wrote a small setup script the naive way, ran it twice, and looked at what was left behind.

The Naive Version, Run Twice

#!/bin/bash
mkdir /tmp/bashtest-dir
useradd bashtestuser
echo "127.0.0.1 bashtest.local" >> /etc/hosts

Nothing about this looks wrong on a first read — it’s exactly the shape of script most setup guides show. First run, clean output, exit code 0. Second run, same script, no changes made to it at all:

$ bash naive.sh
mkdir: cannot create directory '/tmp/bashtest-dir': File exists
useradd: user 'bashtestuser' already exists
$ echo $?
0

Two real errors printed to the terminal — and the script’s own exit code is still 0. Checking /etc/hosts confirms the third line ran anyway, appending a second time:

$ grep bashtest.local /etc/hosts
127.0.0.1 bashtest.local
127.0.0.1 bashtest.local

That’s the actual failure mode worth sitting with: not a crash, not a clear error a monitoring system would catch — a script that reports success while quietly leaving a duplicate DNS override behind, run after run, forever, if this were cron’d or re-run by a provisioning tool on every deploy.

Why the Exit Code Lied

Bash’s default behavior, without set -e, is to keep executing every line in a script regardless of whether an earlier one failed — and the exit code reported at the end is the exit code of the last command that ran, not a summary of everything that happened along the way. mkdir and useradd both failed and printed errors to stderr; the final echo ... >> /etc/hosts succeeded, so that’s the exit code the whole script reports. Anything checking $? after calling this script — a CI pipeline, a cron wrapper, an Ansible shell task without changed_when — sees a clean pass and moves on, having no way to know two of the three actions inside it actually failed.

The Fix, and Why Each Piece Matters

#!/bin/bash
set -euo pipefail

mkdir -p /tmp/bashtest-dir

if ! id bashtestuser &>/dev/null; then
  useradd bashtestuser
fi

grep -qxF "127.0.0.1 bashtest.local" /etc/hosts || echo "127.0.0.1 bashtest.local" >> /etc/hosts

echo "done, exit will be 0"

Run three times in a row against a clean starting state, this version reports exit code 0 every single time, and /etc/hosts ends up with exactly one matching line — not zero, not three. Each change earns its place:

• set -euo pipefail at the top turns “keep going after an error” into “stop immediately and report failure honestly.” -e exits on any command’s non-zero status, -u catches references to unset variables (a common source of scripts silently operating on an empty string instead of the value you meant), -o pipefail makes a pipeline fail if any stage fails, not just the last one — without it, false | true reports success, which is its own quiet trap in any script that pipes through grep or tail.
• mkdir -p instead of plain mkdir — this one’s nearly free. -p makes “directory already exists” a non-event instead of an error, and it also creates any missing parent directories along the way.
• Check-then-act for useradd — there’s no equivalent flag on useradd itself, so the check has to be explicit: id bashtestuser &>/dev/null asks “does this identity already exist” and only creates the user if the answer is no. This is the same pattern every idempotent Ansible module runs internally, just written out by hand.
• grep -qxF before appending — the line that actually stopped the duplication. -x matches the whole line exactly (not a substring), -F treats the search string as literal text instead of a regex (relevant the moment your line contains a dot, which “127.0.0.1” very much does), and -q suppresses output since only the exit code matters here. The || only appends when the check finds nothing.

A Rule of Thumb: Every Action Needs a Check

The pattern generalizes past these three examples: idempotency in bash isn’t a library or a flag you turn on, it’s a discipline of pairing every mutating command with the question “how would I check whether this already happened?” — a file’s existence, an ID’s presence, a grep for a specific line, a package manager’s own “already installed” state. A script with three actions that isn’t idempotent usually has three separate places that discipline was skipped, not one systemic bug, which is exactly why it’s worth checking line by line rather than assuming a script that looks careful actually is.

This Applies to More Than Scripts You Run Manually

The same discipline matters even more once a script stops being something a person runs by hand and starts being embedded somewhere it’ll re-execute unattended and unwatched — cloud-init on every boot of a new instance, a systemd ExecStartPre hook that runs before every service start, a provisioning step inside a CI job triggered by every merge. In all three of those contexts, nobody’s reading the output line by line the way they were the first time the script was written and tested by hand — a duplicate hosts entry or a failed-but-reported-as-successful step just accumulates quietly across however many times the automation fires, and the first sign of a problem is often an unrelated symptom weeks later, not the actual script erroring out where the mistake was made.

The Checklist

• set -euo pipefail at the top of every script that isn’t a one-off — it turns silent partial failure into an honest stop.
• Use built-in idempotency where a flag already exists (mkdir -p, most package managers’ install commands).
• Where no flag exists, write the check by hand: does the user/file/line/resource already exist? Only act if not.
• Run any new script twice in a row against the same starting state before trusting it — a non-zero exit or a changed result on the second run means something in it isn’t actually idempotent yet.