Introduction: The Day I Stopped Being a Manual Operator
Running a Proxmox homelab with a handful of Node.js services managed by PM2 taught me something fast: manual health checks don’t scale, and they don’t sleep. What started as a “quick morning check” across my containers turned into a two-hour ritual that left no time for actual work. Here’s how I moved from manual operator to automation architect, and the concrete scripting strategies that made it possible.
This post is part of our Scripting and Automation series, but it isn’t just about syntax. It’s about the shift in mindset that separates engineers who fight fires from those who prevent them. When we talk about scripting, we aren’t just talking about saving time — we are talking about eliminating the human error that inevitably creeps in when a tired engineer performs the same task for the thousandth time. Below, I want to share the specific Python and Bash strategies that turned a three-hour morning routine into a three-second automated execution.
The “API-First” Mentality in Scripting
One of the biggest mistakes I made early on was writing scripts that were too specific to a single machine. I would write a Bash script that lived on Server-01 and only worked on Server-01. This is what we call “Pet Scripting.” In a modern DevOps environment, we need “Cattle Scripting.” To achieve this, you must adopt an API-first mentality for your automation.
Whether you are using Python or Go, your scripts should interact with your infrastructure via APIs — AWS SDKs, Kubernetes API, or even internal service endpoints — rather than local file manipulation. This allows your scripts to be portable and scalable. For instance, instead of a script that clears logs on one server, I developed a Python-based cleanup utility that queried an inventory API, identified all servers with disk usage above 80%, and triggered a remote cleanup routine via SSH keys. This shift transformed a local fix into a global solution.
Key Principles for Scalable Automation
- Idempotency: Your script should be safe to run multiple times. If the desired state is already achieved, the script should do nothing.
- Error Handling: Never assume a command succeeds. In Bash, use
set -eto exit on error, or better yet, use Python’stry-exceptblocks to log specific failures. - Parameterization: Never hardcode credentials or hostnames. Use environment variables or configuration files to keep your scripts generic and secure.
Here’s a minimal but real example of a parameterized Bash cleanup script that follows all three principles. I use a pattern like this on my Proxmox nodes to prune old Docker logs without touching anything it shouldn’t:
#!/bin/bash
set -euo pipefail
# Configurable via environment — never hardcode paths or thresholds
LOG_DIR="${LOG_DIR:-/var/lib/docker/containers}"
MAX_AGE_DAYS="${MAX_AGE_DAYS:-7}"
DRY_RUN="${DRY_RUN:-false}"
echo "[$(date -Iseconds)] Starting log cleanup in: $LOG_DIR"
find "$LOG_DIR" -name "*-json.log" -mtime "+${MAX_AGE_DAYS}" | while read -r logfile; do
if [ "$DRY_RUN" = "true" ]; then
echo "[DRY RUN] Would remove: $logfile"
else
echo "Removing: $logfile"
rm -f "$logfile"
fi
done
echo "[$(date -Iseconds)] Cleanup complete."
The DRY_RUN flag is not optional polish — it’s essential. Being able to run any destructive script in preview mode before committing is what prevents 3 AM disasters. The set -euo pipefail header ensures the script halts on any unhandled error, undefined variable, or broken pipe, rather than silently continuing into a bad state.
The “Self-Healing” Script: A Real-World Case Study
I once dealt with a legacy payment gateway that would hang every time it received a malformed XML payload. The fix was always identical: kill the process and restart the service. It happened at 3:00 AM at least twice a week. After the third week of interrupted sleep, I realized that if I could describe the fix to a colleague, I could describe it to a script.
I wrote a Python watchdog script that didn’t just check if the process was running — it performed a synthetic transaction. It sent a small, safe request to the health endpoint. If the response took longer than 5 seconds or returned a 500 error, the script would:
- Capture the last 100 lines of the error log and send them to a Slack channel.
- Gracefully stop the service.
- Clear the temp cache.
- Restart the service.
- Verify the health check passed after the restart.
This script didn’t just save my sleep — it provided observability. By sending the logs to Slack before restarting, we finally had the data to identify the malformed XML bug and fix it at the source. That’s the real payoff of self-healing automation: it doesn’t just suppress symptoms, it surfaces the evidence you need to eliminate the root cause.
A similar pattern works well for Node.js services. Here’s a minimal watchdog written in Node.js with axios that I’ve adapted for monitoring Express APIs running under PM2:
import axios from "axios";
import { exec } from "child_process";
import { promisify } from "util";
const execAsync = promisify(exec);
const SERVICE_URL = process.env.SERVICE_URL ?? "http://localhost:3000/health";
const PM2_APP_NAME = process.env.PM2_APP_NAME ?? "my-api";
const TIMEOUT_MS = 5000;
async function checkHealth(): Promise<boolean> {
try {
const response = await axios.get(SERVICE_URL, { timeout: TIMEOUT_MS });
return response.status === 200;
} catch {
return false;
}
}
async function restartService(): Promise<void> {
console.log(`[${new Date().toISOString()}] Health check failed. Restarting ${PM2_APP_NAME}...`);
await execAsync(`pm2 restart ${PM2_APP_NAME}`);
console.log(`[${new Date().toISOString()}] Restart issued. Verifying...`);
// Give the process a moment to come back up
await new Promise((resolve) => setTimeout(resolve, 3000));
const recovered = await checkHealth();
if (recovered) {
console.log("Service recovered successfully.");
} else {
console.error("Service did NOT recover. Manual intervention required.");
process.exit(1);
}
}
(async () => {
const healthy = await checkHealth();
if (!healthy) {
await restartService();
} else {
console.log("Service is healthy.");
}
})();
Running this on a cron schedule or as a PM2-managed process itself gives you a lightweight self-healing loop without any external orchestration tooling. The key detail is the post-restart verification — a restart command that succeeds doesn’t mean the service is actually up. Always check.
Practical Mastery: Choosing Your Weapon
Engineers often ask: “Should I learn Bash, Python, or Go?” My experience suggests that you need a T-shaped skill set. Proficient in one, familiar with all three. Here’s how I categorize their use cases based on years of trial and error.
Bash: The Glue of the Pipeline
Bash is unbeatable for quick tasks inside a CI/CD pipeline or a Dockerfile. If you are moving files, setting environment variables, or calling a few CLI tools, Bash is your best friend. However, once you hit more than two if-else statements or need to parse complex JSON, it’s time to move on. At that point you’re writing a program in a language that wasn’t designed to write programs.
Python: The Swiss Army Knife
Python is the gold standard for general DevOps automation. With libraries like requests for APIs, boto3 for AWS, and pandas for log analysis, it is incredibly powerful. I use Python for anything that requires logic, data manipulation, or integration between different services — like connecting a GitHub webhook to a MongoDB audit log.
Go: The Performance Powerhouse
If you are building a tool that will be distributed as a binary to dozens of servers, or if you need high concurrency like a custom log shipper, Go is the right choice. It compiles to a single static binary with no runtime dependencies, which makes deployment trivially simple. Docker and Kubernetes are written in Go for exactly this reason.
Building a Culture of Automation
Automation can be unsettling for some team members who worry that scripting themselves out of a job means losing that job. In practice, the opposite is true. By automating the mundane, you free yourself to work on the high-value architectural challenges that actually move the needle for the business. No one has ever been let go for making the team faster.
When you write a useful script, don’t keep it on your laptop. Put it in a shared scripts repository, document it with a README.md, and include it in your CI/CD pipelines. When the team sees that the “boring stuff” is handled by the machine, the culture shifts from firefighting to innovating. That shift is worth more than any individual script.
Conclusion: Your Next Steps
Scripting and automation are not about writing code for the sake of code. They are about reclaiming your time and ensuring the stability of your systems. Whether you are starting with a simple .sh file to automate your Git commits or building a Python framework to manage multi-container deployments, every line of code you write to replace a manual task is an investment in your future sanity.
Pick one task you did manually today — just one. Write a script to automate it, even if the script takes longer to write than the task itself. The lessons you learn handling edge cases and errors will compound.
