Running k3s in Proxmox LXC Containers: The Pitfalls Nobody Documents

I’m going to break the usual format of a “how to run k3s in Proxmox LXC” post: I didn’t get it running. I gave it a bounded session on my real Proxmox host, hit a wall that turns out to be a known-but-poorly-documented one, and decided that writing down exactly what happened is more useful than padding out a guide for a setup I haven’t actually gotten to work. This is that writeup — what’s proven, what I ruled out, and what I’d try next.

The Setup

An unprivileged LXC, freshly created, with the two features every guide agrees you need for anything Docker- or Kubernetes-adjacent in a container:

pct create 996 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst 
  --hostname k3s-test --cores 2 --memory 1024 --swap 512 
  --net0 name=eth0,bridge=vmbr0,ip=dhcp 
  --rootfs local-lvm:8 
  --unprivileged 1 --features nesting=1,keyctl=1

Debian 13, 2 cores, 1 GB RAM — deliberately small, since the point was to see what breaks, not to run a real workload. Before touching k3s at all, I checked the container itself was healthy: systemctl is-system-running came back running, zero failed units. Nesting was doing its job; this wasn’t a repeat of the “mount points fail without nesting” problem I ran into provisioning containers with Terraform the same afternoon.

The Install Looked Fine

curl -sfL https://get.k3s.io | sh -

Clean install, no errors: k3s v1.36.3+k3s1, binary in place, systemd unit created and enabled, service started. Nothing here suggested trouble.

Where It Actually Breaks

kubectl get nodes alternated between “server is currently unable to handle the request” and “connection refused” — the API server never settled into a state where it would answer. systemctl status k3s told the real story: the service was crash-looping, cleanly, every ~11-12 seconds:

k3s.service: Deactivated successfully.
k3s.service: Consumed 4.5s CPU time, 247M memory peak.
k3s.service: Scheduled restart job, restart counter is at 12.

“Deactivated successfully” is the detail worth sitting with. That’s systemd’s phrasing for a clean exit, not a crash — no segfault, no OOM-kill, no panic trace anywhere in the journal. Something inside k3s itself was choosing to give up after a few seconds of work, consistently, run after run.

What the Logs Actually Say

Two things stood out on closer reading. First, on every restart:

modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/7.0.14-11-pve
modprobe: FATAL: Module overlay not found in directory /lib/modules/7.0.14-11-pve
level=warning msg="Failed to load kernel module br_netfilter with modprobe"

My first read was “there’s the cause” — except checking my actual Proxmox host’s kernel (lsmod, and /proc/sys/net/bridge/bridge-nf-call-iptables, which only exists if the functionality is active) suggests the capability is already present at the host level, likely built directly into this kernel rather than as a loadable .ko file — which is exactly why modprobe from inside an unprivileged container can’t find a module to load: there’s no module file, because it was never built as one. I’m flagging this as a plausible red herring rather than confirmed, because I didn’t chase it further — but it’s a good example of a log line that looks damning and might not be.

The more convincing lead was this one, appearing right alongside kube-proxy startup on every single restart:

level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 131072"
level=error msg="Failed to set sysctl: open /proc/sys/net/netfilter/nf_conntrack_max: permission denied"

k3s’s bundled kube-proxy wants to write a kernel sysctl on startup, and an unprivileged Proxmox LXC — nesting and keyctl notwithstanding — doesn’t grant write access to that specific /proc/sys path by default. nesting=1 unlocks the mount and cgroup namespacing that Docker and containerd need; it does not unlock arbitrary sysctl writes, which is a separate restriction enforced by the container’s AppArmor profile. That’s the actual gap between “runs Docker fine” and “runs a CNI-based orchestrator fine” inside an LXC, and it’s not obvious until you’re staring at a crash loop with no panic trace.

What I Didn’t Do, and Why

The community-documented fix for this class of problem is a custom AppArmor profile for the container that allows the specific sysctl writes k3s needs (sometimes paired with lxc.apparmor.profile: unconfined, which is a much bigger hammer than the problem calls for). I stopped short of building and applying that for two reasons that are specific to this being a real production Proxmox host and not a disposable lab box: I’d want to scope the AppArmor exception narrowly rather than reach for unconfined, and that’s real engineering, not a five-minute follow-up — worth doing properly in its own session, not squeezed into the same afternoon as everything else I was testing.

Cleanup

pct stop 996 && pct destroy 996 --purge 1

Gone. No lingering LVM volume, no orphaned config.

Update: I Went Back and Tried Privileged Instead

The obvious cop-out from the sysctl wall above is “just make it privileged” — trade the AppArmor engineering for flipping one flag. I said I’d follow up once I’d actually tested something properly, so I gave that exact idea a session of its own on a fresh container rather than leaving it as a plausible-sounding guess. It’s worse than I expected, in an interesting way: privileged mode doesn’t fix this, it swaps one blocker for three different ones, and going through them changes my actual recommendation.

pct create 994 local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst
  --hostname k3s-privileged-test --cores 2 --memory 1024 --swap 512
  --net0 name=eth0,bridge=vmbr0,ip=dhcp
  --rootfs local-lvm:8
  --unprivileged 0 --features nesting=1,keyctl=1

(Debian’s template had moved on to 13.6-1 in the time between this post and the last — worth checking pveam available yourself rather than trusting the version string in any post, this one included.)

Blocker one, same as before, privileged or not: the exact same nf_conntrack_max: permission denied line from the unprivileged run showed up again, unchanged. That one surprised me — I’d assumed privileged meant “full root, no restrictions,” but Proxmox’s default LXC AppArmor profile applies regardless of the privilege bit; privileged changes what the container’s root user can do to itself, not what the profile lets it touch on the host’s /proc. This alone rules out “just tick the privileged box” as the fix I’d half-expected it to be.

Blocker two, new: past that sysctl warning, the service died immediately with an actual error instead of a silent restart:

Error: failed to run Kubelet: failed to create kubelet: open /dev/kmsg: no such file or directory

/dev/kmsg is the kernel’s ring-buffer log device, and LXC doesn’t create it inside a container by default — there’s no kernel to have its own message buffer, so the node doesn’t get one. This is a known WSL2/container gotcha with a one-line folk fix, and it actually worked:

ln -sf /dev/console /dev/kmsg

Symlinking it to the container’s console device gives kubelet something to open, even though it isn’t really the kernel log. Restarted the service, and for the first time all day, it made it past the point everything else had been dying at.

Blocker three, new, and the one that actually ends this:

Failed to start ContainerManager:
  open /proc/sys/kernel/panic: read-only file system,
  open /proc/sys/kernel/panic_on_oops: read-only file system,
  open /proc/sys/vm/overcommit_memory: read-only file system

This is a different kind of wall than the first two. The sysctl and kmsg issues were about permissions and missing device files — fixable with a profile tweak or a symlink. This one is a read-only mount. LXC exposes /proc inside a container in what upstream calls “mixed” mode: most of it is writable, but a deliberately-chosen subset under /proc/sys/kernel and /proc/sys/vm is mounted read-only no matter what the privilege bit says, because those specific knobs affect the whole physical machine, not just one container’s slice of it. There’s no chmod or AppArmor exception for a read-only bind mount; changing it means changing how Proxmox mounts /proc for every container on the host, which is not a per-container fix at all.

Kubelet’s ContainerManager tries to write those three values directly on startup unless you tell it not to via --protect-kernel-defaults=true — which switches its behavior from “set these” to “verify these match, and refuse to start if they don’t.” That’s a real, documented flag, and it might actually get further here if the host’s current values already happen to satisfy what kubelet expects. I didn’t test that variant; by this point I’d gone three layers deep chasing what started as “just try privileged,” which is exactly the kind of scope creep I try to catch before it eats an evening, so I stopped and tore the container down instead.

pct stop 994 && pct destroy 994 --purge 1

What this actually changes: I’d assumed privileged mode was the “give up and use the bigger hammer” fallback if the AppArmor work in the section above didn’t pan out. It isn’t — it trades one AppArmor-enforced restriction for a read-only /proc mount that isn’t a per-container setting at all, on top of a missing device node that’s an easy fix once you know to look for it. The honest ranking, updated: scope the AppArmor profile properly for the unprivileged case (still unsolved, still worth its own session), try --protect-kernel-defaults=true on top of the kmsg symlink as a faster experiment than I gave it credit for, or skip the fight entirely and run the k3s node as a small VM. I’d reach for the VM before I’d reach for privileged mode now, which is the opposite of what I expected going in.