A systemd unit is easy to write and easy to write badly, and the difference only shows up at 3am when the process dies and the unit's opinions about restarting, logging and permissions are all that's left of you. The systemd app generates a unit with the accompanying directives already in place — the ones nobody remembers the spelling of — and this article is the reasoning behind each block, built up in stages.
Stage 1: the minimal unit that works
[Unit]
Description=My service
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=app
Group=app
WorkingDirectory=/srv/app
Environment="NODE_ENV=production"
ExecStart=/usr/bin/node /srv/app/server.js
Two details here already carry weight. ExecStart must be an absolute path — systemd does not consult your $PATH, and the app refuses to generate a unit without one because ExecStart=node server.js fails with an error that mentions neither of those facts. And the network target matters: network.target only means the network stack exists; network-online.target (with the matching Wants=) is the one that waits for an address. This is the entire explanation for the classic symptom "fails on boot, starts fine by hand" in services that bind a specific IP.
Stage 2: restarts, without the lockout
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=5
on-failure restarts on non-zero exit, signals, and timeouts — but not on a clean exit 0. That's usually right: if your process decides to exit cleanly, it presumably meant it. Use always only when a clean exit is itself a failure (some runtimes exit 0 on OOM-adjacent conditions). The app defaults to on-failure with RestartSec=5.
The trap is the start limit. systemd ships a default of 5 starts in 10 seconds; blow through it and the unit enters the failed state and stops being restarted at all — the exact opposite of what you configured Restart= for. With RestartSec=5 you won't hit the default window, but a crash-on-boot bug plus an eager RestartSec=1 will, and then your "self-healing" service is down until a human runs systemctl reset-failed my-service. Set the window deliberately: the pair above allows 5 attempts per 5 minutes, which rides out a transient dependency outage without ever locking out. (StartLimitIntervalSec belongs in [Unit] on older systemd; on anything current it's accepted in [Service] too — systemd-analyze verify will tell you.)
Stage 3: resource guards
MemoryMax=512M
TasksMax=256
MemoryMax is a cgroup hard limit: the service OOM-kills itself instead of taking the box down with it, and paired with Restart=on-failure a slow leak becomes a periodic restart instead of an outage. TasksMax caps threads+processes, which is your fork-bomb and runaway-threadpool guard. Size both from observed usage with headroom, not from hope — an undersized MemoryMax turns into a restart loop, which stage 2 at least keeps bounded.
Stage 4: the sandbox
The app's hardening toggle emits this block, and every line is off by default in systemd:
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
ReadWritePaths=/srv/app
ProtectSystem=strict mounts the entire filesystem read-only for this service. That's the point — a compromised process can't touch /etc or /usr — and it's also the line that breaks things: any path the service writes (logs, uploads, a Unix socket, a SQLite file) must be listed in ReadWritePaths or the first write fails with EROFS. The app names your WorkingDirectory there automatically; add anything else the service writes. PrivateTmp gives it its own /tmp, which also means you can't hand it files through the shared one. NoNewPrivileges kills setuid escalation. Run systemd-analyze security my-service after installing and enjoy the score changing from "UNSAFE" to something you'd admit to.
Stage 5: [Install], and the enable that does nothing
[Install]
WantedBy=multi-user.target
systemctl enable works by creating a symlink in the target's .wants/ directory — and it reads [Install] to find out which target. A unit with no [Install] section gives enable nothing to do; older systemd versions would even exit 0 with just a warning, so the service runs perfectly all day and then simply isn't there after the reboot. WantedBy=multi-user.target is correct for a system daemon; default.target for user units.
Stage 6: stopping as cleanly as you start
Two directives round the unit out. Type= tells systemd when to consider the service started: simple assumes started the instant the process forks, which is fine until something is ordered After= your service and races its socket. Type=notify (for daemons that call sd_notify) or a readiness healthcheck at a higher layer fixes that; oneshot is for scripts, and is the right type when you pair the unit with a timer. On shutdown, systemd sends SIGTERM, waits TimeoutStopSec (default 90s), then SIGKILL — so your process must treat SIGTERM as "finish in-flight work and exit 0", or every deploy is a small outage. If your runtime wants a different signal (nginx wants SIGQUIT for graceful), set KillSignal= explicitly rather than teaching your deploy script to bypass systemd.
The full unit
[Unit]
Description=My service
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=app
Group=app
WorkingDirectory=/srv/app
Environment="NODE_ENV=production"
ExecStart=/usr/bin/node /srv/app/server.js
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=5
MemoryMax=512M
TasksMax=256
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
ReadWritePaths=/srv/app
[Install]
WantedBy=multi-user.target
Install to /etc/systemd/system/my-service.service, then systemctl daemon-reload — systemd does not see edits without it — then systemctl enable --now my-service.
3am triage
systemctl status my-service # state, last exit, recent log tail
journalctl -u my-service -e # jump to the end of the unit's log
journalctl -u my-service --since -1h # the last hour
journalctl -u my-service -p err -b # errors only, this boot
journalctl -u my-service -f # follow live
systemctl reset-failed my-service # clear a start-limit lockout
The one that saves you at 3am is -p err -b: the stack trace from the first crash of the loop, not the four hundred restart lines after it.