Ship It, 2009

What you will do

1. read the systemd unit file for the app service cat /etc/systemd/system/app.service

   A unit file has three sections: [Unit] (metadata/ordering), [Service] (how to run it), [Install] (how it hooks into boot targets). systemd replaced SysVinit in most distros around 2011-2015.

2. check whether the app service is currently running systemctl status app

   A unit file existing on disk means nothing until systemd loads and starts it — "inactive (dead)" is the honest default state of anything you haven't started yet.

3. start the service right now systemctl start app

   start affects only THIS boot. It says nothing about what happens after a reboot — that is a separate, deliberate decision (see the next mission).

4. confirm the service is now active (running) systemctl status app

   Active/enabled are two independent axes: a service can be running-but-not-enabled (won't survive reboot) or enabled-but-not-running (will start on the NEXT boot only). Always check both.

5. make the service start automatically on every future boot systemctl enable app

   enable just drops a symlink into a systemd target's .wants/ directory — no daemon reload of its own logic, just "start me when this target is reached."

6. audit every service unit currently running on the host systemctl list-units --type=service

   Every running service is attack surface and a maintenance burden. Real ops teams audit the running list, since the installed list alone hides what is actually live.

7. view the existing cron schedule crontab -l

   cron's five-field syntax (minute hour day month weekday) has been unchanged since 1975 — one of the oldest still-live interfaces in computing.

8. install a nightly backup cron job crontab -e

   -e opens your crontab in $EDITOR; saving atomically installs the whole file. There is no "append one line" — you always edit the complete schedule.

9. confirm the new job actually took crontab -l

   Never trust that an edit worked — verify. A typo'd cron schedule fails silently at 2am with nobody watching; reading it back is the whole safety net.

10. read the reverse-proxy site config cat /etc/nginx/sites-enabled/app.conf

    nginx (Igor Sysoev, 2004) popularized sites-available/sites-enabled: configs live in one dir, a symlink in the other turns them on — enable/disable a site without deleting its config.

11. find which port nginx is listening on grep listen /etc/nginx/sites-enabled/app.conf

    grep a config instead of eyeballing it — the habit scales from a 10-line site.conf to a 10,000-line one identically.

12. find the backend nginx is forwarding requests to grep proxy_pass /etc/nginx/sites-enabled/app.conf

    proxy_pass is the whole reverse-proxy pattern in one directive: nginx is the public-facing door, the app process behind it never touches the internet directly.

13. validate the config syntax before ever reloading it nginx -t

    nginx -t before every reload is the single habit that prevents the most common self-inflicted outage: a typo'd config taking down a working server.

14. generate a deploy keypair for CI/CD ssh-keygen -t ed25519

    Ed25519 (2013) keys are shorter and faster to verify than RSA-4096 at equivalent security — the modern default whenever the remote end supports it.

15. view the public half of the key you just generated cat ~/.ssh/id_ed25519.pub

    The .pub file is meant to be shared freely — it is what you paste into a server's authorized_keys or a Git host's settings. The private half never leaves this machine.

16. check which public keys this host already trusts for login cat ~/.ssh/authorized_keys

    authorized_keys is the whole of SSH public-key auth on the server side: any private key matching a line in this file can log in as this account — no password needed.

17. see the releases directory and the "current" symlink ls -la /opt/releases

    Symlink-swap deploys (popularized by Capistrano, ~2008) are the classic pattern: unpack a new release into its own directory, then repoint one symlink. The switch is atomic.

18. check disk space before you ship anything df -h

    A full disk breaks deploys, crashes databases mid-write, and fills logs with cascading errors that hide the real problem. Cheapest check that prevents a 2am page.

19. see how much space old releases are eating du -sh /opt/releases

    Keep N releases, prune the rest — the same symlink pattern that makes rollback instant also quietly fills a disk if nobody ever cleans up.

20. list every release this app's deploy tool knows about deploy list

    From here on, "deploy" is this host's own small release tool — same symlink-swap idea as the ls above, wrapped so a human (or CI) never has to touch the symlink by hand.

21. see the current live release and its health, before touching anything deploy status

    Status-first is the ops equivalent of "look both ways" — know what state you're starting from before you change it.

22. ship a normal, healthy release deploy release v1.4

    This is the boring, uneventful path most deploys should be. Remember what "normal" looks like — it makes the abnormal easy to spot later.

23. confirm v1.4 is actually healthy after shipping it deploy health

    Always verify after you ship. "The deploy script didn't error" and "the service actually works" are two different claims — only one of them matters to users.

24. preview infrastructure changes before applying them terraform plan

    Terraform (HashiCorp, 2014) is declarative IaC: you describe the desired end state, and plan shows the diff between that and reality — a dry run for infrastructure.

25. apply the planned infrastructure change terraform apply

    Idempotency is the whole promise of IaC: re-running apply against unchanged config converges to "no changes" rather than creating duplicate resources. Try it again later — it will say exactly that.

26. run a configuration-management playbook ansible-playbook site.yml

    Ansible (Michael DeHaan, 2012) tasks are meant to be idempotent too: "changed" means the task actually did something this run, "ok" means it checked and the system already matched — same playbook, safe to run daily.

27. create the service account that owns the app and its home directory useradd -m deploy

    A dedicated service account instead of running everything as root applies least-privilege to daemons the same as humans. If the app is compromised, it isn't root.

28. grant deploy just enough privilege to manage the service usermod -aG sudo deploy

    -aG APPENDS to a user's groups. Drop the -a by mistake and plain -G REPLACES every group membership the user had — a classic footgun that silently kicks someone out of every other group.

29. ship the big one — release v2.0 deploy release v2.0

    Shipping is the easy part. Everything from here is the actual job: noticing it broke, finding out why, and fixing it without panic.

30. see the failure reflected in deploy status deploy status

    status is the fastest lie-detector you have: it will tell you plainly whether "deployed" and "working" are the same thing right now. Here, they are not.

31. read the application logs to find out WHY it is failing deploy logs

    Logs are ground truth; guessing is not. A missing environment variable crashing the process on boot is one of the most common real-world "it deployed but it's down" causes.

32. rule out nginx itself — confirm the config is syntactically fine nginx -t

    Layered diagnosis: nginx is what users SEE fail (502), but that doesn't mean nginx is at fault. Its config is fine — the process it proxies to isn't running. Don't fix the wrong layer.

33. ROLL BACK to the last known-good release deploy rollback

    Rollback is a legitimate first response, not an admission of failure: restore service now with a release you already know works, root-cause the broken one after, with nobody waiting on you.

34. Boss missionFINAL — verify health is actually restored after the rollback deploy health

    The full incident loop, once: ship, detect, diagnose, roll back, verify. Every real production incident is some version of exactly these five steps, in exactly this order.

Certificate

Nearby eras

All 25 eras in the Terminal Academy