sharenet_iac

Infrastructure as Code for Sharenet, using OpenTofu and Ansible.

Stacks

Stack	Purpose	Required
dns	sharenet.sh zone — prerequisite for all other stacks	Always
backup	B2 backup buckets — prerequisite for staging and prod	Always
forgejo	Forgejo git instance at git.sharenet.sh	Always
ci	Forgejo Actions runner at ci.sharenet.sh	Optional (requires forgejo)
staging	Sharenet app at staging.sharenet.sh	Optional (requires forgejo, ci, backup)
prod	Production at sharenet.sh / sh.sharenet.sh / blog.sharenet.sh	Optional (requires backup)
demo	Public demo at try.sharenet.sh	Optional (requires forgejo)

---

Shared prerequisites

All of the following must be in place before running any scripts.

Prerequisite	Notes
tofu installed locally	OpenTofu
ansible installed locally	Ansible
jq installed locally	Required by secrets.sh for Bitwarden JSON parsing
bws CLI installed (optional)	Bitwarden Secrets Manager CLI; if absent, set secrets as env vars instead
tailscale installed and connected locally	Required for post-hardening Ansible runs
DigitalOcean Spaces bucket exists	Used for OpenTofu state backend; create manually via the DO console before first run
DigitalOcean Spaces access key + secret key	Stored as TOFU_STATE_ACCESS_KEY_ID / TOFU_STATE_ACCESS_SECRET_KEY
DigitalOcean API token	Stored as DIGITALOCEAN_ACCESS_TOKEN
sharenet.sh delegated to DigitalOcean nameservers	Configure at your registrar: ns1.digitalocean.com, ns2.digitalocean.com, ns3.digitalocean.com
SSH key pair generated locally	Public half becomes BOOTSTRAP_SSH_PUBLIC_KEY; set BOOTSTRAP_SSH_KEY_PATH to the private key path (defaults to ~/.ssh/id_ed25519)
Tailscale account with tag:server in ACL	Declare the tag in the Tailscale admin ACL before tailscale up --advertise-tags=tag:server will work
Tailscale ACL permits SSH from your machine to tag:server	Without this ACL entry, --ssh on the node will not grant your machine access

---

Shared secrets pattern

Each stack declares its required secrets in <stack>/tofu/secrets.yml. The secrets.sh script reads that file and fetches only the listed secrets from Bitwarden (or checks they are set as environment variables).

A user deploying only the dns stack needs no Forgejo, Tailscale, or CI secrets at all — only DIGITALOCEAN_ACCESS_TOKEN and the state backend credentials.

To use environment variables instead of Bitwarden, export each variable before running any script:

export DIGITALOCEAN_ACCESS_TOKEN="..."
export TOFU_STATE_ACCESS_KEY_ID="..."
# etc.

---

Bootstrap SSH access

Applies to the forgejo, ci, staging, and demo stacks (any stack with a Droplet).

During the initial provision.sh + configure.sh run, Ansible must reach the Droplet over SSH before Tailscale is installed. To allow this without permanently exposing port 22 to the internet, the DO firewall opens port 22 to a single IP of your choosing.

Set bootstrap_ssh_cidr in <stack>/tofu/terraform.tfvars to your public IP in CIDR notation:

bootstrap_ssh_cidr = "203.0.113.5/32"

How to find your public IP: curl -s ifconfig.me — but read the caveats below before using it.

Important caveats:

• VPN: If you are connected to a VPN, curl ifconfig.me returns the VPN exit node's IP, not your ISP-assigned IP. Whether to use the VPN IP or your real IP depends on how you will be running the scripts — use whichever IP the SSH connection to the Droplet will originate from.
• Split-tunnel VPN: If your VPN is split-tunnel, traffic to DigitalOcean may or may not go through the VPN. Check your routing table if unsure.
• Dynamic IPs: If your ISP assigns a dynamic IP and it changes between provision.sh and configure.sh, the SSH connection will be refused. Re-run provision.sh with the updated IP and then retry configure.sh.
• After configure.sh: configure.sh automatically removes the SSH rule from the DO firewall once Ansible completes successfully (by re-applying Tofu with bootstrap_ssh_cidr cleared). You do not need to do anything manually.

---

Stack: dns

Purpose

Owns the sharenet.sh zone. Run once at initial setup, essentially never destroyed. No Ansible component.

Secrets

# dns/tofu/secrets.yml
secrets:
  - DIGITALOCEAN_ACCESS_TOKEN
  - TOFU_STATE_ACCESS_KEY_ID
  - TOFU_STATE_ACCESS_SECRET_KEY

First-time setup

cp dns/tofu/backend.hcl.example dns/tofu/backend.hcl
cp dns/tofu/terraform.tfvars.example dns/tofu/terraform.tfvars
# Edit both files with real values

Run sequence

./scripts/plan.sh dns
./scripts/provision.sh dns
# No configure.sh for dns

---

Stack: forgejo

Purpose

Self-hosted Forgejo git instance at git.sharenet.sh on a DigitalOcean Droplet. External access via ports 80/443 only; SSH access is exclusively via Tailscale.

Secrets

# forgejo/tofu/secrets.yml
secrets:
  - DIGITALOCEAN_ACCESS_TOKEN
  - TOFU_STATE_ACCESS_KEY_ID
  - TOFU_STATE_ACCESS_SECRET_KEY
  - TAILSCALE_FORGEJO_AUTH_KEY
  - FORGEJO_ADMIN_PASSWORD
  - BOOTSTRAP_SSH_PUBLIC_KEY

Prerequisites

Prerequisite	Notes
dns stack deployed	provision.sh dns must have completed successfully
Ansible collections installed	ansible-galaxy collection install -r forgejo/ansible/requirements.yml
Tailscale reusable auth key with tag:server	Stored as TAILSCALE_FORGEJO_AUTH_KEY
forgejo/tofu/backend.hcl created from example	cp forgejo/tofu/backend.hcl.example forgejo/tofu/backend.hcl
forgejo/tofu/terraform.tfvars created from example	cp forgejo/tofu/terraform.tfvars.example forgejo/tofu/terraform.tfvars
forgejo/ansible/vars.yml created from example	cp forgejo/ansible/vars.yml.example forgejo/ansible/vars.yml
bootstrap_ssh_cidr set in terraform.tfvars	Your public IP in CIDR notation. See Bootstrap SSH access.

Run sequence

./scripts/plan.sh forgejo
./scripts/provision.sh forgejo
./scripts/configure.sh forgejo

After provisioning

After the first successful configure.sh forgejo run:

1. The tailscale role installs and authenticates Tailscale on the Droplet
2. The tailscale role verifies the Tailscale tunnel is running
3. The tailscale role writes forgejo/ansible/inventory/hosts-tailscale.yml with the Droplet's Tailscale IP
4. The tailscale role removes the bootstrap SSH key from authorized_keys, stops and disables sshd, and removes the UFW port 22 rule
5. configure.sh re-applies Tofu with bootstrap_ssh_cidr="", removing the SSH inbound rule from the DO firewall

Subsequent configure.sh runs automatically detect hosts-tailscale.yml and connect via Tailscale SSH — no manual intervention required.

Stack details

Component	Detail
Host	DigitalOcean Droplet (Ubuntu 24.04)
URL	https://git.sharenet.sh
Reverse proxy	Caddy (automatic TLS via Let's Encrypt)
Git service	Forgejo binary (systemd service)
Database	SQLite3
SSH access	Tailscale only (post-hardening)
Firewall	Ports 80, 443, UDP 41641; port 22 restricted to bootstrap_ssh_cidr during initial setup only

The Forgejo instance has a sharenet organisation with four teams (owners, developers, contributors, members) and four repositories (sharenet, sharenet_iac, website, blog), created automatically on first configure run.

---

Stack: ci

Purpose

Forgejo Actions runner at ci.sharenet.sh. Runs CI jobs in ephemeral rootless Podman containers. No public web presence — no Caddy, no inbound 80/443.

Secrets

# ci/tofu/secrets.yml
secrets:
  - DIGITALOCEAN_ACCESS_TOKEN
  - TOFU_STATE_ACCESS_KEY_ID
  - TOFU_STATE_ACCESS_SECRET_KEY
  - TAILSCALE_CI_AUTH_KEY
  - FORGEJO_RUNNER_REGISTRATION_TOKEN
  - BOOTSTRAP_SSH_PUBLIC_KEY

Prerequisites

Prerequisite	Notes
dns stack deployed	provision.sh dns must have completed successfully
forgejo stack fully deployed and healthy	configure.sh forgejo must have completed successfully
Runner registration token generated	Go to https://git.sharenet.sh/-/admin/runners, click "Create Registration Token", store as FORGEJO_RUNNER_REGISTRATION_TOKEN in Bitwarden or as env var
TAILSCALE_CI_AUTH_KEY added	New reusable Tailscale auth key with tag:server — same process as TAILSCALE_FORGEJO_AUTH_KEY
Ansible collections installed	ansible-galaxy collection install -r ci/ansible/requirements.yml
ci/tofu/backend.hcl created from example	cp ci/tofu/backend.hcl.example ci/tofu/backend.hcl
ci/tofu/terraform.tfvars created from example	cp ci/tofu/terraform.tfvars.example ci/tofu/terraform.tfvars
ci/ansible/vars.yml created from example	cp ci/ansible/vars.yml.example ci/ansible/vars.yml
forgejo_url set in ci/ansible/vars.yml	URL of the Forgejo instance
act_runner_version set in ci/ansible/vars.yml	Pin to a specific release
act_runner_base_image pinned to digest in ci/ansible/vars.yml	Replace tag with tag@sha256:... for reproducibility
bootstrap_ssh_cidr set in ci/tofu/terraform.tfvars	Same pattern as forgejo stack

Run sequence

./scripts/plan.sh ci
./scripts/provision.sh ci
./scripts/configure.sh ci

configure.sh ci internally:

1. Runs Ansible (harden -> tailscale -> runner)
2. Reads ci/ansible/outputs/tailscale_ip (written by the tailscale role)
3. Runs a second tofu apply to create the ci.sharenet.sh DNS A record pointing to the Tailscale IP and seal the firewall

After provisioning

The runner is registered with Forgejo and running as a systemd service under the runner user. CI jobs run in ephemeral rootless Podman containers.

Subsequent configure.sh ci runs are fully idempotent: the Tailscale IP is stable, the DNS record already exists, and the second apply is a no-op.

Stack details

Component	Detail
Host	DigitalOcean Droplet (Ubuntu 24.04), separate VPC from forgejo
DNS	ci.sharenet.sh A record -> Tailscale IP
Runner software	act_runner (Forgejo Actions)
Container engine	Rootless Podman, ephemeral per-job
Runner labels	ubuntu-latest
Concurrency	1 (sequential)
SSH access	Tailscale only (post-hardening)
Firewall	UDP 41641 (Tailscale); port 22 restricted to bootstrap_ssh_cidr during initial setup only. No inbound 80/443.

---

Stack: backup

Purpose

Provisions Backblaze B2 buckets and scoped application keys for database backups. Like dns, this is a shared prerequisite — not environment-specific — and should essentially never be destroyed. Run before any stack that needs backup credentials.

Secrets

# backup/tofu/secrets.yml
secrets:
  - TOFU_STATE_ACCESS_KEY_ID
  - TOFU_STATE_ACCESS_SECRET_KEY
  - B2_MASTER_APPLICATION_KEY_ID   # B2 master key ID (for Tofu provider auth)
  - B2_MASTER_APPLICATION_KEY      # B2 master key secret

The B2 provider expects B2_APPLICATION_KEY_ID and B2_APPLICATION_KEY environment variables. secrets.sh reads B2_MASTER_APPLICATION_KEY_ID / B2_MASTER_APPLICATION_KEY from Bitwarden and aliases them to the provider's expected names — the same pattern as DIGITALOCEAN_ACCESS_TOKEN → TF_VAR_digitalocean_token.

Prerequisites

Prerequisite	Notes
Backblaze B2 account	Master application key generated in the B2 console
B2_MASTER_APPLICATION_KEY_ID and B2_MASTER_APPLICATION_KEY in Bitwarden	Master key credentials for the B2 Tofu provider
backup/tofu/backend.hcl created from example	cp backup/tofu/backend.hcl.example backup/tofu/backend.hcl
backup/tofu/terraform.tfvars created from example	cp backup/tofu/terraform.tfvars.example backup/tofu/terraform.tfvars

First-time setup

cp backup/tofu/backend.hcl.example backup/tofu/backend.hcl
cp backup/tofu/terraform.tfvars.example backup/tofu/terraform.tfvars
# Edit backend.hcl with real values (terraform.tfvars has no values to change)

Run sequence

./scripts/plan.sh backup
./scripts/provision.sh backup
# No configure.sh for backup

After provisioning

Copy the outputs into Bitwarden for use by the relevant stack:

cd backup/tofu
source ../../scripts/secrets.sh backup
tofu output -raw staging_backup_key_id      # → store as B2_STAGING_BACKUP_KEY_ID
tofu output -raw staging_backup_key_secret  # → store as B2_STAGING_BACKUP_KEY_SECRET
tofu output -raw prod_backup_key_id         # → store as B2_PROD_BACKUP_KEY_ID
tofu output -raw prod_backup_key_secret     # → store as B2_PROD_BACKUP_KEY_SECRET
cd ../..

Stack details

Component	Detail
Provider	Backblaze B2
Staging bucket	Private, 7-day lifecycle
Staging key	Write-only, scoped to staging bucket
Prod bucket	Private; 30-day db retention, 30-day asset versioning
Prod key	Read + write + delete, scoped to prod bucket (required by rclone sync)

---

Stack: staging

Purpose

Sharenet application at staging.sharenet.sh. Runs the backend, frontend, and Postgres as rootless Podman containers behind a Caddy reverse proxy. Database backups run nightly to B2.

Secrets

# staging/tofu/secrets.yml
secrets:
  - DIGITALOCEAN_ACCESS_TOKEN
  - TOFU_STATE_ACCESS_KEY_ID
  - TOFU_STATE_ACCESS_SECRET_KEY
  - TAILSCALE_STAGING_AUTH_KEY
  - BOOTSTRAP_SSH_PUBLIC_KEY
  - STAGING_DB_PASSWORD
  - STAGING_DO_SPACES_KEY
  - STAGING_DO_SPACES_SECRET
  - B2_STAGING_BACKUP_KEY_ID        # output from backup stack, stored in Bitwarden
  - B2_STAGING_BACKUP_KEY_SECRET    # output from backup stack, stored in Bitwarden
  - STAGING_JWT_SECRET

Prerequisites

Prerequisite	Notes
dns stack deployed	Zone must exist before A record can be created
backup stack deployed	B2 bucket and key must exist; credentials stored in Bitwarden before configure
forgejo stack deployed	Container images are hosted on the Forgejo registry
ci stack deployed	CI builds and pushes :latest images to the Forgejo registry
Forgejo package visibility set to public	Manual step in Forgejo UI: go to https://git.sharenet.sh/sharenet/sharenet → Packages → set sharenet-backend-api-postgres and sharenet-frontend to Public
CI workflow has pushed :latest images	Run a CI build before first staging deploy so images exist in the registry
TAILSCALE_STAGING_AUTH_KEY added	New reusable Tailscale auth key with tag:server
DO Spaces bucket created manually	Create a Spaces bucket in the DO console (same region as the Droplet)
DO Spaces Limited Access key created	Scoped to the staging Spaces bucket. Store as STAGING_DO_SPACES_KEY / STAGING_DO_SPACES_SECRET in Bitwarden
Ansible collections installed	ansible-galaxy collection install -r staging/ansible/requirements.yml
staging/tofu/backend.hcl created from example	cp staging/tofu/backend.hcl.example staging/tofu/backend.hcl
staging/tofu/terraform.tfvars created from example	cp staging/tofu/terraform.tfvars.example staging/tofu/terraform.tfvars
staging/ansible/vars.yml created from example	cp staging/ansible/vars.yml.example staging/ansible/vars.yml
bootstrap_ssh_cidr set in terraform.tfvars	Same pattern as forgejo and ci stacks
All staging secrets in Bitwarden	See secrets table above

First-time setup

cp staging/tofu/backend.hcl.example staging/tofu/backend.hcl
cp staging/tofu/terraform.tfvars.example staging/tofu/terraform.tfvars
cp staging/ansible/vars.yml.example staging/ansible/vars.yml
# Edit all three files with real values
ansible-galaxy collection install -r staging/ansible/requirements.yml

Run sequence

./scripts/plan.sh staging
./scripts/provision.sh staging
./scripts/configure.sh staging

configure.sh staging internally:

1. Runs Ansible (harden → tailscale → app)
2. Runs a final tofu apply with bootstrap_ssh_cidr="" to seal the firewall

Unlike ci, staging does not need a second apply for DNS — the staging.sharenet.sh A record points to the Droplet's public IP, which is known at provision time.

After provisioning

The application is running at https://staging.sharenet.sh. To deploy new images:

# SSH into staging VPS via Tailscale
ssh staging

# Run podman auto-update as the app user
sudo -u app podman auto-update

# Verify services are healthy
sudo -u app systemctl --user status backend frontend postgres

podman auto-update checks containers with AutoUpdate=registry, pulls newer images, and restarts affected units. If the new image fails to start, Podman rolls back automatically.

Subsequent configure.sh staging runs are fully idempotent.

Stack details

Component	Detail
Host	DigitalOcean Droplet (Ubuntu 24.04), separate VPC from forgejo and ci
URL	https://staging.sharenet.sh
Reverse proxy	Caddy (automatic TLS via Let's Encrypt)
Containers	Rootless Podman quadlets under app user (Postgres, backend, frontend)
Container images	Public on Forgejo registry (git.sharenet.sh), pulled without credentials
Image tag	:latest
Database	Postgres 16 container, local named volume
App assets	DO Spaces bucket (manually created, configured in vars.yml), CDN enabled
Backups	Nightly pg_dump via systemd timer, compressed, uploaded to B2 via rclone
Backup retention	7 daily backups (B2 lifecycle rule)
Deployment	Manual podman auto-update over Tailscale SSH
SSH access	Tailscale only (post-hardening)
Firewall	Ports 80, 443, UDP 41641; port 22 restricted to bootstrap_ssh_cidr during initial setup only

---

Stack: demo

Purpose

Public-facing demo at try.sharenet.sh. Runs the same application stack as staging but tracks a manually promoted :demo image tag instead of :latest. State is synthetic and seed-reproducible — no backups needed. A nightly reset restores the database to a known-good state.

Secrets

# demo/tofu/secrets.yml
secrets:
  - DIGITALOCEAN_ACCESS_TOKEN
  - TOFU_STATE_ACCESS_KEY_ID
  - TOFU_STATE_ACCESS_SECRET_KEY
  - TAILSCALE_DEMO_AUTH_KEY
  - BOOTSTRAP_SSH_PUBLIC_KEY
  - DEMO_DB_PASSWORD
  - DEMO_JWT_SECRET

Prerequisites

Prerequisite	Notes
dns stack deployed	Zone must exist before A record can be created
forgejo stack deployed	Container images are hosted on the Forgejo registry
:demo tag exists in registry	Promote with ./scripts/promote-demo.sh before first deploy
crane installed locally	Required by promote-demo.sh for registry retagging
TAILSCALE_DEMO_AUTH_KEY added	New reusable Tailscale auth key with tag:server
DEMO_DB_PASSWORD added	Alphanumeric Postgres password for the demo app user
DEMO_JWT_SECRET added	JWT signing secret for the demo app instance
Ansible collections installed	ansible-galaxy collection install -r demo/ansible/requirements.yml
demo/tofu/backend.hcl created from example	cp demo/tofu/backend.hcl.example demo/tofu/backend.hcl
demo/tofu/terraform.tfvars created from example	cp demo/tofu/terraform.tfvars.example demo/tofu/terraform.tfvars
demo/ansible/vars.yml created from example	cp demo/ansible/vars.yml.example demo/ansible/vars.yml
bootstrap_ssh_cidr set in terraform.tfvars	Same pattern as forgejo, ci, and staging stacks

First-time setup

cp demo/tofu/backend.hcl.example demo/tofu/backend.hcl
cp demo/tofu/terraform.tfvars.example demo/tofu/terraform.tfvars
cp demo/ansible/vars.yml.example demo/ansible/vars.yml
# Edit all three files with real values
ansible-galaxy collection install -r demo/ansible/requirements.yml

# Promote :latest to :demo (requires staging to have validated the build)
./scripts/promote-demo.sh

Run sequence

./scripts/plan.sh demo
./scripts/provision.sh demo
./scripts/configure.sh demo

configure.sh demo internally:

1. Runs Ansible (harden -> tailscale -> demo)
2. Runs a final tofu apply with bootstrap_ssh_cidr="" to seal the firewall

Like staging, demo does not need a second apply for DNS — the try.sharenet.sh A record points to the Droplet's public IP, which is known at provision time.

After provisioning

The application is running at https://try.sharenet.sh.

Image promotion (staging -> demo):

# Promote :latest to :demo
./scripts/promote-demo.sh

# Or promote a specific version
./scripts/promote-demo.sh 0.1.0

# Apply immediately (instead of waiting for podman auto-update timer)
ssh <demo-tailscale-ip> systemctl --user start podman-auto-update.service

Manual reset (restore database to seed state):

./scripts/reset-demo.sh

Nightly automated reset runs at 03:00 UTC via a systemd timer. The reset stops the backend and frontend, restores the database from seed.sql against the running Postgres, and restarts the app containers.

Subsequent configure.sh demo runs are fully idempotent.

Stack details

Component	Detail
Host	DigitalOcean Droplet (Ubuntu 24.04), separate VPC from all other stacks
URL	https://try.sharenet.sh
Reverse proxy	Caddy (automatic TLS via Let's Encrypt)
Containers	Rootless Podman quadlets under app user (Postgres, backend, frontend)
Container images	Public on Forgejo registry (git.sharenet.sh), pulled without credentials
Image tag	:demo (manually promoted from :latest)
Database	Postgres 16 container, local named volume
App assets	Local disk (no object storage)
Backups	None — seed file in repo is the recovery path
Nightly reset	03:00 UTC, systemd timer restores from seed.sql
Deployment	./scripts/promote-demo.sh + podman auto-update
SSH access	Tailscale only (post-hardening)
Firewall	Ports 80, 443, UDP 41641; port 22 restricted to bootstrap_ssh_cidr during initial setup only

---

Stack: prod

Purpose

Production Sharenet application at sharenet.sh, sh.sharenet.sh, and blog.sharenet.sh. Runs backend, frontend, and Postgres as rootless Podman containers behind a Caddy reverse proxy. Static sites (www.sharenet.sh, blog.sharenet.sh) are served directly by Caddy from rsync-published directories. Database and asset backups run nightly to B2.

Secrets

# prod/tofu/secrets.yml
secrets:
  - DIGITALOCEAN_ACCESS_TOKEN
  - TOFU_STATE_ACCESS_KEY_ID
  - TOFU_STATE_ACCESS_SECRET_KEY
  - TAILSCALE_PROD_AUTH_KEY
  - BOOTSTRAP_SSH_PUBLIC_KEY
  - PROD_DB_PASSWORD
  - PROD_DO_SPACES_KEY
  - PROD_DO_SPACES_SECRET
  - B2_PROD_BACKUP_KEY_ID
  - B2_PROD_BACKUP_KEY_SECRET
  - PROD_JWT_SECRET

Prerequisites

Prerequisite	Notes
dns stack deployed	Zone must exist before A records can be created
backup stack deployed	B2 bucket and key must exist; credentials stored in Bitwarden before configure
Tailscale ACL updated	See table below — must be done before configure.sh prod, which scrubs bootstrap SSH
TAILSCALE_PROD_AUTH_KEY added	New reusable Tailscale auth key with tag:server
DO Spaces bucket created manually	Create a Spaces bucket in the DO console (same region as the Droplet)
DO Spaces Limited Access key created	Scoped to the prod Spaces bucket. Store as PROD_DO_SPACES_KEY / PROD_DO_SPACES_SECRET in Bitwarden
PROD_DB_PASSWORD generated	Alphanumeric only (no special chars). Store in Bitwarden
PROD_JWT_SECRET generated	openssl rand -hex 64. Store in Bitwarden
Ansible collections installed	ansible-galaxy collection install -r prod/ansible/requirements.yml
prod/tofu/backend.hcl created from example	cp prod/tofu/backend.hcl.example prod/tofu/backend.hcl
prod/tofu/terraform.tfvars created from example	cp prod/tofu/terraform.tfvars.example prod/tofu/terraform.tfvars
prod/ansible/vars.yml created from example	cp prod/ansible/vars.yml.example prod/ansible/vars.yml
bootstrap_ssh_cidr set in terraform.tfvars	Same pattern as other stacks
All prod secrets in Bitwarden	See secrets table above

Tailscale ACL entries — add these three entries before running configure.sh prod:

From	To	User	Purpose
Your machine	prod (tag:server)	app	promote-prod.sh deploys app images
Your machine	prod (tag:server)	website-publisher	Rsyncs site content (managed in separate repo)
Publisher's machine	prod (tag:server)	blog-publisher	Rsyncs blog content (managed in separate repo)

First-time setup

cp prod/tofu/backend.hcl.example prod/tofu/backend.hcl
cp prod/tofu/terraform.tfvars.example prod/tofu/terraform.tfvars
cp prod/ansible/vars.yml.example prod/ansible/vars.yml
# Edit all three files with real values
ansible-galaxy collection install -r prod/ansible/requirements.yml

Run sequence

./scripts/plan.sh prod
./scripts/provision.sh prod
./scripts/configure.sh prod

# First app deploy
./scripts/promote-prod.sh <tag>

configure.sh prod internally:

1. Runs Ansible (harden → tailscale → caddy → app → website → blog → backup)
2. Runs a final tofu apply with bootstrap_ssh_cidr="" to seal the firewall

Like staging, prod does not need a second apply for DNS — all four A records (@, www, sh, blog) point to the Droplet's public IP, which is known at provision time.

The backend and frontend quadlets are installed with placeholder image tags and are not started by configure. Run promote-prod.sh to deploy the first real image.

After provisioning

Before running promote-prod.sh, the container images must exist in the Forgejo registry at the tag you want to deploy. This means:

1. A CI build has run and pushed images tagged with that version to git.sharenet.sh
2. The Forgejo packages for sharenet-backend-api-postgres and sharenet-frontend are set to Public (Forgejo UI: package settings → visibility)
3. Your machine is connected to Tailscale and the ACL entry permitting SSH to tag:server as app is in place

Then deploy the app — from the sharenet repo (or anywhere you have the script):

./promote-prod.sh <tag>

# If Tailscale MagicDNS does not resolve 'prod', pass the IP explicitly:
PROD_HOST=<prod-tailscale-ip> ./promote-prod.sh <tag>

Static sites are served as soon as configure.sh completes. Website and blog deployment scripts live in their respective repos.

Subsequent configure.sh prod runs are fully idempotent.

Stack details

Component	Detail
Host	DigitalOcean Droplet (Ubuntu 24.04), separate VPC from all other stacks
URLs	https://sharenet.sh → redirect, https://www.sharenet.sh (website), https://sh.sharenet.sh (app), https://blog.sharenet.sh (blog)
Reverse proxy	Caddy (automatic TLS via Let's Encrypt)
Containers	Rootless Podman quadlets under app user (Postgres, backend, frontend)
Database	Postgres 16 container, local named volume
App assets	DO Spaces bucket (manually created, configured in vars.yml)
Static sites	/srv/website/ and /srv/blog/, published via rsync-restricted SSH users
DB backups	Nightly pg_dump via podman exec at 02:00 UTC, compressed, uploaded to B2 via rclone
Asset backups	Nightly rclone sync at 03:00 UTC from DO Spaces to B2
Backup retention	30-day database snapshots; 30-day asset versioning
Deployment	./scripts/promote-prod.sh <tag>
SSH access	Tailscale only (post-hardening)
Firewall	Ports 80, 443, UDP 41641; port 22 restricted to bootstrap_ssh_cidr during initial setup only

---

Migration: existing forgejo installation

If you have an existing forgejo stack deployed before the dns stack was introduced, follow these steps in order.

Step 1: Create forgejo/tofu/secrets.yml

Required for the refactored secrets.sh. Create the file with the forgejo stack's required secrets (listed in the forgejo secrets section above).

Step 2: Extract DNS zone into dns stack

The sharenet.sh zone resource moves from the forgejo stack into the new dns stack.

First, verify the exact state resource path:

cd forgejo/tofu
tofu state list | grep domain

Then run the migration:

# 1. Stand up the dns stack, importing the existing zone
cd dns/tofu
source ../../scripts/secrets.sh dns
tofu init -backend-config=backend.hcl
tofu import -var-file=terraform.tfvars digitalocean_domain.root sharenet.sh

# 2. Verify dns state is clean
tofu plan -var-file=terraform.tfvars

# 3. Remove the zone resource from forgejo state (without destroying it)
cd ../../forgejo/tofu
tofu state rm module.domain.digitalocean_domain.root   # use path confirmed above

# 4. Verify forgejo plan is clean
tofu plan -var-file=terraform.tfvars

No DNS downtime occurs.

---

Full first-time run sequence (from scratch)

# 0. One-time local setup
ansible-galaxy collection install -r forgejo/ansible/requirements.yml
ansible-galaxy collection install -r ci/ansible/requirements.yml
ansible-galaxy collection install -r staging/ansible/requirements.yml
ansible-galaxy collection install -r demo/ansible/requirements.yml
ansible-galaxy collection install -r prod/ansible/requirements.yml

# 1. Shared infrastructure (order matters)
./scripts/plan.sh dns
./scripts/provision.sh dns

./scripts/plan.sh backup
./scripts/provision.sh backup
# → Copy backup stack outputs to Bitwarden:
cd backup/tofu
source ../../scripts/secrets.sh backup
tofu output -raw staging_backup_key_id      # → B2_STAGING_BACKUP_KEY_ID
tofu output -raw staging_backup_key_secret  # → B2_STAGING_BACKUP_KEY_SECRET
tofu output -raw prod_backup_key_id         # → B2_PROD_BACKUP_KEY_ID
tofu output -raw prod_backup_key_secret     # → B2_PROD_BACKUP_KEY_SECRET
cd ../..

# 2. Forgejo stack
./scripts/plan.sh forgejo
./scripts/provision.sh forgejo
./scripts/configure.sh forgejo
# → Set package visibility to public in Forgejo UI for sharenet-backend and sharenet-frontend

# 3. Generate runner registration token (manual)
#    -> https://git.sharenet.sh/-/admin/runners
#    -> Click "Create Registration Token"
#    -> Store as FORGEJO_RUNNER_REGISTRATION_TOKEN in Bitwarden or export as env var

# 4. CI stack
./scripts/plan.sh ci
./scripts/provision.sh ci
./scripts/configure.sh ci
# → Run CI workflow to build and push :latest images to Forgejo registry

# 5. Staging stack
./scripts/plan.sh staging
./scripts/provision.sh staging
./scripts/configure.sh staging

# 6. Demo stack
./scripts/promote-demo.sh           # Promote :latest to :demo
./scripts/plan.sh demo
./scripts/provision.sh demo
./scripts/configure.sh demo

# 7. Prod stack
# → Before running: add three Tailscale ACL entries (app, website-publisher, blog-publisher)
# → See Stack: prod Prerequisites for full checklist
./scripts/plan.sh prod
./scripts/provision.sh prod
./scripts/configure.sh prod
./scripts/promote-prod.sh <tag>     # First app deploy