A bastion host, Terraform, and a five-hour SSH mystery

Today's goal: stop SSH to the main server being reachable directly, and route it through a bastion host instead — a small jump box that's the only thing allowed to reach the real server's SSH port.

Provisioning it with Terraform

Wrote a small Terraform config locally: a security group scoped to my own IP, an SSH key pair generated by Terraform itself, and a t3.micro instance wired to both. First apply forgot the key and security group — neither can be added to a running instance afterward, so the fix meant a full replace, which Terraform flagged and handled cleanly.

The SSH mystery

Then hours of SSH timing out despite everything on the AWS side checking out — security group, NACL, route table, all correct, confirmed end-to-end by AWS's own Reachability Analyzer. The instance itself was healthy too: system log showed sshd installed, my key correctly in authorized_keys. EC2 Instance Connect and Session Manager both failed for unrelated reasons (missing agent, missing IAM role). The real cause: my home ISP's route to us-east-1. Switched to phone data and it connected instantly. Added a security-group rule for that IP and moved on.

Locking the real server down

Edited the original server's security group so port 22 only accepts traffic from the bastion's private IP. Direct SSH from my machine now times out, exactly as intended — the only way in is through the bastion, one hop or via a single ProxyCommand so the real key never touches the jump box:

ssh -i EC2.pem -o "ProxyCommand=ssh -i bastion-key.pem -W %h:%p ec2-user@<bastion-ip>" ubuntu@<private-ip>

What I actually learned

A correct security group doesn't guarantee a connection — routing, NACLs, and ISP path are separate failure domains, and AWS gives tools to check each one without the SSH access you're debugging. "Times out from one network, works from another" is a real, common VN-to-us-east-1 issue, not a config bug. And a bastion is only as good as the rule that closes the real door behind it.