Secure Server Deployments in Hostile Territory
Kyle Rankin
Director of Engineering Operations
NCC Group Domain Services
Agenda
- Introduction
- Traditional Security Assumptions
- Security Challenges in AWS
- AWS Security Model
- Puppet
- Handling Secrets
- Manage Dynamic IPs
- Overall Best Practices
- Questions?
Introduction
- I've built private clouds and deployed on public
- Security has always been part of the design
- Security tradeoff of restriction vs. risk
- First engineer hire at a security startup
- My task: design and build secure, reliable, scalable infrastructure on EC2
- Threat level: People's Liberation Army
- Oh and a serious hack could be a company-ending event
- This is not a complete list of security measures.
Traditional Security Assumptions
- The external network is the main threat
- Tight control over IP space
- Control of hypervisor
- Control of server/network hardware
- Servers aren't shared with other companies
- Servers aren't shared with hackers
- Out-of-band remote console.
Security Challenges in AWS
- Hive of scum and villany
- Can't take anything for granted
- Internal network is an equal threat
- Can't assume neighbors are friendly
- Can't assume your server will be around tomorrow
- Actually a good proving ground for security practices.
Security Challenges in AWS cont.
- Server shares hardware with others
- IP address is dynamic, doesn't persist shutdown
- Data on local storage left behind after shutdown
- No remote console: ssh admin only
- Amazon sysadmin control hypervisor
- Amazon network engineers control network
- Security Group changes disrupt network
- Access to API account = game over.
AWS Security Model
IAM
- IAM lets you create Access Control Lists for API/UI access
- Can restrict accounts to specific AWS services (EC2, S3, ELB, etc.)
- Can restrict levels of access within services (read-only accounts, etc.)
- Like most ACLs, allows pretty sophisticated rules, has some canned templates
- Like most ACLs, complicated. Potential for human error
- Each user creates their own cert pair, uploads public cert to AWS
- Amazon never has private key
- Optional Two-Factor Auth (2FA).
AWS Security Model
Our IAM Practices
- Minimal use of IAM
- Separate AWS (not IAM) accounts for dev/prod
- Single admin user who can manage users
- Each sysadmin has superuser account with AWS "root"
- Require 2FA for user accounts
- Puppetmaster has account with EC2 "root".
AWS Security Model
Security Groups
- Security Groups act as a sort of VLAN
- Allow you to set firewall rules for a group of hosts
- Unlike traditional VLANs, lets you control intra-security group traffic
- Hosts can be in more than one security group
- Security groups can be assigned only at host creation
- Default security group closed to world, completely open internally
- Only controls ingress traffic. Egress requires EC2 Virtual Private Cloud.
AWS Security Model
Our Security Group Practices
- Each server role has its own security group
- Also member of "default" group with minimal permissions
- For instance: VPN granted access to SSH in default group
- Each environment (dev/dr/prod) has own set of groups, own VPN
- Restrict access between security groups
- Most groups allow no access from outside world.
Puppet
- Servers spawned from Puppetmaster, official Debian 7 AMI + post-install for Puppet
- AMI admin account disabled by Puppet
- Take advantage of Puppetmaster/client certs
- No blanket auto-signing, clients provided with PSK embedded in CSR
- Each client cert customized with Subject Alt Name = $role.$domain
- All clients trust Puppetmaster CA, can re-use Puppet certs for internal TLS
- Puppet config checked into git (audit trail).
Handling Secrets
- No secrets in EC2 userdata post-install script!
- Hard to avoid some plaintext secrets on hosts
- Where possible, generate secrets on host (GPG/SSH keys, shared secrets)
- Export generated shared secrets back to Puppet to import on hosts
- Store secrets in /dev/shm ramdisk (custom Puppet module)
- All other secrets stored with Puppet config using hiera-gpg
- GPG decryption key per-environment, Dev can't decrypt Prod secrets
- Admin add Puppetmaster GPG keys to keychain, use wrapper script to encrypt files.
Manage Dynamic IP
- Dynamic IPs are kind of a pain
- Use internal BIND w/ dynamic DNS (DDNS) configured
- In controlled network, may authorize entire subnet, not here
- BIND master generates shared secret to allow A/TXT record DDNS
- Puppet manages exporting shared secrets to hosts
- Each host confirms its DDNS records are in place each time Puppet runs
- Manages record removal during planned shutdown.
Overall Best Practices
- Internal, limited package mirrors w/ signed packages
- Internal repos for custom-built packages, also signed
- Require manual package/config syncing in DR/Prod
- TLS between all internal services and DB connections
- TLS mutual auth everywhere possible
- Restricted access to prod (sysadmin only, separate VPN)
- Host-based IDS (ossec), remote logging (Logstash)
- Key-based SSH auth only, require admin to use password-protected SSH keys
- Encrypt sensitive data on disk.
Questions?
Additional Resources