Run a mock round

Work from the outside in and use metrics before guessing.

1. Confirm scope: is it the instance, the network, the application, or a dependency? Check CloudWatch CPU, network, and EBS metrics and the ALB target response time.
2. Network: use VPC Flow Logs, check for packet loss, and test with ping, traceroute, or curl timing from a bastion.
3. Instance: SSH in and run top, vmstat, iostat, and sar to see CPU steal, memory pressure, and disk I/O.
4. EBS: burst-credit exhaustion on gp2 or throttled IOPS on gp3 shows as high await.
5. Application: enable tracing, and inspect slow queries, thread pools, and GC pauses.

sudo iostat -x 1 5
sudo ss -s
curl -o /dev/null -s -w '%{time_total}\n' https://example.com

Right-size the instance, move to gp3, or fix the query. Check whether the bottleneck is a downstream dependency.

Both provide block storage to EC2, but with different durability and lifecycle.

• Instance store is physically attached NVMe or SSD storage on the host. It offers very high IOPS and low latency, but data is ephemeral: it is lost on stop, terminate, or host failure. It cannot be detached or snapshotted.
• EBS is network-attached, replicated within an Availability Zone, and persists independently of the instance. Volumes can be detached, reattached, and snapshotted to S3. It supports encryption, resizing, and volume types tuned for IOPS (io2), throughput (gp3), or cold data (sc1).

Choose instance store for caches, scratch space, and temporary buffers. Choose EBS for boot volumes, databases, and anything that must survive a reboot. Many workloads combine them: instance store for a cache layer and EBS for durable data.

Design around an RTO and RPO first; they determine active-active versus active-passive.

• DNS and traffic: Route 53 health checks with failover or latency routing, plus Global Accelerator for TCP.
• Compute: deploy identical stacks in two or more regions using infrastructure as code, keeping services stateless where possible.
• Data is the hard part. DynamoDB global tables give multi-region writes, Aurora Global Database offers cross-region replicas with fast promotion, and S3 cross-region replication handles objects. A relational database with a single writer needs careful promotion.
• Messaging: replicate queues or use idempotent consumers.

Test with game days and chaos experiments. Watch for split-brain, replication lag, and cost. Active-active improves availability but multiplies spend and complexity; active-passive is cheaper but has a failover delay.

S3 storage classes trade retrieval cost and latency against storage cost.

• S3 Standard is for frequent access with low latency.
• S3 Intelligent-Tiering automatically moves objects between tiers based on access.
• S3 Standard-IA and One Zone-IA are for infrequent access: cheaper storage with a retrieval fee.
• S3 Glacier Instant Retrieval, Flexible Retrieval, and Deep Archive are archival, with retrieval times from milliseconds to hours.

Lifecycle policies transition or expire objects automatically.

{
  "Rules": [{
    "ID": "archive-logs",
    "Status": "Enabled",
    "Filter": {"Prefix": "logs/"},
    "Transitions": [{"Days": 30, "StorageClass": "STANDARD_IA"},
                    {"Days": 90, "StorageClass": "GLACIER"}],
    "Expiration": {"Days": 365}
  }]
}

Match the class to the access pattern. Minimum storage durations of 30 to 180 days can make early transitions costly.

A role is an identity with permission policies but no permanent credentials. A trusted principal calls AWS STS AssumeRole and receives temporary credentials.

Flow:

1. The role has a trust policy naming who may assume it, such as an account, user, service, or federated identity.
2. The caller invokes sts:AssumeRole and STS returns an access key, secret key, and session token that expire.
3. Requests signed with those temporary credentials are evaluated against the role's permission policy.

aws sts assume-role --role-arn arn:aws:iam::111122223333:role/Deploy \
  --role-session-name deploy-1

Common uses include EC2 instance profiles, ECS and Lambda execution roles, cross-account access, and web identity federation. Prefer roles over long-lived access keys because credentials rotate automatically and can be scoped further with a session policy.

Both are JSON documents that grant permissions, but they attach to different things.

• IAM policies are attached to identities (users, groups, roles) and define what those identities can do.
• S3 bucket policies are resource-based and attached to the bucket; they define who from any account can access it.

{
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::111122223333:root"},
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::my-bucket/*"
}

When both apply, a request must be allowed by the identity policy and not explicitly denied by the bucket policy. Use bucket policies for cross-account access and public-read hosting, and IAM policies to control what your own principals can do. An explicit Deny in either document always wins.

SQS is a queue; SNS is a publish and subscribe topic. They solve different problems and are often combined.

• SQS is point-to-point. A message is stored until a consumer processes and deletes it. It supports visibility timeouts, dead-letter queues, and FIFO ordering. One message is handled by one consumer.
• SNS is fan-out. A publisher sends a message to a topic and it is pushed to all subscribers, including SQS queues, Lambda, HTTP endpoints, and email. There is no storage or retry beyond the delivery policy.

aws sns publish --topic-arn arn:aws:sns:us-east-1:111122223333:orders \
  --message '{"orderId":"42"}'

A common pattern is an SNS topic that fans out to several SQS queues so independent services each process a copy. Use SQS alone for work queues and SNS alone for notifications.

Both control traffic in a VPC, but at different layers.

• Security groups are stateful and attached to ENIs or instances. Return traffic is automatically allowed. Rules are allow-only and evaluated as a whole.
• Network ACLs are stateless and attached to subnets. You must allow both inbound and outbound traffic, including ephemeral ports. They support allow and deny rules and are evaluated in numbered order, lowest first.

aws ec2 describe-security-groups --group-ids sg-0abc123
aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=subnet-123

By default, security groups deny all inbound and allow all outbound; the default NACL allows everything. A typical design uses security groups for fine-grained instance rules and NACLs as a coarse subnet-level guardrail.

Both are Elastic Load Balancers, but they operate at different OSI layers.

• Application Load Balancer (layer 7) understands HTTP and HTTPS, supports host and path-based routing, sticky sessions, and WebSockets, and integrates with WAF. It terminates TLS and adds X-Forwarded-For headers.
• Network Load Balancer (layer 4) handles TCP, UDP, and TLS, offers extremely high throughput and ultra-low latency, and preserves the client source IP. It can have a static IP per Availability Zone.

Choose ALB for web applications and microservices that route by URL. Choose NLB for non-HTTP protocols, gaming, IoT, or when you need static IPs or millions of requests per second.

aws elbv2 create-listener --load-balancer-arn arn:aws:elasticloadbalancing:... \
  --protocol HTTPS --port 443 --certificates CertificateArn=arn:aws:acm:...

Both support health checks and cross-zone load balancing.