Working with AWS CloudTrail

Pre-incident hardening — SSH restricted to "My IP"

• Located the Café Web Server instance and opened its security group.
• The SG started with a single inbound rule: HTTP/80 from anywhere — expected for a public website.
• Added an inbound rule for SSH/22 with source My IP, so the CIDR was a /32, not 0.0.0.0/0. This becomes important later: anything wider than /32 on port 22 is a red flag.
• Browsed http://<WebServerIP>/cafe/ to confirm the site rendered normally. Baseline established.

Enable CloudTrail and observe the defacement

• From the CloudTrail console, created a trail with the exact name monitor (the lab depends on this name).
• Storage: new S3 bucket monitoring#### (four random digits, in this run monitoring0423).
• Encryption: a new KMS alias of the form <initials>-KMS.
• Log events: kept the default (management events), then created the trail.
• Returned to the Café browser tab, did a hard refresh (Shift + reload). After about a minute the bakery cake photo was replaced by a monkey illustration. The site had been defaced.

• Back in the EC2 console → Security tab → the SG now had a second inbound rule: SSH/22 from 0.0.0.0/0. Not added by the operator.
• The attack chain was now visible at two layers: a static asset under /var/www was swapped, and the SG was modified through the AWS control plane.

First-pass analysis with grep and the AWS CLI

• SSH'd into the Café Web Server as ec2-user (using the /32 rule added earlier — note the irony of investigating from inside the compromised instance).
• Created a working directory and downloaded the entire trail bucket recursively:

mkdir ctraillogs && cd ctraillogs
aws s3 ls
aws s3 cp s3://monitoring0423/ . --recursive
# Logs land in AWSLogs/<account-num>/CloudTrail/<Region>/<yyyy>/<mm>/<dd>/
cd AWSLogs/.../...
gunzip *.gz

• Inspected one entry to learn the schema (each event is a single JSON document):

cat <filename>.json | python -m json.tool

Standard fields surfaced: awsRegion, eventName, eventSource, eventTime, requestParameters, sourceIPAddress, userIdentity.

• Filtered all events by sourceIPAddress and eventName across every file with a bash for loop:

for i in $(ls); do echo $i && cat $i | python -m json.tool | grep sourceIPAddress; done
for i in $(ls); do echo $i && cat $i | python -m json.tool | grep eventName; done

The output was noisy — a flood of Describe*, List* and the occasional Update* action. Useful to confirm the trail was working, but not enough to pinpoint the attacker. Time to switch tools.

• Moved to the AWS CLI's CloudTrail lookup, filtered by resource type and then by the specific SG ID:

# Discover region from the EC2 instance metadata service
region=$(curl http://169.254.169.254/latest/dynamic/instance-identity/document \
  | grep region | cut -d '"' -f4)

# Resolve the SG ID of the Café Web Server by tag
sgId=$(aws ec2 describe-instances \
  --filters "Name=tag:Name,Values='Cafe Web Server'" \
  --query 'Reservations[*].Instances[*].SecurityGroups[*].[GroupId]' \
  --region $region --output text)
echo $sgId

# Console logins on the account (returned essentially just the current user)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=ConsoleLogin

# All actions on any SG, then narrowed to the Café SG
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::EC2::SecurityGroup \
  --region $region --output text | grep $sgId

The CLI lookups returned useful but still hard-to-read text blocks. For a wider search across fields like useridentity.userName, SQL on Athena is the better tool.

Breakthrough — Athena SQL over the CloudTrail logs

• From the CloudTrail console → Event history → Create Athena table, pointing at the same monitoring0423 bucket. Athena generated a CREATE EXTERNAL TABLE that maps each JSON field to a column. The useridentity column is a struct, and resources is an array.
• In the Athena Query Editor, set the query results location to s3://monitoring0423/results/ under Settings → Manage.
• Started with a small exploratory query:

SELECT useridentity.userName, eventtime, eventsource, eventname, requestparameters
FROM cloudtrail_logs_monitoring0423
LIMIT 30;

Useful for understanding the column layout. Not enough to find the culprit. Followed the challenge hints and built up the filter step by step until landing on the query that solved the case:

SELECT DISTINCT useridentity.userName, eventName, eventSource
FROM cloudtrail_logs_monitoring0423
WHERE from_iso8601_timestamp(eventtime) > date_add('day', -1, now())
ORDER BY eventSource;

Two complementary filters made the answer obvious. A tighter version using WHERE eventsource = 'ec2.amazonaws.com' AND eventname LIKE '%Security%' isolates only SG-related EC2 events, and the unauthorized AuthorizeSecurityGroupIngress stands out clearly with a userName that is not the current operator.

Containment, eradication and recovery

Identification was only step one. The remediation is the part that actually stops the bleeding — and it has to happen across three different layers: the OS, the IAM control plane, and the static content of the website.

5.1 — Evict the OS attacker

• sudo aureport --auth confirmed an unknown account chaos-user had recent auth records.
• who showed chaos-user was still logged in over SSH.
• First sudo userdel -r chaos-user failed — the user was active — but the error returned the live PID.
• sudo kill -9 <PID> dropped the session, then sudo userdel -r chaos-user succeeded.
• sudo cat /etc/passwd | grep -v nologin confirmed only standard Amazon Linux accounts (root, sync, shutdown, halt, ec2-user) remained.

5.2 — Harden SSH

• sudo ls -l /etc/ssh/sshd_config showed the file had been modified today — also part of the attack.
• Edited /etc/ssh/sshd_config in vi:

# Before
PasswordAuthentication yes
#PasswordAuthentication no

# After
#PasswordAuthentication yes
PasswordAuthentication no

• Restarted the daemon with sudo service sshd restart. From now on, only key-pair authentication is accepted — a guessed/leaked password is no longer enough.

5.3 — Close the network door

Removed the rogue inbound rule SSH/22 0.0.0.0/0 from the Café Web Server SG. Only the original HTTP/80 and the SSH/22 from /32 remained:

5.4 — Restore the website

The attacker conveniently kept a backup of the original image:

cd /var/www/html/cafe/images/
ls -l
sudo mv Coffee-and-Pastries.backup Coffee-and-Pastries.jpg

A hard refresh of the Café URL showed the legitimate bakery photo again.

5.5 — Remove the IAM user

• IAM console → Users → selected chaos → Delete (typed the user name to confirm).
• Result: only awsstudent remained as the active IAM user.

Downloading and inspecting CloudTrail logs from EC2

• aws s3 ls — list buckets to recover the monitoring#### name.
• aws s3 cp s3://monitoring0423/ . --recursive — pull every object from the trail bucket.
• gunzip *.gz — CloudTrail writes each event batch as a gzipped JSON file.
• cat <file>.json | python -m json.tool — pretty-print one event to learn the schema.

Cross-file filtering with bash + grep

• Iterate every JSON file in the current directory and print the field of interest:

for i in $(ls); do
  echo $i
  cat $i | python -m json.tool | grep sourceIPAddress
done

Same pattern works for eventName, userIdentity, etc. Useful for a first pass; breaks down once the logs grow into thousands of events.

AWS CLI — cloudtrail lookup-events

• By event name:

  aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=EventName,AttributeValue=ConsoleLogin
  

• By resource type, scoped to a region:

  aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::EC2::SecurityGroup \
    --region $region --output text
  

• Narrow to a specific SG by piping into grep:

  aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::EC2::SecurityGroup \
    --region $region --output text | grep $sgId
  

• Region discovered straight from the IMDS on the EC2 instance:

  region=$(curl http://169.254.169.254/latest/dynamic/instance-identity/document \
    | grep region | cut -d '"' -f4)
  

  Note: this is IMDSv1. In production, IMDSv2 (token-based) is the secure default.

Athena — SQL over CloudTrail

Once Athena was pointed at the CloudTrail bucket, the same investigation became a series of declarative queries instead of bash plumbing.

-- Explore the schema
SELECT useridentity.userName, eventtime, eventsource, eventname, requestparameters
FROM cloudtrail_logs_monitoring0423
LIMIT 30;

-- All users active in the last day and the distinct actions they took
SELECT DISTINCT useridentity.userName, eventName, eventSource
FROM cloudtrail_logs_monitoring0423
WHERE from_iso8601_timestamp(eventtime) > date_add('day', -1, now())
ORDER BY eventSource;

-- Tight filter: only EC2 SG-related events
SELECT useridentity.userName, eventtime, eventname, sourceIPAddress
FROM cloudtrail_logs_monitoring0423
WHERE eventsource = 'ec2.amazonaws.com'
  AND eventname LIKE '%Security%'
ORDER BY eventtime DESC;

OS-level eviction and hardening

• sudo aureport --auth — recent authentication records.
• who — currently logged-in sessions.
• sudo userdel -r chaos-user — delete OS user and home directory.
• sudo kill -9 <PID> — terminate the live session before delete.
• sudo cat /etc/passwd | grep -v nologin — verify no other login-capable users remain.
• sudo vi /etc/ssh/sshd_config → set PasswordAuthentication no.
• sudo service sshd restart — apply the change.