42 Vulnerabilities and an Intern's AWS Key

  • 42 vulnerabilities patched
  • Production restored same day
  • Access controls rebuilt
  • 42 Critical vulnerabilities remediated
  • <2 hrs Time from Pineido involvement to production restore
  • 8 Stale credentials from former employees revoked

What changed

Critical vulnerabilities remediated
42
Time from Pineido involvement to production restore
<2 hrs
Stale credentials from former employees revoked
8
Publicly exposed S3 buckets secured
6
Separate AWS environments created
3
Production incidents from access issues since remediation
0

What they told us

What a load of spaghetti with sauce. We had no idea how exposed we were until someone showed us the full picture. That audit probably saved us from something much worse than a five-hour outage.
TariqEngineering Manager, Growing Tech Company

What we walked into

A growing tech company with 60 employees and a product serving thousands of daily active users experienced a sudden production outage. The application went down during business hours, and customer-facing traffic started erroring out within minutes. The engineering team had been troubleshooting for over four hours with no resolution. No one was sure what had changed, who had touched what, or where to start looking. They restarted services, rolled back the last two deploys, and combed through application logs, but the failure did not match any code path anyone could find. The CEO called Pineido after the fourth hour because the internal team was running in circles and the cost of every additional down-hour was climbing. The deeper issue, which the outage merely exposed, was that the company had no way to answer the most basic incident question: who did what, and when. Without that answer, four engineers can chase the wrong layer for hours.

What was actually broken

The immediate problem was the outage, but the underlying problem was far worse. The company had grown from 8 engineers to 25 in eighteen months. During that growth, no one had maintained the access control system. AWS IAM roles had been created ad hoc, usually by copying an existing policy that already worked rather than scoping a new one, so permissions only ever accumulated and never shrank. New hires were given broad permissions because it was faster than scoping them properly. This is not an unusual failure: in practice most IAM roles drift toward over-permissioned because nobody circles back to prune them, and unlike a code bug an excessive permission produces no visible error to force the issue. Contractors who had left months ago still had active long-lived access keys, the exact class of standing credential that AWS now recommends eliminating in favor of short-lived, federated access. The intern who joined two weeks before the outage had been given the same access level as the CTO because the onboarding script had not been updated since the company had 8 people.

The outage itself was caused by an accidental deletion of a production database table. The intern had been experimenting with a data migration script in what they believed was a development environment. It was not. The company had no environment isolation. Development, staging, and production all shared the same AWS account with the same IAM policies, so the only thing standing between a test script and live customer data was a connection string. The intern's script ran against the production database because that connection string defaulted to production and no one had documented the difference. Nothing about the setup made the dangerous action feel dangerous, which is precisely how a single keystroke from the newest person on the team took down a product serving thousands of users.

When the engineering team started troubleshooting, they could not determine the root cause because there were no access logs configured for the database. CloudTrail was enabled, as it is by default, but no one had ever opened it, so the one record that would have ended the investigation in minutes sat untouched while four engineers searched everywhere else. The team spent four hours checking application code, restarting services, and reviewing deployment logs before Pineido suggested checking IAM activity. CloudTrail records the calling identity, the source IP, and the exact timestamp for management events like a table deletion, which is exactly what an incident like this turns on. Within 30 minutes of reviewing CloudTrail, we identified the exact API call that had dropped the table, confirmed from the calling identity that it was accidental rather than an intrusion, and traced it to the intern's credentials. The database was restored from a backup that was, fortunately, only 90 minutes old. The outage total was approximately five hours, and the single largest contributor to that number was not the deletion itself but the hours spent looking in the wrong place for the answer.

Step by step

Incident triage and root cause identification

We started with CloudTrail logs and IAM activity reports rather than application code, because a sudden outage with no matching code change is an access-and-identity question, not a bug. Within 30 minutes we found the exact API call that caused the outage, read the calling identity off the event to rule out a compromised account, and traced it to the intern's credentials. The production database was restored from the most recent automated backup, which was only 90 minutes old, so data loss was negligible. Total time from our involvement to production restore: under two hours, against the four the internal team had already spent searching the wrong layer.

Full IAM and infrastructure audit

Over the next five days we audited every IAM role, policy, and credential in the AWS account, using last-accessed data and CloudTrail history to see which permissions were actually exercised rather than just granted. We found 42 critical vulnerabilities: 11 overprivileged roles with admin-level access that should have been read-only, 8 sets of credentials belonging to former employees and contractors that were still active, 6 S3 buckets with public read access containing customer data, 9 security groups with unrestricted inbound rules, 5 Lambda functions with hardcoded API keys in environment variables, and 3 unencrypted database instances. Most of these were not new. They were the silent residue of eighteen months of growth, each one individually defensible at the moment it was created and collectively a map of every way the company could have been breached without an outage ever forcing the question.

Environment isolation

We created separate AWS accounts for development, staging, and production using AWS Organizations, so the boundary between a test script and live customer data became an account boundary rather than a string in a config file. Each environment received its own IAM policies, network configuration, and database instances. Connection strings were moved to AWS Secrets Manager with environment-specific access controls, and a development role was given no path to reach a production secret at all. The scenario that caused the outage became structurally impossible: a script run from a dev context can no longer see, let alone delete, a production table, regardless of who runs it or what the connection string says.

Access policy implementation

We built a role-based access control matrix that mapped each job function to its minimum-required permissions, so access could always be traced back to a documented business reason rather than to whatever policy someone happened to copy. Every employee was re-provisioned from scratch against that matrix with scoped credentials, defaulting to read-only and escalating only where the role genuinely required it. A formal access request process was introduced: any permission change required manager approval and was logged in a central register, giving a clean audit trail of who approved what and why. Production access required two-person approval, a deliberate separation of duties so that no single person, junior or senior, could push an irreversible change alone. The intern was given a development-only role with no production visibility.

Monitoring and alerting

We configured CloudTrail log analysis with automated alerts for high-risk API calls (table deletions, IAM policy changes, security group modifications, and any attempt to stop or delete logging itself, since tampering with the audit trail is the clearest signal that an incident is malicious rather than accidental). A weekly automated scan was set up to detect new overprivileged roles or public-facing resources before they could drift back into the same accumulation pattern that produced the original 42 findings. The engineering team received a runbook for incident response that starts where this incident should have: pull the identity and timestamp from CloudTrail first, so the next outage, if it occurs, follows a structured investigation path instead of four hours of guessing.

42 Critical vulnerabilities remediated

Get in touch

Tell us what you want built. We tell you where to start and what it takes.

The questions that decide if we are right for you

How long does a full security audit take?

The initial audit took five working days, covering every IAM role, policy, credential, S3 bucket, security group, and database instance in the account. Remediation of all 42 vulnerabilities took an additional two weeks, sequenced so the highest-impact exposures (the publicly readable customer-data buckets and the still-active former-employee credentials) were closed in the first days, not the last. The total engagement was three weeks from incident to fully hardened infrastructure. The scope scales with how much sprawl has accumulated, but for a company of this size and growth history, three weeks is typical.

Should we do a security audit if we have not had an incident?

Yes. Most of the vulnerabilities we found had existed for over a year. The outage was the trigger, but the exposure predated it significantly: the public S3 buckets and the live credentials of people who had already left the company were quietly waiting for anyone who happened to look. That is the nature of access-control risk. It produces no error message and no failed deploy, so nothing forces it to the surface until something goes wrong. A proactive audit is far cheaper than an incident-driven one, both in dollars and in the customer trust an actual breach would cost. The right time to run one is before you have a reason to.

What about compliance requirements like SOC 2?

The remediation work we did covered approximately 70% of the technical controls required for SOC 2 Type II. The role-based access matrix, the documented approval-and-logging process, the two-person production approvals, the environment isolation, and the CloudTrail monitoring map directly onto the access-control and change-management criteria auditors examine most closely (CC6 and CC8). What remains is mostly the operating-effectiveness evidence that Type II requires over time: the recurring access reviews, recertifications, and approval records that have to accumulate across the audit window. If the company pursues certification, the hardest infrastructure work is already done, and the rest is running the process we left behind long enough to show it works.

How do you handle the audit without disrupting the team?

The audit runs in parallel with normal development. We need read-only access to AWS and about 30 minutes with two senior engineers for context on how the environment is wired. From there the work is largely passive analysis of logs, policies, and configuration, none of which interrupts a deploy. The team does not stop shipping while the audit is in progress. Remediation steps that do require change, such as splitting environments or re-provisioning credentials, are scheduled with the team and rolled out in low-risk windows so production stays stable throughout.

ByFounder and Imaginator

Published