Analysis of the BrowserStack breach - A classic example of "Pivoting in the Clouds"

BrowserStack was recently breached and it was all over the news as is the case with almost all breaches these days.

In this blog post, I will briefly describe what happened to make everybody aware that things can go really wrong in the Cloud if proper measures are not taken.

The Tl;DR version:

BrowserStack's infrastructure is hosted on the Amazon Web Services (AWS).

They had one particular machine (virtual instance in this case) on the AWS that was not patched against the ShellShock vulnerability.

The attacker leveraged that to pivot through the various moving parts within their AWS setup and steal some information from their production database.

The attacker then used the stolen data and credentials of their AWS SES (See below) service to send emails to some BrowserStack users stating that BrowserStack is shutting down. Ouch!!

The longer version:

The sequence of events:

• Attackers took advantage of the un-patched instance

• Logged in that instance

• Created an IAM user (See below) and generated a key-pair by using the secret keys stored on that instance

• Spawned a new instance using the newly created credentials

• Mounted one of the production backup disks to this instance

• Retrieved config file with database password from this backup

• Copied database tables partially and stole some data. While the database tables were being copied, it triggered an alert and the BrowserStack folks acted immediately blocking the IP.

But, by this time, the attacker already had stolen some data and the SES (See below) credentials which helped them send a fake email to some BrowserStack users.

IAM

This is the AWS Identity Access Management solution where you can create multiple users in an organization and assign the appropriate access rights to them following the minimum privilege access model. In other words, just give the amount of access to an individual that the individual's role demands. Nothing more than that.

SES

This is the AWS Simple Email Service which is a service for sending out emails.

Below is purely my analysis on this incident. Please feel free to comment/ask questions/criticize. Some of the poor practices done by BrowserStack on the AWS Cloud:

1. AWS Secret keys were stored on the un-patched instance. Secret keys should be stored securely following the AWS Best Practices.

2. I don't think they had an inventory of all their running AWS instances. Maybe, they did because it could be obtained from their AWS Console. But, I cannot be sure. Assuming they knew about this running instance, they should have really patched it against ShellShock. This was the root cause and could have prevented the breach all together even if other protections didn't exist.

3. They did have some alerts but they should have really built a lot more alerts like while creating a new IAM user, creating key pairs, etc.

4. Allowing IAM users to be created with elevated privileges. This is an educated guess. If they allowed the newly created IAM user to start a new instance, mount a backup to it, etc., I am guessing this IAM user had elevated privileges. Was this really necessary?

5. 2-factor authentication. AWS provides the capability to implement 2-factor authentication which I don't think was being leveraged here.

6. Storage of sensitive information. The database password was stored in a config file that was readable. This could have been locked down better. Was the backup disk the only place where they stored the database password?

7. There is no mention of how the attacker obtained the SES credentials. I am guessing that was stored on the backup disk as well.

Having talked about the poor practices, there were some good things that BrowserStack did as well:

1. Passwords hashed using bcrypt. This is a biggie!! Never store passwords in cleartext.

2. Alerts triggered at some point. Due to the alert that got triggered while copying database tables, the magnitude of impact was reduced drastically so that's good.

3. They mention auditing by AWS Cloud Trail that helped them track the attacker's movements.

4. Credit Card data processed through 3rd party so the credit card details were not stored on their instances. Again, this is a biggie when it comes to dealing with Credit Card data on the cloud. Leveraging a 3rd party to do this often helps as evident from this case.

5. Locking database when copied. A good fail-over mechanism which helped them in this case to some extent.

6. Other instances were patched against ShellShock so the attack surface was reduced.

7. Instances protected by OS firewall in addition to network firewalls. Defense in depth

8. They mention implementing "security groups" which is a AWS good practice. This helps segregate and isolate different moving parts.

9. Most importantly, they were pretty quick in responding to this breach. So, that was a big plus!

10. They did some more improvements as mentioned in the link below like encrypting backups, auditing logs more, revoking all existing AWS keys, improving monitoring and requesting a 3rd party to conduct a security audit. All these things are definitely going to improve their security posture.

Reference URL:

http://www.nextbigwhat.com/browserstack-hack-attack-explanation-297/