DigitalOcean locking down droplets

I’m having the time of my life today after Digital Ocean conveniently locking down one of my droplets today. The support was completely clueless and worse, some of them weren’t even reading the replies. They take 1-2hr between replies and apparently it’s always not the same guy so the next guy would just briefly skim through the thread and insert another canned reply.

I’m left with no access at all to the server, not even to backup the data and having to deal with the terrible people at customer support. Seriously, I’ve had better experience with 5$/mth shared hosting.

The latest one-line reply I get was asking me to completely destroy the droplet… Seriously?

Does this sound like a familiar experience to anyone else?

what was the first reason they locked your droplet down?

tokopress said

what was the first reason they locked your droplet down?

Yeah, I’m interested too. I’m want to move my stuff from (mt) to Digital Ocean.

The guy who locked down my droplet said that my droplet was sending out ddos attacks and accused that it was a vulnerability with the elasticsearch used on my server. Apparently since I didn’t even have access to my server logs there is no way I could verify whatever he said.

And he gave me this link - http://bouk.co/blog/elasticsearch-rce/

The first line on that article reads:

Elasticsearch has a flaw in its default configuration which makes it possible for any webpage to execute arbitrary code on visitors with Elasticsearch installed.

… which is already a clear indication that this vulnerability on exists on DEVELOPERS machine, not production servers. As far as I know, ES was already configured for closed system operation only so it’s not possible to send DDOS attacks - unless its sending the packets to our app itself, which is very likely when you do a lot of searches.

I still have my account at MT, wouldn’t recommend moving away to DO if you want to save yourself the sufferings. Already wasted ~6 hours trying to communicate with the support.

Hi Syamil,

Really sorry to hear that…

This is exactly what happened to us also…

I noticed that our websites were down yesterday. Our big team has a big droplet on DigitalOcean to host all of our websites…

I talked to our team and our friend who handle the server said that our droplet was suspended… :frowning:

Okay, stay calm…

The first email said that DO got complaint from Bank of America said that one of our websites host phishing page of them…

The second email said that one or some of our websites become DDOS attack source, possibly because of elasticsearch vulnerability… We also do not believe that we have elasticsearch installed…

Then, we jump to conclusion that one of our websites was hacked! And the hacker uses our website to generate bad traffic for his bad purposes…

We keep communicating with them, and in our conversation we find that the phishing page is on one of our “old” WordPress demo website… This subdomain is no longer maintained, so it still use an old version of WordPress…

We know that old WordPress version is full of vulnerabilities… So, it can be a door for hacker to hack our website and use it for bad purposes…

Our dropplet was still suspended, but DO give access via console… Then our team can “remove” this subdomain to fix this issue…

And, finally… our websites are back online again last night before Netherland vs Argentina match… :smiley:

Lesson Learned

  1. DigitalOcean (and also Linode) is “Unmanaged” VPS hosting… So, we are fully responsible with what happened with our server… We are glad that we have member in our team that has good capability on sysadmin…

  2. We need to deactivate / remove our unused / inactive websites…

  3. We need to keep our WordPress up to date…

  4. We need to keep our WordPress secure…

I am not sysadmin, so I am not the expert on this case, I only share what I heard and I knew… And I hope this reply is useful for you… :wink:

I’ve been managing VPS for quite some time from small to larger ones, but this is the first time I was locked down and asked to completely destroy everything just because they suspected vulnerabilities that they didn’t even care to investigate properly.

The droplet that was locked down was a completely closed system for an app with maximum 5 users, so it is very tiny. If such a small app gave me this much problems, I would definitely stay away from DO for any larger scale websites.

I can live with unmanaged VPS, but not with this type of terrible support.

Hi Syamil,

Correct…

It is common when a company has bigger customer base, they need more resources for support… For DO case, their support is less responsive lately, for sure…

I also can feel that “elastic search” email is a generic answer template from their support…

For my personal stuff, I moved from DO to Linode last month… Their price is competitive now… Their documentation / library is better (DO is community-driven documentation)… And, their support is better, I think…

And i was thinking to move to DO too… well after this, i might need to reconsider that.

I found this link, https://www.digitalocean.com/community/questions/my-droplet-is-locked-by-support-staff-because-because-of-an-outgoing-flood-or-ddos-what-do-i-do

I know it’s two months old, but I thought someone would find vultr useful. It’s also cheap but much better than DO.

You can check out this post for more info.

Dear. This is old but I think it’s useful for you. I am using $20/m pack, and with this pack, you can run only about 3-4 demo sites for themes. More site can make it down (I mean down time).

About locking down a droplet, maybe we must check all scripts that are running. If have one scam or fraud script, your droplet may be locked down.

tiennguyenvan said

Dear. This is old but I think it’s useful for you. I am using $20/m pack, and with this pack, you can run only about 3-4 demo sites for themes. More site can make it down (I mean down time).

About locking down a droplet, maybe we must check all scripts that are running. If have one scam or fraud script, your droplet may be locked down.

That’s why you use nginx instead of Apache :slight_smile:

I am using Digital Ocean it is looking good for me. I configured the server (Apache MPM Worker, PHP-FPM, FastCGI, APC…) for live preview it is fast (my opinion) you can check http://routewp.com hmm i am writing this message for who think to use DO.

About the down droplets, i have not any idea really, have you not a snapshot ? Run your server within 30 sec.

Regards, Codestar

I’ve read many complaints regarding droplets being shut down without any prior notice, awful & unprofessional support, security holes etc. Another major turn off for me is that DO allows minecrafts. So basically, you can just cross your fingers and hope no miner lands on your node, otherwise you’ll almost certainly experience performance degradation.

Just my 2 cents

We’ve been using DO since they opened. We actually use many different services similar to this.

one of the biggest problems with this type of thing is it lends itself to people opening up droplets that might not be quite as familiar with security as a server administrator.

We’ve been in the business of hosting since 1996, and do all facets of it from Cloud, to Grid, to co-locations, to VPS. We scaled back 5 years ago and only host for large clients now. Why? security security security. it became a full time job just maintaining hardware and after evaluating it determined the cost was too high to continue.

There is no question that every single thing that happens on those droplets is the responsibility of the owner. Hardening a server is not a luxury, it’s a requirement, and not only that, it requires someone with substantial experience in that area to do it. This is one reason I think it’s not so great that these services are getting so popular. They are greatly contributing to the amount of DDoS attacks and that’s why you’re seeing places like DO take a no tolerance stance and shut down servers getting reported.

I honestly don’t think people understand the depth of knowledge it takes to EFFECTIVELY administer a server.

If you’re going to spin up instances anywhere on the web, it’s going to be the same story. if you’re not very familiar with security, you need to HIRE someone to harden your server, and that is not a one time job. it requires frequent maintenance, and repeated effort. The reality is by far, MOST people running VPS type services should be hiring people to manage them

Just the mere task of updating packages on your instances can make the difference in vulnerability, not even taking into account actual firewall work.

you simply cannot just spin up an instance and be off to the races. The moment your server hits the web it is INSTANTLY being attacked.

I completely agree with you about security. On the other hand, company that shuts your dropplet down because you’ve been using same CC, which was actually company’s credit card, on two accounts before letting you know is anything but professionall! Seriously, shut down the service and then ask questions?

Actually yes, that’s standard procedure.

If someone reports a server spamming, or sending out DDoS, or even simple brute force attacks, the first cause of action is always to remove the server from the network.

Mitigating damage is always priority 1. They have to protect themselves.

Now, if they didn’t verify it was happening, bad on them, but if it was confirmed, immediate removal from the network is appropriate.

This goes back to the problem. These aren’t managed services. They are self managed. So the NOC doesn’t even have authority to investigate by logging in and even looking. Their action is limited. That’s the whole point of unmanaged servers. That’s why they’re dirt cheap

I understand your point, believe me I do. But think of it the other way around. I call your NOC because my sites/servers are being attacked by a machine on your network. You confirm it, and your response to me is ‘Well we notified the owner’ was the response? The first thing i’d be doing is contacting a lawyer.
Once they confirm the illegal activity they’ve been put on notice. from that point on THEY are liable. The onus is on them as a whole anyway in reality, i mean things trickle down. but if they know there is illegal traffic and they don’t immediately stop it once notified, that’s a very bad thing

the proper response is this:

Contact is made alerting them attacks are coming from their facility.
They investigate and confirm of find no problem.
if they find a problem, immediately remove the server from the network and alert the customer.

There should be an escalation procedure for these things.