“Since the user opened a ticket with us this past Sunday, we’ve been actively researching this situation. Initially, we thought it might have resulted from a DDoS attack, which we stated in our first response. After some investigating, it looks as though the spike in traffic was not caused by a DDoS after all,” Dorian Kendal, CMO at Netlify, told Cybernews.
Instead, now they believe that this was a sustained download event of an mp3 file over a stretch of multiple days.
“We’re working directly with the user to better understand what’s happening on their end, so we can uncover what caused the dramatic increase in downloads,” Kendal said.
I’m confused, what is this supposed to mean? Some sort of non-distributed DOS attack? How would working with the customer help there? If they’re susceptible to a denial of service, isn’t that entirely an internal problem?
They are saying that it wasn’t a ddos at all but organic use. The user was notified but did nothing. So they think their notifying stuff isn’t good enough.
Sorry, but what exactly is a “sustained download event” supposed to be? It sounds like they’re describing some sort of DOS-like attack that isn’t a DDOS, where a user manages to force the server to serve up way more data over a sustained period of time than would be reasonable for downloading a single MP3 for normal use.
But maybe that’s not what they mean. It’s very unclear.
Sorry, but what exactly is a “sustained download event” supposed to be?
I’m pretty sure they’re describing something akin to what many small site owners have referred to as ‘the hug of death’. If you’re a small site that blows up on the front page of lemmy (or an actually large community site), you’re going to experience sustained traffic that your site isn’t capable of handling (be that at the computer resource or financial level in this case).
Normally the hug of death’ just takes you offline when your provider can’t handle the load or you blow past your providers thresholds. In this case, that threshold didn’t appear to exist and it just kept adding to the bill.
I am too. Is the agreement to charge per mb downloaded? Do they not have some sort of "turn it off if I hit this max?* feature?
I usually avoid hosting solutions like this just because of this shit. I wanna know how much I’ll owe before the month starts even. Anything else feels like gambling.
Of course they do but they can make 104k if they don’t turn it on.
There are plenty of bandwidth restricted hosting sites out there. Sounds like that is what you want. Maximum speed regardless if that’s used 24/7 or not. If more users request your site than that bandwidth allows - oh well.
I use Netlify to host my frontend projects and portfolio. Does anyone have a way to prevent something like this?
Not use a hosting provider that charges by the amount of traffic?
This appears to be an extreme edge case but overall there is nothing preventing you from waking up to such a huge bill if your site turns into the most popular page on the internet over night.
I didn’t even think commercial host providers would do this.
The only service I knew about that had limit to transferred amount of data was grex.org, a non-commercial public unix shell. It had limit of 10MB/day for your web page, but it also didn’t allow stuff like images.
However, that wasn’t anything commercial. And I think before the shutdown it was just a single computer sitting in someone’s basement.
Not that it helps but the CEO claims they forgive for this type of attack/event. https://news.ycombinator.com/item?id=39521986
Netlify CEO here.
Our support team has reached out to the user from the thread to let them know they’re not getting charged for this.
It’s currently our policy to not shut down free sites during traffic spikes that doesn’t match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
Apologies that this didn’t come through in the initial support reply.
And later they were asked if they would have responded if it didn’t go viral. https://news.ycombinator.com/item?id=39522029
Question:
There are only two questions everyone have:
Would Netlify forgive the bill if this didn’t go viral?
How do you plan to address this issue so that it never happens again?
Everyone here knew someone from Netlify would come and say OP wouldn’t have to pay. That was a given. Now we want to know the important answers.
Answer by CEO:
Yes. We’ve forgiven lots and lots of bills over the last 9 years and they haven’t gone viral
While I’ve always favored erring towards keeping people’s sites up we are currently working on changing the default behavior to never let free sites incur overages
You can put the site behind cloudflare for DDOS protection. Unfortunately, it’s not good for user privacy and it will make the site difficult to access over VPNs, proxies, and TOR.
Netlifiy is very expensive for bandwidth and the free bandwidth can be exceeded very quickly. I would look for something with a hard bandwidth cap. Then your site will just go offline if the bandwidth is exceeded.
Unfortunately, it’s not good for user privacy and it will make the site difficult to access over VPNs, proxies, and TOR.
Difficult, but not impossible (unless the site owner also goes and futher implements additional measures like ASN blocking for known proxies/VPNs/etc), just solve a captcha and you should be on your way pretty much.
The most expensive mp3 of his life…
I find it astonishing that Netlify had no safety mechanism in place to prevent this.
Saddling customers with unbounded liability is irresponsible; arguably negligent.
Definitely negligent, I still remember the young adult who killed himself when he thought his Robinhood account was negative nearly 3 quarters of a million dollars.
“After looking into this, it seems you have a hit song on your site,” the email from Netlify customer support reads. “Maan Bou Jan Sang Lou by Teresa Tang. I was not aware of her, but she seems to be a popular Taiwanese singer. This song is 99% of your bandwidth usage over the past 30 days.”
The letter further explained that a lot of bandwidth was generated from user agents that “are quite ancient using Google Cloud addresses”.
“This would include devices such as circa 2010 iPads, Windows 98 & Windows 6 computers. So either you have a fanbase with a passion for older technology, or this was likely a DDoS attack. To me, this seems to be the latter,” the email continued and suggested hosting such files on third-party platforms, such as YouTube or SoundCloud.
After explaining the standard practice of reducing the bill to 20% after such attacks, which would be $20,900 in this case, the Netlify support team offered a better deal.
“I’ve currently reduced it to about 5%, which is $5,225. I know this is still a lot of money, and I apologize for the inconvenience. If you like, I can raise this internally to see what else can be done.”
The user wasn’t happy with that and decided not to pay but post their story on Reddit and Hacker News instead.
"Netlify CEO here.
Our support team has reached out to the user from the thread to let them know they’re not getting charged for this.
It’s currently our policy to not shut down free sites during traffic spikes that doesn’t match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
Apologies that this didn’t come through in the initial support reply."
This was posted 4 days ago in hackernews.
And that’s all he posted. I think he responed to one of two comments and then ignored everything else. I really dislike that ‘CEO here’, ‘I’m important, listen to me’ means that always ends up on HN. Then they disappear like their mere response to the post is enough. After all, they’re very important people.
If he thought ‘damaged control done’ he was sadly mistaken.
I host my site on Netlify. I’m moving. If they think that it’s acceptable to bill somebody $104k for a small site, at all, without it tripping some alarm for a human to look at before it goes out, then they’re doing it wrong. Something that says ‘Month 1 bill = $20; Month 2 bill = $104,000’ could be a problem isn’t difficult to do. And that they have ‘done this often’ (my words) highlights it’s a problem.
There are many bullshit hosting companies out there I can use who don’t do this sort of thing. Why is Netlify special.
What’s the questions-to-answers ratio to make it acceptable to write “CEO here” in a topic, in your opinion?
How much and what would he need to post so damage control would have been done (and successfull) in your eyes?
Just asking in case I become a CEO and see someone having a solvable issue with my company’s sercice.
What’s the questions-to-answers ratio to make it acceptable to write “CEO here” in a topic, in your opinion?
Not everything is an algorithm or ratios. Hopefully you know that.
How much and what would he need to post so damage control would have been done (and successfull) in your eyes?
See above.
Just asking in case I become a CEO and see someone having a solvable issue with my company’s sercice.
Theres a solvable issue when your site doesn’t deploy. And then theres repeated issues that cause you to send life changing / threatening bills to customers.
If you think they should be treated the same, let me know when you become CEO so I can avoid your business as well.
Good luck with your career though.
You don’t think the person who’s ultimately responsible for a company’s policies is important in a discussion of those policies? There’s nothing arrogant about knowing you’re the one at the center of a news story.
I do. Very important. That’s not what happened here.
I don’t believe showing up, making a statement and then fucking off again is any kind of ‘discussion’ or ‘taking responsibility’. It’s PR.
Paying lip service to recurring issues (and this seems to have escaped people: this has happened multiple times) that are damaging people, is not taking responsibility. It’s PR.
