I actually really respect their policy. Keep the site active and then forgive stupid bills if there was an error.
To shut down or disconnect a cloud service is terrible as usually it’s in error. The errs on the side of the user knowing their stuff better than the hoster which is what I want in a provider.
That “policy” was to reduce his bill from 104k to 5k initially. It was not “forgiven” until his story went viral.
So their actual policy was to send a user paying $0/month a bill for 5k for malicious behavior they didn’t cause on the site. Thats not something to respect.
The CEO playing at “no no, we always forgive these” was not at all what he was told until after thousands if not hundreds of thousands of their potential customers saw how screwed he was by their non existent tools to rate limit bandwidth.
A 9yr old hosting company not having any programtic tools to limit biling is an intentional choice, not some oversight. It’s clearly their buisness model to have people go over the free tier unintentionally, just not unintentionally enough to go viral.
I find it astonishing that Netlify had no safety mechanism in place to prevent this.
Saddling customers with unbounded liability is irresponsible; arguably negligent.
Definitely negligent, I still remember the young adult who killed himself when he thought his Robinhood account was negative nearly 3 quarters of a million dollars.
“After looking into this, it seems you have a hit song on your site,” the email from Netlify customer support reads. “Maan Bou Jan Sang Lou by Teresa Tang. I was not aware of her, but she seems to be a popular Taiwanese singer. This song is 99% of your bandwidth usage over the past 30 days.”
The letter further explained that a lot of bandwidth was generated from user agents that “are quite ancient using Google Cloud addresses”.
“This would include devices such as circa 2010 iPads, Windows 98 & Windows 6 computers. So either you have a fanbase with a passion for older technology, or this was likely a DDoS attack. To me, this seems to be the latter,” the email continued and suggested hosting such files on third-party platforms, such as YouTube or SoundCloud.
After explaining the standard practice of reducing the bill to 20% after such attacks, which would be $20,900 in this case, the Netlify support team offered a better deal.
“I’ve currently reduced it to about 5%, which is $5,225. I know this is still a lot of money, and I apologize for the inconvenience. If you like, I can raise this internally to see what else can be done.”
The user wasn’t happy with that and decided not to pay but post their story on Reddit and Hacker News instead.
"Netlify CEO here.
Our support team has reached out to the user from the thread to let them know they’re not getting charged for this.
It’s currently our policy to not shut down free sites during traffic spikes that doesn’t match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
Apologies that this didn’t come through in the initial support reply."
This was posted 4 days ago in hackernews.
And that’s all he posted. I think he responed to one of two comments and then ignored everything else. I really dislike that ‘CEO here’, ‘I’m important, listen to me’ means that always ends up on HN. Then they disappear like their mere response to the post is enough. After all, they’re very important people.
If he thought ‘damaged control done’ he was sadly mistaken.
I host my site on Netlify. I’m moving. If they think that it’s acceptable to bill somebody $104k for a small site, at all, without it tripping some alarm for a human to look at before it goes out, then they’re doing it wrong. Something that says ‘Month 1 bill = $20; Month 2 bill = $104,000’ could be a problem isn’t difficult to do. And that they have ‘done this often’ (my words) highlights it’s a problem.
There are many bullshit hosting companies out there I can use who don’t do this sort of thing. Why is Netlify special.
What’s the questions-to-answers ratio to make it acceptable to write “CEO here” in a topic, in your opinion?
How much and what would he need to post so damage control would have been done (and successfull) in your eyes?
Just asking in case I become a CEO and see someone having a solvable issue with my company’s sercice.
What’s the questions-to-answers ratio to make it acceptable to write “CEO here” in a topic, in your opinion?
Not everything is an algorithm or ratios. Hopefully you know that.
How much and what would he need to post so damage control would have been done (and successfull) in your eyes?
See above.
Just asking in case I become a CEO and see someone having a solvable issue with my company’s sercice.
Theres a solvable issue when your site doesn’t deploy. And then theres repeated issues that cause you to send life changing / threatening bills to customers.
If you think they should be treated the same, let me know when you become CEO so I can avoid your business as well.
Good luck with your career though.
You don’t think the person who’s ultimately responsible for a company’s policies is important in a discussion of those policies? There’s nothing arrogant about knowing you’re the one at the center of a news story.
I do. Very important. That’s not what happened here.
I don’t believe showing up, making a statement and then fucking off again is any kind of ‘discussion’ or ‘taking responsibility’. It’s PR.
Paying lip service to recurring issues (and this seems to have escaped people: this has happened multiple times) that are damaging people, is not taking responsibility. It’s PR.
Is it possible to avoid that hotliking of a file? in this case was a heavy mp3 file, but it easly could be a heavy image or a video.
I use Netlify to host my frontend projects and portfolio. Does anyone have a way to prevent something like this?
Not use a hosting provider that charges by the amount of traffic?
This appears to be an extreme edge case but overall there is nothing preventing you from waking up to such a huge bill if your site turns into the most popular page on the internet over night.
I didn’t even think commercial host providers would do this.
The only service I knew about that had limit to transferred amount of data was grex.org, a non-commercial public unix shell. It had limit of 10MB/day for your web page, but it also didn’t allow stuff like images.
However, that wasn’t anything commercial. And I think before the shutdown it was just a single computer sitting in someone’s basement.
Not that it helps but the CEO claims they forgive for this type of attack/event. https://news.ycombinator.com/item?id=39521986
Netlify CEO here.
Our support team has reached out to the user from the thread to let them know they’re not getting charged for this.
It’s currently our policy to not shut down free sites during traffic spikes that doesn’t match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
Apologies that this didn’t come through in the initial support reply.
And later they were asked if they would have responded if it didn’t go viral. https://news.ycombinator.com/item?id=39522029
Question:
There are only two questions everyone have:
Would Netlify forgive the bill if this didn’t go viral?
How do you plan to address this issue so that it never happens again?
Everyone here knew someone from Netlify would come and say OP wouldn’t have to pay. That was a given. Now we want to know the important answers.
Answer by CEO:
Yes. We’ve forgiven lots and lots of bills over the last 9 years and they haven’t gone viral
While I’ve always favored erring towards keeping people’s sites up we are currently working on changing the default behavior to never let free sites incur overages
You can put the site behind cloudflare for DDOS protection. Unfortunately, it’s not good for user privacy and it will make the site difficult to access over VPNs, proxies, and TOR.
Netlifiy is very expensive for bandwidth and the free bandwidth can be exceeded very quickly. I would look for something with a hard bandwidth cap. Then your site will just go offline if the bandwidth is exceeded.
Unfortunately, it’s not good for user privacy and it will make the site difficult to access over VPNs, proxies, and TOR.
Difficult, but not impossible (unless the site owner also goes and futher implements additional measures like ASN blocking for known proxies/VPNs/etc), just solve a captcha and you should be on your way pretty much.
