Before everyone gets their pitchforks out - Person from the image posted on Hacker News, CEO replied and said this charge shouldn't have happened and they wouldn't be charging the client anything.
Phew, good to know that if this ever happens to me as a customer, I just need to go viral on HN. What a relief.
Yeah that's is an attack on Netlify and not on him. It's them that should have protections against this. I argue that the customer can't even effectively defend against this themselves if they're using Netlify, which is turn means a court would likely get them off the hook for anything that can easily be classified as a DDOS attack.
Hm yes and no. The user might have angered someone with their website and it might well have been targeted to them instead of Netlify as a whole? I can imagine them using that point in a court if that was the case.
If I were to host on such a service I'd probably put cloudflare in front. Especially as it seems to be static content. But I wouldn't host on a service with unlimited pricing anyway. I'd much rather see my hobby site go down than to have world-class uptime and pay 100k :P
But how do you go from 10GB monthly to 190TB without it raising any flags? Apparently their site had been up for 4 years and suddenly the usage spikes by nearly 2 million percent, and nobody thinks to check up on why, or to notify the user that they're using an extreme amount of data, way beyond what they usually do.
You’d think a competent company would have bots to scour this data and raise alarms, yet here we are.
Hell even AWS isn't this bad. You can go in and set the maximum data you're prepared to allow and then it'll simply just block any connection attempt after that point and send you an alert.
You just have to be aware that you might need to keep an eye on things and be ready to increase bandwidth occasionally in case of something like Black Friday, assuming that kind of thing is relevant to your site.
The user might have angered someone with their website and it might well have been targeted to them instead of Netlify as a whole? I can imagine them using that point in a court if that was the case.
They wouldn't really get anywhere with that claim though, even if it were true and they could find evidence, because the company claims that they actively scan for and protect against this sort of thing, and even they admit that it was a DDoS attack.
Good to hear but it sounds like if the person hadn't gathered so much traction on HN they might still have been screwed.
CEO said that forgiving bills for this kind of a thing is a standard practice, but how come this was the customer support's first reaction:
We normally discount these kinds of attacks to about 20% of the cost, which would make your new bill $20,900. I've currently reduced it to about 5%, which is $5,225.
If the customer support has authority to give 20%/5% discounts, this seems to me like the standard practice, and the CEO is probably just doing damage control because this became public.
In this case, customer service is giving roughly 80% / 95% discounts. Which I think bolsters your point even further.
When I worked in customer service I think the largest i was ever able to issue was a 10% discount. Even with managerial approval I don't think I ever saw anyone get more than a 25% discount, and that was for legitimate complaints, not the Karen style made up whining.
Cases like that (even if the CEO takes pity and resolves it) are why I will always prefer to host on a VPS with a hard egress limit. Hit the limit (legitimately or from DDoS), network access is suspended until you take action (either wait for next billing cycle or buy another 10 TB for like $5-10). Can still log into virtual console to investigate, setup mitigations, etc.
Or better, unmetered. OVH might be a bit of a mess in many areas, but my server is unmetered. Doesn't matter if a VM starts mining crypto or if I get DDoS'd or someome just wants to waste my bandwidth. Network can be pegged 24/7 for all I care, same price in the end.
Hosting companies know they can make a lot of money with on demand pricing like that, and they love it because for the most part you can't do anything about it. If this was a company and not an individual, and the CEO didn't have pity, I'm sure they'd have tried their best to extract that 5k, maybe even 20k or whatever the sales representative thinks they can get out of you. It's crazy how the discounts become plentiful when it's obvious there's no way you can pay it all.
I've heard mixed reviews about OVH and heard they over-provision really badly. Does that cause you any issues / is that inaccurate in your experience?
Only one of my VPS's is truly unmetered, and it's the one I've had since like 2013 and is very, very grandfathered into a lot of perks . lol. I'm holding on to that one as long as I can. It's also got a squeaky-clean IPv4 address that has only ever been used by me.
The rest of the ones I run have a fairly high cap and only meter egress traffic. I think they're like 4 or 5 TB/mo on most of my plans. I've never hit anywhere near that limit even with one of them acting as my Lemmy CDN. Highest I've ever gotten was 75% which was enough to trigger a warning email.
I have a dedicated server, so they can't possibly overprovision that. Just load up the OS over IPMI and I manage the VMs and all. Been using them before AWS was a thing and couldn't be bothered, I like having all the control I can. I have a nice /29 of clean IPs with it that I've owned for 8 years as well.
OVH's IPv6 is total ass though. Don't even try, it's essentially unusable especially if you want to use more than like 8 single IPs of your /56. The routers crap themselves and forget about the rest because it's not a routed prefix, it sees it as if you have a single box claiming 8 IPs on itself.
I'm not sure I would use their cloud offerings. Renting old baremetal from them for cheap is much more price effective, especially if you can snag it on sale. And it also reduces waste by stopping those old boxes from being trashed and putting them to good use.
Renting old baremetal from them for cheap is much more price effective, especially if you can snag it on sale.
Going to have to look into that. I didn't know that was even an option. Good tip!
And it also reduces waste by stopping those old boxes from being trashed and putting them to good use.
In all fairness, they usually end up on eBay and then in my basement 😂
I suggest checking out their discount brands Kimsufi and SoYouStart. I pay like C$12/month for a dedicated server with a few cores, 8GB of RAM, and 2TB of hard drive space.
They used to link to my dig wrapper on my homepage for having their clients debug DNS problems for many years - even with translations of my UI in the various language help sites. I always found it amusing that a hoster of their size does that, instead of spending a lunchbreak to throw something together that integrates with their help page.
There also was a non significant number of users which didn't understand that my homepage had nothing to do with OVH, and ended up mailing me about their DNS problems.
Yeah but then you have a customer calling and screaming at you "We just launched our big sale of the year and our site has been down for an hour!!!!".
If you let them burst and bill them, you end up with angry clients. If you don't, you end up with angry clients. Letting them burst and being forgiving with the bill is the better approach IMHO.
You advice probably doesn't apply to the OP in the image, as a "simple static site" is probably their blog or project wiki. It's very unlikely they even have clients. For that case just having a hard limit and waiting is much safer.
I mean, I get email notifications as I'm approaching the threshold so I'm never caught off guard unless I ignore those. If everything's legit (e.g. no DDoS), I can just add extra egress bandwidth with no interruptions.
Is there a customer involved here?
After all if it's for a customer it might be better to just give them the choice since the bill is on them.
Why pay for a small VPS to host your static site when you can host it for free with a small chance of bankruptcy?
Seems like the person in the OP was using the free tier of that service, which had always been fine.
Active DDoS mitigation
Netlify monitors for traffic pattern anomalies and spikes, and effectively controls for them as needed.
https://www.netlify.com/security/
So is this just a lie? I have never used them and after this post I'm not going to be trying that anytime soon, if ever
Sounds like they violated contract to me.
Unfortunately, I think the way to make this right under our legal system is pay them, then sue them in small claims court to get the $5k back.
Which doesn’t work if OP doesn’t have $5k in cash ready to go.
Sounds like this was "resolved" on HN and CEO said this was an error, but I'm not so sure. The CEO's response seems to imply that that communication to/from service reps is true and not made up. The original post shows they have a business practice for cases like this. Plus if the company was willing to settle from their business practice of 20% down to 5% (which in this case was 15k) then that very likely isn't a decision a service rep could make, so you had some mid to upper level manager make that approval to write-off the $15k and decide that $5k was still owed to the company.
As far as I can tell the only error here is that someone posted about it.
Not to mention the CEO's response from HN just says this shouldn't have happened on free accounts, but that begs the question of would this have been any different on non-free accounts where Netlify failed to mitigate a DDoS as advertised?
Netlify has the highest network data transfer fees. If you're going to use them, you must have a CDN.
Yeah this is why I pay only for services with fixed fees (or that allow me to set a hard limit). Wow, 100k would bankrupt me completely.
Fear of this is why I've never relied on free tiers unless there's some kind of spend circuit breaker available.
Yeah it seems like such an obvious feature. May be there was one and the OOP didn't enable that.
I'm sure there not being one is the feature. Trapping people into the free tier and getting them on overages.
Of course for a hobby site that will never manage to pay this is not a good business model but I can see how this works for more moderate corporate use.
Without knowing any specifics of the TOS or the exact setup beyond what I could gather in this thread: generally speaking they could still send you a bill through email or otherwise.
After that, if you're not paying up, they might be able to successfully get the money out of you through court regardless, depending on a few factors. What's more likely for smaller sums is that they'll just drop it and ban you though.
IANAL of course.
Just because you are trying the free samples at a store, doesn't mean you can also take other food off the shelf without paying just because you left your wallet at home. Bandwidth still costs money.
I mean, I am fine with my hobby website being taken down if it starts to consume an unreasonable amount of bandwidth.
They don't need to give a free tier and I am not entitled to getting anything out of them. Since they have decided to offer one, I do expect to be consulted before sending a bill worth a ferrari.
Now's a great time to learn about rate limiting and honeypots. Even so, I hope you can talk them down further seeing as you don't sound like a large company.
