•

Nginx gets forked by core developer

https://mailman.nginx.org/pipermail/nginx-devel/2024-February/K5IC6VYO2PB7N4HRP2FUQIBIBCGP4WAU.html

33 comments

RobotToaster

•

Seems like open source can't go a week without drama caused by c-suite lately.

slazer2au

•

Seems like corporate greed can't go a week without enshitting on a open source project.

ysjet

•

Nah, c suite was pretty clearly in the right here. Dude left because he was pissed that a vulnerability got assigned a CVE instead of just... Not informing anyone so they could quietly fix it.

Bene7rddso

•

It's an experimental feature. It doesn't need a bugfix release because you're not supposed to run it in production, and it's just a DoS, not privilege escalation or something

ysjet

•

Experimental features are explicitly defined as requiring CVEs. You are supposed to run them in production, that's why they're available as expiermental features and not on a development branch somewhere. You're just supposed to run them carefully, and examine what they're doing, so they can move out of experiment into mainline.

And that requires knowledge about any vulnerabilities, hence why it's required to assigned CVEs to experimental features.

And I'm not sure why you think a DoS isn't a vulnerability, that's literally one of the most classic CVEs there are. A DoS is much, much more severe than a DDoS.

Bene7rddso

•

If you do examine what it's doing you will catch this as soon as an attacker exploits it, and can disable it. Also, you should maybe not run the entire production with experimental features enabled. In a stable feature this would absolutely be a CVE, but this is marked experimental because it might not work right or even crash, like here

ysjet

•

Correct, I agree you run it with an eye on it (which you should probably do anyway) instead of firing and forgetting (which, to nginx's credit, is typically stable enough you can do that just fine).

That said, nginx treats experimental as something you explicitly run in production- when they announced they added it into experimental they actually specifically say to run it in prod in an A/B setup.

https://www.nginx.com/blog/our-roadmap-quic-http-3-support-nginx/

Bene7rddso

•

If you run large‑scale Internet services,

That means if you're large enough that A can pick up the slack if B shits the bed. The only impact would be that you have to use HTTP2

ieatpillowtags

•

You’re completely full of shit, as it’s just simply not the case that you’re “supposed” to run experimental features in production. But run it carefully??? What does that even mean? You obviously have zero experience on either side of the equation.

ysjet

•

Nnnnnot really? Nginx literally asks for people to do that. Here they are announcing that experimental QUIC/HTTP3 support: https://www.nginx.com/blog/our-roadmap-quic-http-3-support-nginx/

You'll notice they literally outright ask

If you run large‑scale Internet services, experiment with A/B testing by deploying nginx-quic for some users or some services.

They literally ask for you to try it in production, so... yeah, you're supposed to use this experimental feature in prod. As for 'run it carefully', just that- don't just fire it off and assume all is good, keep an eye on it like the say. A/B deploy it, see how it works, and let NGINX know if you happen to run into anything.

Tartas1995

•

Have you looked into the CVE? Apparently it is a non issue. You could use it to dos a service that have an experimental feature enabled, which is disabled by default, on a non stable Version. I understand the dev. CVE should be for serious issues. And they alerted their users over an email list

It can be used for dos, as it is crashing workers, but they will be restarted anyway.

ysjet

•

There is an astounding number of lies in your post, good lord.

It is an issue. A DoS is a fairly serious vulnerability, and very much is a vulnerability.
Experimental features are explicitly defined to require their vulnerabilities to be assigned CVEs.
It is not just available on the stable version, but both commercially and via the open source version.
CVEs are not just for serious issues, they are for vulnerabilities. All vulnerabilities. It is a number that allows you to reference an vulnerability, nothing more, nothing less.
Mentioning a CVE on the mailing list is the absolute least they should be doing.
'workers can just be restarted anyway' shows a deep misunderstanding of what a worker does. Any pending or active transactions that worker had now hangs, meaning that the service is still being denied. Trying to recover automatically from a DoS does not mean the DoS is not happening- it just means that the DoS is slower to get rolling, or intermittently seems to work mid-DoS.

Tartas1995

•

There is an astounding number of lies/misrepresentations in your post, good lord.

I never said it isn't an issue. Dos is the issue. It is a vulnerability.
No. CVE are not required. Like never. There is no legal requirements. The c in CVE stands for common btw... You know what is not common, Experimental features on non stable releases.
The stables are not affected. To quote from https://www.nginx.com/blog/updating-nginx-for-the-vulnerabilities-in-the-http-3-module/ about cve-2024-24989, "NGINX Open source mainline version 1.25.4. (The latest NGINX Open source stable version 1.24.0 is not affected.)" And about CVE-2024-24990, "NGINX Open source mainline version 1.25.4. (The latest NGINX Open source stable version 1.24.0 is not affected.)"
Yes and no. Remember the c in cve?
How is it a lie to say that they informed people through a mail list, when they did that? Remember you said I was lying? Also didn't you say they wanted to keep it quiet to fix in secret, while they inform the public? Isn't that a lie? (Also, you call it a cve in this point, well the dev didn't think of it as one and he alerted the users. So they satisfied your "least" requirement for a cve while not thinking of it as a cve.)
My statement is once again not a lie. But let's talk about your stuck transaction. Your transaction isn't "stuck" if you use transactions in your database, but besides that you used an experimental feature on a non stable release on a publicly facing service and the "stuck" transaction is your issue? You are fucking without a condom, my friend. And That experimental feature might just crash randomly, due to memory leaks or what not, and your transaction is stuck too.

Where were my lies? I mean I showed you yours.

ysjet

•

I'm glad we agree a DoS is a vulnerability.
CVE best practices state that CVEs are required to be assigned to experimental features. F5's company policy is that CVE best practices are followed. F5 is the company that owns nginx. Therefore, it was required. Nice 'legal requirement' strawman. Also, 'Common' in this situation is not defined as 'Widespread; prevalent,' it's defined as 'Of or relating to the community as a whole; public.'
That was a typo regarding 'stable,' my bad. I meant to say 'It is just not available on stable, but is both via commercially and via the open source version.' However, it's still available on commercial versions and open source, and 'non-stable' versions are not inherently unstable, they're just called 'mainline'. Proof: https://nginx.org/en/download.html Stable is basically just 'long term support/LTS' versions of nginx.
Again, you are intentionally misusing the definitions of the word common. Lets see what MITRE has to say about it, hmm?

Common Vulnerabilities and Exposures (CVE) is a dictionary of common names (i.e., CVE Identifiers) for publicly known information security vulnerabilities. CVE's common identifiers make it easier to share data across separate network security databases and tools, and provide a baseline for evaluating the coverage of an organization's security tools. If a report from one of your security tools incorporates CVE Identifiers, you may then quickly and accurately access fix information in one or more separate CVE-compatible databases to remediate the problem.

Source: https://cve.mitre.org/about/

Yes, I would consider notifying the development mailing list as 'quietly' fixing it, as most all companies using it will not be on the development mailing list. It's meant to be an area for developers to discuss things. They didn't inform the public, they informed the devs.
Where are you getting database from? You've randomly pivoted into talking about database transactions then started babbling about how you somehow think using a production mainline release with production options on a fully supported commercial binary is somehow inherently unsafe, as though it wouldn't still be in dev or test.

Since you seem to have no idea about how web servers work, or indeed, experimental features, I'll let you in on a secret- The only difference between a non-experiemntal option in nginx and an experimental option is that they're unsure if they want that feature in nginx, and are seeing how many people are actually using it/interested in, or they think that usage patterns of the feature might indicate another, better method of implementation. "Experimental" does not mean "unfinished" or "untested."

If you know nothing about programming, CVEs, or even web engines, please stop embarrassing yourself by trying to trumpet ill-thought out bad takes on subjects you don't understand.

davel

•

Please don’t complain to us mod/admins about someone making things personal, when you’re the one calling someone a liar and a know-knowing about their field of work.

ysjet

•

Really dude? I never once devolved to name calling, I stated that s/he lied when s/he made false statements. What else am I supposed to say there?

I also don't understand how saying they doesn't know what the subject matter s/he's taking a stance on is 'know-knowing' either? S/He's straight up said they doesn't know what a CVE is, doesn't know what experimental means, and while they claims to be in this field of work, they doesn't know what a web worker is and confused a web transaction with a database transaction.

Sure, I could have been nicer about it when they started escalating, but I never made it personal, and have no intentions of doing so either.

EDIT: realized I was assuming their gender.

Tartas1995

•

Dude, can you be less rude? Calling me a liar, without point out a lie. At best, you found a misunderstanding of cve on my end which wouldn't be a lie and isn't in the part that you called a lie. Also I don't think that there was a misunderstanding on my end of what cve means. Then you call me basically a clueless idiot for not having a clue about web servers. While I actually currently am working for a multi billion dollars companies as a backend dev and never worked anything but web dev. Then you complain about a straw man when you don't bother to express what your actual argument was and I had to guess.

You might realize that I am not bothering to argue your points, there is a simple reason why, you are being a dick. Make your points clearly like you did just a moment ago and don't be rude while doing it and you get an interesting conversation.

In case, you are curious, I am actually rather neutral on whether or not, it should be cves. I see the devs reasons and think they are reasonable and I understand why f5 would report it. A new fork seems to be an overreaction though. I bet you didn't expect me to hold this position because you were busy being a dick instead of having a conversation

BaumGeist

•

Context:

TLDR: The devs don't like bugs in released software being assigned CVEs, which requires a special security update instead of a standard bugfix included in the regular update cycle.

:The most recent "security advisory" was released despite the fact
: that the particular bug in the experimental HTTP/3 code is
: expected to be fixed as a normal bug as per the existing security
: policy, and all the developers, including me, agree on this.
:
: And, while the particular action isn't exactly very bad, the
: approach in general is quite problematic.

There was no public discussion. The only discussion I'm aware of
happened on the security-alert@ list, and the consensus was that
the bug should be fixed as a normal bug. Still, I was reached
several days ago with the information that some unnamed management
requested an advisory and security release anyway, regardless of
the policy and developers position.

And nginx's announcement about these CVEs

Historically, we did not issue CVEs for experimental features and instead would patch the relevant code and release it as part of a standard release. For commercial customers of NGINX Plus, the previous two versions would be patched and released to customers. We felt that not issuing a similar patch for NGINX Open Source would be a disservice to our community. Additionally, fixing the issue in the open source branch would have exposed users to the vulnerability without providing a binary.

Our decision to release a patch for both NGINX Open Source and NGINX Plus is rooted in doing what is right – to deliver highly secure software for our customers and community. Furthermore, we’re making a commitment to document and release a clear policy for how future security vulnerabilities will be addressed in a timely and transparent manner.

cybersandwich

•

I...agree with the "company" I think. This sounds like dev sour grapes but what the company was asking them to do seems better from the customer pov and for cyber security I'm general.

Maybe I'm missing something.

fernandofig

•

This sounds like dev sour grapes but what the company was asking them to do seems better from the customer pov and for cyber security I'm general.

As a developer myself (though not on the level of these guys): sorry, but just, no.

The key point is this:

[...] we did not issue CVEs for experimental features and instead would patch the relevant code and release it as part of a standard release.

Emphasis mine. In software, features marked as "experimental" usually are not meant to be used in a production environment, and if they are, it's in a "do it at your own risk" understanding. Software features in an experimental state are expected to be less tested and have bugs - it's essentially a "beta" feature. It has a security bug? Though - you weren't supposed to be using it in a security-sensitive environment in the first place, it sounds perfectly reasonable to me that it should be addressed in a normal release as opposed to an out-of-band one.

We can argue if forking the project is or isn't extreme, but the devs absolutely have good reason to be pissed. This is typical management making decisions without understanding technical nuances and - from what is being told by the devs - not talking it through before doing it.

BaumGeist

•

For the record I agree with @fernandofig@reddthat.com, but I also want to add that a DoS is not necessarily a security risk. If it can be leveraged to expose sensitive information, then yes, that's a vulnerability; this isn't that.

Digging into the CVEs:

CVE-2024-24989:

#Security Advisory Description

When NGINX Plus or NGINX OSS are configured to use the HTTP/3 QUIC module, undisclosed requests can cause NGINX worker processes to terminate. (CVE-2024-24989)

Note: The HTTP/3 QUIC module is not enabled by default and is considered experimental. For more information, refer to Support for QUIC and HTTP/3.

#Impact

Traffic is disrupted while the NGINX process restarts. This vulnerability allows a remote unauthenticated attacker to cause a denial-of-service (DoS) on the NGINX system. There is no control plane exposure; this is a data plane issue only.

CVE-2024-24990 basically says the same.

Some choice clauses:

undisclosed requests can cause NGINX worker processes to terminate

Traffic is disrupted while the NGINX process restarts.

So it doesn't take down the server nor the parent process, it kills some threads which then... restart.

Note: The HTTP/3 QUIC module is not enabled by default and is considered experimental

I was able to find that the affected versions:

NGINX Plus R30 P2 and R31 P1
Open source subscription R5 P2 and R6 P1 Open source mainline version 1.25.4

but most importantly:

The latest NGINX Open source stable version 1.24.0 is not affected.

And saving me the hassle of linking and quoting all 5 of the version history pages for the affected products, the uniting factor is: they're all based on Open Source versions 1.25.*

None of them are using the latest stable version.

It's not even going to affect most sites, and definitely not ones for whom downtime is a major issue: they would not be using the non-stable version, much less enabling experimental features in a non-stable version.

But the part that irks me the most is the dillution of what a CVE is. Back in the day, it meant "something that can lead to security breaches," now it just seems to mean "hey guys, I found a bug." And that's bad because now you have one of two outcomes: 1. unnecessarily panicking users by leading them to believe their software is a security risk when it isn't, or 2. compromising the integrity and usability of CVE reports by drowing the important ones in waves of "look guys, the program crashes when I can leverage root privileges to send it SIGKILL!"

If this was just a bug hunter trying to get paid, that's one thing, but these were internally assigned and disclosed. This was an inside job. And they either ignored or never consulted the actual experts, the ones they have within their own staff: the devs.

Why? To what end? Did they feel left out, what with not having any CVEs since 2022? Does this play some internal political struggle chess move? Do they just hate the idea of clear and unambiguous communication of major security holes to the general public? Are they trying to disrupt their own users' faith in their paid products? Does someone actually think a DoS is the worst thing that can happen? Is there an upper level manager running their own 1.25 instance that needs this fixed out-of-band?

It's just all so asinine.

chameleon

•

Note: The HTTP/3 QUIC module is not enabled by default and is considered experimental

Do note that despite not being enabled by default, it is enabled in the official binary packages.

There's a funny amount of layers to this thing but as far as I'm concerned, if it's a feature you ship in the default binary packages on your site, that is definitively enough for a CVE even if it's disabled by default.

fernandofig

•

Thank you for digging this out. Turns out it's even worse than what I gleaned from my surface-level take.

NaoPb

•

Thanks for this. I get now why the devs are pissed.

ysjet

•

You're not missing anything, dude just threw a hissy fit because he's not the king of his fiefdom anymore.

merthyr1831

•

the CVE thing seems to be a straw that broke the camel's back if anything. it seems a bit fucky to expect a core maintainer to work on your project without pay because you wanted to look virtuous by firing them during the initial invasion of Ukraine.

I'm sure if they, yaknow, paid him, the corporate procedures he was still bound to wouldn't be so bad.

doubt freegnix will get far, mind you, but I don't think it's entirely fair to call his reaction "sour grapes"

cyborganism

•

Huh. I didn't even know F5 was Russian. I didn't even know there were behind nginx.

I'm so disconnected.

I'm also surprised to see F5 technologies being used even though it's Russian.

double_oh_walter

•

I don’t think f5 is Russian, but Maxim might be?

Also, they did not create nginx, but bought it a while back.

agent_flounder

•

You're not alone in any of that.

synae

•

Stuff like this is a great reminder about the power of Open Source. Even if it's inconvenient for the downstream user(/admin/etc), it contributes to strengthening software as a whole