Apache HTTPd Devs Considered Harmful
Joe Schaefer
Background
For the past 25 years, I have been the lead developer of the apreq subproject within the Apache HTTPd Server Parent Project. The original idea of libapreq, as a safe/performant HTML form submission and Cookie parsing library, came out of a collaboration between Lincoln Stein and Doug MacEachern in the late 90s.
It was my vision back then to transform the library into a generic, non-Perl related C library that would support language bindings from other programming languages, which is why I pushed for the project to be homed under the HTTPd umbrella instead of the Apache-Perl project.
With the advent of httpd-2.X, an entirely new I/O Filter architecture emerged from httpd core, as well as the complete separation of APR from the core itself as a more general purpose POSIX-like portability runtime for C projects like Subversion. In fact, libapreq2 is more closely aliged with the Apache APR project in that spirit, and its Perl API reflects that as part of its APR::Request buildout. It has a built-in CGI mode for standalone operation, outside of the httpd runtime, which makes unit testing a breeze.
Yet the key component of apreq2 has always been the mod_apreq2 Apache module, which was first conceived by Bill Wrowe in the early 2000s. What he designed, during a brainstorming session with me (in person), was a single parser library internal to httpd, that shared the submitted request body with every key stakeholder module in the runtime. That meant providing parsed data to modules hooked into the request processing engine before, during, and after the content handler runs. And it needed to work for subrequests as well, regardless of whether or not the content handler consumed the parsed data, or consumed and reparsed the raw request body itself.
I explained the design goals several times over the years, even in 2012 on dev@httpd. But it was always like talking to the wind with these guys; they just never cared.
Storm Clouds Gathering
While this vision was wildly successful, with language bindings available for several languages like Perl, PHP, TCL, R, etc., ever since about 2010 it has proven tragic for the existing user community consisting of all of them, not just the members of the Perl community.
What happened? Philip Gollucci, a Perl/FreeBSD colleague of mine at the time, started agitating that we promote the project to be released from inside the HTTPd server itself. What Philip didn’t know very well back then was how utterly peevish, vapid, and territorial that team had become, which would have meant having to collaborate with them directly on user-facing decisions about the code base.
In 2012, Philip got what he wanted and I stopped resisting, so he forked the existing project and copied the C library components into HTTPd core.
Fallout
In 2018 I resigned from the Foundation en masse. You can guess the reasons.
In 2020 or so, Google’s Security Team took advantage of an alpha release of httpd 2.5 by fuzzing its 8 year old copy of apreq2. They found a few hotspots that needed repair.
Instead of having the courtesy of reaching out to Philip, Issac Goldstand, Max Kellermann (@MaxKellermann), myself (@joesuf4), or anyone else involved in the development of libapreq2, a junior engineer on the HTTPd team went about the business of “bug fixing” the vulnerabilities Google found. You can see a record of his trial and error work in every release since then.
But the coup de grace was the 2022 release of 2.17, wherein the rookie developer purposely introduced a fatal bug into the codebase, breaking a nineteen year old regression test.
Postmortem
If you are wondering how something with a broken regression test winds up on CPAN, you’ll have to look into how RELENG is done in the server project.
Long story short, they commented out the test and shipped it anyway, and called it a Security Release that fixed a vulnerability every prior release was susceptible to.
Why do I care now? Because I’m the sucker users reach out to for answers as a known subject matter expert.
This sucks, but I’m sorry to tell you that my days wearing the Superman cape at Apache ended about a decade ago.