Some web servers (like azure.com) are throttling us unless we claim
to be a more recent version of Chrome.
Before this change, every HTTP request would take up to 30 seconds
before receiving a response. After the change, azure.com responds
instantly.
This is very silly, but it's the game we have to play.
Instead of passing RequestServer and ImageDecoder socket FDs as
command-line arguments to WebContent, send them over the main IPC
channel after launch. This unifies initial connection and reconnection
into a single code path.
We currently attach HTTP cookie headers from LibWeb within Fetch. This
has the downside that the cookie IPC, and the infrastructure around it,
are all synchronous. This blocks the WebContent process entirely while
the cookie is being retrieved, for every request on a page.
We now attach cookie headers from RequestServer. The state machine in
RequestServer::Request allows us to easily do this work asynchronously.
We can also skip this work entirely when the response is served from
disk cache.
Note that we will continue to parse cookies in the WebContent process.
If something goes awry during parsing. we limit the damage to that
process, instead of the UI or RequestServer.
Also note that WebSocket requests still have cookie headers attached
attached from LibWeb. This will be handled in a future patch.
In the future, we may want to introduce a memory cache for cookies in
RequestServer to avoid IPC altogether as able.
The cookie RFC strongly suggests that cookies only contain ASCII, and
that non-ASCII values be encoded with e.g. base64. Web reality differs,
however, and browsers are expected to support UTF-8 encoded cookies.
This aligns with document.cookie.
This fixes the following WPT tests:
/cookies/encoding/charset.html
/cookiestore/encoding.https.any.html
No test added here because we don't have a mechanism yet to set an
HTTP cookie and inspect it via our file:// URL test infra.
It's possible for the cookie value from a Set-Cookie header to contain
invalid UTF-8. We must isomorphic decode this header.
This fixes the /cookies/domain/domain-attribute-idn-host.sub.https.html
WPT test. The test added here is a crash test rather than a text test
because we cannot access the received Set-Cookie header from JS on the
file:// test URL.
We recently added "Chrome/140.0.0.0" to our User-Agent string which
fixed an issue with some web servers ignoring or throttling us.
Unfortunately we were still being served crappy version of many
major websites.
This patch improves the situation by adding more lies, specifically
"AppleWebKit/537.36 Safari/537.36". This gives us modern versions
of sites like Google, GMail, Instagram, and many more.
If the cache mode is no-store, we must not interact with the cache at
all.
If the cache mode is reload, we must not use any cached response.
If the cache-mode is only-if-cached or force-cache, we are permitted
to respond with stale cache responses.
Note that we currently cannot test only-if-cached in test-web. Setting
this mode also requires setting the cors mode to same-origin, but our
http-test-server infra requires setting the cors mode to cors.
This is a sad compatibility hack, but it's unfortunately necessary for
many prominent websites to work in Ladybird.
With our previous UA string, we've observed things like:
- Getting HTTP 403 for every request
- Being served "your browser is not supported" pages
- Being served outdated/broken content
- Extreme network throttling
- Other annoyances that magically go away if we change the UA string
By playing along with the silly game and putting Chrome in the string,
most such problems disappear instantly and anyone trying to use Ladybird
has a much better time.
Note that Ladybird *is* still in the UA string as well, so you can
easily identify us if you want to.
Propagate the request initiator type (e.g., "xmlhttprequest", "fetch",
"script", "stylesheet") from LibWeb through the IPC layer to DevTools.
This enables Firefox DevTools to correctly identify XHR/fetch requests
and display appropriate cause types in the Network panel's "Initiator"
column.
This adds support for viewing request payloads (POST data) and response
bodies in the Firefox DevTools network panel.
Request bodies are captured when network requests start and passed
through IPC to the NetworkEventActor, which returns them via the
getRequestPostData protocol method.
Response bodies are streamed via a new IPC message as data is received,
accumulated in NetworkEventActor (with a 10MB size limit to prevent
memory issues), and returned via getResponseContent. Text content is
returned as UTF-8, while binary content (images, etc.) is base64.
Hook ResourceLoader to emit network request lifecycle events through
IPC to the UI process, where FrameActor creates NetworkEventActor
instances that serialize requests using Firefox's Remote Debug Protocol.
The Network panel now shows requests with method, URL, status, MIME
type, size, and timing information. Several features remain stubbed
(POST data, response content, cause detection) marked with FIXMEs.
The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP
caching) implementation. We currently have one implementation in LibWeb
for our in-memory cache and another in RequestServer for our disk cache.
The implementations both largely revolve around interacting with HTTP
headers. But in LibWeb, we are using Fetch's header infra, and in RS we
are using are home-grown header infra from LibHTTP.
So to give these a common denominator, this patch replaces the LibHTTP
implementation with Fetch's infra. Our existing LibHTTP implementation
was not particularly compliant with any spec, so this at least gives us
a standards-based common implementation.
This migration also required moving a handful of other Fetch AOs over
to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP
folder, so perhaps it makes sense for LibHTTP to be the implementation
of that entire set of facilities.)
The only thing in HTTPResponse being used is reason_phrase_for_code,
which is just a static helper method. Move it to its own file and remove
HTTPResponse.
This is just one less thing to have to port to an upcoming HTTP header
refactor.
We could probably do with removing LoadRequest altogether. But this just
removes unused methods for now to make an upcoming HTTP header change a
bit simpler.
Disallow calling `StringBase::bytes()` on temporaries to avoid returning
`ReadonlyBytes` that outlive the underlying string.
With this change, we catch a real UAF:
`load_result.data = maybe_response.release_value().bytes();`
All other updated call sites were already safe, they just needed to use
an intermediate named variable to satisfy the new lvalue-only
requirement.
Previously, unbuffered requests were only available as a special mode
for EventSource. With this change, they are enabled by default, which
means chunks can be read from the stream as soon as they arrive.
This unlocks some interesting possibilities, such as starting to parse
HTML documents before the entire response has been received (that, in
turn, allows us to initiate subresource fetches earlier or begin
executing scripts sooner), or start rendering videos before they are
fully downloaded.
Co-authored-by: Timothy Flynn <trflynn89@pm.me>
Global Privacy Control aims to be a replacement for Do Not Track. DNT
ended up not being a great solution, as it wasn't enforced by law. This
actually resulted in the DNT header serving as an extra fingerprinting
data point.
GPC is becoming enforced by law in USA states such as California and
Colorado. CA is further working on a bill which requires that browsers
implement such an opt-out preference signal (OOPS):
https://cppa.ca.gov/announcements/2025/20250911.html
This patch replaces DNT with GPC and hooks up the associated settings.
JavaScript module requests (in a non-worker context) always have CORS
enabled. However, CORS requests are only allowed for same-origin or
HTTP/S requests. This patch extends this to allow resource:// requests
from opaque origins (e.g. about: URLs).
We must also set the Access-Control-Allow-Origin header to "null" to
ensure that the response is accepted by the CORS checks. This does not
affect requesting resource:// URLs from resource:// URLs as those are
same-origin and skip CORS checks.
This ultimately enables requesting resource:// JS modules from the
about:settings page.
Start work on a speculative HTML Parser in Swift. This component will
walk ahead of the normal HTML parser looking for fetch() requests to
make while the normal parser is blocked. This work exposed many holes in
the Swift C++ interop component, which have been reported upstream.
This commit:
- Prevents path traversal via the about: scheme
- Prevents loading about:inspector
- Requires about: URIs to be opaque paths
- Prevents crashes with invalid percent encoded paths
