We need to store request headers in order to handle Vary mismatches.
(Note we should also be using BLOB for header storage in sqlite, as they
are not necessarily UTF-8.)
If the cache mode is no-store, we must not interact with the cache at
all.
If the cache mode is reload, we must not use any cached response.
If the cache-mode is only-if-cached or force-cache, we are permitted
to respond with stale cache responses.
Note that we currently cannot test only-if-cached in test-web. Setting
this mode also requires setting the cors mode to same-origin, but our
http-test-server infra requires setting the cors mode to cors.
This is expected by WPT. For this to work, we must be able to determine
the network partition key for shared worker environments. So we now set
a top-level origin for these environments, with a FIXME to implement it
in accordance with the Client-Side Storage Partitioning spec.
We had skipped some steps in the spec and were:
* Always broadcasting an old value of null, instead of what it
actually was previously.
* Still broadcasting a storage event even if the value had
not changed in storage compared to the last value.
Fix both issues by returning what the old value is in the setter and
implementing the missing logic.
We currently disable the disk cache because the WPT runner will run more
than one RequestServer process at a time. The SQLite database does not
handle this concurrent read/write access well.
We will now enable the disk cache with a per-process database. This is
needed to ensure that WPT Fetch cache tests are sufficiently handled by
RequestServer.
This adds support for intercepting network requests and serving local
file content instead. When a URL matches an entry in the substitution
map, the local file is served while preserving the original URL's
origin for cross-origin checks.
Usage:
Ladybird --resource-map=/path/to/map.json
The JSON file format is:
{
"substitutions": [
{
"url": "https://example.com/script.js",
"file": "/path/to/local/script.js",
"content_type": "application/javascript",
"status_code": 200
}
]
}
Fields:
- url (required): Exact URL to intercept (query string and fragment
are stripped before matching)
- file (required): Absolute path to local file to serve
- content_type (optional): Override Content-Type header (defaults to
guessing from filename)
- status_code (optional): HTTP status code (defaults to 200)
This is incredibly useful for debugging production websites: you can
intercept any script, stylesheet, or other resource and replace it with
a local copy containing your own debug instrumentation, console.log
statements, or experimental fixes - all without modifying the actual
site or setting up a local dev server.
- Add WindowManagement to PolicyControlledFeature enum
- Add screen_count() virtual method to PageClient
- Store all screen rects in WebContent::PageClient, derive both
screen_rect() and screen_count() from stored data
- Implement screen_count() overrides in SVGPageClient and PageHost
- Replace FIXME stub in Screen.cpp with spec-compliant implementation
Add support for WPT test variants, which allow a single test file to be
run multiple times with different URL query parameters. Tests declare
variants using `<meta name="variant" content="?param=value">` tags.
When test-web encounters a test with variants, it expands that test into
multiple runs, each with its own expectation file using the naming
convention `testname@variant.txt` (e.g., `test@run_type=uri.txt`).
Implementation details:
- WebContent observes variant meta tags and communicates them to the
test runner via a new `did_receive_test_variant_metadata` IPC call
- test-web dynamically expands tests with variants during execution,
waking idle views after each test completion to pick up new work
- Use index-based test tracking to avoid dangling references when the
test vector grows during variant expansion
- Introduce TestRunContext to group test run state, and store a static
pointer to it for signal handler access
This enables proper testing of WPT tests that use variants, such as the
html5lib parsing tests (which test uri, write, and write_single modes)
and the editing/bold tests (which split across multiple ranges).
To avoid unnecessary IPC traffic, we now only send network response
bodies when a DevTools client is connected.
This requires tracking DevTools connection state in ViewImplementation
so we can propagate it to new WebContent processes created during
cross-site navigation.
Propagate the request initiator type (e.g., "xmlhttprequest", "fetch",
"script", "stylesheet") from LibWeb through the IPC layer to DevTools.
This enables Firefox DevTools to correctly identify XHR/fetch requests
and display appropriate cause types in the Network panel's "Initiator"
column.
This adds support for viewing request payloads (POST data) and response
bodies in the Firefox DevTools network panel.
Request bodies are captured when network requests start and passed
through IPC to the NetworkEventActor, which returns them via the
getRequestPostData protocol method.
Response bodies are streamed via a new IPC message as data is received,
accumulated in NetworkEventActor (with a 10MB size limit to prevent
memory issues), and returned via getResponseContent. Text content is
returned as UTF-8, while binary content (images, etc.) is base64.
Previously, console messages were sent using an index-based system where
DevTools would be notified of new message indices and then request them
in batches. This created synchronization issues during page navigation
when the WebContent process resets while DevTools still has stale index
state.
This changes to a push-based model where console messages are sent
immediately as resources when they are logged, matching how Firefox
DevTools handles console messages. Each message is pushed through IPC
and forwarded to DevTools as a "console-message" or "error-message"
resource.
This eliminates the need for index tracking in FrameActor and simplifies
the entire console message pipeline from WebContent through to DevTools.
Hook ResourceLoader to emit network request lifecycle events through
IPC to the UI process, where FrameActor creates NetworkEventActor
instances that serialize requests using Firefox's Remote Debug Protocol.
The Network panel now shows requests with method, URL, status, MIME
type, size, and timing information. Several features remain stubbed
(POST data, response content, cause detection) marked with FIXMEs.
In the next commit, we will rely on global scope's URL being
initialized even earlier in the process of setting up the environment
for the Worker global. Move this hack just before that.
If the cURL request completes with anything other than CURLE_OK, we must
not keep the cache entry. For example, if the server's connection closes
while transferring data, we receive CURLE_PARTIAL_FILE. We don't want
this cache entry to be treated as valid in a subsequent request.
Implement PlaybackStream using WASAPI. The design is similar to
PlaybackStreamAudioUnit in that it uses a task queue. A high priority
thread is used to render the stream. All the stream controls save for
the exit being requested which happens on destruction of the stream are
managed by the render thread.
Due to the design of the windows audio mixer the audio we receive must
be resampled to match the sample rate of the mixer. We use a float based
interleaved PCM stream which matches both our existing code and the
audio mixer which internally usues floats.
Having to use a mutex around a queue for the task queue is suboptimal,
in a future PR a MPSC queue could be added to AK and used instead.
This directive allows our disk cache to serve stale responses for a time
indicated by the directive itself, while we revalidate the response in
the background.
Issuing requests that weren't initiated by a client is a new thing for
RequestServer. In this implementation, we associate the request with
the client that initiated the request to the stale cache entry. This
adds a "background request" mode to the Request object, to prevent us
from trying to send any of the revalidation response over IPC.
Not super important, but this will match an upcoming ID for stale-while-
revalidate requests. It also would techinically be UB if this ID had
ever overflowed. Let's make it 64-bit while we are here to also avoid
the possibility that it ever will overflow.
The bug fixed here is that in RequestServer's ConnectionFromClient, we
stored the websocket ID as i32, despite it being i64 everywhere else.
Let's make it unsigned while we are here to be consistent with how all
other RS-related IDs will be soon.
Computing the font for an element in `compute_font` is premature since
we are yet to apply animated properties - instead we should compute the
value on the fly (with a cache to avoid unnecessary work) to ensure we
are respecting the latest values
The system uses ssize_t so it can return -1 in case of an error. But in
our case, we will transform that to an AK::Error, thus we never return
-1. Let's return size_t instead.
This is largely editorial. One behaviour change is that events are now
sent from a global task on the DOMManipulation task source.
Somewhat awkwardly, the spec refers to `this` before the Worker exists.
As it's for getting the relevant global object / settings object, I've
had to work around that.
Corresponds to:
917c2f6a73
SocketAddressWindows.h contains (re)definitions of a bunch of system-
level types. This leads to a bunch of conflicts when trying to include
proper Windows headers in SystemWindows.cpp. This patch removes this
inclusion and just forward-declares the couple of types we need.
We currently have two ongoing implementations of RFC 9111, HTTP caching.
In order to consolidate these, this patch moves the implementation from
RequestServer to LibHTTP for re-use within LibWeb.
The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP
caching) implementation. We currently have one implementation in LibWeb
for our in-memory cache and another in RequestServer for our disk cache.
The implementations both largely revolve around interacting with HTTP
headers. But in LibWeb, we are using Fetch's header infra, and in RS we
are using are home-grown header infra from LibHTTP.
So to give these a common denominator, this patch replaces the LibHTTP
implementation with Fetch's infra. Our existing LibHTTP implementation
was not particularly compliant with any spec, so this at least gives us
a standards-based common implementation.
This migration also required moving a handful of other Fetch AOs over
to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP
folder, so perhaps it makes sense for LibHTTP to be the implementation
of that entire set of facilities.)
If multiple cross-document navigations are queued on
SessionHistoryTraversalQueue, running the next entry before the current
document load is finished may result in a deadlock. If the new document
has a navigable element of its own, it will append steps to SHTQ and
hang in nested spin_until.
This change uses promises to ensure that the current document loads
before the next entry is executed.
Fixes timeouts in the imported tests.
Co-authored-by: Sam Atkins <sam@ladybird.org>
We currently have a FIXME to validate cached data with a crc32. But this
is sort of a non-starter, because we never actually have the cached data
in-memory - we transfer it to the WebContent process via system calls,
and it never reaches userspace in RequestServer.
Chrome makes a bit of an educated gamble here. They assume cosmic bit
blips are extremely unlikely, thus the cached data does not get verified
with a hash. Instead, they store non-cryptographic hashes of some select
fields, and they validate just those hashes.
Here, we store a hash of the cache key in the cache header, and a hash
of the cache header in the cache footer. With these validations, along
with other validations already in-place, we can be reasonably sure we
are not sending corrupt data to the WebContent process.
This effectively reverts 9b8f6b8108.
I misunderstood what Chrome was doing here - they will issue a range
request only for what they call "sparse" cache entries. These entries
are basically used to cache partial large file, e.g. a multi-gigabyte
video. If they hit a legitimate read error, they will fail the request
with a ERR_CACHE_READ_FAILURE status.
We will now (again) fail with a network error when a cache read fails.
For example, we will want to be able to test that a cached object was
expired after N seconds. Rather than waiting that time during testing,
this adds a testing-only request header to internally advance the clock
for a single HTTP request.
This mode allows us to test the HTTP disk cache with two mechanisms:
1. If RequestServer is launched with --http-disk-cache-mode=testing, it
will cache requests with a X-Ladybird-Enable-Disk-Cache header.
2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response
header indicating how the response was handled by the cache. There is
no standard way for a web request to know what happened with respect
to the disk cache, so this fills that hole for testing.
This mode is not exposed to users.
This is causing misaligned reads with Address Sanitizer enabled. We
could maintain the packed attribute, and deal with alignment - but we
ended up not actually needing these to be packed anyways. The only thing
we need to know is the serialized size of the header, which we can just
determine differently.
We previously excluded headers exempted from storage when we serialized
the headers into the database. However, we stored the original headers
in-memory. So when a subsequent request hit CacheIndex::find_entry, we
would return an entry with response headers that should have been
excluded.
