We can use Duration as-is without coercing values to seconds. This
probably doesn't make much difference in real-world scenarios, but when
we hammer the cache during tests, this truncation will cause flakey
behavior.
The Windows RequestPipe implementation uses a non blocking local socket
pair, which means the non-fatal "resource is temporarily unavailable"
error that can occur in the non-blocking HTTP Response data writes can
be retried. This was seen often when loading https://ladybird.org.
While the EAGAIN errno is defined on Windows, WSAEWOULDBLOCK is the
error code returned in this scenario, so we were not detecting that we
could retry and treated the failed write attempt as a proper error.
We now detect WSAEWOULDBLOCK and convert it into the errno equivalent
EWOULDBLOCK. There is precedent for doing a similar conversion in the
Windows PosixSocketHelper::read() implementation.
Finally, we retry when we receive either EAGAIN or EWOULDBLOCK error
codes on all platforms. While POSIX allows these 2 error codes to have
the same value, which they do on Linux according to
https://www.man7.org/linux/man-pages/man3/errno.3.html, it is not
guarenteed. So we now ensure platforms that return EWOULDBLOCK with a
value different than EAGAIN also perform write retries.
This commit extends is_cacheable() to allow storage of responses that
rely on heuristic cacheability, including status codes defined as
heuristically cacheable.
We implement heuristic freshness lifetime calculation based on the
Last-Modified header as guidance, and apply it when no explicit
expiration information is present.
We were treating Content-Type as exempt from header updates during cache
revalidation and incorrectly allowing Content-Length to get overwritten
when handling an HTTP 304 response.
This caused cached entries to end up with a mismatched Content-Length
that described the validating response instead of the stored body.
We currently store ByteString as TEXT, but that is interpreted by SQLite
as UTF-8. All string-related operations will try to operate on the text
as if it is stored with UTF-8. We should use BLOB instead, which does
not make any encoding assumptions.
The only user of ByteString in the database currently is the HTTP disk
cache. So we bump its version here to remove existing data. Again, we
will want to handle changes like this more gracefully eventually.
Per RFC 9111, we're allowed to cache HTTP responses that don't have a
"Cache-Control" header, provided they do have an "Expires" header.
This lets us cache JavaScript resources on https://x.com/ and makes
it load much faster when cached.
This change moves the initial alpha premultiplication step for all
decoded images from WebContent to the ImageDecoder process. This
doesn't reduce the overall amount of work, but it can make sites with a
lot of images more responsive.
Prevents observably calling Trusted Types, which can run arbitrary JS,
cause crashes due to use of MUST and allow arbitrary JS to modify
internal elements.
We set bInheritHandles to TRUE for all child processes we spawn. Some
of the types of objects that support handle inheritence is all of the
STD handles (STD_INPUT_HANDLE, STD_OUTPUT_HANDLE, STD_ERROR_HANDLE) and
the console screen buffer. This means if Ladybird and all the child
service processes it launches/communicates with our console apps, only
a single console needs to be allocated and all child process output
their logs to that single console.
When a request becomes stale, we will now issue a revalidation request
(if the response indicates it may be revalidated). We do this by issuing
a normal fetch request, with If-None-Match and/or If-Modified-Since
request headers.
If the server replies with an HTTP 304 status, we update the stored
response headers to match the 304's headers, and serve the response to
the client from the cache.
If the server replies with any other code, we remove the cache entry.
We will open a new cache entry to cache the new response, if possible.
We currently store response headers in the cache entry file, before the
response body. When we implement cache revalidation, we will need to
update the stored response headers with whatever headers are received
in a 304 response. It's not unlikely that those headers will have a size
that differs from the stored headers. We would then have to rewrite the
entire response body after the new headers.
Instead of dealing with those inefficiencies, let's instead store the
response headers in the cache index. This will allow us to update the
headers with a simple SQL query.
This currently just holds a cache version. If we make a breaking change
to the cache format, we can increment the version here to wipe the cache
database. This is certainly a blunt hammer approach; we will want to
handle such changes more gracefully when we can.
In the initial work that got Ladybird running on Windows, there were
some DLLs that WebContent implicitly depended on which was causing
runtime errors when launching as they didn't exist in libexec.
So the workaround was to explicitly link the targets that had issues to
WebContent and use lagom_copy_runtime_dlls() to ensure they got copied
to libexec.
But given libexec is not a standard Windows convention, in a later
review we made sure Services got output to the bin folder, but those
initial workarounds were not removed even though they were now
unnecessary.
The function currently has 2 purposes: (1) To copy dependent dlls for
executables to output binary directory. This ensures that these helper
processes can be ran after a build given not all DLLs from vcpkg libs
get implicitly copied to the bin folder. (2) Allow fully background
and/or GUI processes to use the Windows Subsystem. This prevents
unnecessarily launching a console for the process, as we either require
no user interaction or the user interaction is all handled in the GUI.
The Win32 API equivalent to pipe2() is CreatePipe(), which creates read
and write anonymous pipe handles that we can set to non-blocking via
SetNamedPipeHandleState(); however, this initial approach caused issues
as our Windows infrastructure assumes socket-based handles/fds and that
we don't use Windows pipes at all, see Core::System::is_socket() in
SystemWindows.cpp. So we use socketpair() to keep our current
assumptions true.
Given that Windows uses socketpair() and Unix uses pipe2(), this
RequestPipe abstraction avoids ifdef soup by hiding the details about
how the read/write fds pair is created and how response data is written
to the client.
Instead of having ExecutionContext track function names separately,
we give FunctionObject a virtual function that returns an appropriate
name string for use in call stacks.
We previously had no protection against the same URL being requested
multiple times at the same time. For example, if a URL did not have any
cache entry and became requested twice, we would open two cache writers
concurrently. This would result in both writers piping the response to
disk, and we'd have a corrupt cache file.
We now hold back requests under certain scenarios until existing cache
entries have completed:
* If we are opening a cache entry for reading:
- If there is an existing reader entry, carry on as normal. We can
have multiple readers.
- If there is an existing writer entry, defer the request until it is
complete.
* If we are opening a cache entry for writing:
- If there is an existing reader or writer entry, defer the request
until it is complete.
This object will be needed in a future commit to store requests awaiting
other requests to finish. Doing this in a separate commit just to make
that commit less noisy.
We previously waited until we received all response headers before we
would create the cache entry. We now create one immediately, and handle
writing the headers in its own function. This will allow us to know if
a cache entry writer already exists for a given cache key, and thus
prevent creating a second writer at the same time.
We currently manage request lifetime as both an ActiveRequest structure
and a series of lambda callbacks. In an upcoming patch, we will want to
"pause" a request to de-duplicate equivalent requests, such that only
one request goes over the network and saves its response to the disk
cache.
To make that easier to reason about, this adds a Request class to manage
the lifetime of a request via a state machine. We will now be able to
add a "waiting for disk cache" state to stop the request.
This will allow more easily using these from other files. This also lets
us hide the Windows.h header necessity in a single location, instead of
needing to remember to include it everywhre we would otherwise include
<curl/curl.h>.
This is used by tests to set the default time zone to UTC.
This is because certain tests create JavaScript Date objects, which are
in the current timezone.
If transferring a cached response body fails for any reason, we will now
issue a network request instead of failing the request outright.
The catch here is that we will have already transferred the response
code and headers to the client, and potentially some of the body. So we
attempt to only request the remaining data over the network using a
range request. This feels a bit sketchy, but this is also how Chromium
behaves.
However, the server may or may not support range requests. If they do,
we can expect an HTTP 206 response with the bytes we need. If not, we
will receive an HTTP 200 (assuming the request succeeded), along with
the entire object's body. In this case, we also behave like Chromium,
and internally drop number of bytes we had already transferred.
If we are unable to pipe the response body from a cache file to the
client, let's take the extra safe approach of deleting the cache file
for now. We already remove the file if we weren't able to read its
metadata during initialization.
This is a bit of a blunt hammer, but this hooks an action to clear the
HTTP disk cache into the existing Clear Cache action. Upon invocation,
it stops all existing cache entries from making further progress, and
then deletes the entire cache index and all cache files.
In the future, we will of course want more fine-grained control over
cache deletion, e.g. via an about:history page.
This adds a disk cache for HTTP responses received from the network. For
now, we take a rather conservative approach to caching. We don't cache a
response until we're 100% sure it is cacheable (there are heuristics we
can implement in the future based on the absence of specific headers).
The cache is broken into 2 categories of files:
1. An index file. This is a SQL database containing metadata about each
cache entry (URL, timestamps, etc.).
2. Cache files. Each cached response is in its own file. The file is an
amalgamation of all info needed to reconstruct an HTTP response. This
includes the status code, headers, body, etc.
A cache entry is created once we receive the headers for a response. The
index, however, is not updated at this point. We stream the body into
the cache entry as it is received. Once we've successfully cached the
entire body, we create an index entry in the database. If any of these
steps failed along the way, the cache entry is removed and the index is
left untouched.
Subsequent requests are checked for cache hits from the index. If a hit
is found, we read just enough of the cache entry to inform WebContent of
the status code and headers. The body of the response is piped to WC via
syscalls, such that the transfer happens entirely in the kernel; no need
to allocate the memory for the body in userspace (WC still allocates a
buffer to hold the data, of course). If an error occurs while piping the
body, we currently error out the request. There is a FIXME to switch to
a network request.
Cache hits are also validated for freshness before they are used. If a
response has expired, we remove it and its index entry, and proceed with
a network request.
I noticed the existing code would end up calling
`computed_properties->property(PropertyID::Custom)`
so let's actually ask for the custom property instead.
Global Privacy Control aims to be a replacement for Do Not Track. DNT
ended up not being a great solution, as it wasn't enforced by law. This
actually resulted in the DNT header serving as an extra fingerprinting
data point.
GPC is becoming enforced by law in USA states such as California and
Colorado. CA is further working on a bill which requires that browsers
implement such an opt-out preference signal (OOPS):
https://cppa.ca.gov/announcements/2025/20250911.html
This patch replaces DNT with GPC and hooks up the associated settings.
