This state will indicate to the media element that it's not guaranteed
to have a frame yet, for the purposes of determining the ready state.
JavaScript should be sure that video elements with a ready state of
HAVE_CURRENT_DATA or greater represent the current video frame already.
To allow the state to be exited if audio is disabled, audio tracks are
now only added to the buffering set on enable if the audio sink exists,
since without the sink starting the data provider, it will never be
removed.
This is a step towards making video ref tests.
This allows us to differentiate between having no data available yet,
having current data, and having future data. The main purpose of this
is to allow a new starting state to explicitly force HAVE_METADATA
instead of >= HAVE_CURRENT_DATA.
Note that the SeekingStateHandler returns Current instead of None. This
is deliberate, since the buffered ranges from the demuxer(s) can be
used to inform whether the possibly-current data is actually available
at the seek target.
Doing this in FFmpegVideoDecoder meant that the fallback to BT.709 with
unspecified transfer characteristics was taking precedence. Instead of
overriding unspecified to sRGB there, change the switch statement in
ColorSpace::from_cicp() so that the unspecified fallback and override
both live in the same place.
This should fix the contrast on a lot of YouTube videos.
Note that this now also affects images, which is consistent with
Chromium.
Storing these was pointless, since they're only used briefly to provide
the info needed to upload the buffers to the GPU.
For BT.2020 coefficients, we can just say that the bit depth is 16 bits
since we're always bit replicating to that for Skia's shaders.
This fixes a rare flake in HTMLMediaElement-load-after-decode-error,
where if the demuxer managed to get a block successfully but have its
reads in get_frames() aborted by the initial seek, it would skip the
single bad frame and never fire the error event.
Having PlaybackManager start in Buffering was causing us to report
a media element readyState of HAVE_CURRENT_DATA. HAVE_CURRENT_DATA
doesn't make a whole lot of sense for local files, since we should have
all the data immediately when we process the metadata. This is
reflected in the buffered attribute, so let's not limit the ready state
unecessarily.
Use mimalloc for Ladybird-owned allocations without overriding malloc().
Route kmalloc(), kcalloc(), krealloc(), and kfree() through mimalloc,
and put the embedded Rust crates on the same allocator via a shared
shim in AK/kmalloc.cpp.
This also lets us drop kfree_sized(), since it no longer used its size
argument. StringData, Utf16StringData, JS object storage, Rust error
strings, and the CoreAudio playback helpers can all free their AK-backed
storage with plain kfree().
Sanitizer builds still use the system allocator. LeakSanitizer does not
reliably trace references stored in mimalloc-managed AK containers, so
static caches and other long-lived roots can look leaked. Pass the old
size into the Rust realloc shim so aligned fallback reallocations can
move posix_memalign-backed blocks safely.
Static builds still need a little linker help. macOS app binaries need
the Rust allocator entry points forced in from liblagom-ak.a, while
static ELF links can pull in identical allocator shim definitions from
multiple Rust staticlibs. Keep the Apple -u flags and allow those
duplicate shim symbols for LibJS and LibRegex links on Linux and BSD.
This fixes a crash when a track is enabled and then disabled while a
seek is in progress.
The logic in SeekingStateHandler is reworked to keep track of the
tracks that are currently being seeked, and when a track is disabled,
it is no longer counted against the seek completion. Any seek
completion callback that was instated is cleared by calling seek with
a null callback.
It may be worth making a separate function on the data providers to
clear the current seek instead, to avoid the extra work of seeking, but
this scenario is a very rare one unless someone intentionally triggers
it, and the cost is minimal unless the toggles are spammed.
A crash test is included, which both tests for the crash, and would
also time out if the failing VERIFY in on_track_enabled() was avoided
with the previous seeking implementation, due to the originally-enabled
video track's seek callback being clobbered by on_track_enabled()'s
seek.
Audio output on macOS was consuming Core Audio resources until the
PlaybackStream creation took well over the timeout for some tests.
This was observed in media-source-buffered.html, where it would time
out due to the long-running callback on the main thread to create the
PlaybackStream for AudioMixingSink.
However, the AudioUnit init should definitely not be blocking the main
thread, so I've added a FIXME there.
This is delegated to the state handlers, but it essentially amounts to
`state() != Buffering && state() != Seeking`. If the PlaybackManager is
in either state, we know that there is no future data yet, as it should
exit those states as soon as the data is ready.
PlaybackManager then intersects all enabled tracks' buffered time
ranges. This will be used by the media element for the buffered
attribute and to update the ready state.
Apparently this function uses a bitrate heuristic to determine which
track is best. We don't want or need that, so just select the first
track with default disposition (e.g. FlagDefault=1 in Matroska).
...giving tracks a kind attribute, and renaming name to label.
Demuxers will need to determine the kind attribute, since the spec for
sourcing tracks requires us to select based on info we don't expose.
Most WebM files don't have their default duration defined, so we need
to parse the Opus frame header to determine the duration. This is
needed for buffered range calculation.
Instead of using a single track entry for all blocks in the file, use a
lookup to get the info needed to calculate the timestamp for the
specific track a block belongs to. No change in behavior for
SampleIterator, since that only returns blocks from the track that was
passed. This will be useful for MSE, since it demuxes all tracks at
once.
This will allow us to pass in a class implementing Demuxer for each
track owned by a MediaSource.
We'll also use the new ThreadPool class instead of a dedicated media
initialization thread. We shouldn't spin up a new thread for such a
trivial operation.
As soon as the promise is resolved, we need to unref it, so move it to
the deferred_invoke() callback. This allows PulseAudio streams to be
released properly and gives TestPlaybackStream a chance to pass instead
of timing out.
This was causing a flake on CI due to the race between the teardown of
the event loop and the rejection of the promise from another thread.
Regressed in 39d865b.
Previously, we would just listen to the single video track for
buffering, so if for some reason the audio data runs ahead of the
video, we would drop some audio until the video buffered. Instead,
stop playing audio at the last available sample when any provider is
blocked.
Also, PlaybackManager now starts in the Buffering state, so that it can
wait for enough data to be ready to play without interruption. When the
end of the stream is reached, the buffering state is exited to ensure
that we don't get stuck buffering at the end of a media file.
Hook up a callback in AudioMixingSink to notify PlaybackManager if the
output fails to be initialized. Then, when that happens, swap out the
time provider for GenericTimeProvider and continue without audio.
Fixes#8071
This allows us to avoid returning a PlaybackStream in cases where the
async initialization fails.
This is a step towards more graceful fallbacks when audio fails in
AudioMixingSink.
d146adf made the fetch callbacks use the media element via weak
references. This caused the `error` event not to fire on media elements
that are detached from the document and go out of scope, if the GC got
to them before the fetch completed.
Instead of relying on weak references in the callbacks, we can stop the
ongoing fetch when the document becomes inactive to allow it to be GCed
after that point. By storing the FetchData on the media element, we're
able to resume the fetch where it left off if the document becomes
active again.
We could potentially figure out a way to make elements with no event
handlers and no parent stop their fetches in order to be GCed sooner,
but that is probably a bit fiddly, so may not be worth it for now.
Fixes a rare flake in WPT's `html/semantics/embedded-content/media-
elements/error-codes/error.html` test. A test to force the bug using
`Internals::gc()` has been added.
