Add support for WPT test variants, which allow a single test file to be
run multiple times with different URL query parameters. Tests declare
variants using `<meta name="variant" content="?param=value">` tags.
When test-web encounters a test with variants, it expands that test into
multiple runs, each with its own expectation file using the naming
convention `testname@variant.txt` (e.g., `test@run_type=uri.txt`).
Implementation details:
- WebContent observes variant meta tags and communicates them to the
test runner via a new `did_receive_test_variant_metadata` IPC call
- test-web dynamically expands tests with variants during execution,
waking idle views after each test completion to pick up new work
- Use index-based test tracking to avoid dangling references when the
test vector grows during variant expansion
- Introduce TestRunContext to group test run state, and store a static
pointer to it for signal handler access
This enables proper testing of WPT tests that use variants, such as the
html5lib parsing tests (which test uri, write, and write_single modes)
and the editing/bold tests (which split across multiple ranges).
Introduce the HTMLSelectedContentElement and integrate it into
<select>, <option> and HTMLParser.
See whatwg/html#10548.
There are two bugs with WPT tests which causes the third subtest
in selectedcontent.html and selectedcontent-mutations.html fail.
See whatwg/html#11882, web-platform-tests/wpt#55849.
This implements parsing part of customizable <select> spec update.
See whatwg/html PR #10548.
Two failing subtests in `html5lib_innerHTML_tests_innerHTML_1.html`
and `customizable-select/select-parsing.html` are due to the spec
still disallowing `<input>` inside `<select>`, even though Chrome
has already implemented this behavoir (see whatwg/html#11288).
If multiple cross-document navigations are queued on
SessionHistoryTraversalQueue, running the next entry before the current
document load is finished may result in a deadlock. If the new document
has a navigable element of its own, it will append steps to SHTQ and
hang in nested spin_until.
This change uses promises to ensure that the current document loads
before the next entry is executed.
Fixes timeouts in the imported tests.
Co-authored-by: Sam Atkins <sam@ladybird.org>
There are two changes happening here: a correctness fix, and an
optimization. In theory they are unrelated, but the optimization
actually paves the way for the correctness fix.
Before this commit, the HTML tokenizer would attempt to look for named
character references by checking from after the `&` until the end of
m_decoded_input, which meant that it was unable to recognize things like
named character references that are inserted via `document.write` one
byte at a time. For example, if `∉` was written one-byte-at-a-time
with `document.write`, then the tokenizer would only check against `n`
since that's all that would exist at the time of the check and therefore
erroneously conclude that it was an invalid named character reference.
This commit modifies the approach taken for named character reference
matching by using a trie-like structure (specifically, a deterministic
acyclic finite state automaton or DAFSA), which allows for efficiently
matching one-character-at-a-time and therefore it is able to pick up
matching where it left off after each code point is consumed.
Note: Because it's possible for a partial match to not actually develop
into a full match (e.g. `¬indo` which could lead to `⋵̸`),
some backtracking is performed after-the-fact in order to only consume
the code points within the longest match found (e.g. `¬indo` would
backtrack back to `¬`).
With this new approach, `document.write` being called one-byte-at-a-time
is handled correctly, which allows for passing more WPT tests, with the
most directly relevant tests being
`/html/syntax/parsing/html5lib_entities01.html`
and
`/html/syntax/parsing/html5lib_entities02.html`
when run with `?run_type=write_single`. Additionally, the implementation
now better conforms to the language of the spec (and resolves a FIXME)
because exactly the matched characters are consumed and nothing more, so
SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN.
The new approach is also an optimization:
- Instead of a linear search using `starts_with`, the usage of a DAFSA
means that it is always aware of which characters can lead to a match
at any given point, and will bail out whenever a match is no longer
possible.
- The DAFSA is able to take advantage of the note in the section
`13.5 Named character references` that says "This list is static and
will not be expanded or changed in the future." and tailor its Node
struct accordingly to tightly pack each node's data into 32-bits.
Together with the inherent DAFSA property of redundant node
deduplication, the amount of data stored for named character reference
matching is minimized.
In my testing:
- A benchmark tokenizing an arbitrary set of HTML test files was about
1.23x faster (2070ms to 1682ms).
- A benchmark tokenizing a file with tens of thousands of named
character references mixed in with truncated named character
references and arbitrary ASCII characters/ampersands runs about 8x
faster (758ms to 93ms).
- The size of `liblagom-web.so` was reduced by 94.96KiB.
Some technical details:
A DAFSA (deterministic acyclic finite state automaton) is essentially a
trie flattened into an array, but it also uses techniques to minimize
redundant nodes. This provides fast lookups while minimizing the
required data size, but normally does not allow for associating data
related to each word. However, by adding a count of the number of
possible words from each node, it becomes possible to also use it to
achieve minimal perfect hashing for the set of words (which allows going
from word -> unique index as well as unique index -> word). This allows
us to store a second array of data so that the DAFSA can be used as a
lookup for e.g. the associated code points.
For the Swift implementation, the new NamedCharacterReferenceMatcher
was used to satisfy the previous API and the tokenizer was left alone
otherwise. In the future, the Swift implementation should be updated to
use the same implementation for its NamedCharacterReference state as
the updated C++ implementation.
There's a quirk in HTML where the parser should ignore any line feed
character immediately following a `pre` or `textarea` start tag.
This was working fine when we could peek ahead in the input stream and
see the next token, but didn't work in character-at-a-time parsing with
document.write().
This commit adds the "can ignore next line feed character" as a parser
flag that is maintained across invocations, making it work in this
parsing mode as well.
20 new passes in WPT/html/syntax/parsing/ :^)
Instead of always inserting a new text node, we now continue appending
to an extisting text node if the parser's character insertion point is
a suitable text node.
This fixes an issue where multiple invocations of document.write() would
create unnecessary sequences of text nodes. Such sequences are now
merged automatically.
19 new passes in WPT/html/syntax/parsing/ :^)
We were neglecting to return after handling the `frameset` start tag,
which caused us to process it twice, once properly and once generically.
54 new passes in WPT/html/syntax/parsing/ :^)
Before this change, the explicit EOF inserted by document.close() would
instantly abort the parser. This meant that parsing algorithms that ran
as part of the parser unwinding on EOF would never actually run.
591 new passes in WPT/html/syntax/parsing/ :^)
This exposed a problem where the parser would try to insert a root
<html> element on EOF in a document where someone already inserted such
an element via direct DOM manipulation. The parser now gracefully
handles this scenario. It's covered by existing tests (which would
crash without this change.)
Since we don't support the "variant" meta tag stuff in WPT, I've simply
copied the test files here, and then test.js looks at the filename to
figure out which test function to use.
This incrases our coverage of the HTML parser substantially by also
invoking it via document.write() one-shot, and character-at-a-time.
Previously, if the NumericCharacterReferenceEnd state was reached when
current_input_character was None, then the
DONT_CONSUME_NEXT_INPUT_CHARACTER macro would restore back before the
EOF, and allow the next state (after the SWITCH_TO_RETURN_STATE) to
proceed with the last digit of the numeric character reference.
For example, with something like `ї`, before this commit the
output would incorrectly be `<code point with the value 1111>1` instead
of just `<code point with the value 1111>`.
Instead of putting the `if (current_input_character.has_value())` check
inside NumericCharacterReferenceEnd directly, it was instead added to
DONT_CONSUME_NEXT_INPUT_CHARACTER, because all usages of the macro
benefit from this check, even if the other existing usage sites don't
exhibit any bugs without it:
- In MarkupDeclarationOpen, if the current_input_character is EOF, then
the previous character is always `!`, so restoring and then checking
forward for strings like `--`, `DOCTYPE`, etc won't match and the
BogusComment state will run one extra time (once for `!` and once
for EOF) with no practical consequences. With the `has_value()` check,
BogusComment will only run once with EOF.
- In AfterDOCTYPEName, ConsumeNextResult::RanOutOfCharacters can only
occur when stopping at the insertion point, and because of how
the code is structured, it is guaranteed that current_input_character
is either `P` or `S`, so the `has_value()` check is irrelevant.
This allows us to disable test output, which performs expensive assert
tracking. This was making our imported tests run significantly slower
than tests run via `WPT.sh`.
Formatting the output ourselves also allows us to remove unnecessary
information from the test output.
This commit also rebaselines all existing imported WPT tests to follow
the new format.
This isn't directly in the spec, but since replaceChild is implemented
in terms of remove + insert, the removal step may cause arbitrary code
to execute, and so we have to verify that the replaceChild inputs still
make sense afterwards, before doing the insertion.
This roughly matches what WebKit does, and makes a bunch of HTML parsing
tests in WPT stop asserting.
If we reach the insertion point at the same time as we switch to another
tokenizer state, we have to bail immediately without proceeding with the
next code point. Otherwise we'd fetch the next token, get an EOF marker,
and then proceed as if we're at the end of the input stream, even though
more data may be coming (with more calls to document.write()..)
We were neglecting to check the namespace when looking for a specific
type of element on the stack of open elements in many cases.
This caused us to confuse HTML and SVG elements.
Element::tag_name() returns an uppercased string for HTML elements,
which is usually not what's expected by the parser algorithms that look
at tag names.
It's actually possible for there to be no adjusted current node, when
the stack of open elements is empty. This was covered by one of the WPT
parsing tests.
