Update the GB18030 encoder to spec-compliantly handle old PUA code
points via a direct byte lookup table (spec step 5). Bake the 18
GB18030-2022 code point updates into indexes.json and remove the
now-unnecessary patching logic from the code generator. Drop the
redundant hardcoded switch in the decoder's range function, as the
range formula already produces correct values.
Import WPT tests for gb18030 decoder, gb18030 encoder, and gbk
encoder, and register the worker variant in TestConfig.ini.
When the encoder encounters an unencodable code point while in jis0208
state, the spec says to emit ESC ( B (0x1B 0x28 0x42) to switch to
ASCII mode before returning an error. The encoder was incorrectly
emitting ESC ( J (0x1B 0x28 0x4A) which selects Roman mode instead.
This caused form submission using ISO-2022-JP to produce incorrect
escape sequences when replacing unencodable characters with numeric
character references.
Also imports the WPT iso2022jp-encode-form-errors-stateful test.
Introduce a StreamingDecoder wrapper that lets callers feed bytes to a
Decoder one chunk at a time. It buffers any incomplete trailing byte
sequence at the end of a chunk and prepends it to the next chunk, so a
multi-byte code point split across a chunk boundary is decoded correctly
once the next chunk arrives.
To support that, add an incomplete_tail_length() virtual on Decoder
returning the number of trailing bytes that form an incomplete sequence
per the Encoding Standard's decoder handler byte ranges, with overrides
for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP,
Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps
single-byte legacy decoders correct.
This is the foundation for the upcoming incremental HTML parser, which
needs to decode network response bodies as they arrive.
This is preparation for removing the endianness override, since it was
only used by a single client: LibTextCodec.
While here, add helpers and make use of simdutf for fast conversion.
Instead of asserting when you call TextCoded::decoder_for() with a
non-standard encoding name, let's be nice and see if we can't find a
decoder for the standardized version of the encoding name.
We're starting with a very basic decoding API and only ISO-8859-1 and
UTF-8 decoding (and UTF-8 decoding is really a no-op since String is
expected to be UTF-8.)
