regex engine - compile tries using utf8 octets by demerphq · Pull Request #24712 · Perl/perl5

demerphq · 2026-08-15T19:48:38Z

The trie compiler now builds trie transitions from the octets of the
UTF-8 representation of the pattern. This gives every trie the same
256-octet alphabet and removes the need to represent Unicode codepoints
as entries in a potentially very large transition alphabet.

There is a special case for patterns whose codepoints are all ASCII: the
ASCII and UTF-8 representations use the same octets, so they naturally
share the same trie representation. Other low-page and native-string
cases are handled as encoding details around this common UTF-8-byte
representation, rather than as different trie formats.

Encoding-specific matching rules, including native versus UTF-8 behavior
and case folding, are handled independently of the fact that trie
transitions are stored as UTF-8 octets. This keeps the representation
uniform while preserving the existing matching semantics.

Trie construction now uses the list representation throughout and
converts it directly to the compressed transition form used by the
executor. The old flat table construction path, character maps, reverse
character map, wide character map, and inline-charclass trie op variants
are removed. This eliminates a substantial amount of special handling
for Unicode transitions and makes the implementation easier to maintain
and prove correct.

Trie indexes and related counters are widened to U32 where required,
while the debug bitmap is kept as U8 data. Prefix extraction tracks
encoded-byte and source-codepoint lengths so that UTF-8 prefixes are not
split in the middle of a codepoint. Debug trie dumps now describe the
octet alphabet and omit columns that are not present in the pattern.

The regular expression internals version now identifies the octet-trie
implementation, and the perldelta and regex internals documentation
describe the new representation. Tests cover native and UTF-8 byte
tries, updated diagnostics, and benchmark the scaling of large
alternations with different amounts of padding.

This set of changes requires a perldelta entry, and it is included.

For now anyway. Maybe in the future we should just switch to double compiling, or maybe compiling on demand, and generate both a utf8 octet trie and a latin-1 version as well.

The trie compiler now builds trie transitions from the octets of the UTF-8 representation of the pattern. This gives every trie the same 256-octet alphabet and removes the need to represent Unicode codepoints as entries in a potentially very large transition alphabet. There is a special case for patterns whose codepoints are all ASCII: the ASCII and UTF-8 representations use the same octets, so they naturally share the same trie representation. Other low-page and native-string cases are handled as encoding details around this common UTF-8-byte representation, rather than as different trie formats. Encoding-specific matching rules, including native versus UTF-8 behavior and case folding, are handled independently of the fact that trie transitions are stored as UTF-8 octets. This keeps the representation uniform while preserving the existing matching semantics. Trie construction now uses the list representation throughout and converts it directly to the compressed transition form used by the executor. The old flat table construction path, character maps, reverse character map, wide character map, and inline-charclass trie op variants are removed. This eliminates a substantial amount of special handling for Unicode transitions and makes the implementation easier to maintain and prove correct. Trie indexes and related counters are widened to U32 where required, while the debug bitmap is kept as U8 data. Prefix extraction tracks encoded-byte and source-codepoint lengths so that UTF-8 prefixes are not split in the middle of a codepoint. Debug trie dumps now describe the octet alphabet and omit columns that are not present in the pattern. The regular expression internals version now identifies the octet-trie implementation, and the perldelta and regex internals documentation describe the new representation. Tests cover native and UTF-8 byte tries, updated diagnostics, and benchmark the scaling of large alternations with different amounts of padding. On this system, the normal seven-case corpus was measured with dumbbench using an untouched default configuration in both blead and this branch. The measured compile and match phase means were 0.109703 and 0.027088 seconds for blead, versus 0.103755 and 0.026573 seconds for the branch. This is a change of -5.4% and -1.9%, respectively. The branch was also measured with its trie disabled. The branch trie versus off change was +25.2% for compilation and -17.9% for matching. These numbers describe this small normal corpus; the large-alternation workload is where the asymptotic difference is most clear. On the branch-only large-alternation corpus, with 8 alternation counts from 10 through 1280 and 4 padding lengths from 10 through 80, the off model was dominated by the interaction term (n_alt * n_pad), with R^2 = 1.000 and an interaction t-statistic of 131.14. The trie model was dominated by const, with R^2 = 0.502 and an interaction t-statistic of -0.12. At 1280 alternations and 80 padding characters, off took 1.149000 seconds per child and trie took 0.012024 seconds, a 95.56x speedup. The benchmark uses dumbbench and optionally Statistics::Regression. When these are not available, it reports the corresponding cpanm command needed to install them.

khwilliamson

Here's the EXACTish node I created to work around trie constraints EXACTFAA_NO_TRIE - Like EXACTFAA, (string not UTF-8, folded except: MICRO,
SHARP S; folded length <= unfolded, not currently trie-able) */

I was wrong earlier about the existence of TRIEC in comments. Sorry for the noise.

I didn't trace through the code looking for unhandled edge cases. That would have taken a lot of time.

khwilliamson · 2026-08-22T02:26:10Z

        else if (REGNODE_TYPE(OP(first)) == TRIE &&
-                ((reg_trie_data *)RExC_rxi->data->data[ TRIE_DATA_SLOT(first) ])->minlen > 0)
+                ((reg_trie_data *)RExC_rxi->data->data[ TRIE_DATA_SLOT(first) ])->minlen > 0
+                && ((reg_trie_data *)RExC_rxi->data->data[ TRIE_DATA_SLOT(first) ])->byte_trie != 2)


As a reader, I would prefer a mnemonic, like an enum, for the possible byte_trie values; I kept having to remember what means what.

Sure, willdo!

khwilliamson · 2026-08-22T02:45:36Z

+{
+    const U8 *end;
+
+    return is_utf8_string_loclen(uc, (STRLEN)(e - uc), &end, NULL)


Why not plain is_utf8_string(). It returns false unless the whole length is valid. Both function versions return true on plain ASCII

khwilliamson · 2026-08-22T10:47:34Z

+}
+
+/* EXACT_REQ8 and the UTF-8 folding nodes contain UTF-8 source bytes even
+ * when the pattern itself is not marked UTF-8.  Treat those nodes as UTF-8


I don't think this is true. I looked around, and couldn't find a case where one of these types of nodes could occur in a non-UTF8 pattern

khwilliamson · 2026-08-22T10:54:54Z

+ * here; otherwise a wide character is read one byte at a time and then
+ * encoded a second time when the byte trie is built. */
+#define TRIE_SOURCE_UTF8(noper)                                             \
+    (UTF || OP(noper) == EXACT_REQ8 || OP(noper) == EXACTFU_REQ8            \


This made me consider if tries could apply to /l matching. I didn't look to see how that is handled; I presume that tries aren't generated because of the complications involved. But I suppose they could be if the pattern didn't have folding and all branches didn't have anything like [[:alpha:]] in them. But it's probably not worth the effort

Historically we couldn't trie locale specific patterns because the meaning of the pattern can change depending on the current locale. So the trie table would have to be remapped.

khwilliamson · 2026-08-22T12:36:19Z

    */


+    /* UTF-8 byte tries can have up to UTF8_MAXBYTES transitions for each


If we restricted tries to running only on Unicode characters, we could save space with likely no one being affected. The symbol for that is MAX_UNICODE_UTF8_BYTES

Why would we need to restrict tries to running only on unicode characters? What we do is we build a trie contining utf8 representations of the data, and then translate the awkward codepoints on the fly. so from the tries point of view all data is utf8.

khwilliamson · 2026-08-22T12:51:18Z

+                        U8 *ep;
+                        U8 *bp;
+                        if (!TRIE_SOURCE_UTF8(noper))
+                            uvc = NATIVE_TO_UNI(uvc);


I think this breaks on EBCDIC. There may be other similar places as well. This changes uvc for code outside this block. And it's trying to construct a trie based on not the native values, but the official Unicode ones. I don't understand why. Mostly EBCDIC just works these days. One codes without thinking about it except in rare cases. The native value is what one wants

I dont remember this part, ill review it and get back to you.

khwilliamson · 2026-08-22T12:52:09Z

+         * initial unused element.  Preserve the theoretical packed size
+         * before using transcount as the growable allocation size. */
+        const STRLEN transition_count = transcount - 1;
+        PERL_UNUSED_VAR(transition_count);


I don't understand why this is needed

Thanks, ill fix.

khwilliamson · 2026-08-22T13:25:01Z

+                        else
+                            break;
+                    }
+                    else if (utf8_prefix[bytes] < 0x80


The mnemonic ! UTF8_IS_CONTINUATION should be used.

khwilliamson · 2026-08-22T13:42:08Z

+                        break;
+
+                    if (!need) {
+                        if (utf8_prefix[bytes] < 0x80)


Maybe this is why you translated native to unicode. But again EBCDIC generally just works as long as you use the mnemonic macro (which is clearer to the reader anyway).

Why not use UTF8SKIP() - 1 instead of this series of conditionals? I think that would mean this would automatically extend to above-Unicode code points, without the break currently done

The new code depends on being able to efficiently convert non-utf8 to utf8 to handle the awkward codepoints. that is the whole point here, the data it always stored in utf8. always. (it could be stored as EBCDIC utf8 or whatever if there is such a thing.). Where can we test such a platform?

demerphq added 3 commits August 14, 2026 11:28

regex engine - get rid of 16 bit values, use 32 bit values for charid

b56123c

For now anyway. Maybe in the future we should just switch to double compiling, or maybe compiling on demand, and generate both a utf8 octet trie and a latin-1 version as well.

Widen trie word indexes to U32

846d5cf

demerphq requested a review from khwilliamson August 15, 2026 19:48

github-actions Bot added the hasConflicts label Aug 20, 2026

khwilliamson reviewed Aug 22, 2026

View reviewed changes

		*/


		/* UTF-8 byte tries can have up to UTF8_MAXBYTES transitions for each

Uh oh!

Conversation

demerphq commented Aug 15, 2026

Uh oh!

khwilliamson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants