Regular expressions in Power Fx

[gregli-msft]

It's time to officially add regular expression support to Power Fx. Power Apps have had them for a while in the form of Match, MatchAll, and IsMatch functions.

Our challenge has been that .NET and JavaScript have significantly different regular expression engines. And add on to this that Excel has integrated PCRE2. Ideally, we'd like Power Fx to have a common regular expression language that can be used on any host and gives the same results. That has required a three pronged approach:

1. Limit the language to a common set of features that can be transpiled to .NET, JavaScript, and PCRE2. Make this a compile time check so that it is consistent (since we have one compiler for all hosts, even JavaScript) and makers get good information early.
2. Transpile as needed. For example, .NET can be transpiled to accept \r as a newline character, JavaScript can be transpiled for freespacing.
3. Accept that it won't be perfect. For example, the capture group rules for JavaScript with zero-match quantifiers are just different from the rest. Fortunately, this appears to be the only major area where things are different and most makers shouldn't run into it.

I'd love your feedback on the regular expression language choices made. We have been opinionated in some areas. For example, explicit capture is the default and named and numbered captures can't be used together (.NET and JavaScript disagree on numbering of named captures, explicit capture is a best practice and higher performance, and named captures are easier to read and maintain and we should be promoting their use). We have also tried to make things clearer, for example requiring a literal ^ to be escaped in a character class, rather than depending on location wihin the character class to determine the meaning.

Please review the draft regular expressions documentation.

The three most important pieces of code to review:

• Draft compiler regular expression validation code, which itself is two large regular expressions that generate lots and lots of errors. Note that this code only returns a bool, it makes no modifications.
• Draft runtime transpiler for .NET
• Draft runtime transpiler for JavaScript

And if you are so inclined, here is the PR including this test file which may be worth reviewing.

Thanks for your input!

[PowerAppsDarren]

I love it, Greg! Thank you!

All I need now is the ability to take all the new imperative code I'm stuffing in App.Formulas into separate class files!! 🙏

...just like the golden era in VB 4, 5, & 6! 🤠

Bring Back OOP! 😉

[nathanroseakl]

Greg - I think this is a great start. I'll get hands on with the expressions over the next few days and see how I get on. Question - I didn't see anything about compiling to SQL - this could be super useful for Dataverse formula columns Common issue - user enters a phone number and we use the regex in a formula column to reformat the number to the standard needed for our autodialer, sms gateway etc.

[nathanroseakl]

I've been playing around with this in the CLI and it's seriously impressive - note I've submitted an update to the docs to make it clear the predefined patterns must be expressed as 'Match.PredefinedPattern' so it's a bit easier for makers to run with it. Quick question I'm trying to use the enum match type with MatchAll - in this example I'd expect to find 2 email addresses but it's only returning the full string. When I match on a specific word it behaves as expected. What am I doing wrong?