Introduction

Regular expressions (“regexes”) allow defining a pattern and executing it against strings. Substrings which match the pattern are termed “matches”.

A regular expression is a sequence of characters that define a search pattern.

Regex finds utility in:

• input validation
• find-replace operations
• advanced string manipulation
• file search or rename
• whitelists and blacklists
• …

Simultaneously, regular expressions are ill-suited for other kinds of problems:

• parsing XML or HTML
• exactly matching dates
• …

There are several regex implementations—regex engines—each with its own quirks and features. This book will avoid going into the differences between these, instead sticking to features that are, for the most part, common across engines.

The example blocks throughout the book use JavaScript under the hood. As a result, the book may be slightly biased towards JavaScript’s regex engine.

Basics

Regular expressions are typically formatted as /<rules>/<flags>. Often people will drop the slashes and flags for brevity. We’ll get into the details of flags in a later chapter.

Let’s start with the regex /p/g. For now, please take the g flag for granted.

As we can see, /p/g matches all lowercase p characters.

Instances of the regex pattern found in an input string are termed “matches”.

Character Classes

It’s possible to match a character from within a set of characters.

/[aeiou]/g matches all vowels in our input strings.

Here’s another example of these in action:

We match a p, followed by one of the vowels, followed by a t.

There’s an intuitive shortcut for matching a character from within a continuous range.

We can combine ranges and individual characters in our regexes.

Our regex /[A-Za-z0-9_-]/g matches a single character, which must be (at least) one of the following:

• from A-Z
• from a-z
• from 0-9
• one of _ and -.

We can also “negate” these rules:

The only difference between the first regex of this chapter and /[^aeiou]/g is the ^ immediately after the opening bracket. Its purpose is to negate the rules defined within the brackets. We are now saying:

“match any character that is not any of a, e, i, o, and u”

Examples

Prohibited username characters

Unambiguous characters

Character Escapes

Character escapes act as shorthands for some common character classes.

Digit character — \d

The character escape \d matches digit characters, from 0 to 9. It is equivalent to the character class [0-9].

\D is the negation of \d and is equivalent to [^0-9].

Word character — \w

The escape \w matches characters deemed “word characters”. These include:

• lowercase alphabet — a–z
• uppercase alphabet — A–Z
• digits — 0–9
• underscore — _

It is thus equivalent to the character class [a-zA-Z0-9_].

Whitespace character — \s

The escape \s matches whitespace characters. The exact set of characters matched is dependent on the regex engine, but most include at least:

• space
• tab — \t
• carriage return — \r
• new line — \n
• form feed — \f

Many also include vertical tabs (\v). Unicode-aware engines usually match all characters in the separator category.

The technicalities, however, will usually not be important.

Any character — .

While not a typical character escape, . matches any¹ character.

Escapes

In regex, some characters have special meanings as we will explore across the chapters:

• |
• {, }
• (, )
• [, ]
• ^, $
• +, *, ?
• \
• . — Literal only within character classes.¹
• - — Sometimes a special character within character classes.

When we wish to match these characters literally, we need to “escape” them.

This is done by prefixing the character with a \.

Examples

JavaScript in-line comments

Asterisk-surrounded substrings

The first and last asterisks are literal since they are escaped — \*.

The asterisk inside the character class does not necessarily need to be escaped¹, but I’ve escaped it anyway for clarity.

The asterisk immediately following the character class indicates repetition of the character class, which we’ll explore in chapters that follow.

Groups

Groups, as the name suggests, are meant to be used to “group” components of regular expressions. These groups can be used to:

• Extract subsets of matches
• Repeat groups an arbitrary number of times
• Refer to previously matched substrings
• Enhance readability
• Allow complex alternations

We’ll see how to do a lot of this in later chapters, but learning how groups work will allow us to study some great examples in these later chapters.

Capturing groups

Capturing groups are denoted by ( … ). Here’s an expository example:

Capturing groups allow extracting parts of matches.

Using your language’s regex functions, you would be able to extract the text between the matched braces for each of these strings.

Capturing groups can also be used to group regex parts for ease of repetition of said group. While we will cover repetition in detail in chapters that follow, here’s an example that demonstrates the utility of groups.

Other times, they are used to group logically similar parts of the regex for readability.

Backreferences

Backreferences allow referring to previously captured substrings.

The match from the first group would be \1, that from the second would be \2, and so on…

Backreferences cannot be used to reduce duplication in regexes. They refer to the match of groups, not the pattern.

Here’s an example that demonstrates a common use-case:

This cannot be achieved with a repeated character classes.

Non-capturing groups

Non-capturing groups are very similar to capturing groups, except that they don’t create “captures”. They take the form (?: … ).

Non-capturing groups are usually used in conjunction with capturing groups. Perhaps you are attempting to extract some parts of the matches using capturing groups. You may wish to use a group without messing up the order of the captures. This is where non-capturing groups come handy.

Examples

Query String Parameters

We match the first key-value pair separately because that allows us to use &, the separator, as part of the repeating group.

(Basic) HTML tags

As a rule of thumb, do not use regex to match XML/HTML.¹²³⁴

However, it’s a relevant example:

Names

Find: \b(\w+) (\w+)\b

Replace: $2, $1⁵

Before

John Doe
Jane Doe
Sven Svensson
Janez Novak
Janez Kranjski
Tim Joe

After

Doe, John
Doe, Jane
Svensson, Sven
Novak, Janez
Kranjski, Janez
Joe, Tim

Backreferences and plurals

Find: \bword(s?)\b

Replace: phrase$1⁵

Before

This is a paragraph with some words.

Some instances of the word "word" are in their plural form: "words".

Yet, some are in their singular form: "word".

After

This is a paragraph with some phrases.

Some instances of the phrase "phrase" are in their plural form: "phrases".

Yet, some are in their singular form: "phrase".

Repetition

Repetition is a powerful and ubiquitous regex feature. There are several ways to represent repetition in regex.

Making things optional

We can make parts of regex optional using the ? operator.

Here’s another example:

Here the s following http is optional.

We can also make capturing and non-capturing groups optional.

Zero or more

If we wish to match zero or more of a token, we can suffix it with *.

Our regex matches even an empty string "".

One or more

If we wish to match one or more of a token, we can suffix it with a +.

Exactly x times

If we wish to match a particular token exactly x times, we can suffix it with {x}. This is functionally identical to repeatedly copy-pasting the token x times.

Here’s an example that matches an uppercase six-character hex colour code.

Here, the token {6} applies to the character class [0-9A-F].

Between min and max times

If we wish to match a particular token between min and max (inclusive) times, we can suffix it with {min,max}.

At least x times

If we wish to match a particular token at least x times, we can suffix it with {x,}. Think of it as {min,max}, but without an upper bound.

A note on greediness

Regular expressions, by default, are greedy. They attempt to match as much as possible.

Suffixing a repetition operator (?, *, +, …) with a ?, one can make it “lazy”.

Here, this could also be achieved by using [^"] instead of . (as is best practice).

[…] Lazy will stop as soon as the condition is satisfied, but greedy means it will stop only once the condition is not satisfied any more

—Andrew S on StackOverflow

Examples

Bitcoin address

Youtube Video

We can adjust this to not match the last broken link using anchors, which we shall encounter soon.

Alternation

Alternation allows matching one of several phrases. This is more powerful than character classes, which are limited to characters.

Delimit the set of phrases with pipes—|.

One of foo, bar, and baz

If only a part of the regex is to be “alternated”, wrap that part with a group—capturing or non-capturing.

Try followed by one of foo, bar, and baz

Matching numbers between 100 and 250:

This can be generalized to match arbitrary number ranges!

Examples

Hex colours

Let’s improve one of our older examples to also match shorthand hex colours.

It is important that [0-9A-F]{6} comes before [0-9A-F]{3}. Else:

Roman numerals

Flags

Flags (or “modifiers”) allow us to put regexes into different “modes”.

Flags are the part after the final / in /pattern/.

Different engines support different flags. We’ll explore some of the most common flags here.

Global (g)

All examples thus far have had the global flag. When the global flag isn’t enabled, the regex doesn’t match anything beyond the first match.

(Case) Insensitive (i)

As the name suggests, enabling this flag makes the regex case-insensitive in its matching.

Multiline (m)

The multiline flag has to do with the regex’s handling of anchors when dealing with “multiline” strings—strings that include newlines (\n). By default, the regex /^foo$/ would match only "foo".

We might want it to match foo when it is in a line by itself in a multiline string.

Let’s take the string "bar\nfoo\nbaz" as an example:

bar
foo
baz

Without the multiline flag, the string above would be considered as a single line bar\nfoo\nbaz for matching purposes. The regex ^foo$ would thus not match anything.

With the multiline flag, the input would be considered as three “lines”: bar, foo, and baz. The regex ^foo$ would match the line in the middle—foo.

Dot-all (s)

The . typically matches any character except newlines. With the dot-all flag, it matches newlines too.

Unicode (u)

In the presence of the u flag, the regex and the input string will be interpreted in a unicode-aware way. The details of this are implementation-dependent, but here are some things to expect:

• Character classes may match astral symbols.
• Character escapes may match astral symbols and may be unicode-aware.
• The i flag may use Unicode’s case-folding logic.
• The use of some features like unicode codepoint escapes and unicode property escapes may be enabled.

Whitespace extended (x)

When this flag is set, whitespace in the pattern is ignored (unless escaped or in a character class). Additionally, characters following # on any line are ignored. This allows for comments and is useful when writing complex patterns.

Here’s an example from Advanced Examples, formatted to take advantage of the whitespace extended flag:

^                   # start of line
    (
        [+-]?       # sign
        (?=\.\d|\d) # don't match `.`
        (?:\d+)?    # integer part
        (?:\.?\d*)  # fraction part
    )
    (?:             # optional exponent part
        [eE]
        (
            [+-]?   # optional sign
            \d+     # power
        )
    )?
$                   # end of line

Anchors

Anchors do not match anything by themselves. Instead, they place restrictions on where matches may appear—“anchoring” matches.

You could also think about anchors as “invisible characters”.

Beginning of line — ^

Marked by a caret (^) at the beginning of the regex, this anchor makes it necessary for the rest of the regex to match from the beginning of the string.

You can think of it as matching an invisible character always present at the beginning of the string.

End of line — $

This anchor is marked by a dollar ($) at the end of the regex. It is analogous to the beginning of the line anchor.

You can think of it as matching an invisible character always present at the end of the string.

The ^ and $ anchors are often used in conjunction to ensure that the regex matches the entirety of the string, rather than merely a part.

Let’s revisit an example from Repetition, and add the two anchors at the ends of the regex.

In the absence of the anchors, http/2 and shttp would also match.

Word boundary — \b

A word boundary is a position between a word character and a non-word character.

The word boundary anchor, \b, matches an imaginary invisible character that exists between consecutive word and non-word characters.

There is also a non-word-boundary anchors: \B.

As the name suggests, it matches everything apart from word boundaries.

Examples

Trailing whitespace

Markdown headings

Without anchors:

Lookaround

Lookarounds can be used to verify conditions, without matching any text.

You’re only looking, not moving.

• Lookahead
  • Positive — (?=…)
  • Negative — (?!…)
• Lookbehind
  • Positive — (?<=…)
  • Negative — (?<!…)

Lookahead

Positive

Note how the character following the _ isn’t matched. Yet, its nature is confirmed by the positive lookahead.

After (?=[aeiou]), the regex engine hasn’t moved and checks for (?=\1) starting after the _.

Negative

Without the anchors, this will match the part without the # in each test case.

Negative lookaheads are commonly used to prevent particular phrases from matching.

Lookbehind

Positive

Negative

Examples

Password validation

Lookarounds can be used verify multiple conditions.

Quoted strings

Without lookaheads, this is the best we can do:

Advanced Examples

Javascript comments

[\s\S] is a hack to match any character including newlines. We avoid the dot-all flag because we need to use the ordinary . for single-line comments.

24-Hour Time

Meta

Replace: <Example regex={/$1/$2}>

I performed this operation in commit d7a684f.

Floating point numbers

• optional sign
• optional integer part
• optional decimal part
• optional exponent part

The positive lookahead (?=\.\d|\d) ensures that the regex does not match ..

Latitude and Longitude

See also: Floating Point Numbers

MAC Addresses

UUID

IP Addresses

HSL colours

Integers from 0 to 360

• 360
• 300 to 359 — 3, [0-5], any digit
• 0 to 299
  • optionally 1 or 2 as the hundreds digit
  • optionally any tens digit
  • a units digit

Percentages

• 100, optionally followed by .000…
• one or two digit integer, optionally followed decimal part

Bringing it all together

Next Steps

Congratulations on getting this far!

Obligatory xkcd:

If you’d like to read more about regular expressions and how they work:

• awesome-regex
• regex tag on StackOverflow
• StackOverflow RegEx FAQ
• r/regex
• RexEgg
• Regular-Expressions.info
• Regex Crossword
• Regex Golf

Thanks for reading!