Fueling Your Coding Mojo

Buckle up, fellow PHP enthusiast! We're loading up the rocket fuel for your coding adventures...

Popular Searches:

PHP: tokenizing, using a regular Expression (mostly there)

Hey everyone,

I've been working on tokenizing a string in PHP and I seem to be stuck with using regular expressions. I've made some progress, but I could really use some help to fine-tune my solution.

Here's what I've come up with so far:

$string = "Hello, (world). I'm tokenizing this string.";

// Tokenizing by splitting on whitespace and punctuation
$tokens = preg_split('/[\s[:punct:]]+/', $string, -1, PREG_SPLIT_NO_EMPTY);

My goal is to tokenize the string by splitting it into separate words and punctuation marks, while keeping the punctuation marks as separate tokens. So in the case of the above example, I would like to end up with an array of tokens like this:

[0] => Hello
[1] => ,
[2] => (
[3] => world
[4] => )
[5] => .
[6] => I'm
[7] => tokenizing
[8] => this
[9] => string
[10] => .

However, the problem with my current solution is that it also splits some words incorrectly. For instance, the word "I'm" is currently split into two tokens: "I" and "m".

I believe this is due to the regex pattern I'm using, which splits on any whitespace and any punctuation mark. Ideally, I'd like to modify the pattern so that it doesn't split words like "I'm" or "don't".

I'm not very experienced with regular expressions, so any help or suggestions on how I can modify the pattern to achieve the desired result would be greatly appreciated.

Thanks in advance for your assistance!

All Replies


Hey there!

I had a similar issue with tokenizing strings in PHP using regular expressions. I found a solution that might work for your case and prevent breaking words like "I'm" or "don't".

Instead of using `preg_split`, you can use `preg_match_all` with a modified regex pattern. Here's an example:

$string = "Hello, (world). I'm tokenizing this string.";

// Tokenizing by matching words and punctuation
preg_match_all("/[\p{L}']+|[[:punct:]]/", $string, $matches);
$tokens = $matches[0];


In this modified pattern, `[\p{L}']+` will match any letter or group of letters (including the apostrophe for contractions like "I'm"), and `[[:punct:]]` will match any punctuation mark.

This approach should give you the desired result by preserving words while breaking down punctuation marks into separate tokens.

Give it a try and let me know if it works for you!

User 1


Hey folks,

I had a similar predicament with tokenizing strings in PHP using regular expressions. I stumbled upon a different approach that worked wonders for me.

Instead of relying solely on regular expressions, I utilized the `str_word_count()` function, which intelligently identifies words within a string.

Here's an example of how you can implement it:

$string = "Hello, (world). I'm tokenizing this string.";

// Tokenizing using str_word_count
$words = str_word_count($string, 1);

// Adding punctuation marks as separate tokens
foreach (str_split($string) as $char) {
if (!ctype_space($char) && !in_array($char, $words)) {
$words[] = $char;


By using `str_word_count()` with the parameter `1`, it returns an array containing all the words present in the string. Then, I looped through each character in the string and checked if it's not a space and not already in the words array. If that's the case, it adds the punctuation mark as a separate token.

Give this approach a shot and let me know if it helps you achieve the desired tokenization!

User 2


Hey everyone,

I faced a similar challenge while working on a project that required string tokenization in PHP. To achieve the desired result, I took a slightly different approach using the `strtok()` function.

Here's an example of how you can utilize `strtok()` for tokenizing your string:

$string = "Hello, (world). I'm tokenizing this string.";

$delimiters = " \t\n\r\0\x0B"; // Set the delimiters as spaces and punctuation marks

$tokens = [];

$token = strtok($string, $delimiters);
while ($token !== false) {
$tokens[] = $token;
$token = strtok($delimiters);


In this approach, we first specify the delimiters we want to use for tokenization. Here, I included spaces and common punctuation marks like tabs, newlines, and brackets. You can customize the `$delimiters` variable according to your requirements.

Next, we initialize an empty array to store our tokens. The `strtok()` function is then used to extract tokens from the string iteratively. It takes two parameters: the string to tokenize and the delimiters.

Within the `while` loop, we add each token to the `$tokens` array and update the value of `$token` using `strtok($delimiters)` in order to fetch the next token.

Give this approach a try, and I hope it helps you achieve the desired tokenization outcome!

User 3

New to LearnPHP.org Community?

Join the community