The basic facts

The key element when using a multibyte character set in PHP is to know exactly what you’re doing. If you don’t, you can easily end up with partially corrupted text and wrong results. The one big reason for this is the fact that PHP by default does not support anything other than byte-sized character set. In fact, strictly speaking, PHP doesn’t really know what a character set is. All it sees are bytes, not characters. This means that every string-function in PHP works on the assumption that a byte is a character. When dealing with for instance UTF-8 this is no longer true. The result is that strlen reports the number of bytes in the string, not the number of characters. Similarly, strpos will give you the position in bytes, not characters, and many of the other string-functions have similar problems. So what to do?

First off, a very handy fact about UTF-8 is it’s ASCII-compatible (7bit ASCII that is), meaning these characters are binary represented as 0xxx xxxx (where x are ASCII bits). Another handy fact about valid UTF-8 strings is that any encoded Unicode character has a unique byte sequence, meaning it can’t be confused with a part of another character. This means that if you encounter a byte 00100000 (20 hex or 32 dec) it can not be anything other than a space character, or else it’s not a valid UTF-8 string. For those interested in how this is archived, here’s the binary representation of UTF-8 characters (skip if you’re not into that type of stuff ;o)

1 byte:  0xxx xxxx
2 bytes: 110x xxxx 10xx xxxx
3 bytes: 1110 xxxx 10xx xxxx 10xx xxxx
4 bytes: 1111 0xxx 10xx xxxx 10xx xxxx 10xx xxxx

Well, enough with talk about UTF-8 in general. Here’s a step-by-step guild how to use UTF-8 in PHP.

1.  Find you what you have available

If you’re going to use UTF-8 in PHP you’ll first need to find you what’s available to you. Ideally you should have the PHP extension mbstring install and it should not be set to overload str-functions. With mbstring you’ll have a set of functions that are multibyte aware. If you don’t have mbstring available, check for iconv (also an PHP extension). In PHP 5 and later this extension will give you some very simple functions to work with (strlen, strpos, strrpos, substr and validation). If you have neither mbstring nor iconv available at your host (I’m assuming that you’re going to run something in a hosted server), you should strongly consider change host and/or your need for Unicode/UTF-8, as you’re going to have to make some native functions that works properly on UTF-8 encoded strings. For the sake of simplicity, I’m going to assume you have mbstring or iconv installed.

2. Store your files as UTF-8

This should be somewhat obvious. If you’re going to use UTF-8, you should store your files as it too. The simple reason is that any strings you have stored in your scripts will be UTF-8 too, and should be outputed properly given you’ve done everything else correct.

3. Define input and output as UTF-8

This is quite depending on what you’re doing, but chances are that you’re dealing with the HTTP-protocol and HTML. If so you have to add the following function call before any output (if you’re not using output buffering).

header('Content-Type: text/html; charset=utf-8');

and add the following to the head-part of your HTML-document

<meta http-equiv=Content-Type content="text/html; charset=utf-8">

This will state that any output is html and UTF-8. If you’re not dealing with html, just replace text/html with whatever you’re using (and proably drop the meta tag too). This should also ensure that input from most browsers is UTF-8, but to be really sure it might be a good idea to add the attribute accept-charset to any forms you might have, like this:

<form accept-charset="UTF-8" method="post">
<!-- Input stuff here -->
</form>

This attribute can also be used with a comma-separated list of character set your script accept. To keep it simple you should just use UTF-8 as the only accepted “character set” (strictly speaking, UTF-8 is an encoding of the Unicode character set, and not a charset by itself).

If you don’t have control over the input, for instance RSS feed from 3. party server, transform it to Unicode and encode to UTF-8. This can be done like this:

//mbstring
$UTF8string = mb_convert_encoding($string, 'other charset', 'UTF-8');
// or call
mb_internal_encoding('UTF-8');
// once to use UTF-8 as default and and drop last parameter like this:
$UTF8string = mb_convert_encoding($string, 'other charset');
 
//iconv
$UTF8string = iconv('other charset', 'UTF-8', $string);

Consult the PHP manual for supported character sets.

4. Validate your input

I can not stress this point enough. Check that your input actually is UTF-8, or the multibyte aware functions might not work as expected. Also, if not validated, those who view or store your data might get security problems like SQL-injection (though it would be their fault it’s happening …). This can be done as follows for mbstring:

//last parameter can still be dropped as showed in last example
$validUTF8 = mb_check_encoding($string, 'UTF-8'); //will give true if valid, false if not

for iconv you’ll have to go for a bit more dirty solution

function validateUTF8_iconv($before)
{
    $after = iconv('UTF-8', 'UTF-8', $before);
    return ($before === $after);
}

The reason this works is because any non-valid characters are removed or changed, and thus the strings will not be equal anymore.

5. Store your data correct

Now this is really a pit a lot of people fall into. Storing UTF-8 on disc is no problem, and is done just as before. Storing data in a database however, is in the case of MySQL definitely not as before. First of you’ll need MySQL 4.1 or later as the versions before don’t support what we’re about to do. The big problem with PHP and MySQL is that the connection is by default set to latin1, also known as ISO-8859-1, and a lot of people then actually store there data as a UTF-8-transformed ISO-8859-1. That is, MySQL thinks your input is ISO-8859-1 and then  convert it to Unicode and encode it as UTF-8. This will lead to problems when you view your data in for instance phpMyAdmin which connects to MySQL the proper way when dealing with UTF-8. The correct way to connect to MySQL is now

$link = mysql_connect('localhost', 'mysql_user', 'mysql_password');
mysql_query("SET NAMES 'utf8' COLLATE 'utf8_general_ci'");

The difference is the second line which states what charset is being used for further talk on the connection. The collate part can be dropped if you’re using the default one (utf8_general_ci). In addition to this, the table and fields should be defined with collation utf8_[something] and to save space don’t use char-fields as they will use 3x number of characters you define. Instead you should use varchar. Please also note that MySQL only supports Unicode 3.0, that is 2 byte Unicode up to code-point FFFE (BOM) or 3 byte UTF-8. If you need any code-points above that, you’re unfortunately in for some trouble …

This is however only the case of MySQL. For other database systems you should check what is the default charset and how to change it. The documentation/manual of the system is a good place to start to find this type of information.

6. Functions operating on UTF-8

The points above should make sure that you’re using UTF-8. The last thing is that you have to remember that any str-functions might not work correctly. Try to use str-functions defined in mbstring or iconv (see documentation). Some functions do however work as expected. Strcmp does only a binary compare and still works, strcasecmp however don’t. Str_replace will also work on valid UTF-8 strings as any given character has a unique byte sequence, but the case-less version str_ireplace don’t. In general any str-function that is not case-less and don’t need to operate on the number of characters in the string, should work just fine as long as any input to the function is valid UTF-8.

There are also some functions that are not part of mbstring or iconv that do support UTF-8. Htmlspecialchars, htmlentities and preg_* (with u modifier) are examples of this. There are also functions that operate on a purley binary level without any regards to charset. Examples of this is the md5 and sha1 functions.

That concludes this part of the howto. You should now be able to use UTF-8 the right way. If you have any questions, please leave a comment and I’ll happily answer. The next part will hopefully deal with some common pitfalls and how to debug and solve them.

First of all, you shouldn’t be using a foreign charset or encoding without knowing it’s risk. Unicode requires a mulibyte encoding, which has it’s own set of problems compared to singlebyte charsets like ISO-8859-1 and simliar. As I’m only really familiar with UTF-8, I’m going to use that as an example. I could proably write a book about the problems, but here’s a small list:

• BOM (U+FEFF) and “reverse”-BOM (U+FFFE). This is not really a problem in UTF-8 as it doesn’t have the issues with little and big endian (wikipedia about endianness), but if you convert to for instance UTF-16, having a “reverse”-BOM at the start of your document would be a disaster.
• Str-functions aren’t multibyte aware and might “break” your strings.
• Non-shortest form. There’s a bit of math behind this one, which I’m not going to explain today, so you’ll just have to trust me on this one. Non-shortest form is an illegal representation of a Unicode character. It basicly is represented using more bytes than necessary and in UTF-8 this results in ‘ , which in hex terms is 0×27, could be represented by 0xC0A7. If those who developed for example the RDBMS you’re using is just as unknowing as alot of people out there, they will decode or recode this to UTF-16, UCS-2 or UCS-4 and get a single quote ( ‘ ) which might just result in a SQL-injection …
• Surrogates. In UTF-8 surrogates (U+D800-U+DFFE) are not allowed, and is actually regared as a potential security risk. Allowing these will result in a � or a similar “this is an illegal byte”-character being displayed.
• Illegal combination of bytes resulting in a � or a similar “this is an illegal byte”-character being displayed.

In addition to those above, there is also the possibility of security problems when you’re actually escaping input with functions not aware of multibyte charsets (here’s an example). With all this in mind I can tell you why mbstring is such a bad piece of coding.

First of all it doesn’t have a complete replacement for every str-function. We’re missing *sort, *trim, strcasecmp, str_ireplace, ucfirst, ucwords and wordwrap just to mention some. The second thing is that mbstring is writen by a Japanese or something, which is quite obvious considering strcasecmp is missing. … and no, using mb_strtolower and strcasecmp is not a valid way as the process requires either simple or full case folding which has some differences from the “to lower”-process. Anyway, mbstring is writen more or less to enable Japanese, Chinese and similar languages to work. The thing is, these languages don’t have case as most other languages which explains the lack of the above.

Third, there is some bugs in mbstring which result in invalid validaton. Here are som examples:

• Anything behind a null-byte isn’t validated
• Surrogates are allowed
• “Reverse”-BOM is allowed
• Title case isn’t working

Fourth and final, the overloading abilites make the str-functions behave different from the original str-functions. These are undocumented differences, possibly breaking applications expecting str-functionality. For instance, if 2. paramter in strrchr is an int, is converted to a character. In mb_strrchr however, this will give an error.