We’ve all seen it, captcha or pictures that protects alot of web-forms against those nasty spam-bots while at the same time blissfully destroying the usability of the web (10 “good” examples). While a lot of attempts have been made to make it more usable for more people (pictures aren’t exactly user-friendly for blind people …), it still boils down to some requirements either for the browser (e.g. javascript) or the user (typing into an input field). While requiring the browser to support something isn’t that bad, requiring the user to do something definitely isn’t usability at it’s best. Wouldn’t it be nice to just instant see whether or not the current request is made from a bot or a browser? Well, you might be able …

For the last two weeks I’ve created my personal pet project logging spam attempts in some scripts running at Ascended Development. After more than 300 attempts and above 100 unique spam-bots (or IPs anyway), the results shock me … a lot. The spam-bots that has visited the site is incredible stupid. In simplest terms they don’t even seem to be able to handle cookies (e.g. thereby also session in PHP), little less the pictures or javascript. The most ridiculous thing is that they actually do send data in the antispam input field. The problem is that it’s 2 to 4 times longer than the number of characters in the picture, and yes, it didn’t supply the cookie for the session meaning it wouldn’t succeed even if the wild guess was right.

So what can we do without requiring user input or javascript? Certainly not add a hidden input field in the form and hope that the bot add something to it. All bots (yeah, actually every single one of them) either provide the default value or didn’t provide the field at all. Actually out of the 300 attempts only 50 didn’t provide the field, and a massive 250 attempts had the default value. None tried any other values, and it seems that those that didn’t provide the field tried again with the default value. Simply put, a hidden field don’t work, at least not on those bots visiting our site at the logged locations.

So here’s the trick: Simply read the HTTP-headers. Has to be too good to be true? Well, the log is equally clear on this matter too. None of the bots provided the fields Accept-Language and Accept-Encoding, both of which quite frankly any decent browser sends out these days (Opera, Firefox, Konqueror, Chrome, Safari and even IE). Even lynx, a text browser, does send these headers, and I tested it with a two year old release. It does make sense if you think about it. The browsers will add the Accept-Language so pages can be correctly localized and Accept-Encoding so that compression can be used. Both things is benefitial to the user, and therefor present despite both being optional. The spambots on the other hand seem to be using libaries like libcurl to build their HTTP-client, and by default these libraries don’t seem add Accept-Language or Accept-Encoding. Add the fact that few, if any, sort spam-bots from browsers this way, and we can see that it’s not really such a surprise after all.

This isn’t without it’s flaws though. I’ve only encountered simple, general spam-bots and not the ones attacking widespread software like phpBB or vBulletin. Also, the site isn’t subject to targeted attacks. That been said, I wouldn’t be surprised if they too fail to provide these HTTP-headers, and for the time being I’m quite confident that this method is about as efficient, or better, than the current widespread method of using pictures. It should also continue to be that way until this type of checking is more widely used. So until then, I believe this is a good way to avoid spam in you web applications:

<?php
if (isset($_SERVER['HTTP_ACCEPT_ENCODING']) === false ||
    isset($_SERVER['HTTP_ACCEPT_LANGUAGE']) === false)
{
    echo 'Spambot';
}
else
{
    echo 'Browser';
}
?>

If only this was working against email spam too …

The basic facts

The key element when using a multibyte character set in PHP is to know exactly what you’re doing. If you don’t, you can easily end up with partially corrupted text and wrong results. The one big reason for this is the fact that PHP by default does not support anything other than byte-sized character set. In fact, strictly speaking, PHP doesn’t really know what a character set is. All it sees are bytes, not characters. This means that every string-function in PHP works on the assumption that a byte is a character. When dealing with for instance UTF-8 this is no longer true. The result is that strlen reports the number of bytes in the string, not the number of characters. Similarly, strpos will give you the position in bytes, not characters, and many of the other string-functions have similar problems. So what to do?

First off, a very handy fact about UTF-8 is it’s ASCII-compatible (7bit ASCII that is), meaning these characters are binary represented as 0xxx xxxx (where x are ASCII bits). Another handy fact about valid UTF-8 strings is that any encoded Unicode character has a unique byte sequence, meaning it can’t be confused with a part of another character. This means that if you encounter a byte 00100000 (20 hex or 32 dec) it can not be anything other than a space character, or else it’s not a valid UTF-8 string. For those interested in how this is archived, here’s the binary representation of UTF-8 characters (skip if you’re not into that type of stuff ;o)

1 byte:  0xxx xxxx
2 bytes: 110x xxxx 10xx xxxx
3 bytes: 1110 xxxx 10xx xxxx 10xx xxxx
4 bytes: 1111 0xxx 10xx xxxx 10xx xxxx 10xx xxxx

Well, enough with talk about UTF-8 in general. Here’s a step-by-step guild how to use UTF-8 in PHP.

1.  Find you what you have available

If you’re going to use UTF-8 in PHP you’ll first need to find you what’s available to you. Ideally you should have the PHP extension mbstring install and it should not be set to overload str-functions. With mbstring you’ll have a set of functions that are multibyte aware. If you don’t have mbstring available, check for iconv (also an PHP extension). In PHP 5 and later this extension will give you some very simple functions to work with (strlen, strpos, strrpos, substr and validation). If you have neither mbstring nor iconv available at your host (I’m assuming that you’re going to run something in a hosted server), you should strongly consider change host and/or your need for Unicode/UTF-8, as you’re going to have to make some native functions that works properly on UTF-8 encoded strings. For the sake of simplicity, I’m going to assume you have mbstring or iconv installed.

2. Store your files as UTF-8

This should be somewhat obvious. If you’re going to use UTF-8, you should store your files as it too. The simple reason is that any strings you have stored in your scripts will be UTF-8 too, and should be outputed properly given you’ve done everything else correct.

3. Define input and output as UTF-8

This is quite depending on what you’re doing, but chances are that you’re dealing with the HTTP-protocol and HTML. If so you have to add the following function call before any output (if you’re not using output buffering).

header('Content-Type: text/html; charset=utf-8');

and add the following to the head-part of your HTML-document

<meta http-equiv=Content-Type content="text/html; charset=utf-8">

This will state that any output is html and UTF-8. If you’re not dealing with html, just replace text/html with whatever you’re using (and proably drop the meta tag too). This should also ensure that input from most browsers is UTF-8, but to be really sure it might be a good idea to add the attribute accept-charset to any forms you might have, like this:

<form accept-charset="UTF-8" method="post">
<!-- Input stuff here -->
</form>

This attribute can also be used with a comma-separated list of character set your script accept. To keep it simple you should just use UTF-8 as the only accepted “character set” (strictly speaking, UTF-8 is an encoding of the Unicode character set, and not a charset by itself).

If you don’t have control over the input, for instance RSS feed from 3. party server, transform it to Unicode and encode to UTF-8. This can be done like this:

//mbstring
$UTF8string = mb_convert_encoding($string, 'other charset', 'UTF-8');
// or call
mb_internal_encoding('UTF-8');
// once to use UTF-8 as default and and drop last parameter like this:
$UTF8string = mb_convert_encoding($string, 'other charset');
 
//iconv
$UTF8string = iconv('other charset', 'UTF-8', $string);

Consult the PHP manual for supported character sets.

4. Validate your input

I can not stress this point enough. Check that your input actually is UTF-8, or the multibyte aware functions might not work as expected. Also, if not validated, those who view or store your data might get security problems like SQL-injection (though it would be their fault it’s happening …). This can be done as follows for mbstring:

//last parameter can still be dropped as showed in last example
$validUTF8 = mb_check_encoding($string, 'UTF-8'); //will give true if valid, false if not

for iconv you’ll have to go for a bit more dirty solution

function validateUTF8_iconv($before)
{
    $after = iconv('UTF-8', 'UTF-8', $before);
    return ($before === $after);
}

The reason this works is because any non-valid characters are removed or changed, and thus the strings will not be equal anymore.

5. Store your data correct

Now this is really a pit a lot of people fall into. Storing UTF-8 on disc is no problem, and is done just as before. Storing data in a database however, is in the case of MySQL definitely not as before. First of you’ll need MySQL 4.1 or later as the versions before don’t support what we’re about to do. The big problem with PHP and MySQL is that the connection is by default set to latin1, also known as ISO-8859-1, and a lot of people then actually store there data as a UTF-8-transformed ISO-8859-1. That is, MySQL thinks your input is ISO-8859-1 and then  convert it to Unicode and encode it as UTF-8. This will lead to problems when you view your data in for instance phpMyAdmin which connects to MySQL the proper way when dealing with UTF-8. The correct way to connect to MySQL is now

$link = mysql_connect('localhost', 'mysql_user', 'mysql_password');
mysql_query("SET NAMES 'utf8' COLLATE 'utf8_general_ci'");

The difference is the second line which states what charset is being used for further talk on the connection. The collate part can be dropped if you’re using the default one (utf8_general_ci). In addition to this, the table and fields should be defined with collation utf8_[something] and to save space don’t use char-fields as they will use 3x number of characters you define. Instead you should use varchar. Please also note that MySQL only supports Unicode 3.0, that is 2 byte Unicode up to code-point FFFE (BOM) or 3 byte UTF-8. If you need any code-points above that, you’re unfortunately in for some trouble …

This is however only the case of MySQL. For other database systems you should check what is the default charset and how to change it. The documentation/manual of the system is a good place to start to find this type of information.

6. Functions operating on UTF-8

The points above should make sure that you’re using UTF-8. The last thing is that you have to remember that any str-functions might not work correctly. Try to use str-functions defined in mbstring or iconv (see documentation). Some functions do however work as expected. Strcmp does only a binary compare and still works, strcasecmp however don’t. Str_replace will also work on valid UTF-8 strings as any given character has a unique byte sequence, but the case-less version str_ireplace don’t. In general any str-function that is not case-less and don’t need to operate on the number of characters in the string, should work just fine as long as any input to the function is valid UTF-8.

There are also some functions that are not part of mbstring or iconv that do support UTF-8. Htmlspecialchars, htmlentities and preg_* (with u modifier) are examples of this. There are also functions that operate on a purley binary level without any regards to charset. Examples of this is the md5 and sha1 functions.

That concludes this part of the howto. You should now be able to use UTF-8 the right way. If you have any questions, please leave a comment and I’ll happily answer. The next part will hopefully deal with some common pitfalls and how to debug and solve them.

<?php
$res = array_fill(0, 5, 0);
for($i = 0; $i < 5; $i++)
{
    ob_start();
    $start = microtime(true);
    for ($j = 0; $j < 1000000; $j++)
        echo ".";
    $end = microtime(true);
    $res[$i] = $end-$start;
    ob_end_clean();
}
for ($i = 0; $i < 5; $i++)
    echo $res[$i]."\n";
?>

and

<?php
$res = array_fill(0, 5, 0);
for($i = 0; $i < 5; $i++)
{
    ob_start();
    $start = microtime(true);
    for ($j = 0; $j < 1000000; $j++)
        print ".";
    $end = microtime(true);
    $res[$i] = $end-$start;
    ob_end_clean();
}
 
for ($i = 0; $i < 5; $i++)
    echo $res[$i]."\n";
?>

As you can see I’ve tried to make this as simple as possible. I’ve used output buffer to remove the huge variables also known as browser and network (yes, they’re really a bottleneck here, trust me or try comparing firefox with lynx). Testing was done in the 64bit version of Fedora 9 (which is a linux distro) with PHP 5.2.6 on an AMD Athlon 64 X2 4600 (dual core CPU running at standard 2.4GHz).

Running this gives me an average of 0.333 seconds for echo and 0.346 seconds for print. Doing some calculation you’ll see that the average difference between each echo and print is 13ns or 0.000000013 seconds. Even though this is a 3.8% difference, it’s not even a millionth of a second, but 13 billionth! Hardly any difference if you ask me. I know, it’s just testing with a single, silly punctuation mark. To prevent people from going “There must be a bigger difference with a longer string!”, I ran the test with the first paragraph of lorem ipsum.

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Nunc luctus arcu vehicula est. Donec facilisis iaculis magna. Mauris neque dui, varius in, fermentum id, scelerisque et, massa. Nam ac velit nec odio molestie pellentesque. Nulla dolor mauris, tempus ultrices, cursus in, ultrices sollicitudin, orci. Sed at ligula. Sed id erat id nisl molestie tempus. Vestibulum nibh dolor, vulputate nec, dictum non, sollicitudin nec, nibh. Sed vitae diam eget felis dignissim tempor. Aenean vel risus. Integer consectetuer nibh. Ut eu nunc. Donec at sapien.

This is 551 characters, which should satisfy most people. Results? 0.987 seconds for echo and 1.014 seconds for print, meaning echo is 2.6% faster. Again, hardly any difference as this is 27ns or 0.000000027 seconds. Conclusion: Myth busted! You’ll have to do a ridicules amount of outputing to make any actual difference.

PS: If you run this on your own computer, I would love to hear about your results (including type of CPU, OS and PHP version). Remember this though: Please bar in mind that you’ll have to disable any CPU-throttling and run it in separate files to get accurate results.

First of all, most of these tips are undocumented and presented as obvious facts. This makes them hard to disprove or confirm as the reality is a bit different and more complicated than what these people tend to think. A good example of such a clam is “a foreach-loop is faster than a for-loop” or vica versa. I’ve tested this a couple of times, and though I get a consistent result in favor of for-loop, just changing the platform from Linux to Windows seems to reverse the result. This displays a major problem. Even the slightest difference in OS, software version, hardware etc. can change the results significantly. This is never mentioned by people presenting optimization “tips”.

Second, even though the result is correct, it’s often pointless to actually make the change because you’ll never actually save considerable amount of time. Let’s look at the first point in both my links.

If a method can be static, declare it static. Speed improvement is by a factor of 4.

Sounds good, doesn’t it? Speed improvements by a factor of 4! That must be significant, right? No, not necessarily. It can very well be an improvement by a factor of 4 (even though my testing seem to show about equal speed), but that’s only as significant as the time it initially takes. If one execution takes, say 1µs, saving 0.75µs isn’t that significant (µs = microsecond = 10^-6 s = 0.000001) considering the function itself might using 100µs on each execution. This is very often the case when these people present their “tips” if they’re actually correct on what’s the faster part.

Third, with the previous stuff in mind. Even if the clam is correct and even if there is some performance to be gained, is it worth it? Not necessarily. There are several things to keep in mind here. Primarily optimization degrades readability, and there’s also the issue of someone actually have to make these changes. Is it really that worth it if you have to spend an hour changing alot of code just to have a cumulative saving of about 1ms? I don’t think so …

The final nail in the coffin for these “tips” is the fact that it’s never ever tested with any load. Load is important as that’s the reality. It doesn’t matter if solution A is x times faster than solution B saving y amount of seconds, if solution A hits a bottleneck when it’s run in 10 parallel requests making it much slower than solution B under the same conditions. Here’s where real optimization come in handy. Caching stuff, using a PHP accelerator, having a thought through algorithm, actually knowing what’s the perfomance hit and so on is always going to beat these type of lists hands down anytime. That’s the reality of optimization, not whether or not you use single or double quotes.

Because PHP basically fail to distinct between for instance foo::bar(); as in namespace foo with function bar vs. class foo with static function bar they’ve just decided to scrap the whole :: thing and instead use backslash. I can’t tell you how stupid I think this really is. Instead of actually fixing the problem, faulty look-up, they’ve taken the very easy way out. This just adds to the inconsistencies for PHP. There are several programing languages that I know of that don’t have these problems, and none of them have namespace separation in this way. Actually, I can’t even remember ever seeing any language using backslash for anything other than escaping, but that might just have something to do with other languages being consistent. This quote from a comment on slashdot really says it all:

Java:
Attribute/Method access: foo.bar
Static method access: Foo.bar
Package access: foo.bar.baz

C#:
Attribute/Method access: foo.bar
Static method access: Foo.bar
Namespace access: foo.bar.baz

Python:
Attribute/Method access: foo.bar
Static method access: Foo.bar
Module access: foo.bar.baz

PHP:
Attribute/Method access: $foo->bar
Static method access: Foo::bar
Namespace access: foo\bar\baz

Now I’m not quite done here. If you look at the RFC at PHP about the change they list “IDE compatibility” as criterion for the new namespace separator. This is just plain stupid. Who are you designing a language for, humans or IDEs? I would say it’s for humans, but it seems the PHP devs thinks otherwise …

First of all, you shouldn’t be using a foreign charset or encoding without knowing it’s risk. Unicode requires a mulibyte encoding, which has it’s own set of problems compared to singlebyte charsets like ISO-8859-1 and simliar. As I’m only really familiar with UTF-8, I’m going to use that as an example. I could proably write a book about the problems, but here’s a small list:

• BOM (U+FEFF) and “reverse”-BOM (U+FFFE). This is not really a problem in UTF-8 as it doesn’t have the issues with little and big endian (wikipedia about endianness), but if you convert to for instance UTF-16, having a “reverse”-BOM at the start of your document would be a disaster.
• Str-functions aren’t multibyte aware and might “break” your strings.
• Non-shortest form. There’s a bit of math behind this one, which I’m not going to explain today, so you’ll just have to trust me on this one. Non-shortest form is an illegal representation of a Unicode character. It basicly is represented using more bytes than necessary and in UTF-8 this results in ‘ , which in hex terms is 0×27, could be represented by 0xC0A7. If those who developed for example the RDBMS you’re using is just as unknowing as alot of people out there, they will decode or recode this to UTF-16, UCS-2 or UCS-4 and get a single quote ( ‘ ) which might just result in a SQL-injection …
• Surrogates. In UTF-8 surrogates (U+D800-U+DFFE) are not allowed, and is actually regared as a potential security risk. Allowing these will result in a � or a similar “this is an illegal byte”-character being displayed.
• Illegal combination of bytes resulting in a � or a similar “this is an illegal byte”-character being displayed.

In addition to those above, there is also the possibility of security problems when you’re actually escaping input with functions not aware of multibyte charsets (here’s an example). With all this in mind I can tell you why mbstring is such a bad piece of coding.

First of all it doesn’t have a complete replacement for every str-function. We’re missing *sort, *trim, strcasecmp, str_ireplace, ucfirst, ucwords and wordwrap just to mention some. The second thing is that mbstring is writen by a Japanese or something, which is quite obvious considering strcasecmp is missing. … and no, using mb_strtolower and strcasecmp is not a valid way as the process requires either simple or full case folding which has some differences from the “to lower”-process. Anyway, mbstring is writen more or less to enable Japanese, Chinese and similar languages to work. The thing is, these languages don’t have case as most other languages which explains the lack of the above.

Third, there is some bugs in mbstring which result in invalid validaton. Here are som examples:

• Anything behind a null-byte isn’t validated
• Surrogates are allowed
• “Reverse”-BOM is allowed
• Title case isn’t working

Fourth and final, the overloading abilites make the str-functions behave different from the original str-functions. These are undocumented differences, possibly breaking applications expecting str-functionality. For instance, if 2. paramter in strrchr is an int, is converted to a character. In mb_strrchr however, this will give an error.