We’ve all seen it, captcha or pictures that protects alot of web-forms against those nasty spam-bots while at the same time blissfully destroying the usability of the web (10 “good” examples). While a lot of attempts have been made to make it more usable for more people (pictures aren’t exactly user-friendly for blind people …), it still boils down to some requirements either for the browser (e.g. javascript) or the user (typing into an input field). While requiring the browser to support something isn’t that bad, requiring the user to do something definitely isn’t usability at it’s best. Wouldn’t it be nice to just instant see whether or not the current request is made from a bot or a browser? Well, you might be able …

For the last two weeks I’ve created my personal pet project logging spam attempts in some scripts running at Ascended Development. After more than 300 attempts and above 100 unique spam-bots (or IPs anyway), the results shock me … a lot. The spam-bots that has visited the site is incredible stupid. In simplest terms they don’t even seem to be able to handle cookies (e.g. thereby also session in PHP), little less the pictures or javascript. The most ridiculous thing is that they actually do send data in the antispam input field. The problem is that it’s 2 to 4 times longer than the number of characters in the picture, and yes, it didn’t supply the cookie for the session meaning it wouldn’t succeed even if the wild guess was right.

So what can we do without requiring user input or javascript? Certainly not add a hidden input field in the form and hope that the bot add something to it. All bots (yeah, actually every single one of them) either provide the default value or didn’t provide the field at all. Actually out of the 300 attempts only 50 didn’t provide the field, and a massive 250 attempts had the default value. None tried any other values, and it seems that those that didn’t provide the field tried again with the default value. Simply put, a hidden field don’t work, at least not on those bots visiting our site at the logged locations.

So here’s the trick: Simply read the HTTP-headers. Has to be too good to be true? Well, the log is equally clear on this matter too. None of the bots provided the fields Accept-Language and Accept-Encoding, both of which quite frankly any decent browser sends out these days (Opera, Firefox, Konqueror, Chrome, Safari and even IE). Even lynx, a text browser, does send these headers, and I tested it with a two year old release. It does make sense if you think about it. The browsers will add the Accept-Language so pages can be correctly localized and Accept-Encoding so that compression can be used. Both things is benefitial to the user, and therefor present despite both being optional. The spambots on the other hand seem to be using libaries like libcurl to build their HTTP-client, and by default these libraries don’t seem add Accept-Language or Accept-Encoding. Add the fact that few, if any, sort spam-bots from browsers this way, and we can see that it’s not really such a surprise after all.

This isn’t without it’s flaws though. I’ve only encountered simple, general spam-bots and not the ones attacking widespread software like phpBB or vBulletin. Also, the site isn’t subject to targeted attacks. That been said, I wouldn’t be surprised if they too fail to provide these HTTP-headers, and for the time being I’m quite confident that this method is about as efficient, or better, than the current widespread method of using pictures. It should also continue to be that way until this type of checking is more widely used. So until then, I believe this is a good way to avoid spam in you web applications:

<?php
if (isset($_SERVER['HTTP_ACCEPT_ENCODING']) === false ||
    isset($_SERVER['HTTP_ACCEPT_LANGUAGE']) === false)
{
    echo 'Spambot';
}
else
{
    echo 'Browser';
}
?>

If only this was working against email spam too …

The basic facts

The key element when using a multibyte character set in PHP is to know exactly what you’re doing. If you don’t, you can easily end up with partially corrupted text and wrong results. The one big reason for this is the fact that PHP by default does not support anything other than byte-sized character set. In fact, strictly speaking, PHP doesn’t really know what a character set is. All it sees are bytes, not characters. This means that every string-function in PHP works on the assumption that a byte is a character. When dealing with for instance UTF-8 this is no longer true. The result is that strlen reports the number of bytes in the string, not the number of characters. Similarly, strpos will give you the position in bytes, not characters, and many of the other string-functions have similar problems. So what to do?

First off, a very handy fact about UTF-8 is it’s ASCII-compatible (7bit ASCII that is), meaning these characters are binary represented as 0xxx xxxx (where x are ASCII bits). Another handy fact about valid UTF-8 strings is that any encoded Unicode character has a unique byte sequence, meaning it can’t be confused with a part of another character. This means that if you encounter a byte 00100000 (20 hex or 32 dec) it can not be anything other than a space character, or else it’s not a valid UTF-8 string. For those interested in how this is archived, here’s the binary representation of UTF-8 characters (skip if you’re not into that type of stuff ;o)

1 byte:  0xxx xxxx
2 bytes: 110x xxxx 10xx xxxx
3 bytes: 1110 xxxx 10xx xxxx 10xx xxxx
4 bytes: 1111 0xxx 10xx xxxx 10xx xxxx 10xx xxxx

Well, enough with talk about UTF-8 in general. Here’s a step-by-step guild how to use UTF-8 in PHP.

1.  Find you what you have available

If you’re going to use UTF-8 in PHP you’ll first need to find you what’s available to you. Ideally you should have the PHP extension mbstring install and it should not be set to overload str-functions. With mbstring you’ll have a set of functions that are multibyte aware. If you don’t have mbstring available, check for iconv (also an PHP extension). In PHP 5 and later this extension will give you some very simple functions to work with (strlen, strpos, strrpos, substr and validation). If you have neither mbstring nor iconv available at your host (I’m assuming that you’re going to run something in a hosted server), you should strongly consider change host and/or your need for Unicode/UTF-8, as you’re going to have to make some native functions that works properly on UTF-8 encoded strings. For the sake of simplicity, I’m going to assume you have mbstring or iconv installed.

2. Store your files as UTF-8

This should be somewhat obvious. If you’re going to use UTF-8, you should store your files as it too. The simple reason is that any strings you have stored in your scripts will be UTF-8 too, and should be outputed properly given you’ve done everything else correct.

3. Define input and output as UTF-8

This is quite depending on what you’re doing, but chances are that you’re dealing with the HTTP-protocol and HTML. If so you have to add the following function call before any output (if you’re not using output buffering).

header('Content-Type: text/html; charset=utf-8');

and add the following to the head-part of your HTML-document

<meta http-equiv=Content-Type content="text/html; charset=utf-8">

This will state that any output is html and UTF-8. If you’re not dealing with html, just replace text/html with whatever you’re using (and proably drop the meta tag too). This should also ensure that input from most browsers is UTF-8, but to be really sure it might be a good idea to add the attribute accept-charset to any forms you might have, like this:

<form accept-charset="UTF-8" method="post">
<!-- Input stuff here -->
</form>

This attribute can also be used with a comma-separated list of character set your script accept. To keep it simple you should just use UTF-8 as the only accepted “character set” (strictly speaking, UTF-8 is an encoding of the Unicode character set, and not a charset by itself).

If you don’t have control over the input, for instance RSS feed from 3. party server, transform it to Unicode and encode to UTF-8. This can be done like this:

//mbstring
$UTF8string = mb_convert_encoding($string, 'other charset', 'UTF-8');
// or call
mb_internal_encoding('UTF-8');
// once to use UTF-8 as default and and drop last parameter like this:
$UTF8string = mb_convert_encoding($string, 'other charset');
 
//iconv
$UTF8string = iconv('other charset', 'UTF-8', $string);

Consult the PHP manual for supported character sets.

4. Validate your input

I can not stress this point enough. Check that your input actually is UTF-8, or the multibyte aware functions might not work as expected. Also, if not validated, those who view or store your data might get security problems like SQL-injection (though it would be their fault it’s happening …). This can be done as follows for mbstring:

//last parameter can still be dropped as showed in last example
$validUTF8 = mb_check_encoding($string, 'UTF-8'); //will give true if valid, false if not

for iconv you’ll have to go for a bit more dirty solution

function validateUTF8_iconv($before)
{
    $after = iconv('UTF-8', 'UTF-8', $before);
    return ($before === $after);
}

The reason this works is because any non-valid characters are removed or changed, and thus the strings will not be equal anymore.

5. Store your data correct

Now this is really a pit a lot of people fall into. Storing UTF-8 on disc is no problem, and is done just as before. Storing data in a database however, is in the case of MySQL definitely not as before. First of you’ll need MySQL 4.1 or later as the versions before don’t support what we’re about to do. The big problem with PHP and MySQL is that the connection is by default set to latin1, also known as ISO-8859-1, and a lot of people then actually store there data as a UTF-8-transformed ISO-8859-1. That is, MySQL thinks your input is ISO-8859-1 and then  convert it to Unicode and encode it as UTF-8. This will lead to problems when you view your data in for instance phpMyAdmin which connects to MySQL the proper way when dealing with UTF-8. The correct way to connect to MySQL is now

$link = mysql_connect('localhost', 'mysql_user', 'mysql_password');
mysql_query("SET NAMES 'utf8' COLLATE 'utf8_general_ci'");

The difference is the second line which states what charset is being used for further talk on the connection. The collate part can be dropped if you’re using the default one (utf8_general_ci). In addition to this, the table and fields should be defined with collation utf8_[something] and to save space don’t use char-fields as they will use 3x number of characters you define. Instead you should use varchar. Please also note that MySQL only supports Unicode 3.0, that is 2 byte Unicode up to code-point FFFE (BOM) or 3 byte UTF-8. If you need any code-points above that, you’re unfortunately in for some trouble …

This is however only the case of MySQL. For other database systems you should check what is the default charset and how to change it. The documentation/manual of the system is a good place to start to find this type of information.

6. Functions operating on UTF-8

The points above should make sure that you’re using UTF-8. The last thing is that you have to remember that any str-functions might not work correctly. Try to use str-functions defined in mbstring or iconv (see documentation). Some functions do however work as expected. Strcmp does only a binary compare and still works, strcasecmp however don’t. Str_replace will also work on valid UTF-8 strings as any given character has a unique byte sequence, but the case-less version str_ireplace don’t. In general any str-function that is not case-less and don’t need to operate on the number of characters in the string, should work just fine as long as any input to the function is valid UTF-8.

There are also some functions that are not part of mbstring or iconv that do support UTF-8. Htmlspecialchars, htmlentities and preg_* (with u modifier) are examples of this. There are also functions that operate on a purley binary level without any regards to charset. Examples of this is the md5 and sha1 functions.

That concludes this part of the howto. You should now be able to use UTF-8 the right way. If you have any questions, please leave a comment and I’ll happily answer. The next part will hopefully deal with some common pitfalls and how to debug and solve them.

<?php
$res = array_fill(0, 5, 0);
for($i = 0; $i < 5; $i++)
{
    ob_start();
    $start = microtime(true);
    for ($j = 0; $j < 1000000; $j++)
        echo ".";
    $end = microtime(true);
    $res[$i] = $end-$start;
    ob_end_clean();
}
for ($i = 0; $i < 5; $i++)
    echo $res[$i]."\n";
?>

and

<?php
$res = array_fill(0, 5, 0);
for($i = 0; $i < 5; $i++)
{
    ob_start();
    $start = microtime(true);
    for ($j = 0; $j < 1000000; $j++)
        print ".";
    $end = microtime(true);
    $res[$i] = $end-$start;
    ob_end_clean();
}
 
for ($i = 0; $i < 5; $i++)
    echo $res[$i]."\n";
?>

As you can see I’ve tried to make this as simple as possible. I’ve used output buffer to remove the huge variables also known as browser and network (yes, they’re really a bottleneck here, trust me or try comparing firefox with lynx). Testing was done in the 64bit version of Fedora 9 (which is a linux distro) with PHP 5.2.6 on an AMD Athlon 64 X2 4600 (dual core CPU running at standard 2.4GHz).

Running this gives me an average of 0.333 seconds for echo and 0.346 seconds for print. Doing some calculation you’ll see that the average difference between each echo and print is 13ns or 0.000000013 seconds. Even though this is a 3.8% difference, it’s not even a millionth of a second, but 13 billionth! Hardly any difference if you ask me. I know, it’s just testing with a single, silly punctuation mark. To prevent people from going “There must be a bigger difference with a longer string!”, I ran the test with the first paragraph of lorem ipsum.

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Nunc luctus arcu vehicula est. Donec facilisis iaculis magna. Mauris neque dui, varius in, fermentum id, scelerisque et, massa. Nam ac velit nec odio molestie pellentesque. Nulla dolor mauris, tempus ultrices, cursus in, ultrices sollicitudin, orci. Sed at ligula. Sed id erat id nisl molestie tempus. Vestibulum nibh dolor, vulputate nec, dictum non, sollicitudin nec, nibh. Sed vitae diam eget felis dignissim tempor. Aenean vel risus. Integer consectetuer nibh. Ut eu nunc. Donec at sapien.

This is 551 characters, which should satisfy most people. Results? 0.987 seconds for echo and 1.014 seconds for print, meaning echo is 2.6% faster. Again, hardly any difference as this is 27ns or 0.000000027 seconds. Conclusion: Myth busted! You’ll have to do a ridicules amount of outputing to make any actual difference.

PS: If you run this on your own computer, I would love to hear about your results (including type of CPU, OS and PHP version). Remember this though: Please bar in mind that you’ll have to disable any CPU-throttling and run it in separate files to get accurate results.

First of all, most of these tips are undocumented and presented as obvious facts. This makes them hard to disprove or confirm as the reality is a bit different and more complicated than what these people tend to think. A good example of such a clam is “a foreach-loop is faster than a for-loop” or vica versa. I’ve tested this a couple of times, and though I get a consistent result in favor of for-loop, just changing the platform from Linux to Windows seems to reverse the result. This displays a major problem. Even the slightest difference in OS, software version, hardware etc. can change the results significantly. This is never mentioned by people presenting optimization “tips”.

Second, even though the result is correct, it’s often pointless to actually make the change because you’ll never actually save considerable amount of time. Let’s look at the first point in both my links.

If a method can be static, declare it static. Speed improvement is by a factor of 4.

Sounds good, doesn’t it? Speed improvements by a factor of 4! That must be significant, right? No, not necessarily. It can very well be an improvement by a factor of 4 (even though my testing seem to show about equal speed), but that’s only as significant as the time it initially takes. If one execution takes, say 1µs, saving 0.75µs isn’t that significant (µs = microsecond = 10^-6 s = 0.000001) considering the function itself might using 100µs on each execution. This is very often the case when these people present their “tips” if they’re actually correct on what’s the faster part.

Third, with the previous stuff in mind. Even if the clam is correct and even if there is some performance to be gained, is it worth it? Not necessarily. There are several things to keep in mind here. Primarily optimization degrades readability, and there’s also the issue of someone actually have to make these changes. Is it really that worth it if you have to spend an hour changing alot of code just to have a cumulative saving of about 1ms? I don’t think so …

The final nail in the coffin for these “tips” is the fact that it’s never ever tested with any load. Load is important as that’s the reality. It doesn’t matter if solution A is x times faster than solution B saving y amount of seconds, if solution A hits a bottleneck when it’s run in 10 parallel requests making it much slower than solution B under the same conditions. Here’s where real optimization come in handy. Caching stuff, using a PHP accelerator, having a thought through algorithm, actually knowing what’s the perfomance hit and so on is always going to beat these type of lists hands down anytime. That’s the reality of optimization, not whether or not you use single or double quotes.

We start by setting up the paths to our new domin. It might be a good idea to put new domains in the home directory. Though you are free to put them where you want.

I personally use the home directory, and will base my short guide on doing so.

Start by creating the domain as a folder in the home directory.

# mkdir /home/example.tld

Next give apache owner rights over the folder

# chown apache:apache /home/example.tld

Change the directory to the example.tld.

# cd /home/example.tld

We want to make two directories, one to store our web page to example.tld, and one to store subdomains.

# mkdir htdocs
# mkdir subdomains
# chown apache:apache htdocs
# chown apache:apache subdomains

Since we are going to add one subdomain, we will add our subdomain “admin” to our subdomain folder.

# mkdir subdomains/admin
# chown apache:apache subdomains/admin

And at last create a htdocs folder in the admin subdomain folder.

# mkdir subdomins/admin/htdocs
# chown  apache:apache subdomains/admin/htdocs

We are almoust ready now. Put your web page files into the htdocs folders, or just create a dummy index.html with “Hello World” or something. If you get many folders and files and need th change the owner on all you can run this command to change all files and dirs to the apache user and group.

# chown apache:apache -R /home/example.tld

Now we are ready to edit the apache configuration.

Open the configuration file (you need to be root or have sudo privileges)

#vi /etc/httpd/conf/httpd.conf

Go to the end of the configuration file and go into insert mode. Add two virtual hosts in this file at the far end.

First Add the virtual host if it’s not present

NameVirtualHost *:80

Then add the virtual host for the main domain, example.tld

<VirtualHost *:80>
DocumentRoot /home/example.tld/htdocs
ServerName example.tld
<Directory “/home/example.tld/htdocs”>
allow from all
Options +Indexes
</Directory>
</VirtualHost>

Next add the virtual host for the subdomain. Repeat this one for every subdomain you want

<VirtualHost *:80>
DocumentRoot /home/example.tld/subdomains/admin/htdocs
ServerName admin.example.tld
<Directory “/home/example.tld/subdomains/admin/htdocs”>
allow from all
Options +Indexes
</Directory>

Save the configuration and restart apache

# service httpd restart

Finished!

Well, its actually going to be a bit nastier for those who get banned.

Your forum account will be directly tied to your Master EA Account, so if we ban you on the forums, you would be banned from the game as well since the login process is the same. And you’d actually be banned from your other EA games as well since its all tied to your account. So if you have SPORE and Red Alert 3 and you get yourself banned on our forums or in-game, well, your SPORE account would be banned to. It’s all one in the same, so I strongly reccommend people play nice and act mature.

Now this is really crazy! Apparently, if EA don’t like what you do on their forum or anywhere else for that matter, you’ll be locked out of your own games! If this is really true, then I’ve never ever seen such a distasteful treatment of customers in my entire life. True, the victims here are most likely not mom’s angel, but they’ve still got rights, and this isn’t mentioned in the TOS. Not that I’m a lawyer or anything, but I think it’s illegal to steal, and if EA regards piracy as stealing, then this is definitely stealing too! So congrats EA, you just became a thief!

Update:  Thankfully, it turns out that this is just a misunderstanding. EA will NOT ban you from you game.

I know subdomains can be setup as wildcards, CNAME and A-records, but right now I use A-record for the subdomains. To just get it going we will consentrate of just getting it working with the following configuration.

Domain: example.tld
Subdomains: mail, admin, mysql
IP: 1.2.3.4

Now we will start by adding the domain to the named.conf – Though it might not be present at this path you might need to search for it. (Using for example webmin would help you show all configuration files).

# vi /etc/named.conf

Add a new zone to named.conf

zone “example.tld” {
type master;
file “/var/named/example.tld.hosts”;
};

Save the configuration

Push escape, then : x

The next step is creating the DNS records.  Add a new file to bind.

# vi /var/named/example.tld.hosts

In this file you will create your DNS-records.

$ttl 15m ; TTL Might be smart to have at 15 minutes until your conf are ok
example.tld. IN SOA ns.example.tld. user.example.tld. (
123456789 ;An unique number
86000 ;Refresh rate in seconds
7200 ;Update Retry in seconds
3600000 ;Expiry in seconds
600 ; minimum in seconds
)

example.tld. IN NS ns.example.tld.
example.tld. IN A 1.2.3.4
www.example.tld. IN CNAME example.tld
mail IN A 1.2.3.4
admin IN A 1.2.3.4
mysql IN A 1.2.3.4
mail.example.tld. IN MX 1 example.tld.

Now save the file and restart BIND

# service bind restart

Seriously, my first impression was “KDE on Vista”. This is the same copycat behavour we’ve seen before from the Redmond-area. For instance, that taskbar looks quite like that in KDE 4, and MS have been “creative” enought to let those widg … oh sorry, I of course mean gadgets, float on the desktop instead of beeing confined to the sidebar. “New!” Riiiiiight … Then there’s the new Explorer, which has it’s resemblance of Dolphin.

I’m all aware of the fact that MS, Apple, KDE etc. are all sort of stealing ideas from each other, but everyone except MS actually does bring something new to the table now and then. When the heck i MS going to do that? When is MS actually going to be innovative instead of constantly being a copycat ending up with a bad copy? Gotta wonder …

The installation of trac is an easy if you got the correct access. Anyhow will I not go through the install process in this guide, but you can find a tutorial on doing that at the trac home page.

When you got plesk installed you should have  you subdomains as virtual hosts. This should be installed automatic from your plesk admin panel.

Lets say your site is named example.tld and your subdomain named trac.

Site name: example.tld
Sub domain: trac

Now log into a SSH client and navigate to the trac subdomain folder

# cd /var/www/vhosts/example.tld/subdomains/trac/

As we just can’t put the project files from trac into the htdocs folder just make a project in this folder so you got a sub folder “project” in the trac directory (or another place).

When you’ve got your project folder in place you are ready to setup the trac for your subdomain. In the subdomain folder locate the folder “conf”. This folder will locate a file that extend the apache configuration, vhost.conf – If not, vi will create it

#vi vhost.conf

Not much configuration necessary here. Add this information (replace the example.tld and path with your environment. And myproject with your project folder for the www-path)

<Location /myproject>
SetHandler mod_python
PythonInterpreter main_interpreter
PythonHandler trac.web.modpython_frontend
PythonOption TracEnv /var/www/vhosts/example.tld/subdomains/trac/project/
PythonOption TracUriRoot /myproject
SetEnv PYTHON_EGG_CACHE /tmp/cache
</Location>

Now restart the apache web server and you are good to go

#service httpd restart
or
#/etc/rc.d/httpd (or apache) restart