Feb 25

php — how to detect spider/crawler with php

I am going to write a PHP script today that detects whether a search engine spider is crawling a page.With PHP, we can check the value of $_SERVER['HTTP_USER_AGENT'],a predefined variables which store the contents of the User-Agent: header from the current request, if there is one.

$pattern = ‘/(google|msnbot|MSNPTC|yahoo|Slurp|overture.com|del.icio.us|DoCoMo|DoCoMo)/i’;
if ( preg_match($pattern, $_SERVER['HTTP_USER_AGENT'] === FALSE) { // not a spider

} else { // spider

}

You can find a list of search engine spiders/crawlers from  http://www.user-agents.org/.

Feb 15

magento — translate words into other language for payment module

my new mangento payment module need multiple language support.Magento interface is translated with CSV localization files which was stored at the location of app/locale/[lang_folder],ie app/locale/en_us.OR you can use the built-in Inline translation tool which store the translation pair into database.it’s much more convenience to translate via Csv localization files than the inline translation tool when you want to translate lots of string at a time.

First, i create an empty csv file named Mzcart_Pagbrasil.csv in app/locale/en_us/.I add a translation for test purpose:

“Powered by Pagbrasil”,”Powered by Pagbrasil—-”

Second,to make the new added csv localization file visible to magento, i need to update the config xml file of this module.add the config below to config.xml file under  config >> frontend path.

<translate>
<modules>
<Mzcart_Pagbrasil>
<files>
<default>Mzcart_Pagbrasil.csv</default>
</files>
</Mzcart_Pagbrasil>
</modules>
</translate>

However,  the string “Powered by Pagbrasil” was not translated. I did clear the cache. After googling for a long time without any clue, i added a new string and it was translated,but the first string still not changed.

“Powered by Pagbrasil”,”Powered by Pagbrasil—-”        >>>>>>> still not translated.
“Please pay your order now”,”Please pay your order now@@@@@@@@@@@@”

what’s the problem?the problem is that php does not support utf8 with bom. a BOM-ed UTF-8 string will start with the three following bytes EF BB BF. php will treat this bytes as file content.this lead to the mismatch of the first string. you may find that all lang file provided by magento all encoded in utf8 without bom.

For more info you may refer to http://stackoverflow.com/questions/2223882/whats-different-between-utf-8-and-utf-8-without-bom

Feb 13

magento — how to get Vat number/tax id

i am developing a new payment module under magento.Everything going through well as usual.However, this new payment module require a cutomer tax id. by default,magento didn’t show this field for customer.To make it available for all customer,log into magento backend, goto

System > Configuration > Customers > Customer Configuration > Create New Account Options [tab]

change the config of “Show VAT Number on Frontend” to “yes”.

so how to get the new added filed — customer vat number/taxt id.you need to get billing or shipping model first.the code below as an example.

$billing  = $this->getOrder()->getBillingAddress();
$billing->getVatId();

 

 

Feb 12

apache — password protect your web resource only to external user

Think about  that an IT company would like to protect their example project from Internet user,but allow their customer to access with password.however,the developer would not like to input password.

if you wanted to let people on your network have unrestricted access to a portion of your website, but require that people outside of your network provide a password, you could use a configuration similar to the following:

Require valid-user
Allow from 192.168.1
Satisfy Any

you can put the config directive into a .htaccess file or enclose the config diretive with apache Directory directive.With the Satisfy Any directive the client will be granted access if they either pass the host restriction or enter a valid username and password.But please keep in mind that the config may not work if your server allow all user access by default.we should tweak the config to be more restricted.

Order Allow,Deny
AuthType Basic
AuthName “Restricted Resource”
AuthUserFile /var/users
Require valid-user

Allow from 192.168.1
Satisfy Any

[OR]

Order Deny,Allow
Deny from All
AuthType Basic
AuthName “Restricted Resource”
AuthUserFile /var/users
Require valid-user

Allow from 192.168.1
Satisfy Any

 

 

Nov 27

remove server info and PHP info from response header

Below is a comm http response header:

HTTP/1.1 200 OK
Date: Wed, 27 Nov 2013 01:18:27 GMT
Server: Apache/2.0.55 (Debian) PHP/5.1.2
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Encoding: gzip
Vary: Accept-Encoding
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8

To keep your server from attack, you should hide all unnecessary information about your system.With apache web server,this was control by ServerTokens  and ServerSignature config directive.From apache manual,we find out  all available config value of ServerTokens and its sample out.

ServerTokens Prod[uctOnly]
Server sends (e.g.): Server: Apache
ServerTokens Major
Server sends (e.g.): Server: Apache/2
ServerTokens Minor
Server sends (e.g.): Server: Apache/2.0
ServerTokens Min[imal]
Server sends (e.g.): Server: Apache/2.0.41
ServerTokens OS
Server sends (e.g.): Server: Apache/2.0.41 (Unix)
ServerTokens Full (or not specified)
Server sends (e.g.): Server: Apache/2.0.41 (Unix) PHP/4.2.2 MyMod/1.2

Obviously, we should turn our config to ServerTokens Prod.

To stop apache from exposing info in related error page,we need to turn off ServerSignature.

ServerSignature off

To hide the php related information,locate your php config file, find or add expose_php.

expose_php off

Nov 17

linux – how to replace special string in multiple files

As an administrator, you will need to fix an error for all user.I found that there are some users use [NC] flag with -f,-d,-l in their RewriteCond directive in zen cart .htaccess file. This misconfiguration is not so critical but it will force the busy server stop  to write to disk and generate huge big error log.i can not just sent a  notice to all the related customers about fixing their error.you know the customer is the god.

i don’t want to correct all the .htaccess file one by one.So,first i need to locate all the file contain
RewriteCond %{REQUEST_FILENAME} !-f [NC]

The command below will list all the .htaccess file containing ‘!-f [NC]‘ from all users’ document root.
# grep -ril ‘\!-f \[NC\]‘ /home/*/www/.htaccess

Now,with the single command below, i will replace all ‘!-f [NC]‘ with ‘!-f‘ in all related .htaccess files found.
#sed -i ‘s/\!-f \[NC\]/\!-f/g’ `grep -ril ‘\!-f \[NC\]‘ /home/*/www/.htaccess`

 

Nov 16

Apache – RewriteCond: NoCase option for non-regex pattern xx is not supported

we got lots of ‘NoCase option not supported’ warning like below from our apache server error log

[Fri Nov 15 20:46:14 2013] [warn] RewriteCond: NoCase option for non-regex pattern ‘-f’ is not supported and will be ignored.

As you may have guessed the problem is with your rewrite rule and one or more lines RewriteCond in your .htaccess file. From the mod_rewrite documentation,i found this

‘nocase|NC’ (no case)
This makes the test case-insensitive – differences between ‘A-Z’ and ‘a-z’ are ignored, both in the expanded TestString and the CondPattern. This flag is effective only for comparisons between TestString and CondPattern. It has no effect on filesystem and subrequest checks.

Pay attention to the last sentence It has no effect on filesystem and subrequest checks. this means that ‘nocase|NC’  is no need when you use ‘-d’ (directory) ‘-f’ (regular file) and ‘-l’ (symbolic link)  in your code pattern in the RewriteCond directive. they will just be ignored.

Unfortunately it is not explicitely written but these CondPatterns will only work in case sensitive mode when it contains filesystem related flags(-d,-f,-l).so keep in minds that you can not test insensitive mode in linux server.This means

If there is a file name A.php in domain(abcdomain.com) root it can be reached via http://www.abcdomain.com/A.php but not http://www.abcdomain.com/a.php

Someone may find that the warning messages still showing up that the waring show up in the error log file after the removed the NC flag.i thought it was because the error log was global error log,so the warning messages will continue showing up before you fix all website on the server.

 

 

Oct 23

whm/cpanel – SoftException in Application.cpp: Directory / is not owned by

in a sudden,all website throw out “500 internal server error”. i got those message from apache error log:

[Tue Oct 22 16:28:27 2013] [error] [client 95.108.217.252] SoftException in Application.cpp:592: Directory / is not owned by falc0961
[Tue Oct 22 16:28:27 2013] [error] [client 95.108.217.252] Premature end of script headers: index.php

Yes, the php is runnning in su_php mode. so this maybe caused by permission.i got no clue after i check the doc root of the related website. the owner and permission is ok.

At last i put my attention on the error log

—- SoftException in Application.cpp:592: Directory / is not owned by falc0961.

i don’t know why it check the root directory.but when i found out that the root is belong to some user id other than root user.so i fix the problem with the simple chown command as:

chown root.root /

(it was casued by copying some directory content to root directory,which overwrite the . file.)

 

Oct 19

apache — 301 redirect from long domain www.abcdomain.com to short abcdomain.com

For search engine, www.abcdomain.com and abcdomain.com are different domain.For seo purpose,we need to do 301 redirect to tell search engine to  treat the two domain as one.

To redirect long form domain with www to the domain without www,you can use the rewrite rule below.this can help you save time if you have lots of websites.

RewriteCond %{HTTP_HOST} ^([a-z0-9-]+)\.([a-z]+)$
RewriteRule (.*) http://www\.%1\.%2/$1 [R=301,L]

OR

RewriteCond %{HTTP_HOST} ^([a-z0-9-]+)\.([a-z]+)$
RewriteRule (.*) http://www\.{HTTP_HOST} [R=301,L]

On the other way,if you prefer to keep short form domain,you can adapt the rule to:

RewriteCond %{HTTP_HOST} ^www.([a-z0-9-]+)\.([a-z]+)$
RewriteRule (.*) http://%1\.%2/$1 [R=301,L]

Oct 17

linux — forward user email

To forward user’s email,just put an .forward file with the email addr in the user’s home diretory.If this not work for your server,Make sure the .forward file is belong to the user you want to forward email.for example,if you want to forward all email for user testuser, change current working directory to /home/testuser,touch a .forward file.

# cd /home/testuser
# touch .forward
# echo ‘abctest@abc.com’ >> .forward
# chown testuser.testuser .forward
# chmod 664 .forward

if this not work,you may check the log file located at /var/log/maillog for more info.

And keep in mind,only in a Postfix/Sendmail system it is sendmail command that utilies this file.For other situation,you may check the related document.