Web Hosting Talk : Web Hosting Main Forums : Programming Discussion : Need help with NGINX/Roboo/Perl syntax

[infinitnet]

Hello WHT,

I'm trying to integrate a whitelist for Search Engine Spiders for the NGINX module "Roboo", written in Perl.

From README:

Roboo_whitelist         -   "UA('Googlebot'),IP(127.0.0.0/8),URI('^/ajax/')"
        - UA('pcre_string') and URI('pcre_string') syntax whitelists a user-agent or URI string using PCRE
        - IP(0.0.0.0/0) syntax whitelists a CIDR network

It works, if I write it as following:

set $Roboo_whitelist "UA('user-agent-1'),UA('user-agent-2')";

However NGINX gives me a 500 error, if I try to include full user agent strings with brackets and semicolon:

set $Roboo_whitelist "UA('user-agent-1 (+http://www.google.com/adsbot.html)'),UA('user-agent-2 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)')";

Can anyone help me with this? I'm not really a coder.

[barbus]

What is in error log?

[infinitnet]

Got it myself. Some special characters need to be escaped with \, so it should look like:

set $Roboo_whitelist "UA('user-agent-1 \(\+http\:\/\/www\.google\.com\/adsbot\.html\)'),UA('user-agent-2 \(compatible\; Googlebot\/2\.1\; \+http\:\/\/www\.google\.com\/bot\.html\)')";

Or similar. Will be much fun to write every search engine spider like this.