Short of manually checking each user's directory is there an easier way to scan the user's directories for illegal content/keywords?

You could do a filename search in the shell. I really dont go looking unless there's a reason to (like i'm contacted). I'm interested in the privacy of my customers and have no need to check what files they have (in most cases) that aren't publicly available.

I want to search within files also, not just the file names. Privacy isn't much of a concern for me because of the greater risk of fraud and illegal activity that comes with running a free web host.

I suppose if it is a free web host its ok to check and that it would be more of a risk. I can't think of anything that comes to mind as I dont check what my users are hosting.

I know iPanel has an illegal file scanner, I have no experience with it though.

Unfortunately that scanner does not work with suPHP. :(

No offense Joe, but I would stay away from sniffing through you're clients files unless you have a good reason to, it could break a lot of privacy laws, not to mention make a lot of unhappy clients.

While I can respect the thought of wanting to assure your clients are within the laws, just digging through files can pose more problems then it's worth in a lot of cases, and in some extreme cases, legal action against you.

Unless you are running a free host, or target the warez/questionable adult/rips/etc.. demographic then you should be just fine.

And it probably isn't the best idea in the world to post on a public forum that many potential clients read on an hourly basis that your customer's privacy isn't much of a concern to you.

it could break a lot of privacy laws, not to mention make a lot of unhappy clients.

It would not break any laws as it is your server, but for sure, i would not be a happy customer

It would not break any laws as it is your server, but for sure, i would not be a happy customer


Seems to me it would have to break at least one or two laws. It's basically the same as a landlord going into your house or apartment when you aren't there and looking around. Yes they own the property but the space that you pay for is yours to do what you want with as long as it isn't illegal and falls within the rental lease (or in this case, the AUP).

Either way you might be right, I'm not too savvy on privacy laws as I don't go snooping around. ;)

I'm sorry to get so off track but our privacy policy is spelled out perfectly in our terms of service. Privacy is a big concern for us in the sense that anybody who does not have access to the information will not get the information.

I am not looking for a way to harvest all of my client's e-mails, I'm looking for a way to automate the process of finding illegal content before it costs me a lot of money and so I do not have to keep manually digging through their files.

Unfortunately because it is a free web host we are a major target for phishing sites, warez, and other illegal activity. We manually check all of our sign-ups but considering the staff is all voluntary and we get at least 50 sign-ups a day, some slip through the cracks and when somebody finds out before us it can get ugly (I just read a thread on here a day ago about a data center only giving a client 15 minutes to respond to a abuse report before getting shut down, I don't want to get that kind of e-mail while I'm sleeping.)

So again, I understand all of your concerns but our client's privacy is respected up to the point that is required for us to ensure the quality of our service.

I would not go through users files unless you have a specific reason, such as an abnormal and sudden spike in resources, disk use or bandwidth.

I have heard of people (requires root access) doing a search by extension. Some common extensions for abuse are
.iso
.mpeg
.mpg
.avi
.vob
.exe
.wmv
,mp3

etc

You forgot to list .PHP. ;)

You forgot to list .PHP. ;)

That could be a lot of PHP files to go through...just use php.ini to disable functions you don't want.

You forgot to list .PHP. ;)

Sure, but that would make ALL of my clients sites die...

But PHP and HTML files are the ones I am worried about, not the others.

Once you start policing your clients files, you may also open a can of liability on your part. Not necessarily to your clients but to content management it's self. Say for example your customer has illegal content and it becomes common knowledge that you scan for it but for one reason or another, this nefarious content gets missed. Because you make it your business to scan, it also becomes your business to police and protect from illegal content and hence you are liable for the content on your servers. Ignorance is bliss, knowledge is power but ignorant power is just insanity.

It's really an uphill battle running a free webhost and trying to make sure these sort of files aren't being used. I think what it comes down to is taking an overview of your clients and making sure obvious scams don't get through. Then just try to find a provider that is willing to just pass along the complaints to you to let you deal with them. I had one provider null my IP after one complaint, but after I explained what kind of hosting it was and told them just to forward any complaints to me to deal with them it hasn't been much of a problem since.

Unless you are running a free host, or target the warez/questionable adult/rips/etc.. demographic then you should be just fine.


He already said it was a free web host

Once you start policing your clients files, you may also open a can of liability on your part. Not necessarily to your clients but to content management it's self. Say for example your customer has illegal content and it becomes common knowledge that you scan for it but for one reason or another, this nefarious content gets missed. Because you make it your business to scan, it also becomes your business to police and protect from illegal content and hence you are liable for the content on your servers. Ignorance is bliss, knowledge is power but ignorant power is just insanity.

I am already liable for all content on my servers whether I scan files regularly or not. I am legally and financially responsible for my server regardless of who uploads the files. This is why I am trying to take a proactive approach to combatting illegal usage of my service instead of waiting for somebody to tell me my IPs have been blacklisted or my server is taken offline.

As a paid hosting provider I would have never considered doing this, mainly because the clients also have a financial interest in the service and thus 99.9% of the time wouldn't consider doing anything that would possibly take their site offline.

Free hosting on the other hand is a completely different animal because people would rather use free services for illegal usage, plain and simple. It's not a matter of will they use it, it's a matter of when they will use it. Free clients have nothing vested into a free hosting company and have little reason to respect our TOS or governing laws when using our services.

I've already taken many steps to prevent my servers from being exploited for illegal intent but I am looking for other possible options that can allow me to ensure my clients that I am doing everything possible to make sure that their sites aren't taken offline or comprimised in anyway.

It will be an going battle which will cost you allot of time.

You just said it yourself. Free hosting as people dont pay dont they dont have anything to lose so they take care less about your TOS.

I think the only option you really have is to try to identify who signs up with your service. Automated sign ups or anymous for free hosting is a nightmare. It doesnt matter if you scan the content they will bypass you eventually. Looks at how hackers and spammers get away with it on paid hosts. You are just opening the doors for them. There isnt anything you can do on the server side to prevent this except on the sign ups.

Try to call per phone of make some ID check on sign up. Its the best idea I have. Once a person knows that You know who he is, he will respect the TOS as he will be afraid to be held liable if he uses it for ilegal content.

Right now there isn't much more we can do for sign-up verification without spending a lot of money. The sign-up process is pretty solid but some do sneak through and what I'm looking for will hopefully catch those who do get through.

So back on topic, can anybody suggest a scanning method that can be automated or run manually?

No, I dont think there exists a system to identify ilegal content like you are looking nor an either simple way to do it.

An automated system has no way to know which content is legal or not. You could probably search filenames but all users have to do is rename the files, which most probably already do. So if its not a human looking at it you cannot do it.

A way would be to track which files are downloaded the mosts or have more hits. So ilegal content probably gets allot of traffic. Besides that im sorry but I dont have a clue how it would be done. Any automated system you put in place is going to be cheated pretty easy.

I think you misunderstood what I meant. What I'm looking for in particular is a shell script that I can run as root to scan the /home/ directories. I know there are options out there like grep that will scan within files but I can't for the life of me write a bash script that will scan the /home/ directory and subdirectory for all *.php and *.htm* files with certain keywords (phish, nulled, iso, hack, warez, 0day, etc...) and either print the results on-screen or in a log/e-mail.

By automated I meant that this script can be run as a cron and e-mail/log the results for a staff member to review later.

I think you misunderstood what I meant. What I'm looking for in particular is a shell script that I can run as root to scan the /home/ directories. I know there are options out there like grep that will scan within files but I can't for the life of me write a bash script that will scan the /home/ directory and subdirectory for all *.php and *.htm* files with certain keywords (phish, nulled, iso, hack, warez, 0day, etc...) and either print the results on-screen or in a log/e-mail.

By automated I meant that this script can be run as a cron and e-mail/log the results for a staff member to review later.

find+grep? Eg, something like this:
#!/bin/bash
FINDBIN=/usr/bin/find
EGREPBIN=/bin/egrep
BADWORDS="phish|nulled|warez|0day"
$FINDBIN /home/ -iname \*.php -o -iname \*.htm -o -iname \*.htm* -type f -exec $EGREPBIN -i -H $BADWORDS {} \;That could easily be run from cron, and should be relatively fast as it is only grepping through matching filenames.

You could probably pass the "--exclude-from" argument to egrep to exclude any legit files containing those keywords if needed too.

Awesome! I will give this a try when I get home. :D

Of course, someone could also be using your area as a repo for warez. I'd check filenames on .rar, .zip and .exe files.

If you have suspicious about particular non-licensed scripts that may be in use, you could check with a simple perl script too:

(use the same find command, but put this after -exec)

TOSChecker.pl {} \;

Then in TOSChecker.pl:

#!/usr/bin/perl

@known_code =
(
'some known line of code from commonly unlicensed scripts',
'some other line of code from another script',
'etc'
);

if (@ARGV != 1) {
print "Usage: TOSChecker.pl filename\n";
exit 1;
}

if (!open(FILE, $ARGV[0])) {
print "Error opening file $ARGV[0]: $!\n";
exit 1;
}

@lines = <FILE>;
close FILE;

foreach (@lines) {
if (grep /$_/, @known_code) {
print "Found unlicensed code in $ARGV[0]...setting permissions.\n";
chown 0, 0, $ARGV[0];
chmod 600, $ARGV[0];
}
}

exit 0;

Not quite as fast as the answer above, but you can pretty much confirm with certainty that there is a bad script--and prevent it from running. For the array of known code snippets, I'd either use a common 'require' statement that will exist on almost every file of the unlicensed software, or, a random line from the common include. The first will find more files, the second will pretty much blowout the script by only setting one commonly required file to root:root -rw-------

Of course, no matter what you employ, you'll burn up all your system resources trying to track these scripts down long before you develop a system that is impenetrable. Someone will always get around you, so I tend to agree with the previous posters. There's no way to police it adequately, so I'd avoid the legal responsibilities that may come with making policing your users content your business.

Thanks for the input but I only need to search .php and .htm* files since those will provide all of the information I need. I routinely check the disk space usage for the accounts on my server and if a user hits 40% (400MB) then I usually check to see if they are using the hosting for file storage (which is allowed as long as the file is legal and being linked to from their hosted website and not a remote site).

I understand that policing content is a resource intense process but it's either that or ignore the potential problem that is bound to happen. I figure it doesn't cost me anything to proactively scan for problems but the costs involved if I do not are definately not within my budget.

Yes, users have found ways around my scanning techniques but that is why I am always looking for new ones. So far I've only received 1 DCMA notice... not bad at all compared to the amount of illegal material I did find. :)

Again, I would like to point out that what is hosted on my servers IS my legal responsibilies and cannot be avoided regardless if I scan or not. Ignorance is never an excuse. ;)

I think the problem here is that we don't know the law (or the law hasn't been written yet).

1. It is not the responsibility of a landlord to keep tabs on tenants for illegal activity.

2. It is not the responsibility of a bank to keep tabs on security debosit boxes.

3. It is not the highway department's responsibility to keep tabs on people who drive on its roads, monitoring, say what laws they have broken.

4. It is not the Post Office's responsibility (I think) to make sure that all mail that is sent is not illegal (scams, frauds, illegal materials).

Has it been decided in court that it is the responsibility of an ISP to monitor traffic for illegal activity?

If it has, then is this true for hosts also? I have no idea if this has been "heard" yet (but I don't follow the news for this kind of thing).

And on the other side, it would be illegal for a landlord to snoop around someone's house. It would be illegal for a bank to open safe deposit boxes. It would be illegal for the highway department to pull you over for a search. (It wouldn't be illegal for the PO to open a box though.) So is it actually illegal for a host to go through a clients' files (of course you can create a contract that does allow this).

I see your point but in this instance it is not the same thing as those "examples" you listed. And yes, it is perfectly legal to view content located within my server by me. It would be illegal to use said content for myself but because of the nature of the hosting, it is expected that the content be reviewed by me and my staff to combat fraud and TOS violations.

This thread is turning into a moral/legal debate but I don't see why it should since that wasn't the topic of this thread. I understand you respect your PAYING CUSTOMER'S privacy, I did also. But free web hosting IS A COMPLETELY DIFFERENT THING and thus needs to be handled differently.

If you're being hosted by a company and the servers are taken offline due to 1 client violating some international law, regardless if it was your company's fault or not you still have hundreds of clients who are offline and angry. How do you recover from said action? Now imagine offering a free service that generates no income and needing to purchase a new dedicated server and restoring 350+ backups from your local computer... a very painful process. The need to continously scan the illegal content is there, with the method above all of the results don't even contain any identifiable information except for the cPanel account.

Again, we are not snooping through user's images, we are not scanning through databases, we are not downloading ISOs, RARs, or ZIP files, we are not keeping a record of who hosts what, and we are not even going through people's directories.

The process performed is that when questionable content is found we view their website (which is publicly viewable) and if further investigation is needed we will view the contents of the file.

So in reply to all of those who feel it is better to ignore the problem and claim ignorance, that is not an option because I have a duty to perform that my clients expect of me.

Hey Joe, I did not mean to say that you are at fault here; if it is in your TOS that you can check content then that is up to your TOS.

What I am really asking here is, why do you think it is your responsibility if your clients host illegal content? Why do you think your server would be shut down if your clients were doing illegal things?

I don't mean to hijack your thread but if your server were to be shut down that sounds unfair.

I think the problem here is that we don't know the law (or the law hasn't been written yet).


(of course you can create a contract that does allow this).

The point is that your contract can be used to set a "law", As long as you have some way to force your customers to "Agree" to your contract, then you hold the upper hand. I know for sure most CRM Apps have this built in (WHMCS for example).

You can put close to anything in your TOS

Chapter 11 - Page 3
You and all of your sub resellers MUST chew pink bubblegum while uploading files over FTP. Only mint bubblegum is required if you use the cPanel file manager.

Of course that was a joke, but you can (and should) place a small bit in your TOS saying that as a host, you are authorized to access your customers files under circumstances or to comply with Law Enforcement. Below is something I came up with (your free to use, just change the name).

In general K-Disk Networks does not activally monitor our customers activities, however if we see any abnormal use of resources, we may investigate by accessing your account to look for potentially dangerous files or violations of our terms of service.

K-Disk Networks will cooperate with any search warrents that request access to your account.

It is my responsibility because my clients trust me to provide them with web hosting. If the server is offline then their sites are offline thus I am not providing them with the service I have promised.

I've read multiple threads here of data centers that shutdown servers due to various abuse complaints with little or no notice. I'm not saying my data center is like this but in all fairness I completely understand and would actually prefer my data center to shutdown a problematic server rather than take the chance of it impacting my or anybody else's server.

I may sound paranoid but I am always overly cautious when it comes to my clients.

It is my responsibility because my clients trust me to provide them with web hosting. If the server is offline then their sites are offline thus I am not providing them with the service I have promised.

I've read multiple threads here of data centers that shutdown servers due to various abuse complaints with little or no notice. I'm not saying my data center is like this but in all fairness I completely understand and would actually prefer my data center to shutdown a problematic server rather than take the chance of it impacting my or anybody else's server.

I may sound paranoid but I am always overly cautious when it comes to my clients.

Right thinking, In some cases it is better to loose one problematic customer then 90% of your customer base. In alll honesty, most customers would not mind an automated program that will check their files for "dangerous" content. I cant say I would be comfortable with a human reading some of my internal content, but I have no problems with a program.

For the record, here is a copy of our current Privacy Policy within our TOS:

PRIVACY POLICY
Your personal information will always remain private!* We will never sell or share your personal information with any other person or company for our benefit. All files uploaded to our servers are subject to examination by our staff to ensure the safety and security of our staff, clients, and property. *We will always assist with government or law enforcement requests.

It's pretty straight forward IMO but I'm willing to adjust the wording if you have any better suggestions.

As for the human involvement, with an automated system this removes the need for a human to physically view a file unless it contains certain keywords.