Results 1 to 15 of 15
  1. #1

    Exclamation Dedicated Server For A Search Engine

    Hello everyone, This is only for future refences so dont bother emailing me about it, I would like to know which host is the best for dedicated server due to the fact that i have built a search engine, so it could use alot of CPU and space.

    Thanks in advance.

  2. #2
    Join Date
    Mar 2005
    NY USA

  3. #3
    Join Date
    Feb 2003
    San Francisco, CA
    It's not a matter of plain question & answer. You have to research, read the forums, compare prices & specs...

    "Which car is the best? I wanna drive real fast."
    First Amendment of a webhost: if you fail to prepare, prepare to fail.

  4. #4
    Join Date
    Nov 2003
    It would depend on how busy it is as well. Generally you should start out small then expand as needed. Try creating some kind of model that would allow you to dynamically add new servers to distribute the load easily.

    If you are looking for a specific company then Yaroslav is correct. Although I would personally recommend layeredtech,theplanet, and steadfastnetworks.

    Although "Which car is the best? I wanna drive real fast." isn΄t a good example, the answer is easy, a 1991 BMW 318is, put a e36 M3 engine in it, redo intake, exhuast and suspension, new 16 or 17 inch lightwieght rims. Finally if you have enough money left put the biggest turbo charger you can afford on

  5. #5
    It depends on how extensive your database is and how much visitors using the site.

    You may want to look at a datacenter which would allow you to grow and probably create a server farm where you could continually add servers to the farm as and when your requirement increases.

    If it is something very high end, you may need to look at load balancing, probably a couple of Dual Xeon to handle the front end and a few more to handle the database.

  6. #6
    Well, Each page added to the database is about 993 Bytes and it can index about 300-1,000 pages every minute so it would need alot of space and bandwidth for quite a cheap price and alot of uptime would be helpfull.

  7. #7
    Join Date
    Jun 2003
    San Francisco
    Is it a search engine like google? If so, we can make some assumptions:

    1. Your indexing application will be running 24/7.
    2. Your database will grow steadily. (1M/minute, about 1G/day)
    3. Your visitor traffic will be low initially

    I would use three servers in a private LAN
    1. Database server (running mySQL?)
    2. Web server
    3. Indexing server (where the indexing app runs)

    If you are short on funding, you may combine #1 and #2 in one server at first, but I wouldn't make one server run both the indexing app and web server.

    Your database server needs to be the best you can afford. Xeon, 1 GB RAM, RAID-0 SCSI HD at least 200GB total, which will last you 6 months (according to 1GB/day increase in size). No need to have too much RAM at the beginning because you will have only the indexing connection and few visitor connection. Use motherboard which allows dual CPU, but start with one first, upgrade if traffic gets heavy. Use RAID-0 for the speed and space. Use another server for backup (see below).

    Your indexing server needs to be FAST. Dual Xeon (or AMD). The RAM will depend on your indexing application. It doesn't need fast HD since the HD is hardly used. SATA is fine. It doesn't even need RAID.

    You can also use the indexing server for backup storage. Every two days or so, stop the indexing and copy the database from your database server to the indexing server. So, your indexing server should have bigger HD space than the database server. I would use 1 80GB for the OS and app, and another 300GB for the backup. Since you use SATA, they are cheap.

    If your RAID-0 of your database server ever got corrupted, you can always use the backup. Since it is a search engine, it is OK to have data from few days ago (nobody will notice). That's why there is no need to use RAID-5 for your database.

    Your web server can be a crappy Celeron. Nothing special here. You could make your database server also your web server.

    You will use way more incoming bandwidth than outgoing bandwidth initially. If you want to do co-lo, some co-lo vendors give heavy discount for incoming bandwidth.

  8. #8
    Join Date
    Jun 2003
    San Francisco
    Another idea for you to save money.

    Run the indexing app on a PC at home. Your DSL should have a uplink bandwidth or 100kbps - should be enough to handle the database update, and your dedicated (or co-lo) server doesn't need all the incoming bandwidth.

  9. #9
    You see the indexing app is one 8 KB file, It cant be stopped once it starts unless database is deleted or emptyed, I would probalise only be able to afford 1 server in the beigning.

    should i use dedicated or co-location?

  10. #10
    Rather than using one large machine, it may be more economical and allow for more expansion if you put it across a lot of lower powered machines (much like google does).
    crucialparadigm - Affordable, Reliable, Professional :
    Web Hosting
    • 24/7 Support • Web Hosting • Reseller Hosting • Cloud/VPS Plans • Dedicated Servers •

  11. #11
    How do i go about doing this?

  12. #12
    Join Date
    Jun 2003
    San Francisco

    You need to make your app to stop/restart when needed. There will be time when you have to do that.

    You can have the index app run at home, database and web server on the same dedicated server. I don't think you will get a lot of traffic at the beginning, so the bandwidth requirement is low.

    I think you just need a P4/1GB/200~300GB HDD dedicated server, and then a workstation at home, and you can get this to work.

    dedicated is cheaper for you at the beginning. Once you grow and want to have the indexing server the same location as your database server so they are connected with private LAN for faster update, then you need co-lo.

  13. #13
    Remember that it will take while to build traffic, there are a ton of search engines out there, so be patient building traffic takes a while. Indexing on your home pc is a great cost effective idea. You might even be better starting with a VPS, and move to a Dedicated once the traffic comes, or go with a less expensive dedicated like Layeredtech or something.

  14. #14
    You see i would perfer to start with all the servers, the indexing i cant really do at home, it will be running 24/7/365 and indexing alot very fast.

  15. #15
    Join Date
    Jun 2003
    San Francisco
    1, modify your app so you can control it to stop and restart anytime. You will be kicking yourself if you build a great site, and something happens to your server and you lose everything.

    2, the indexing app will be running 24/7, and that's exactly why you shouldn't put it on the db/web server. Either have it on a seperate server (which you have to pay for), or at home.

    3, your indexing server needs a lot of incoming bandwidth, but less outgoing bandwidth (to the db server). Based on your data - 933 bytes/page * 300 to 1000 page/minute, that translates to 5-15KB/second (outgoing). Your DSL line's uplink is around 12KB/second, cable would be even faster. You wouldn't max out, but it wouldn't be much slower.

    4. There is really little market for another search engine. That's why you should spend as little as possible until you know it is taking off. Yes, VPS would work for you ($40) at the beginning, but most VPS plans come with little storage. 20GB storage will last you only a month. A server with 200GB storage is not cheap - around $200/month.

    5. If you want your indexing server to be in a data center, you should co-lo it. You probably can have the two servers co-loed for $400/month since you need little bandwidth. However, you have to provide the server, so that's a high initial cost (~$2000).

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts