hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : Dedicated Server : Improving Internet Download Speeds On The Server Itself
Reply

Forum Jump

Improving Internet Download Speeds On The Server Itself

Reply Post New Thread In Dedicated Server Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old 12-06-2006, 01:35 AM
nykornman nykornman is offline
Newbie
 
Join Date: Dec 2006
Posts: 12

Improving Internet Download Speeds On The Server Itself


My partner and I run a business which has specific needs. We are not sure of the best way to meet them so we will try to describe our business model to the best of our ability and we are hoping someone will be able to suggest a service which will improve our results. Perhaps it is renting a dedicated server, or upgrading our own internet connection, we dont know.

Our business consists of running software from a computer which is connected to the internet. The software opens a large number of Internet browsers, and controls them. Each browser navigates to the same page, and then each browser refreshes that page continuously, as quickly as the internet connection will support, scraping the screen each time to detect certain changes in the HTML. Naturally, the "faster" the internet connection, the better results we get, since each browser is able to refresh many more times per second, and so the delay between when the HTML on the target page actually changes and when one of our browsers picks up the change is minimized. That is our goal.

We would like to know what technologies will enable us to improve the performance of this program. Please ask if anything is unclear. Thanks very much.



Sponsored Links
  #2  
Old 12-06-2006, 09:44 AM
Xandrios Xandrios is offline
Web Hosting Master
 
Join Date: Jun 2004
Location: Bay Area
Posts: 1,320
Why don't you just download the page source? That reflects each change also, plus it will be much easier to refresh and detect changes. I've never heard of actually having to render the pages in a browser window (To detect changes)?

Anyway, the limiting issue here is probably your CPU and RAM. Each browser window takes a couple of MB ram, and the content of the website does also. Rendering all these websites all the time also costs a great deal of CPU time.

  #3  
Old 12-06-2006, 09:49 AM
TheVoice TheVoice is offline
Web Hosting Master
 
Join Date: Oct 2002
Posts: 702
Don't use a browser as you'll be wasting time rendering pages, making requests to images etc. You need to use a program like wget to which will just download the raw html, disabling reverse dns lookups will help as well, or just making the requets to an IP.

Combine that with putting the server as physically close to the target website you want to scrape and thats about as fast as you can get.

__________________
ServerMatingProject.com
The World's first server mating experiment
We give new meaning to I/O intensive and hot swap

Sponsored Links
Reply

Related posts from TheWhir.com
Title Type Date Posted
European ISPs BT and Alcatel-Lucent Reach 'Fastest Ever' Broadband Speeds Web Hosting News 2014-01-24 14:25:41
HostingCon and the 6 Keys to Improving Sales Results Blog 2013-06-19 09:21:16
Mega Firefox Extension Speeds Up Downloads, Removes File Size Limits Web Hosting News 2013-02-11 11:43:22
PlusServer Launches Unmetered.com Hosting Service for Streaming, Download Web Hosting News 2013-01-10 09:11:41
File Sharing Service RapidShare Changes Traffic Model to Deter Piracy Web Hosting News 2012-11-09 12:16:35


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
Advertisement:
Web Hosting News:



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?