hosted by liquidweb

Go Back   Web Hosting Talk : Web Hosting Main Forums : Dedicated Server : Improving Internet Download Speeds On The Server Itself

Forum Jump

Improving Internet Download Speeds On The Server Itself

Reply Post New Thread In Dedicated Server Subscription
Send news tip View All Posts Thread Tools Search this Thread Display Modes
Join Date: Dec 2006
Posts: 12

Improving Internet Download Speeds On The Server Itself

My partner and I run a business which has specific needs. We are not sure of the best way to meet them so we will try to describe our business model to the best of our ability and we are hoping someone will be able to suggest a service which will improve our results. Perhaps it is renting a dedicated server, or upgrading our own internet connection, we dont know.

Our business consists of running software from a computer which is connected to the internet. The software opens a large number of Internet browsers, and controls them. Each browser navigates to the same page, and then each browser refreshes that page continuously, as quickly as the internet connection will support, scraping the screen each time to detect certain changes in the HTML. Naturally, the "faster" the internet connection, the better results we get, since each browser is able to refresh many more times per second, and so the delay between when the HTML on the target page actually changes and when one of our browsers picks up the change is minimized. That is our goal.

We would like to know what technologies will enable us to improve the performance of this program. Please ask if anything is unclear. Thanks very much.

Sponsored Links
Web Hosting Master
Join Date: Jun 2004
Location: Bay Area
Posts: 1,320
Why don't you just download the page source? That reflects each change also, plus it will be much easier to refresh and detect changes. I've never heard of actually having to render the pages in a browser window (To detect changes)?

Anyway, the limiting issue here is probably your CPU and RAM. Each browser window takes a couple of MB ram, and the content of the website does also. Rendering all these websites all the time also costs a great deal of CPU time.

Web Hosting Master
Join Date: Oct 2002
Posts: 702
Don't use a browser as you'll be wasting time rendering pages, making requests to images etc. You need to use a program like wget to which will just download the raw html, disabling reverse dns lookups will help as well, or just making the requets to an IP.

Combine that with putting the server as physically close to the target website you want to scrape and thats about as fast as you can get.

The World's first server mating experiment
We give new meaning to I/O intensive and hot swap

Sponsored Links

Related posts from
Title Type Date Posted
FCC Redefines Broadband in Bid to Encourage Faster Internet Speeds Web Hosting News 2015-01-29 13:50:26
UK ISPs Deliver Advertised Broadband Speeds to Only 10 Percent of Customers: Report Web Hosting News 2014-11-18 17:04:09
'Room for Improvement' as Most Major US Broadband ISPs Fail to Meet Advertised Speeds Web Hosting News 2014-06-20 12:18:32
AT&T Plans to Bring Fiber-to-the-Home to 21 New US Metro Areas Web Hosting News 2014-05-23 15:17:20
European ISPs BT and Alcatel-Lucent Reach 'Fastest Ever' Broadband Speeds Web Hosting News 2014-01-24 14:25:41

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Log in with your username and password

Forgot Password?
WHT Host Brief Email:

We respect your privacy. We will never sell, rent, or give away your address to any outside party, ever.

Web Hosting News:
WHT Membership
WHT Membership



Welcome to

Create your username to jump into the discussion! is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.

(4 digit year)

Already a member?