Follow-up...

I had my client's server installed Friday, did final configuration and testing over the weekend... and at 1:00am PDT today (Tuesday) we moved all of our users off of (leased) dedicatedhosting.com server and over to (our own server at) Weinbar.

My client's site is esnipe.com -- and it places bids against eBay. It must place the bids to THE SECOND and any Internet or server failure means bidders do not get their auction item. Even a 1 second Internet failure! The site places thousands and thousands of bids a day.

Obviously going with a low-priced provider like Weinbar may not seem ideal for such a critical site... but we are building two complete redundant sites (in different cities)... so we are willing to increase our risk on individual providers. A lot of work is going into our custom software to detect and correct for failures, we are less than a month away from having full replication and automatic failover.

I have experience with "Big Boys" such as Verio, Qwest, and Internap -- and so far I'll say Weinbar is defiantly more down to earth in meeting the needs of a small business. The big boys screw up too! Recently (as discussed on this message forum) Weinbar has shown to lack depth in the staffing... but I surely feel that we are getting more than what we pay for.

So far, Weinbar has delivered to me what they promised. Of course, it has only been 4 days -- but a lot of times ISP's screw up in those first days. There were some minor delays, but they did meet their agreements.

No provider is perfect, here is hope that Weinbar keeps plugging and leverages their good service and continues to improve the weak areas...

Thanks to all who have helped on this great forum.

Stephen

P.S. The www.esnipe.com site keeps extensive records of connection problems to eBay, plus -- once we get our second site up and full replication going in the next few weeks -- we will have each site monitoring the other. I will keep the forums posted on general quality of the Weinbar network.
P.S.S. My goal is to have the 2nd site, East Coast of United States, up and running in 2 weeks. Server is already built and mostly configured. Any co-location facilities with prices / bandwidth similar to Weinbar are what I'm after. Please feel free to email info to Stephen@RoundSparrow.com

It is somewhat refreshing to read your quite objective review about weinbar. Many people complained even about the smallest detail yet you can still see the good side of weinbar that those people need to appreciate more. I sincerely hope your journey with weinbar will always be a satisfactory one.

BTW: have you found/decided on the other host on the east coast yet?

cheers,
:beer:

No, I haven't found my eastcost provider yet. So far the list I came up with 8 or 10 days ago hasn't changed...

(Note that you are on it):
1) interserver.net
2) dtwebworks.com
3) insidecrew.com

What I really need is a low-latency network... and a site that is consistent on the network quality. As we are doing our own server -- I'm not wanting server support (other than reboot and prompt rack dismount/mount).

I'm realistic. ISP's are still in the imature stage, that doesn't mean that proper planning can't overcome the shortcomings. I still believe that server issues (including security) are still more a problem that ISP's... you have to look at the "total cost of ownership" for a web site.


Although I have to admit, despite all my knowledge, I can't figure out how some (dtwebworks.com) and weinbar can offer such great prices when these guys are just "purchasing from someone bigger"? Care to share the secret :) Am I overlooking how I could get the same deal from their upstream facility? i just don't see the quantity discount making that much difference.

I'm only 1/2 serious about that, as I know that some things for competitive reasons they have to keep to themselves.

Basically, we buy in enough bulk that our pricing is reflected down to the customer. If you were to go with the "big boys" and were buying in the quantity that we sell, they would charge you more because it is low for them.

Since we buy in larger blocks, we get it for less, thus we can offer it for less.

It's kind of like why resellers in the phone industry charge less than the people they buy it from. When purchased in quantity, the price is lower.

Michael, you are the straightest shooter in the ISP business I have met. And I have evaluated over 40 ISP's in the last 5 years.

I'm happy to be your customer, and appreciate your ability to deal with sophisticated customers on a budget.

Thanks for the reply.

Originally posted by StephenRS
No, I haven't found my eastcost provider yet. So far the list I came up with 8 or 10 days ago hasn't changed...

(Note that you are on it):
1) interserver.net
2) dtwebworks.com
3) insidecrew.com



Today, I just moved out my server from interserver to Site5/NAC (I will wait several more days to share my experience with them). Nothing is really wrong about interserver, they have been great host. Except my server uptime with them is < 97%, mostly due to VDI's having problem. Really, if you think your client can't afford having some sort of network problem even for a second, then I would say don't go with interserver (again, nothing wrong about them per se, it's VDI problem). I will be more than happy to show you my server latency graph while I was with interserver, but it is not the recent, about 10 days ago... (but, it does not differ much actually since 10 days ago).

Although I haven't been hosted inside NAC.net facility for long (in fact, today will be my first day), I have been monitoring and comparing both NAC and VDI for several days. I have some latency graphs to compare NAC and VDI if you think you need it. If I have to choose, I will choose NAC.net for right now (hence, I moved my server out from interserver). This is solely based on network connectivity. Beside, NAC.net has 15 diverse connectivity for a total of 775 mbps bandwith. I bet you know how important this diverse connectivity for your needs.

Now, if I may make a suggestion (to add some another possibility into your list), I will suggest pegasus (pwebtech.com). The owner, Jay (with WHT member ID jayglate) frequenst this board a lot and although I have no server colocated directly with him (except through Site5), I would suggest you to contact him since he might be able to work a good deal for you.

All the best for you!

cheers,
:beer:

pwebtech.com is now top on the list. The only thing that somewhat bugs me is the "need for a special deal" -- I really liked Weinbar's up-front pricing... as I'm not asking for anything special (1U server with 40 or 50GB a month)...

http://www.pwebtech.com/colocate.html
Their page just says "Sorry This Page is coming soon."

I'll get over it, it sounds like they have what I want.

Jay has been in contact with me, so we are working out details. He told me the setup fee is $50, the same as what Weinbar charges. Things are looking good so far.

Thanks for the pointer.

StephenRS, I am interested in your redundant network setup. I remember reading a post of yours a few weeks ago about the cost/benefit of building redundant systems. Anyhow, I'm not a sysadmin. I just lurk here because every now and then a little more knowledge seeps into my thick scull by way of cathode-ray radiation osmosis or something. I do know that piping all output to /dev/null saves on HDD resources - or something ;)

So, in layman's terms can you explain how this redundant network of yours is going to work - with your east-coast/west-coast server setup. Does this mean you will have a third server that load balances sending visitors randomly to east or west server? Also, how will you mirror the data and keep both servers up-to-date?

Thanks.

Hi Freakysid...

I'm actually working on outlining this technology for future ventures. I even came up with a name for it...

MISTUS -- Multiple Internet Server Technology Using Software. I even registered the domain: www.mistus.org - but haven't had time to put up the documentation yet.

The intention of the name MISTUS is to outline the basic ways to do this type of thing. Just like there are various levels of "RAID disk storage" -- there are different ways you can approach the multi-site problem. Just like you have choices (and advantages/disadvantages) between RAID0 vs RAID5.

My customer (esnipe.com) site is the premier one to use this new technology. I also have plans to do another site when I'm done with esnipe.com -- one of my own to use the same technology (although the site has a different purpose than esnipe.com - it has nothing to do with eBay).

To understand why I'm doing this... I have a background in high-end web hosting -- software design and operating system work. Clustering has always been a fascination of mine... dating back to late 1980's. But in the current marketplace, clustering almost always seem to involve "high end" systems. You talk to companies like Compaq / IBM / Microsoft about clustering... and they send you to their most expensive line of products :)

I'm trying to start from cheap and work my way up. When you have redundancy of everything (different site, different server) -- you don't need everything to be so perfect. Which makes things even cheaper!

I'm trying to get back to the roots of what has made the Internet grow to what it is today -- reliability through protocol and proper design. Look at how DNS and routing works, redundancy is built in from the start. ((You must have two DNS servers for your domain)). But I think too much of this good stuff has been forgotten; most web sites do NOT have good recovery features, as redundancy of databases is considered a "high end" (expensive) feature... I intend to help change that.

Linux folks have been doing this for CPU processing, I'm trying to help do it for web site (Application) hosting.

=============

Your question is very insightful -- you obviously understand that there are different ways to approach this. ((And trying to explain my ideas to other people is what made me use RAID analogies, which lead to me coming up with the MISTUS name.))

How am I doing it... well the whole concept (the "S" in MISTUS) is about software. You put more work into your database code and extra program agents to manage the database sync -- but the payoff is in ISP savings and in the hardware savings. And of course, the reduced downtime and better scalability!

I am developing special programming libraries and agents that help in converting existing web sites to MISTUS-capable ones. eSnipe.com is a Microsoft ASP site (Windows 2000), but the technology should be easily applied to almost any common web platform (Cold Fusion, PHP, JSP, etc.)... the key is in how you make replication work with inexpensive databases (mySQL, Access2000, ProgSQL, etc.). My goal is to even allow multiple platforms... you could replicate your data from a Microsoft ASP site over to a Linux PHP application!

To answer your direct questions
========================
How eSnipe.com is going to work... we are about 3 weeks away from turning it "live". Weinbar has the first of the servers.

As you have seen by this thread, first off I have to get (at least) two dedicated servers at (at least) two different ISP's. As we must be able to run agents and things that you can't run on an ISP's shared servers. We have opted to build our own servers (as I also have extensive experience build my own computers).... but we could have gone with a leased system if we wanted. I put removable drive trays in our custom built 1U servers, and I have an identical system here at my home office -- so I can ship a complete site major upgrade by sending down a 3 pound hard drive via overnight. Of course, we do smaller upgrades over the Internet.

Then we go for total redundancy. for eSnipe.com we registered two new domain names: esnipe-west.com and esnipe-east.com -- but customers have been instructed to use the "main one" esnipe.com -- for which we will round-robin between the multiple sites. The "east" and "west" terminology is mainly to make it easier for our customers to understand what is going on.

Once we go live with MISTUS, the customer will have the option of selecting either site. The data will be fully replicated between both... and if for whatever reason they can't connect to esnipe.com they are free to try the individual sites... or they may prefer one over the other (perhaps they get better response because the server is closer to them in terms of network hops / latency).

We do not use a third server to load balance. Plain old round-robin DNS will do that... or the users themselves by selecting which one.

As for the data replication -- that part is still being developed (I am the lead developer) and I'm not ready to fully explain it all. Our goal is to put as little overhead on the servers as possible, support multiple sites (not just 2) and to allow the timing parameters to be adjusted. Under 5 minutes is what we currently believe is appropriate for esnipe.com -- but we have extensive tests to perform in the next few weeks to validate that.

I currently have 6 Windows 2000 computers running in my house right now -- plus 2 Linux ones while I am doing the development and testing :)

My goal is to post most of the information on www.mistus.org once we are further along... some rough stuff is already done (and my esnipe.com customer has it)... but I'm not ready to publish it in public yet. Documentation takes time...

I hope this answers your question. Please feel free to ask more, I have been planning + thinking this a lot for the last 3 months, I'm interested in any input.

-- Stephen Gutknecht
-- Renton, Washington USA

P.S. This message is the first time I've discussed these ideas and the name MISTUS itself in public. My esnipe.com customer is fully supporting this and will be licensing the technology from me. I will own the rights and all the code developed so that I can use it on future projects.
P.S.S. If you think this is a long message, think about the fact there may be a book coming about this stuff in the future :)

StephenRS,
I would completely agree with you on this topic, maybe I have had a couple of hickups in my hosting service and it has disappointed me, but going back:

Downtime 1 hour after signup
IP Addressing Problem
Low Response Level On Progress Of Server Setup
2 Days to add a domain to DNS (Domain>IP)
A couple of Hickups in Network Service After Setup
7 IP Addresses not Bound To eth0 on reciept of server

Each of these queries have generated massive response on the bulletin board and have also caused a number of others to express their feelings in open.

Although I can fully understand that most of them are trivial, looking back at the occasion, and they are all, most certainly due to the Single Router failure, this is a plain example of the cascade effect on the internet, everything is linked.

Downtime (Router Failure) > IP (because of router failure) > Low Response (Fixing Router) > Couple of Hickups (Fixing Router) > DNS/IP not bound (catching up and trying to load balance the work for other clients and not just myself)

The entire thing is related, as I have done before and as I will do again, is apologise to WeinBar and hope we can have an even closer business relationship.

Looking back, WeinBar have answered my queries, more than compensated me for the problems and sorted out all other clients in the process.

I have got 100% uptime on my server since starting QWKMON and it now looks like WeinBar are back upto the standards of praise people were giving them before this outage.

WeinBar - Keep up the good work, you make me feel ashamed of some of the messages I have previously posted.

FootNote:
I Wish you luck following your visit to hospital.

Hey, I can thank you that you attracted all the problems I might have had :) Good to hear things are back on track.

StephenRS - interesting read. I have some background in highly available system (well and some foreground as well - my current real job uses redundant everything). Actually when I think about it everything I've done has been HA - FAA, telco and financial. You wouldn't believe the design for the air traffic control system. But I digress.

My own website is not yet at the level where I can afford to apply all this stuff there but I hope it will be some day. Anyway a couple questions:

How are you accomplishing the round robin DNS? Are you hosting the DNS server on one of these machines or elsewhere?

Are your sessions stateful? and if so do you replicate the state in real time or somehow ensure a client stays on one server for the duration of the session?

Since you're talking about cross platform database replication, I assume your going to roll your own. Any reason to not use MySQL's built in replication? (besides not being able to do it cross platform) Any ideas on the latency? Are you going to multi-phase commit?

Feel free to respond by e-mail if you'd like - I'm afraid we're getting a bit off topic for this forum. Let us know when you get that mistus site up.

Hi,

I have been planning a redundant web servers system as well, for a special webhosting project I am developing. The idea is to load balance and improve reliability. I do not plan to implement this right now. I am absolutely not an expert in this domain (and I have no idea what I am talking about), but here is my design (please read completly as the best things are at the end):

2 or more servers (DuH).

1 is Master, the rest are slave(s).

I will use DNS servers that allows load balancing (like DJBDNS). Small Time To Live. The DNS server will load balance between the slave(s) and the master.

The slave webservers will deliver standard HTML docs and images to clients. When CGI / PHP / SSI will be requested, a special wrapper will send the request to the master server (the best here is to have the master on a seperate 100 MBPS interface)

All files are synced between Master and Slaves every XX minutes. ALL files, including CGI scripts, data files, not used on slave servers.

If a Slave fails, special software on the dns servers will detect it and stop resolving to it. The Master and the other slaves will take over the load.

If the MASTER fails, a Slave will become Master, automatically. Since all the files are located on both master and slave, it is possible. Some special software will have to be implemented to have the webserver start delivering CGI scripts locally, and various other stuff.

About databases:
All scripts will be configured to access the database server on localhost. All servers will have database server (mySQL) running and data will be kept updated on slave even if completly useless unless master fails.

The ideal would be that the master server only serves dynamic content for the slaves. But in this case, more than 1 slave should be used. The master should also run the FTP service.


Thank you,

Cedric

I have to go to Dallas for the rest of the week, and then eSnipe has a lot of work for me as soon as I return... so I'm not going to be able to discuss MISTUS (site to site replication) more than I have. I'd love to, and maybe I will be able to once I get back.... but for now, further discussion will have to wait. (Earlier in this thread) I tried to write one long message to answer most of the anticipated questions so that I could be done with discussion for a while and complete implementation :)

All the questions are good ones.

Ok, a few quick answers...
=======================
jnestor -- my goal is to force failover to be something that a programmer thinks about. I find most web sites don't try to deal with server failure at all. And those that do seem to try and push it to the backend (like you mention mySQL replication)... where I feel that you have to start with the frontend (the user presentation) and build your code to be "multi-aware" so that it considers "the other site down errors" like you do for other general errors (such typos on a user input field)...

On some MISTUS systems, such as the one we are working for eSnipe -- the goal is NOT TO BE SEAMLESS! The goal is to EXPOSE the seams, so the users understand what is being done...

On how do I deal with sessions between servers, and all those lines of questions. That is the type of difference between a "MISTUS LEVEL1" and "MISTUS LEVEL2" system in my thinking. Think of how different RAID systems explain different ways of organizing disks to yield different properties. That was my thinking, standardize the way we communicate with each others about such design factors.

eSnipe itself is going to be a pretty simple implementation. It won't allow session sharing and so forth... I don't think it needs it. They key at eSnipe is to not miss placing bids on eBay -- and we don't need sessions between servers to do that. For other applications (outside of eSnipe), that may be requirement, and require a "more sophisticated MISTUS design".

But my goal right now is to focus on the simplest of needs, with the simplest of sites... to bring what is consider "high end" to the "low end".


Closing
==============
I'll be up front that most of what I'm discussing and doing has likely been done before... by others. I'm not trying to do a miracle or anything :) The ideas are obvious, but still, I don't see many (existing) sites actually doing it. It is my attempt to actually organize my efforts and coin methods of describing them (with the term "MISTUS") that I believe to be unique... but many times I find out someone else has already started on something (public project) after I have :) The world (Internet) is a big place. Part of the reason I focused on coming up with a name for this thing ("MISTUS") was to make it easier for those of us doing this to find each other. MISTUS is a pretty unique keyword on search engines :)

Stephen -- Renton (Seattle), Washington

WeinBar - where are you? It's been 4 days with no word from you about colocation, and you have all my info.


Andrew

Well thank you Stephen for taking the time to post those explainations. You only error is that you over-estimate my knowledge; for example, I don't have a clue about round-robin DNS. But your posts were very interesting and informative.

Good luck with your trip to Dallas. When you get back, put up a post here and let us know whether you "MISTUS" while you where away :D

haha. OK, off to the airport. I invite you too look up the meaning to the word MISTUS, it is an actual word :) Although don't take it personal...

freakysid -- Round Robin DNS can be done pretty simple. Put two IP addresses on the same hostname. If you have access to NSLOOKUP on one of your machines, I suggest looking at cgi.ebay.com as an example hostname that has 3 IP addresses. The client operating systems (TCP/IP stacks) typically select another IP when one doesn't respond... although some OS (TCP/IP stacks) work better than others...

One of the things I intend to do more research on. I know older versions of Windows (Win98 before the Second Edition) were not so good... they would stick with the same "bad" IP if a server didn't respond. I'll likely do my own tests (those 10 machines I have doesn't hurt).

By all! Off to Dallas. I will be at the First Saturday Swap meet :)