Issues with HTTP requests [Archive]

HostSentry

07-20-2007, 07:14 PM

For some reason I can't quiet get cartikahosting.com to respond properly through port 80 on a TCP connection.

I always get this:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>301 Moved Permanently</TITLE>
</HEAD><BODY>
<H1>Moved Permanently</H1>
The document has moved <A HREF="http://www.cartikahosting.com/">here</A>.<P>
<HR>
<ADDRESS>Apache/1.3.37 Server at cartikahosting.com Port 80</ADDRESS>
</BODY></HTML>

No matter what I try.
I've used cartikahosting.com as the hostname, www.cartikahosting.com (http://www.cartikahosting.com) as the hostname... and it does not change at all.

I've tried all of these GET methods:

GET http://www.cartikahosting.com/\n

GET http://www.cartikahosting.com\n

GET http://cartikahosting.com\n

GET http://www.cartikahosting.com/\n
No luck at all here. Anyone have any thoughts?

mwatkins

07-20-2007, 08:02 PM

You aren't making a proper HTTP/1.1 request. Do at least the following:

% telnet cartikahosting.com 80
GET / HTTP/1.1
Host: www.cartikahosting.com

And you'll get:

Connected to www.cartikahosting.com.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Sat, 21 Jul 2007 00:02:53 GMT
Server: NOYB
X-Powered-By: PHP/4.4.7
Set-Cookie: 17a7a6a1a5164868702bd829667ece39=-; path=/
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Sat, 21 Jul 2007 00:02:56 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
Transfer-Encoding: chunked
Content-Type: text/html

89be
<!DOCTYPE html PUBLIC [snip]

HostSentry

07-20-2007, 08:08 PM

Well I am using a Socket connection, not telnet.

You might be right about sending the wrong request... but it is weird that it works on about 30 other sites I tried.

I'll look into it.

print $socket "GET / HTTP/1.1 http://cartikahosting.com\n";

... returns

HTTP/1.1 400 Bad Request
Date: Sat, 21 Jul 2007 00:08:01 GMT
Server: NOYB
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>400 Bad Request</TITLE>
</HEAD><BODY>
<H1>Bad Request</H1>
Your browser sent a request that this server could not understand.<P>
The request line contained invalid characters following the protocol string.<P>
<P>
<HR>
<ADDRESS>Apache/1.3.37 Server at cartikahosting.com Port 80</ADDRESS>
</BODY></HTML>

mwatkins

07-20-2007, 08:09 PM

Socket request / telnet - doesn't matter. The key is the request you are sending is incorrect. That some HTTP servers give a response is irrelevant, but certainly makes debugging more interesting, doesn't it!

Key points: the GET request doesn't specifiy a fully qualified hostname plus resource path, its just the resource path, i.e. / or /index.html or /users/foo/.

HTTP/1.0 didn't provide the hostname in the request. Its ignorant of the ways of virtual hosts and such.

HTTP/1.1 fixed that by adding a "Host:" header which must be passed, per the specification.

The most common form of Request-URI is that used to identify a
resource on an origin server or gateway. In this case the absolute
path of the URI MUST be transmitted (see section 3.2.1, abs_path) as
the Request-URI, and the network location of the URI (authority) MUST
be transmitted in a Host header field. For example, a client wishing
to retrieve the resource above directly from the origin server would
create a TCP connection to port 80 of the host "www.w3.org" and send
the lines:

GET /pub/WWW/TheProject.html HTTP/1.1
Host: www.w3.org

http://tools.ietf.org/html/rfc2616#section-14.23

A client MUST include a Host header field in all HTTP/1.1 request
messages .

mwatkins

07-20-2007, 08:22 PM

Well I am using a Socket connection, not telnet.
You might be right about sending the wrong request... but it is weird that it works on about 30 other sites I tried.

I think what you are seeing is the result of your "implied" HTTP 1.0 request. Your earlier attempts probably were interpreted as HTTP/1.0 requests by various web servers (since the protocol wasn't specified in your request) and the absence of the Host: header would have also helped a web server come to that conclusion. 1.0 requests can't have host headers.

What you have have seen, in some cases, is generic pages being returned - of the sort that a web server configured to do vhosting returns when someone tries to browse via IP rather than hostname.tld

You can compare... do a 1.0 style and 1.1 style (with the host header) request and raise a message if they are different - suspect you'll find some that are.

HostSentry

07-20-2007, 08:33 PM

Thanks for all of the help!

You are definitely right... but I am having trouble sending the information over the socket.

Basically you think I should send :
[GET / HTTP/1.1
Host: www.cartikahosting.com (http://www.cartikahosting.com)
]
... to the server.

For some reason that just causes the server to not respond at all.

Edit:

Hey!! It worked. I just needed to add another linebreak. It makes sense that that would be the case, otherwise the server would not know if I wanted to add other info.

Thanks!

Edit Two:

I can't help but notice that requests take 5 seconds or so to begin streaming data this way. When doing it "improperly" it would be nearly instant.

Azavia

07-21-2007, 01:00 AM

Probably something with your script; it definitely does not take 5 seconds.

Also, you should generally use CR LF for linebreaks in the HTTP protocol, not LF. Most servers will accept LF, but it is somewhat standardized to use CR LF as far as I know.

mwatkins

07-21-2007, 07:37 AM

I can't see why the request would stall for 5 seconds although if you are seeing a difference between a "default page" (aka a "This web site is under construction" offered up by the web server) and the output of an actual application, there may well be some lag.

If it pauses for some time regardless of where you point your script to (try it on a localhost webserver), then indeed you must have an issue with your code.

cartika-andrew

07-21-2007, 06:21 PM

HostSentry

07-21-2007, 07:47 PM

Hey hostsentry - Im having our network admin take a look at this and will forward you his response - based on your comments, I can almost assure you that you have a combination of issues - 1) code that you are using and 2) our mod_security configuration. In the meantime, would you mind changing the thread title? - I am positive you dont mean to insinuate that cartikahosting is having problems, but, the clients that pointed me to this thread certainly had that impression.

Having said all of this - Love your service and I will certainly get you the answers you need as we definately want to participate in what you are doing....

It was merely an issue with the server not responding to a GET request without HTTP information. Really that is not a problem at all, and it is certainly not your fault.

If a mod would like to change it to "Issues with HTTP requests" that would be great. I can't edit it myself.

cartika-andrew

07-21-2007, 07:50 PM

It was merely an issue with the server not responding to a GET request without HTTP information. Really that is not a problem at all, and it is certainly not your fault.

If a mod would like to change it to "Issues with HTTP requests" that would be great. I can't edit it myself.

Thanks man - dont worry, I didnt take it the wrong way at all... just heard some interesting feedback from this thread....

really appreciate what you are doing and I will drop you an email with the coding you need to use in order to fulfill this requirement on our cluster...

again - I am thrilled that you are adding us to your service and am really happy to be involved....

HostSentry

07-21-2007, 08:27 PM

Thanks man - dont worry, I didnt take it the wrong way at all... just heard some interesting feedback from this thread....

really appreciate what you are doing and I will drop you an email with the coding you need to use in order to fulfill this requirement on our cluster...

again - I am thrilled that you are adding us to your service and am really happy to be involved....
Actually it is going to be used to search for changes on your web site, then archive the newest page for later viewing. Additionally it will grab pricing information, useful contact information, and a whole lot of other stuff. Unfortunately my funding budget of 0.00 dollars won't let me register an account with your company ;). It's not just your company though, but yours was the first that didn't respond to a basic GET request.

cartika-andrew

07-21-2007, 08:30 PM

Actually it is going to be used to search for changes on your web site, then archive the newest page for later viewing. Additionally it will grab pricing information, useful contact information, and a whole lot of other stuff. Unfortunately my funding budget of 0.00 dollars won't let me register an account with your company ;). It's not just your company though, but yours was the first that didn't respond to a basic GET request.

lol - no fear - if you need a free account for monitoring, I have no issue providing it. As for a basic get request - we have some pretty advanced security going on, so, it is no surprise that automated requests are being blocked. We will happily work with you on this stuff

thanks again !

HostSentry

07-21-2007, 08:36 PM

lol - no fear - if you need a free account for monitoring, I have no issue providing it. As for a basic get request - we have some pretty advanced security going on, so, it is no surprise that automated requests are being blocked. We will happily work with you on this stuff

thanks again !

Thanks!

If a moderator would not mind, please lock this thread and change the name to "Issues with HTTP Request".
:lovewht:

Xeentech

07-21-2007, 10:15 PM

What's with every one reimplementing HTTP over again today? Some one else was doing it in This Thread (http://www.webhostingtalk.com/showthread.php?t=621923) too.

This is work that has already been covered :)

Its good to have an understanding of the HTTP spec so you know how your server works. But it's also a good idea just to use a stack that is tested and proven.

See libCurl (http://curl.haxx.se/libcurl/). There are binding for just about every popular language.

HostSentry

07-21-2007, 10:20 PM

Xeentech

07-21-2007, 10:27 PM

Because I could have it up an running in about 15 mins, where as you're still dealing with abstract ideas to do with socket programming and reading and writing buffers, learning HTTP/1.1 etc..

Jump to later on in your apps development when you want to add HTTPS/SSL/TLS.. that is just a flag in cURL and you wouldn't need to change any code at all.

If you're dealing with sockets your self you need to though almost all your work out and start reimplementing now doing RSA and AES etc.

No if you had to port your code to another platform like from UNIX to Win32 or Darwin, socket programming has a few small but annoying differences, but they're already handled by all of the programmers that went before you.

It's almost always a good idea to try and reuse code if possible.

HostSentry

07-21-2007, 10:31 PM

anon-e-mouse

07-22-2007, 06:23 AM

Thanks!

If a moderator would not mind, please lock this thread and change the name to "Issues with HTTP Request".
:lovewht:
The report button gets results quicker than hoping we will see this thread ;) Someone else did it for you this time.