Im trying to set up an IPSec tunnel between a Ubuntu VPS and 2 windows boxes and im having some issues.
First off, im confused about the ports. Do the ports that my windows boxes use need to be opened on the VPS's IpTables? or do they need to be forwarded to my windows boxes instead?
I have the IPSec server fully setup (i followed the instructions here and it appears to be working) but when i try connecting to it from either of my windows boxes; no connection is being made. I opened the ports the tunnel uses so that's not an issue and i even (temporarily) tried the tunnel connection with the firewall disabled and i was still not able to connect.
I happened to come across your post out of the blue and I hope I can be of a little help. IPSEC is an AWESOME tool to use but you must understand there are multiple different "styles" of what people call IPSEC, which many are misnomers. OK, so if you are planning on using the built in Windows VPN client, this is NOT a "Site-To-Site" IPSEC, it is actually using L2TP over IPSEC. (And actually, it also utilizes the Point to Point Protocol (PPP) daemon as well, which must be set up in the linux box also). I think it would help to explain the differenc between Site-to-site and "remote-access" IPSEC first.
Site to site allows you to essentially "bridge" remote subnet('s) to your local subnet('s) as if they were connected to the same switch/router. For instance, let's say you have a local network at home of 192.168.100.0/24 and you have 10.10.10.0/26 behind the DMZ on a remote server. I used different prefixes in the example on purpose, because it doesn't matter if it is a one to one mapping or not. I run several large ESXi vCenter Servers, each with about 75 virtual machines (somewhere around 500 servers and 6 class C private subnets as well as some /26 private subnets in total) and use site-to-site IPSEC with a HARDWARE cisco vpn box locally to offload the high speed and mathematically intensive decryption operations.
So, every one of the private networks that is totally isolated from the WAN is "transparently" bridged to my local domains at my office and home without ever dialing a VPN connection. All of my computers - and every one of my employees - has access (if they are so granted) to the backend of the remote network without any additional work. It truly makes the remote network appear to be local. If that is what you are looking to do, I would HIGHLY suggest purchasing an IPSEC VPN switch that 1) simplifies your life as far as setup goes and 2) offloads the computations into dedicated hardware (trust me, IPSEC is processor intensive, especially when you are talking high speed). You can go with a fairly cheap unit that Netgear or Linksys makes for under $129 on an 8 port gigabit switch. Last I saw, the ~$130 models will do somewhere around 15 separate tunnels simultaneously.
Now, if you are simply talking about an on demand, connect as needed, remote access VPN, I might suggest PPTP if Windows is your main OS. PPTP is much simpler to set up for simple remote access and you will likely get better performance also. If you are set on the IPSEC, again, unless you download an IPSEC VPN client capaple of the phase I and II negotiations (also referred to as the IKE and Security Association / ESP) I don't believe windows will do this for you. The windows firewall does have support to ALLOW certain SA and IKE parameters (e.g. AES128+SHA1 or 3DES+SHA1 or MD5, etc., etc) to PASS THROUGH, but it does not set up the negotiation for you.
There is another option, which is what I did at home, instead of the Cisco VPN. I took an old desktop computer, threw in a couple gigabit NICs and put Vyatta Community edition on it and set up all of the IPSEC site-to-site parameters right there. If you're into networking, have an old computer laying around (and, yes it can be old...vyatta doesn't need more than a few hundred megahertz .. maybe 500MHz and 512MB-1GB ram... less than 1GB hard disk!!) .. that may be your best choice. The system can do virtually any networking setup you need, provided you know how to tell it to do it, and the method of setting up the IPSEC is 10x easier than the way you are proposing now. I know this from experience, believe me.
I'll leave it here for now, if this didn't help you, let me know and I'll see if I can help further. Just don't get frustrated with it, in the end .. especially if you go for the site-to-site, you will be amazed that you ever went without it !
Oh, almost forgot about the ports! Typically you need to have the ports open on the VPN Server side, so, yes, IP Tables needs to allow the ports to be open (e.g. 500 and 4500). Also, make sure you don't have IP tables set with the default drop TCP on
So, this will take and keep you from opening those ports WIDE open and will allow it for the UDP traffic with the specified source and destination ports only. In this case, "eth0" would be your public interface on the linux box that is NOT behind the firewall or on the private space.
Last edited by sietec; 10-13-2013 at 03:58 PM.
Reason: forgot to mention iptables