This page details step by step how I accomplished aggregating two separate internet connections into one, seemingly single, connection to the Internet. This "single" connection has a greater bandwidth capacity that either of the individual connections could yield. As a side affect this also nicely doubles up as a redundant connection. This method can be used for achieving link aggregation for both home and business users alike for a fraction of the cost of commercial connections / aggregation units available that do the same thing and in theory could be expanded to as many links you like however be warned, the more links there are, the more overheads there will be with tunnelling; thus potentially noticeable drops in speed.
For this demonstration I am using two Virtual Machines via VirtualBox. This enables me to experiment with without cutting my servers off from remote administration. Both machines are running Ubuntu 10.04.3 LTS (Lucid) however there is no reason this wont work for other distributions as long as you modify the steps accordingly.
Just to get this out of the way. I am not responsible for anything you might break. Be cautious when playing around with routing tables/network interfaces especially with remote servers as it could potentially cause loss of connectivity which might not be a simple fix.
Firstly, we will cover the requirements needed for this demonstration:
The idea behind this is quite simple and has been around for a long time. Think of it in terms of plumbing. You have a narrow pipe from your house (Your ISP), to the drain outside (The Internet). This pipe can only carry so much water (Information) to its destination at any given moment. If you add another pipe and link them together you are essentially allowing both pipes between them to carry more to the destination, thus increasing throughput.
The only initial problem with this is that each of the connections to your ISP have their own routes and IP addresses. You cant simply have multiple connections to one ISP and have it magically work (Unless your ISP supports Channel Bonding and will set it up for you). What this guide aims to do (continuing from the analogy above), is wrap both pipes together with tape so that from inside the house and outside, they appear as one and carry more down tham then a single pipe on its own.
The solution is to create a VPN tunnel to your fat pipe endpoint via each connection. Once the tunnels have been bonded it will act as one interface. Your remote fat pipe server will appear as one IP address but all traffic to that new IP address is being split down both links and vice versa. Once at the other end, as its VPN tunnelled, your server will see it as originating from the same IP inside your network, regardless of the route it takes. This then enables you to route it to the Internet and back with ease.
This diagram summarizes the setup we are trying to achieve:
First off we'll install the bonding driver onto both servers. This is nice and simple: (Note # represents a command as root)
As we want to aggregate the multiple connections into one seemingly single link, I will be using Bonding mode 1, aka balace-rr. To enable this, we need to edit the /etc/network/interfaces file and add the following:
Note that my local subnet is 10.0.0.0/8 and so it is clear when I am using the tunnel, my VPN subnet will be 172.26.0.0/30
And the remote server:
As a brief explanation, balance-rr does not load balance connections but actually load balances the packets.
This might seem backwards to some people but when all is said and done, we will be forwarding all internet traffic to 172.26.0.1 thus that would be our gateway. As a rule of thumb gateway's are always the first IP in the subnet... (Usually)
For this to work, we need to ensure that we have separate connections to our endpoint. These separate connections need to travel through each internet connection. For arguments sake, I have two ADSL connections and thus two routers on my network - 10.0.0.1 and 10.0.0.254.
My fat pipe endpoint also needs to have two IP addresses. For this example - 126.96.36.199 and 188.8.131.52. These will enable me to ensure that my tunnels are routed via separate ISPs.
On my local server I need to do the following:
Note: If you want to aggregate more than two connections, using this method you would need the same number of remote IP addresses as you do Internet connections to ensure that traffic is sent via each interface.
Please be aware that this is by no means an elegant solution...
I originally decided to use SSH tunnels when I was playing around with bonding as both of my servers already had OpenSSH installed which meant no messing about installing/configuring OpenVPN (SSH is so versatile!). The only caveat to this method is that it requires you to allow SSH Logins via Root. It is also worth noting that using SSH Tunnels for the VPN connections essentially means you are tunnelling TCP over TCP which can potentially have a lot of overheads, especially via SSH due to the encryption and compression should you use it. See this article for some more information.
At this stage I'd like to point out that whilst I did use SSH, you can potentially use any VPN/Tunnelling technology such as OpenVPN - as long as you get a tun/tap interface that you can use with the bonding driver. That is however out of the scope of this article. If you will be using another technology or protocol to create your tun or tap devices then you can skip ahead to Bonding the links
As mentioned at the top of this page, my end point is in a Data Center. It is however a VM running on my server. Primarily this was to prevent me from losing access to my machine if I broke things however it also adds the security of, if the machine is compromised, the intruder doesn't have access to any of my services.
Never the less, we will secure it as much as possible... You need to edit your /etc/ssh/sshd_config file on your Remote Server and change/add the following:
Once that is done, add some keys from your local server to your remote server so that you can ssh to the root account. More information on SSH Keys can be found here or simply by googling.
Once you have confirmed that you can SSH to the root account of the remote server from the root account of your local server you are ready to go.
We will be making use of the -w parameter of ssh. This will create a tunX interface on both ends that we will later bond together.
If this is successful, ifconfig -a on both machines should reveal a tun0 and tun1 interface!
Great! Now we can proceed to bonding them.
NOTE: It might be advisable at this time to check netstat to ensure that you have connections coming in from separate ISPs:
The bold addresses are my tunnels and the other is a terminal I have open to execute netstat. The command I used shows only connections on port 22 (default ssh port) and will only show Established ones. Looks like we are good to go!
This part is relatively easy now all the heavy lifting is done. As mentioned above, we have created a bond0 interface on both servers that have two (or potentially more) slaves, tun0 and tun1. To bring this interface up we simply execute the following on both servers
Presto! Both interfaces are now up and running. You can test to ensure it is working by trying to ping each other
Next you ned to remove the default gateway of your local server (if there is one) and tell it to use the remote server's VPN address as its gateway instead:
All that is left is to enable ipv4 forwarding on both servers and add a route back to your internal network on the remote server and then you should be all set for using the local server as a gateway for the internet!To enable:
And then to tell the remote server how to get back to our network so that our requests get answered:
A pinch of iptables hackery:
Your local server should now be able to access the world wide web. Note that this server should, in this example at least, have an IP in the 10.0.0.0/8 subnet. That is the IP you use for your gateway on all your other machines to allow them to use the aggregated link.
I will at a later stage document automating the whole process as much as I can and have some form of connection monitoring in case the ADSL links stop working. Watch this space!
If everything was done correctly, your bonded link should be up and running. Your remote server should be able to "see" your local server via its VPN IP address and vice versa! If this is the case then most of the leg work has been done for you.
All outgoing connections to the internet from your local server should now be appearing as if they are from your remote server's IP address, in my example, 184.108.40.206. (You can quickly test this by using whatismyip.com). Lets say that you want to access a web page that you have set up on the local server that gives statistics about your setup. We will use iptables to acheive this:We are assuming that your remote server connects to the internet via eth0
Once this is done, any connection to http://220.127.116.11 should be transparently forwarded down your bonded link to the other end which should serve your web page.
This could also be expanded to, for example, allow Windows Remote Desktop to one of your internal computers to be accessible via the bonded link, or any service inside your LAN to be accessible from the publicly routable address of your remote server
NOTE: For the example above for forwarding RDP connections, the PC you are forwarding to will need to have its gateway set to the 10.0.0.0/8 address of the machine hosting the bonded link else it wont know where to send the responses to.
I found Karl Rupp's NAT page to be helpful in explaining NAT with iptables!