Hello, Community!
Some weeks ago a very close friend of mine approached me and asked about an issue in his VyOS installation. He is using several WireGuard tunnels in a star topology connecting his private sites with a central hub. All of his traffic is forced to traverse the hub - which adds additional latency to the connections.
He wanted to change the design so that individual spokes are able to talk to one another, directly. This requires a topology change to a full-mesh when sticking with WireGuard. A fully meshed WireGuard installation with 4 sites will require you to configure 3 WireGuard tunnels per individual site, leading to a total of 12 tunnels—too complex!
I then explained how to use DMVPN (Dynamic Multipoint Virtual Private Network) with VyOS—and as there is a new LTS release on the way, it is time for some in-depth testing of a very old feature! As DMVPN was the initial reason I choose VyOS (1.1.7, more than 5 years ago), it is fun for me to write about the same topic in 2021.
Topology
We both live in Germany and most of the ISPs use PPPoE to terminate customer sessions. Thus this post explains a real-world topology re-created in a lab. Even most ISPs do not let customers choose to opt-in for a static IP address on the PPPoE session, thus all spokes use a dynamic public IPv4 address.
The lab consists of three PPPoE enabled sites (A, B and C) connected to the ISPs BNG platform. The fourth site (D) acts as the DMVPN hub. D is connected to a different port of the BNG platform of the same ISP but this is of no relevance for this setup - we only need D to have a static IPv4 address and be reachable by A, B and C. We also do not look at the BNG configuration as this is not required for this setup/out of scope.
This article was developed using VyOS 1.3-epa2 on a lab hosted by GeFoekoM e.V. (AS12817) which provided me with VMs and IP prefixes for my setups.
Basic connectivity
Customer / spoke sites
We first need to dial our PPPoE connection from all our three CPE (Customer Premises Equipment) devices located on sites A, B and C. I only cover this for CPE-1 as the configuration is identical - except for the authentication data - for CPE-2 and CPE-3.
set interfaces pppoe pppoe1 authentication password 'cpe-1'
set interfaces pppoe pppoe1 authentication user 'cpe-1'
set interfaces pppoe pppoe1 no-peer-dns
set interfaces pppoe pppoe1 source-interface 'eth0.7'
set system host-name 'cpe-1'
The PPPoE session is established via eth0 VLAN7 which is required by the local ISP. We also do not request a DNS server from our peer as we will handle DNS by our self. The connection is immediately dialed and you can verify it:
vyos@cpe-1:~$ show interfaces pppoe
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
pppoe1 100.64.128.17/32 u/u
Our client now uses 100.64.128.17 on the WAN/PPPoE side - this IP address will be used by DMVPN to form the tunnels.
D - our DMVPN hub site
The VyOS instance acting as the DMVPN hub is offered a private leased line by the ISP and connects to the BNG platform. The IP address offered from the ISP to be used on CPE-4 is 100.64.10.1/31. CPE-4 uses a static default route to point all traffic towards the BNG interface at 100.64.10.0 which is directly connected on eth0.
We establish basic reachability by running:
set interfaces ethernet eth0 address '100.64.10.1/31'
set protocols static route 0.0.0.0/0 next-hop 100.64.10.0
set system host-name 'cpe-4'
We now do some ICMP pings from the hub (CPE-4) to one of our remote CPE devices, CPE-1 for instance.
vyos@cpe-1:~$ ping 100.64.10.1
PING 100.64.10.1 (100.64.10.1) 56(84) bytes of data.
64 bytes from 100.64.10.1: icmp_seq=1 ttl=63 time=24.0 ms
vyos@cpe-1:~$ traceroute 100.64.10.1
traceroute to 100.64.10.1 (100.64.10.1), 30 hops max, 60 byte packets
1 100.64.128.1 (100.64.128.1) 2.911 ms 2.276 ms 2.048 ms
2 100.64.10.1 (100.64.10.1) 4.597 ms 5.513 ms 8.167 ms
vyos@cpe-4:~$ ping 100.64.128.17
PING 100.64.128.17 (100.64.128.17) 56(84) bytes of data.
64 bytes from 100.64.128.17: icmp_seq=1 ttl=63 time=4.48 ms
It is now time to do the basic connectivity setup for your CPE-2 and CPE-3 site and check reachability from one site to another. CPE-1 to CPE-2, CPE-2 to CPE-3 ...
DMVPN
This is what Wikipedia tells us:
DMVPN (Dynamic Multipoint Virtual Private Network) provides the capability for creating a dynamic-mesh VPN network without having to pre-configure all possible tunnel end-point peers, including IPsec (Internet Protocol Security) and ISAKMP (Internet Security Association and Key Management Protocol) peers. DMVPN is initially configured to build out a hub-and-spoke network by statically configuring the hubs (VPN headends) on the spokes, no change in the configuration on the hub is required to accept new spokes. Using this initial hub-and-spoke network, tunnels between spokes can be dynamically built on demand (dynamic-mesh) without additional configuration on the hubs or spokes. This dynamic-mesh capability alleviates the need for any load on the hub to route data between the spoke networks.
Tunnels
Every site requires a tunnel interface which is used as the virtual circuit connecting all spokes and the hub in a single point-to-multipoint network. We use the interface tun0 for this. The prefix for tun0 is 192.168.254.0/26 and every site requires one IP address from this prefix. The tunnel interface will later on carry all the encrypted communication from e.g. CPE-1 -> CPE-3 or CPE-2 -> CPE-1.
GRE (Generic Routing Encapsulation) is used to form the site-to-site tunnels.
Hub
As the hub has a static IP address for his WAN connection, we tell the tunnel interface to source all connections from this fixed IP address (100.64.10.1).
set interfaces tunnel tun0 address '192.168.254.62/26'
set interfaces tunnel tun0 encapsulation 'gre'
set interfaces tunnel tun0 multicast 'enable'
set interfaces tunnel tun0 parameters ip key '1'
set interfaces tunnel tun0 source-address '100.64.10.1'
Spoke
All spoke sites use a common configuration for the tunnel interface which carries the site-to-site traffic, except that the tunnel interface IP address needs to be adjusted per each individual CPE.
Common Configuration
set interfaces tunnel tun0 encapsulation 'gre'
set interfaces tunnel tun0 multicast 'enable'
set interfaces tunnel tun0 parameters ip key '1'
set interfaces tunnel tun0 source-address '0.0.0.0'
CPE-1
set interfaces tunnel tun0 address '192.168.254.1/26'
CPE-2
set interfaces tunnel tun0 address '192.168.254.2/26'
CPE-3
set interfaces tunnel tun0 address '192.168.254.3/26'
NHRP setup
As already outlined I am going to use CPE-4 as the DMVPN hub as this is the only site which provides a static IP address. DMVPN relies on the fact that at least the hub uses a static IP address. The static IP address is required so that clients on a dynamic address can query the hub and ask for the dynamic IP address used by a spoke site. This is done by the Next Hop Resolution Protocol (NHRP).
VyOS makes use of OpenNHRP. OpenNHRP implements NBMA Next Hop Resolution Protocol (as defined in the RFC 2332). It makes it possible to create dynamic multipoint VPN Linux router using NHRP, GRE and IPsec.
Hub
set protocols nhrp tunnel tun0 cisco-authentication 'secret'
set protocols nhrp tunnel tun0 holding-time '300'
set protocols nhrp tunnel tun0 multicast 'dynamic'
set protocols nhrp tunnel tun0 redirect
set protocols nhrp tunnel tun0 shortcut
SPOKE
The spoke configuration is identical on each and every spoke system as we only inform our system that if you are looking for the IP address 192.168.254.62/26 it can be reached via 100.64.10.1
set protocols nhrp tunnel tun0 cisco-authentication 'secret'
set protocols nhrp tunnel tun0 holding-time '300'
set protocols nhrp tunnel tun0 map 192.168.254.62/26 nbma-address '100.64.10.1'
set protocols nhrp tunnel tun0 map 192.168.254.62/26 register
set protocols nhrp tunnel tun0 multicast 'nhs'
set protocols nhrp tunnel tun0 redirect
set protocols nhrp tunnel tun0 shortcut
IPSec
Setting up a dynamically, fully-meshed site-to-site VPN network only makes sense if the data passing to the internet is encrypted. VyOS relies on strongSwan to do all the heavy lifting for this.
VyOS needs to know which interface should participate in the the IPSec process and accept encrypted packets to be processed. This needs to be set individually on every system according to the interface used on the WAN side.
The DMVPN hub would use
set vpn ipsec ipsec-interfaces interface 'eth0'
whereas DMVPN spokes would use pppoe1, pppoe2 or pppoe3 - depending on the CPE you are configuring.
set vpn ipsec ipsec-interfaces interface 'pppoeX'
We now have all the basic connectivity configured. It is time to secure our site-to-site links using IPSec. First of all we need to define the cipher suite used for the connection on every system participating in the network, so the cipher configuration must be identical on each and every system.
set vpn ipsec esp-group ESP-DMVPN compression 'disable'
set vpn ipsec esp-group ESP-DMVPN lifetime '1800'
set vpn ipsec esp-group ESP-DMVPN mode 'transport'
set vpn ipsec esp-group ESP-DMVPN pfs 'dh-group2'
set vpn ipsec esp-group ESP-DMVPN proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-DMVPN proposal 1 hash 'sha1'
set vpn ipsec ike-group IKE-DMVPN ikev2-reauth 'no'
set vpn ipsec ike-group IKE-DMVPN key-exchange 'ikev1'
set vpn ipsec ike-group IKE-DMVPN lifetime '3600'
set vpn ipsec ike-group IKE-DMVPN proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-DMVPN proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-DMVPN proposal 1 hash 'sha1'
After deciding for which cipher to go and configuring them on your systems the last thing to get the DMVPN network up and running is to create a new IPSec profile which is used to handle any encryption/decryption in the packets traversing the tun0 interface. This configuration is again identical on every device be it a DMVPN hub or one of the DMVPN spokes.
set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'VyOS-topsecret'
set vpn ipsec profile NHRPVPN bind tunnel 'tun0'
set vpn ipsec profile NHRPVPN esp-group 'ESP-DMVPN'
set vpn ipsec profile NHRPVPN ike-group 'IKE-DMVPN'
Connectivity tests
After having all of our nodes configured to establish a secure connection to each other it is time for a short test-drive and see if we can reach our neighboring devices.
CPE-1 should reach CPE-2, CPE-3 and CPE-4
vyos@cpe-1:~$ ping 192.168.254.2
PING 192.168.254.2 (192.168.254.2) 56(84) bytes of data.
64 bytes from 192.168.254.2: icmp_seq=1 ttl=63 time=18.7 ms
vyos@cpe-1:~$ ping 192.168.254.3
PING 192.168.254.3 (192.168.254.3) 56(84) bytes of data.
64 bytes from 192.168.254.3: icmp_seq=1 ttl=63 time=22.2 ms
vyos@cpe-1:~$ ping 192.168.254.62
PING 192.168.254.62 (192.168.254.62) 56(84) bytes of data.
64 bytes from 192.168.254.62: icmp_seq=1 ttl=64 time=8.94 ms
CPE-2 should reach CPE-1, CPE-3 and CPE-4
vyos@cpe-2:~$ ping 192.168.254.1
PING 192.168.254.1 (192.168.254.1) 56(84) bytes of data.
64 bytes from 192.168.254.1: icmp_seq=1 ttl=64 time=4.02 ms
vyos@cpe-2:~$ ping 192.168.254.3
PING 192.168.254.3 (192.168.254.3) 56(84) bytes of data.
64 bytes from 192.168.254.3: icmp_seq=1 ttl=63 time=20.9 ms
vyos@cpe-2:~$ ping 192.168.254.62
PING 192.168.254.62 (192.168.254.62) 56(84) bytes of data.
64 bytes from 192.168.254.62: icmp_seq=1 ttl=64 time=4.04 ms
... and so on ...
BGP
Dynamic multipoint VPN is only fun if the LAN (Local Area Network) routes are also learned dynamically on our routers (CPE-1 to CPE-4). When sticking with static routes in this example one would require to configure four static routes with their respective gateway per CPE instance. If another CPE is added to the mesh, you will need to install the new route(s) on each other CPE device.
As the second part of this articles headline is BGP we will now configure our required BGP sessions between the hub and spoke sites so routing information is exchange dynamically via the hub.
VyOS makes use of FRRouting as its routing control plane which is abstracted by our CLI commands.
In order to make network extension less of a pain for future workloads this guide assigned a single /16 network to every site. We will only announce this /16 summary route through our BGP sessions. The internal routing of the site is aggregated at the CPE router. In our lab, we will only have one connected /24 network, but if you have multiple VLANs at your site, just draw any network from that particular /16 assigned to your location.
Hub
Our DMVPN hub holds an (e)BGP session to each individual spoke to learn and redistributes routes. The DMVPN hub uses BGP ASN 65000, CPE-1 uses 65001, CPE-2 uses 65002 and so on.
We will use a BGP peer-group for the DMPN spokes at the hub so in case we change something on our configuration we do it for all our DMVPN remote sites at once.
set protocols bgp 65000 neighbor 192.168.254.1 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 192.168.254.1 remote-as '65001'
set protocols bgp 65000 neighbor 192.168.254.2 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 192.168.254.2 remote-as '65002'
set protocols bgp 65000 neighbor 192.168.254.3 peer-group 'DMVPN'
set protocols bgp 65000 neighbor 192.168.254.3 remote-as '65003'
set protocols bgp 65000 parameters default no-ipv4-unicast
set protocols bgp 65000 parameters log-neighbor-changes
set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast
set protocols bgp 65000 timers holdtime 30
set protocols bgp 65000 timers keepalive 10
As the hub site might also have a local area network attached, we also announce the route for the hubs LAN prefix - optional.
# announce our local network into BGP
set protocols static route 172.20.0.0/16 blackhole distance '200'
set protocols bgp 65000 address-family ipv4-unicast network 172.20.0.0/16
Spokes
CPE-1
set protocols bgp 65001 neighbor 192.168.254.62 address-family ipv4-unicast
set protocols bgp 65001 neighbor 192.168.254.62 remote-as '65000'
set protocols bgp 65001 parameters default no-ipv4-unicast
set protocols bgp 65001 parameters log-neighbor-changes
set protocols bgp 65001 timers holdtime 30
set protocols bgp 65001 timers keepalive 10
# announce our local network into BGP
set protocols static route 172.17.0.0/16 blackhole distance '200'
set protocols bgp 65001 address-family ipv4-unicast network 172.17.0.0/16
CPE-2
set protocols bgp 65002 neighbor 192.168.254.62 address-family ipv4-unicast
set protocols bgp 65002 neighbor 192.168.254.62 remote-as '65000'
set protocols bgp 65002 parameters default no-ipv4-unicast
set protocols bgp 65002 parameters log-neighbor-changes
set protocols bgp 65002 timers holdtime 30
set protocols bgp 65002 timers keepalive 10
# announce our local network into BGP
set protocols static route 172.18.0.0/16 blackhole distance '200'
set protocols bgp 65002 address-family ipv4-unicast network 172.18.0.0/16
CPE-3
set protocols bgp 65003 neighbor 192.168.254.62 address-family ipv4-unicast
set protocols bgp 65003 neighbor 192.168.254.62 remote-as '65000'
set protocols bgp 65003 parameters default no-ipv4-unicast
set protocols bgp 65003 parameters log-neighbor-changes
set protocols bgp 65003 timers holdtime 30
set protocols bgp 65003 timers keepalive 10
# announce our local network into BGP
set protocols static route 172.19.0.0/16 blackhole distance '200'
set protocols bgp 65003 address-family ipv4-unicast network 172.19.0.0/16
BGP verification
Now that all our BGP neighborships are established, it is time to validate if the sessions are up and routes are exchanged. For this VyOS comes with the show bgp summary command. The older show ip bgp summary command will also work but is limited to the IPv4 address family.
vyos@cpe-4:~$ show bgp summary
IPv4 Unicast Summary:
BGP router identifier 192.168.254.62, local AS number 65000 vrf-id 0
BGP table version 12
RIB entries 7, using 1344 bytes of memory
Peers 3, using 64 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
192.168.254.1 4 65001 53 63 0 0 0 00:17:49 1 4
192.168.254.2 4 65002 40 44 0 0 0 00:17:31 1 4
192.168.254.3 4 65003 36 42 0 0 0 00:17:07 1 4
The hub tells us that it has three active BGP sessions to every individual DMVPN spoke and received one route per spoke and in addition sent 4 routes to the spoke.
We can now have a look at the routing table on the hub and see if we have successfully learned all the prefixes.
vyos@cpe-4:~$ show ip bgp
BGP table version is 12, local router ID is 192.168.254.62, vrf id 0
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 172.17.0.0/16 192.168.254.1 0 0 65001 i
*> 172.18.0.0/16 192.168.254.2 0 0 65002 i
*> 172.19.0.0/16 192.168.254.3 0 0 65003 i
*> 172.20.0.0/16 0.0.0.0 0 32768 i
We can also check the local routing table of any one of our spoke sites, this example uses CPE-1.
vyos@cpe-1:~$ show ip bgp
BGP table version is 8, local router ID is 192.168.254.1, vrf id 0
Default local pref 100, local AS 65001
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 172.17.0.0/16 0.0.0.0 0 32768 i
*> 172.18.0.0/16 192.168.254.2 0 65000 65002 i
*> 172.19.0.0/16 192.168.254.3 0 65000 65003 i
*> 172.20.0.0/16 192.168.254.62 0 0 65000 i
Testing the newly formed mesh network
We should not forget about to also test endpoint reachability by conducting end-to-end tests from C1, C2 and C3 which simulate our local attached devices/computers
C1 -> C2
C1> ping 172.18.1.100
84 bytes from 172.18.1.100 icmp_seq=1 ttl=62 time=7.746 ms
84 bytes from 172.18.1.100 icmp_seq=2 ttl=62 time=7.124 ms
C1> trace 172.18.1.100
trace to 172.18.1.100, 8 hops max (ICMP), press Ctrl+C to stop
1 172.17.1.1 0.753 ms 0.477 ms 0.517 ms
2 192.168.254.2 6.701 ms 3.331 ms 2.730 ms
3 172.18.1.100 3.775 ms 4.183 ms 3.701 ms
C1 -> C3
C1> ping 172.19.1.101
84 bytes from 172.19.1.101 icmp_seq=1 ttl=62 time=9.368 ms
84 bytes from 172.19.1.101 icmp_seq=2 ttl=62 time=6.211 ms
C1> trace 172.19.1.101
trace to 172.19.1.101, 8 hops max (ICMP), press Ctrl+C to stop
1 172.17.1.1 1.095 ms 0.601 ms 0.886 ms
2 192.168.254.3 4.182 ms 2.565 ms 2.303 ms
3 172.19.1.101 4.039 ms 2.722 ms 2.424 ms
Verification of the spoke-to-spoke communication
This guide/post would not be complete if I would not have checked if the communication really goes from spoke to spoke without the hub forwarding our traffic.
On C3 I set up a ICMP ping using 1200 bytes towards C1. I used this high ICMP payload size to identify the encrypted packets in the Wireshark analysis. On the image below we will see that the IPSec connection really goes from CPE-3 (100.64.128.19) to CPE-1 (100.64.128.17) and data is directly sent.
Recap
I hope you have once more enjoyed reading about one of the VPN features from VyOS - even if this is an ancient one, but still in use and recommended for certain use-cases. As VyOS supports a large variety of different VPN techniques there is no "right" choice to make as there is always another solution.
Cheers!
Comments