Greetings, Community!

I'm back with an exciting topic:

Azure Accelerated Networking and its integration with VyOS Universal Router. In this discussion, we'll delve into how this powerful feature can be leveraged to create high-performance routers in the Azure cloud environment.

Our focus will be on:

• Understanding Azure Accelerated Networking.
• Steps to integrate this feature with VyOS.
• Practical use cases for building efficient, high-speed routers on Azure.
• Potential benefits and considerations.

Whether you're a network engineer, NetDevOps, cloud architect, or VyOS enthusiast, this information will help you optimize your Azure-based networking solutions. Let's dive in and unlock the potential of combining VyOS with Azure's advanced networking capabilities!

What is Azure Accelerated Networking?

Azure Accelerated Networking is a new option for Azure Infrastructure as a Service (IaaS) Virtual Machine (VM) on the NIC level, providing several benefits by enabling single root I/O virtualization (SR-IOV) to a VM, greatly improving its networking performance. This high-performance path bypasses the host from the datapath, reducing latency, jitter, and CPU utilization, for use with the most demanding network workloads on supported VM types. You would typically use this feature with heavy workloads that need to send or receive data at high speed with reliable streaming and lower CPU utilization. It will enable speeds of up to 30 Gbps per Virtual Machine. Best of all, it’s free!

Traditionally, network traffic in virtual environments is processed through the virtual CPU, which can lead to performance bottlenecks. Accelerated Networking circumvents this limitation by allowing network traffic to be processed directly by Azure's network hardware, bypassing the virtual CPU entirely.

For VyOS VMs, the benefits of Accelerated Networking are particularly notable:

• Reduced Latency: Accelerated Networking significantly decreases network latency by processing network traffic at the hardware level. This is crucial for VyOS Universal Router, which often serves as routers and/or firewalls, where low latency is essential for optimal network performance.
• Enhanced Throughput: The direct hardware processing enables higher data throughput between VyOS VMs and other network components. This is particularly beneficial for VyOS appliances handling large volumes of traffic or serving as VPN endpoints.
• Optimized CPU Utilization: With network traffic bypassing the virtual CPU, more processing power becomes available for VyOS's routing, firewall, and other network functions. This leads to improved performance and scalability, especially in high-traffic scenarios.
• Improved Packet Processing: Accelerated Networking can enhance VyOS's ability to handle complex routing scenarios and process packets more efficiently. It is crucial for its role as a versatile network operating system.

For more detailed information about this functionality and its internal workings on Azure, please refer to the official documentation through the following link:

https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-overview?tabs=redhat

How do you enable this Accelerated Networking on VyOS?

Several methods exist to add this functionality to your VyOS virtual machine (VM) on Azure.

1. If we use the marketplace VyOS VM, it has an option called accelerated networking that enables this functionality :
   
   

• First stop/deallocate the VM:

az vm deallocate --resource-group myRG --name myVM

• Once stopped, enable Accelerated Networking on the NIC of your VM:

az network nic update --name myNic --resource-group myRG --accelerated-networking true

• Restart your VM. It's that simple.

az vm start --resource-group myRG --name myVM

3. To minimize human error or misunderstandings, I've developed a Terraform module that automates the VM process: 

                 azure-accelerated-networking-terraform

To check if accelerated networking is enabled using the GUI:

1. In the Azure management console, go to the desired VM, then Networking.
2. Select the desired NIC. In this example, accelerated networking is shown as enabled.

How to check if the interfaces have enabled Accelerated Networking

Check the VF (virtual function) for the VyOS Universal Router VM using the following commands. If you receive output similar to the following sample output, accelerated networking is enabled and working.

# interfaces view on VyOS


azureuser@macsec01:~$ sudo ip -s link show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

RX:  bytes packets errors dropped  missed   mcast

     208268 2432  0   0   0   0

TX:  bytes packets errors dropped carrier collsns

     208268 2432  0   0   0   0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1560 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 00:0d:3a:2c:2e:21 brd ff:ff:ff:ff:ff:ff

RX:  bytes packets errors dropped  missed   mcast

   17329091   78855  0   0   0   0

TX:  bytes packets errors dropped carrier collsns

   22443494   96247  0   0   0   0

3: vf_vf_eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1560 qdisc mq master eth0 state UP mode DEFAULT group default qlen 1000

link/ether 00:0d:3a:2c:2e:21 brd ff:ff:ff:ff:ff:ff

RX:  bytes packets errors dropped  missed   mcast

    1333736 7748  0   0   0   0

TX:  bytes packets errors dropped carrier collsns

   22477352   96254  0   0   0   0

altname enP41680p0s2

altname enP41680s1


# SR-IOV multi-queue, based on VM flavor: 


azureuser@macsec01:~$ sudo ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: n/a
TX: n/a
Other: n/a
Combined: 2
Current hardware settings:
RX: n/a
TX: n/a
Other: n/a
Combined: 2


# ring-buffer on VyOS

azureuser@macsec01:~$ sudo ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 16930
RX Mini: n/a
RX Jumbo: n/a
TX: 2560
TX push buff len: n/a
Current hardware settings:
RX: 8738
RX Mini: n/a
RX Jumbo: n/a
TX: 170
RX Buf Len: n/a
CQE Size: n/a
TX Push: off
RX Push: off
TX push buff len: n/a
TCP data split: n/a

azureuser@macsec01:~$ sudo lspci
a5e7:00:02.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80)

MACsec over WAN 

MACsec is an interesting alternative to existing tunneling solutions that protect Layer 2 by performing integrity, origin authentication, and, optionally, encryption. A normal use case is MACsec between hosts and access switches, between two hosts, or between two switches.

In this use case, we will explore how MACsec can be combined with VXLAN to protect traffic between two remote sites over the internet, similar to a site-to-site VPN at Layer 2. This approach offers a straightforward method to enhance our workload performance using a high-performance VyOS router.

High-level diagram on Azure

• VyOS version to deploy it: VyOS 1.4.0
• There are two remote sites, macsec01 on cloud and macsec02 on-prem, connected over a WAN or the Internet. 
• Azure VM works with accelerated networking and a public ip from Azure cloud.
• We use a private address to check connectivity over the tunnel
• A Layer 2 Tunnel connects the sites -  in our POC, we use a VxLAN tunnel secured by MACsec.
• At the end of this POC, we will set up an IPv4/IPv6 communication using MACsec over VxLAN technology.

Deployment and configuration:

The following high-level steps create the above architecture. As mentioned earlier, this post focuses on the VyOS configuration; if needed, Terraform automation will cover the basic steps.

To get started with this configuration, we will run the following commands:

 VyOS - macsec01(azure cloud-site):

# interfaces deploy on VyOS:

azureuser@macsec01:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address      MAC                VRF        MTU  S/L    Description
-----------  --------------  -----------------  -------  -----  -----  -------------
eth0         10.0.2.4/24     00:0d:3a:2c:2e:21  default   1560  u/u
lo           127.0.0.1/8     00:00:00:00:00:00  default  65536  u/u
             ::1/128

# vxlan interface configuration, we define a static MAC address and a larger TTL:

set interfaces vxlan vxlan1 mac '00:11:22:33:44:01'
set interfaces vxlan vxlan1 parameters ip ttl '64'
set interfaces vxlan vxlan1 remote '185.144.208.198'
set interfaces vxlan vxlan1 source-interface 'eth0'
set interfaces vxlan vxlan1 vni '10'

# macsec configuration parameters: 

set interfaces macsec macsec1 address '192.0.2.1/24'
set interfaces macsec macsec1 address '2001:db8::1/64'
set interfaces macsec macsec1 description 'To on-prem'
set interfaces macsec macsec1 security cipher 'gcm-aes-128'
set interfaces macsec macsec1 security encrypt
set interfaces macsec macsec1 security static key 'ddd6f4a7be4d8bbaf88b26f10e1c05f7'
set interfaces macsec macsec1 security static peer SEC02 key 'eadcc0aa9cf203f3ce651b332bd6e6c7'
set interfaces macsec macsec1 security static peer SEC02 mac '00:11:22:33:44:02'
set interfaces macsec macsec1 source-interface 'vxlan1'

VyOS - macsec02 (on-prem site):

# interfaces deploy on VyOS:

vyos@macsec02-onprem:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address          MAC                VRF        MTU  S/L    Description
-----------  ------------------  -----------------  -------  -----  -----  -------------
eth0         185.144.208.198/24  00:50:56:a4:4b:89  default   1560  u/u    WAN
lo           127.0.0.1/8         00:00:00:00:00:00  default  65536  u/u

# vxlan interface configuration, we define a static MAC address and a larger TTL:

set interfaces vxlan vxlan1 mac '00:11:22:33:44:02'
set interfaces vxlan vxlan1 parameters ip ttl '64'
set interfaces vxlan vxlan1 remote '4.231.10.136'
set interfaces vxlan vxlan1 source-interface 'eth0'
set interfaces vxlan vxlan1 vni '10'

# macsec configuration parameters: 

set interfaces macsec macsec1 address '192.0.2.2/24'
set interfaces macsec macsec1 address '2001:db8::2/64'
set interfaces macsec macsec1 description 'To Azure'
set interfaces macsec macsec1 security cipher 'gcm-aes-128'
set interfaces macsec macsec1 security encrypt
set interfaces macsec macsec1 security static key 'eadcc0aa9cf203f3ce651b332bd6e6c7'
set interfaces macsec macsec1 security static peer SEC01 key 'ddd6f4a7be4d8bbaf88b26f10e1c05f7'
set interfaces macsec macsec1 security static peer SEC01 mac '00:11:22:33:44:01'
set interfaces macsec macsec1 source-interface 'vxlan1'

Packet Analysis - debugging :

Let’s quickly review the traffic secured by MACsec. To verify that the solution is functioning correctly, a packet capture is performed on the WAN interface in the Azure cloud environment. Save the captured data to a file named macsec.pcap.

monitor traffic interface eth0 save macsec.pcap

To confirm that the solution is working, perform a few ping packets:

# generate traffic over macsec


azureuser@macsec01:~$ ping 192.0.2.2
PING 192.0.2.2 (192.0.2.2) 56(84) bytes of data.
64 bytes from 192.0.2.2: icmp_seq=1 ttl=64 time=97.0 ms
64 bytes from 192.0.2.2: icmp_seq=2 ttl=64 time=97.0 ms
64 bytes from 192.0.2.2: icmp_seq=3 ttl=64 time=97.0 ms
64 bytes from 192.0.2.2: icmp_seq=4 ttl=64 time=97.0 ms
64 bytes from 192.0.2.2: icmp_seq=5 ttl=64 time=97.1 ms
^C
--- 192.0.2.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 96.967/97.012/97.091/0.044 ms

azureuser@macsec01:~$ ping 2001:db8::2
PING 2001:db8::2(2001:db8::2) 56 data bytes
64 bytes from 2001:db8::2: icmp_seq=1 ttl=64 time=97.1 ms
64 bytes from 2001:db8::2: icmp_seq=2 ttl=64 time=97.1 ms
64 bytes from 2001:db8::2: icmp_seq=3 ttl=64 time=97.1 ms
^C
--- 2001:db8::2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 95.967/97.015/97.091/0.042 ms


# check the ARP cache

azureuser@macsec01:~$ show arp
Address    Interface    Link layer address    State
---------  -----------  --------------------  ---------
10.0.2.1   eth0         12:34:56:78:9a:bc     REACHABLE
192.0.2.2  macsec1      00:11:22:33:44:02     STALE

# check the IPV6 ND 

azureuser@macsec01:~$ show ipv6 neighbors
Address                   Interface    Link layer address    State
------------------------  -----------  --------------------  -------
fe80::211:22ff:fe33:4402  macsec1      00:11:22:33:44:02     STALE
2001:db8::2               macsec1      00:11:22:33:44:02     STALE

Packet capture open in Wireshark, where we see the encapsulations: 

Interface MACsec on vyos : Interface MACsec on vyos : 

azureuser@macsec01:~$ show interfaces macsec macsec1
6: macsec1: protect on validate strict sc off sa off encrypt on send_sci on end_station off scb off replay off
    cipher suite: GCM-AES-128, using ICV length 16
    TXSC: 0011223344010001 on SA 0
        0: PN 52, state on, key 00000000000000000000000000000000
    RXSC: 0011223344020001, state on
        0: PN 52, state on, key 01000000000000000000000000000000

Conclusion

Azure Accelerated Networking is a powerful technology that enables the creation of high-performance networking solutions. While this post focuses on a specific use case, its benefits extend beyond, enhancing various types of VPN connections such as IPsec and WireGuard.

Our primary focus here is on implementing a straightforward solution to boost the performance of a VyOS router. Our VyOS router handles workloads traveling from on-premises infrastructure to Azure environments. By leveraging Azure Accelerated Networking, we can significantly improve throughput and reduce latency.

Additionally, we'll explore the use of MACsec over WAN connections. MACsec provides an extra layer of security for our tunnels when transmitting data over the internet, ensuring data integrity and confidentiality.
This optimized approach combines the speed advantages of Azure Accelerated Networking with the robust security of MACsec, resulting in a high-performance, secure networking solution for hybrid cloud environments.

Keep an eye out for upcoming content. We're exploring additional cloud networking scenarios relevant to our topic. As always, we're grateful for the support of the VyOS community.