VyOS Networks Blog

Organizations go multi-cloud for good reasons, resilience and pricing leverage among them. What rarely gets budgeted is what happens to the network team once those workloads are actually live in AWS, Azure, and a datacenter at the same time.

Routing tables drift out of sync. VPN tunnels accumulate faster than anyone documents them. And the failover design that looked airtight in the diagram takes three times as long as planned the first time someone actually pulls the plug.

I want to be precise about why this happens, because it is not a monitoring problem, and another dashboard will not fix it. It is a routing architecture problem.

Why Multi-Cloud Routing Becomes Hard to Manage

Each cloud provider builds its networking stack for its own environment first. AWS Transit Gateway, Azure Virtual WAN, Google Cloud's Network Connectivity Center, and your on-premises edge routers were never designed to be one network. They are four capable networks that happen to need interconnecting, and the seams show.

The same patterns come up in nearly every multi-cloud environment I look at:

• Routing behavior that refuses to match. Azure VPN gateways default to ASN 65515. AWS uses 64512 as the Transit Gateway default. Google Cloud Router leaves the ASN choice entirely to you. Route propagation timing, metric handling, and what each side does with your BGP attributes all differ, so a policy you intended to be identical across environments diverges in practice unless you actively force it not to.

• VPN tunnel proliferation. AWS caps a site-to-site VPN tunnel at 1.25 Gbps, so scaling throughput means adding tunnels and ECMP, which means more tunnels. Add a region, a VPC, a VNet, or a site and the count climbs again. Most environments reach dozens of IPsec tunnels before anyone reassesses the topology, and every one of them has to be configured, monitored, and eventually troubleshot.

• Policy drift. Firewall rules and routing policy get configured separately in each environment, by whichever team did that particular deployment. Two years later, nobody can say with confidence what traffic is actually allowed to flow where.

• Blind spots at the boundaries. Native cloud monitoring is good inside its own walls and nearly useless at the seam between environments. One provider reports a tunnel as up while the other reports the same tunnel as down, and that seam is exactly where outages start. A surprising share of "tunnel is up but the application is broken" tickets turn out to be MTU and MSS clamping mismatches across IPsec that no cloud console will surface for you.

• Costs that scale with every connection you add. The native interconnect services all carry meters that architecture diagrams never show. AWS Transit Gateway bills $0.05 per hour per attachment plus $0.02 per GB of data processed, and each site-to-site VPN connection adds another $0.05 per hour. Azure Virtual WAN charges $0.361 per hour per site-to-site VPN scale unit plus $0.05 per hour for every branch connection, with outbound inter-VNet traffic starting at $0.035 per GB. Google Cloud VPN bills $0.05 per hour per tunnel (US-region list prices throughout). None of these numbers looks alarming on its own. Multiplied across dozens of tunnels, attachments in three clouds, and cross-cloud data transfer at $0.02 to $0.09 per GB, the interconnect line item grows faster than the workloads it serves. Stack per-instance or per-throughput appliance licensing on top and every redundant path costs proportionally more, when redundancy is the reason you went multi-cloud in the first place.

What These Gaps Actually Cost

None of this shows up as one visible failure. It shows up as a failover test that runs long, a security review that cannot get a straight answer about traffic paths, or an incident where you are troubleshooting the same tunnel through two unrelated vendor interfaces at once. It also shows up on the bill, as attachment hours, tunnel hours, and per-GB processing fees spread across three consoles, none of them itemized as "interconnect."

The common thread: every cloud-native or vendor-specific tool manages its own segment well and sees nothing beyond it. What multi-cloud networking actually needs is the opposite, a routing and VPN layer that behaves the same no matter which cloud, hypervisor, or box it runs on.

How to Close the Gaps

• Standardize on one network operating system everywhere. When the same OS runs on your cloud instances, your hypervisors, and your hardware, BGP, OSPF, and IPsec stop requiring three separate skill sets, and the debugging session at 2 a.m. works the same in every environment. I will be honest about the tradeoff: running your own routing layer means you own high availability, patching, and capacity planning, which Transit Gateway and Virtual WAN would otherwise handle for you. Past two environments the math favors ownership, because the cost of divergence grows faster than the cost of running the layer. But it is a real tradeoff, not a free win.
• Design routing policy once, centrally. Policy belongs at the architecture level, applied consistently, not reconstructed from memory inside each provider's console. Consistency in the configuration itself is what prevents drift. A policy document that nobody revisits does not.
• Choose your VPN topology deliberately. My default is hub-and-spoke: each cloud and site connects into a small number of consistent endpoints, which is easier to secure and far easier to troubleshoot than a mesh that grew organically. It is a default, not a law. A small full-mesh WireGuard deployment can be perfectly manageable. But fewer, consistently configured tunnels beat a larger number of loosely standardized ones almost every time.
• Make visibility span the boundaries, including cost visibility. Tunnel state and route state should look the same to you whether the far end sits in AWS, Azure, GCP, or your own rack, and native tooling alone will not get you there. Treat spend the same way: per-attachment hours, per-tunnel hours, and per-GB processing fees are scattered across separate billing consoles, so nobody sees the full interconnect cost until it has already grown. Identical operational commands on both ends of every connection solve the state half; a single, regularly reviewed interconnect cost line solves the other.
• Keep per-connection service fees and licensing out of your redundancy decisions. A backup path or an extra region should be an engineering decision. When every attachment, tunnel, and processed gigabyte carries its own meter, and appliance licensing scales per instance or per throughput tier, that decision quietly turns into a procurement negotiation, and the architecture ends up less redundant than it should be. Favor building blocks whose cost does not grow with the number of paths you build.

Where to Start

You do not need to rebuild the network at once. Start with an audit: list every tunnel, note which cloud or site it connects, and check that the routing policy matches on both ends. If you find a tunnel nobody on the team can fully account for, you are in good company. That finding is close to universal, and it is half the point of the exercise.

Then use the next planned region, VPC, or VNet as the pilot for the standardized approach, rather than attempting a wholesale migration in one push.

Where VyOS Fits

Everything above is deliberately vendor-neutral, because the approach matters more than any product. That said, this is the problem VyOS is built for, and since I work on it, here is the short version.

VyOS is the same network operating system, with the same BGP and OSPF implementation, the same IPsec and WireGuard stack, and the same CLI, whether it is deployed from the AWS, Azure, or Google Cloud marketplace, on a hypervisor, or on bare metal. You design the routing and VPN architecture once and run it identically in every environment. That is point one on the list above, made concrete.

On licensing: subscriptions are not tied to per-instance throughput tiers, and the software carries no artificial performance limits. Throughput is bounded by your hardware and the Linux kernel, and with the VPP-based dataplane, the kernel stops being the ceiling too. Adding a backup path costs you the compute it runs on, nothing more: no per-attachment hours, no per-tunnel meter, no per-GB processing fee.

If you are planning a new cloud deployment or auditing what you already have, the multi-cloud networking solutions pages cover site-to-site VPN, BGP, and hub-and-spoke design in more depth.

Deploy VyOS on your cloud platform

Have a specific multi-cloud topology you want a second opinion on?

Talk to our engineers!

Additional Links

• VyOS Solutions