VyOS Project 2019 October update
Everyone is probably aware that VyOS was built upon a codebase that hasn't aged well. VyOS 1.2.0 was all about a (sometimes painful) base system and codebase refresh, but the good thing is that as we get rid of the old code and replace it with modern, redesigned alternatives, new things become possible.
Python library internals refresh
The first implementation of vyos.config, the Python library for reading the config data, used to rely on cli-shell-api calls for every operations. There were two reasons for it. First, it was the simplest thing to do. Second, since the original code inherited from Vyatta was GPLv2 only, everything that links against it would have to be under the same license terms, which excludes even GPLv3. One of the goals of VyOS is to be an open platform rather than a thing in itself, and we believe people should be able to write addons under any license they want, even if it's incompatible with GPL. For this reason all new libraries are under LGPLv2+, which requires that modifications of their own code must also be under LGPLv2 or a later version, but places no restrictions on code that uses those libraries. Calling an external executable and reading its output doesn't count as linking, so we used that loophole—at cost of performance.
In any case, since the Python rewrite movement really gained momentum among the community and the number of Python scripts in the system has grown, that approach started showing its limitations. Many people noticed that commits are now much slower than they used to be in 1.1.x. We are well aware of the problem, and one part of the solution has landed in the rolling release this month.
Calling an external executable is quite some performance overhead already, but the performance of cli-shell-api itself only made it worse. Now, upon initialization, vyos.config retrieves the running and session configs from cli-shell-api, and then uses the libvyosconfig-based library to parse and manipulate it in memory. Retrieving a value from the config is now a literal order of magnitude faster (old implementation takes about 20 seconds to retrieve "system host-name" thousand times, the new one takes only two seconds). Since the real functionality is in libvyosconfig, which is under LGPLv2+, and cli-shell-api is only used at initialization, the solution doesn't create licensing problems.
This is not a complete solution though, and there's room for improvement. One remaining problem is the script startup time. We are planning to use new parser's ability to parse partial configs to make scripts retrieve only the parts they really need. We've also found that jinja2 templates are fast to render, but slow to load, so we are going to look into precompiling them at package build time.
Exporting config to JSON
A long requested and long discussed feature is now here. You can export the whole config or its parts to JSON and use it in your scripts, in any language. The simplest way to do it is to use the | json pipe, but you can also call /usr/bin/vyos-config-to-json directly if you want.
# show interfaces dummy | json
{
"dummy": {
"dum0": {
"address": [
"192.168.168.1/32",
"192.168.168.2/32"
],
"description": "Test interface",
"disable": {},
"ip": {
"ospf": {
"bandwidth": "10000",
"cost": "100",
"network": "non-broadcast"
}
}
},
"dum1": {
"address": [
"203.0.113.76/32",
"203.0.113.79/32"
]
}
}
}
There's also experimental support for retrieving a chunk of config as a dict based on this approach, which may radically simplify commit-time scripts and rid them of long chains of return_value() calls in the future.
HTTP API improvements
The HTTP API is also improving. First, we've fixed a silly oversight: there was no command to save the config. Now where it. The endpoint is /config-file and it supports to operations: save and load. If you don't specify the file when saving, it saves to /config/config.boot. Here's an example:
# curl -X POST -Fkey=qwerty -Fdata='{"op": "save", "file": "/config/config.boot"}' https://192.0.2.1/config-file
{"error": null, "data": "Saving configuration to '/config/config.boot'...\nDone\n", "success": true}
There's also support for installing and removing images:
# curl -X POST -Fkey=qwerty -Fdata='{"op": "add", "url": "http://example.com/vyos.iso"}' https://192.0.2.1/image
# curl -X POST -Fkey=qwerty -Fdata='{"op": "delete", "url": "1.2.0-rc4"}' http://192.0.2.1/image
You can also retrieve the full of partial config through the API now:
# curl -X POST -Fkey=qwerty -Fdata='{"op": "showConfig", "path": ["interfaces", "dummy"], "configFormat": "json"}' https://192.0.2.1/retrieve
{
"error": null,
"data": {
"dummy": {
"dum0": {
"disable": {},
"description": "Test interface",
"address": [
"192.168.168.1/32",
"192.168.168.2/32"
],
"ip": {
"ospf": {
"bandwidth": "10000",
"cost": "100",
"network": "non-broadcast"
}
}
},
"dum1": {
"address": [
"203.0.113.76/32",
"203.0.113.79/32"
]
}
}
},
"success": true
}
In reality that output is not pretty-printed, though we may add a flag for it. The configFormat option defaults to "json", but it also supports "raw" (same as CLI "show" would give you) and "json_ast" values. The "json_ast" is a serialized config syntax tree.
Interface configuration script rewrites
Christian Poessinger et al. made a good progress with rewriting network interface configuration scripts in Python. When the rewrite is complete, it will make them much easier to maintain and extend.
Buster-based rolling release
It's not there yet, but we are planning to switch the rolling release to Debian Buster soon. Those images mostly work, but most serious problems with it have been resolved. If you want to help us with testing it, download images from downloads.vyos.io/?dir=rolling/equuleus/amd64 and report the findings.
Platform-specific images and planned build system overhaul
We once made a point to keep VyOS a single unified image, but that approach is showing its limits. The generic image got quite large, while a person using it on say KVM will never need open-vm-tools, and a person using it on VMware will never need the QEMU guest agent. A person using VyOS on a bare metal machine will never need any of those, but would benefit from platform-specific drivers. There's also an issue of link files for network interface naming.
At the same time, the current makefile of vyos-build contains multiple targets like "vmware" or "kvm" that conflate unrelated things together, namely package lists and image formats. The plan is to make these orthogonal and make it possible to specify the flavor (KVM, vmware, edgecore, dell...) through the ./configure options and select the disk image format with a make target. That way everyone will be able to build, for example, a raw image with EdgeCore drivers for dd'ing onto a HDD.
People with a corporate subscription will have access to the full array of prebuilt images, while the Professional level subscribers will be able to pick a platform. Everyone will be able to build them as usual.
Rolling release images will use a generic flavor that covers most platforms.
FRR scripts rewrite and VRF
The scripts that configure routing protocols is one of the oldest and messiest parts of the system. Many people are asking for VRF support, which requires changes to those scripts. Before we do it, we are going to rewrite that part to make it easier to maintain and extend. That will open a path not only to VRF, but also to adding support for the missing protocols (IS-IS for example) and multicast routing—it's supported by FRR but the CLI is missing.
This requires a serious design discussions, so everyone is welcome to participate. Even telling us which part of the routing protocols CLI you find most annoying is already valuable, so feel free to share your thoughts!
And by the way...
Historically, VyOS has had some kind of a silent community. Many people would install it and keep running it for years, without ever feeling a need to tell anyone anywhere about their success with it. The Amazon machine image has hundreds users—and zero reviews.
It feels strange to even have to say, but social confirmation is important. When we've applied for Discourse hosting for open source projects, they had doubts about the project because our repositories have too few stars on Github. In the long run projects are judged by their code quality and ability to serve user's need, but for initial impressions people rely on stars, likes, and reviews—that's just a fact. The more people are talking about, the more people join and contribute.
So, if you've got a few seconds to spare, star vyos-build or vyos-1x on github, leave a review on Amazon and Azure or Distrowatch. There are other platforms like AlternativeTo or G2
If you've got more time, write a blog post about your configuration. Don't stay silent.
Comments