VyOS - Blog

VyOS 2.0 development digest #9: socket communication functionality, complete parser, and open tasks

daniil@sentrium.io (Daniil Baturin) — Sun, 15 Jan 2017 18:40:40 GMT

Socket communication

A long-awaited (by me, anyway ;) milestone: VyConf is now capable of communicating with clients. This allows us to write a simple non-interactive client. Right now the only supported operaion is "status" (a keepalive of sorts), but the list will be growing.

I guess I should talk about the client before going into technical details of the protocol. The client will be way easier to use than what we have now. Two main problems with CLI tools from VyOS 1.x is that my_cli_bin (the command used by set/delete operations) requires a lot of environment setup, and that cli-shell-api is limited in scope. Part of the reason for this is that my_cli_bin is used in the interactive shell. Since the interactive shell of VyConf will be a standalone program rather than a bash completion hack, we are free to make the non-interactive client more idiomatic as a shell command, closer in user experience to git or s3cmd.

This is what it will look like:

SESSION=$(vycli setupSession)

vycli --session=$SESSION configure

vycli --session=$SESSION set "system host-name vyos"

vycli --session=$SESSION delete "system name-server 192.0.2.1"

vycli --session=$SESSION commit

vycli --session=$SESSION exists "service dhcp-server"

vycli --session=$SESSION commit returnValue "system host-name"

vycli --session=$SESSION --format=json show "interfaces ethernet"

As you can see, first, the top level words are subcommands, much like "git branch". Since the set of top level words is fixed anyway, this doesn't create new limitations. Second, the same client can execute both high level set/delete/commit operations and low level exists/returnValue/etc. methods. Third, the only thing it needs to operate is a session token (I'm thinking that unless it's passed in --session option, vycli should try to get it from an environment variable, but we'll see, let me know what you think about this issue). This way contributors will get an easy way to test the code even before interactive shell is complete; and when VyOS 2.0 is usable, shell scripts and people fond of working from bash rather than the domain-specific shell will have access to all system functions, without worrying about intricate environment variable setup.

The protocol

As I already said in the previous post, VyConf uses Protobuf for serialized messages. Protobuf doesn't define any framing, however, so we have to come up with something. Most popular options are delimiters and length headers. The issue with delimiters is that you have to make sure they do not appear in user input, or you risk losing a part of the message. Some programs choose to escape delimiters, other rely on unusual sequences, e.g. the backend of OPNSense uses three null bytes for it. Since Protobuf is a binary protocol, no sequence is unusual enough, so length headers look like the best option. VyConf uses 4 byte headers in network order, that are followed by a Protobuf message. This is easy enough to implement in any language, so it shouldn't be a problem when writing bindings for other languages.

The code

There is a single client library that can be used by all of the non-interactive client and the interactive shell. It will also serve as the OCaml bindings package for VyConf (Python and other languages wil need their own bindings, but with Protobuf, most of it can be autogenerated).

Parser improvements

Inactive and ephemeral nodes

The curly config parser is now complete. It supports the inactive and ephemeral properties. This is what a config with those will look like:

protocols {

  static {

    /* Inserted by a fail2ban-like script */

    #EPHEMERAL route 192.0.2.78/32 {

      blackhole;

    }

    /* DIsabled by admin */

    #INACTIVE route 203.0.113.128/25 {

      next-hop 203.0.113.1;

    }

  }

}

While I'm not sure if there are valid use cases for it, nodes can be inactive and ephemeral at the same time. Deactivating an ephemeral node that was created by scritp perhaps? Anyway, since both are a part of the config format that the "show" command will produce, we get to support both in the parser too.

Multi nodes

By multi nodes I mean nodes that may have more than one value, such as "address" in interfaces. As you remember, I suggested and implemented a new syntax for such nodes:

interfaces {

  ethernet eth0 {

    address [

      192.0.2.1/24;

      192.0.2.2/24;

    ];

  }

}

However, the parser now supports the original syntax too, that is:.

interfaces {

  ethernet eth0 {

    address 192.0.2.1/24;

    address 192.0.2.2/24;

  }

}

I didn't intend to support it originally, but it was another edge case that prompted me to add it. For config read operations to work correctly, every path in the tree must be unique. The high level Config_tree.set function maintains this invariant, but the parser gets to use lower level primitives that do not, so if a user creates a config with duplicate nodes, e.g. by careless pasting, the config tree that the parser returns will have them too, so we get to detect such situations and do something about it. Configs with duplicate tag nodes (e.g. "ethernet eth0 { ... } ethernet eth0 { ... }") are rejected as incorrect since there is no way to recover from this. Multiple non-leaf nodes with distinct children (e.g. "system { host-name vyos; } system { name-server 192.0.2.1; }") can be merged cleanly, so I've added some code to merge them by moving children of subsequent nodes under the first on and removing the extra nodes afterwards. However, since in the raw config there is no real distinction between leaf and non-leaf nodes, so in case of leaf nodes that code would simply remove all but the first. I've extended it to also move values into the first node, which equates support for the old syntax, except node comments and inactive/ephemeral properties will be inherited from the first node. Then again, this is how the parser in VyOS 1.x behaves, so nothing is lost.

While the show command in VyOS 2.0 will always use the new syntax with curly brackets, the parser will not break the principle of least astonishment for people used to the old one. Also, if we decide to write a migration utility for converting 1.x configs to 2.0, we'll be able to reuse the parser, after adding semicolons to the old config with a simple regulat expression perhaps.

Misc

Node names and unquoted values now can contain any characters that are not reserved, that is, anything but whitespace, curly braces, square brackets, and semicolons.

What's next?

Next I'm going to work on adding low level config operations (exists/returnValue/...) and set commands so that we can do some real life tests.

There's a bunch of open tasks if you want to join the development:

T254 is about preventing nodes with reserved characters in their names early in the process, at the "set" time. There's a rather nasty bug in VyOS 1.1.7 related to this: you can pass a quoted node name with spaces to set and if there is no validation rule attached to the node, as it's with "vpn l2tp remote-access authentication local-users", the node will be created. It will fail to parse correctly after you save and reload the config. We'll fix it in 1.2.0 of course, but we also need to prevent it from ever appearing in 2.0 too.

T255 is about adding the curly config renderer. While we can use the JSON serializer for testing right now, the usual format is also just easier on the eyes, and it's a relatively simple task too.

VyOS 2.0 development digest #8: vote for or against the new tag node syntax, and the protobuf schema

daniil@sentrium.io (Daniil Baturin) — Thu, 05 Jan 2017 22:10:17 GMT

Tag node syntax

The change in tag node format I introduced in the previous post turned out quite polarizing and started quite some discussion in the comments. I created a poll in phabricator for it: https://phabricator.vyos.net/V3 , please cast your vote there.

If you missed the post, or found the explanation confusing, here's what it's all about. Right now in config files we format tag nodes (i.e. nodes that can have children without predefined names, such as interfaces and firewall rules) differently from other nodes:

/* normal node */

interfaces {

  /* tag node */

  ethernet eth0 {

    address 192.0.2.1/24

  }

  /* tag node */

  ethernet eth1 {

    address 203.0.113.1/24

  }

}

It looks nice, but complicates the parser. What I proposed and implemented in the initial parser draft is to not use any custom formatting for tag nodes:

/* normal node */

interfaces {

  /* actually a tag node, but rendering is now the same as for normal */

  ethernet {

    eth0 {

      address 192.0.2.1/24;

    }

    eth1 {

     address 203.0.113.1/24

    }

  }

}

This makes the parser noticeable simpler, but makes the syntax more verbose and adds more newlines.

If more people vote against this change than for it, I'll take time to implement it in the parser.

Note: This change only affects the config syntax, and has no effect on the command syntax. The command for the example above would still be "set interfaces ethernet eth0 address 192.0.2.1/24", in user input and in the output of "show configuration commands". Tag nodes will also be usable as edit levels regardless of the config file syntax, as in "edit interfaces tunnel; copy tun0 to tun1".

Protobuf schema

Today I wrote an initial draft of the protobuf schema that VyConf daemon will use for communication with clients (shell, CLI tool, and HTTP bridge). You can find it here: https://github.com/vyos/vyconf/blob/master/data/vyconf.proto

Right now it defines the following operations:

SetupSession

Setup the VyConf communication session. Takes two optional arguments: ClientApplication and OnBehalfOf. ClientApplication is purely informational, it will show up in the commit log and other accounting logs just like we see "root by boot-config-loader" or "vyos by cli" in "run show system commit" now.

OnBehalfOf is more interesting, it's meant for HTTP bridge and possibly other API bridges (netconf etc.) that use their own authentication mechanism. The interactive shell and the CLI client when run locally will use SO_PEERCRED for authentication, because no one will ever want to enter their password in the VyOS shell after entering it to login via SSH, but the only alternative to SO_PEERCRED is full-fledged single sign-on through Kerberos. Since the HTTP bridge is a separate process that takes requests from non-local users, VyConf cannot take the UID from socket credentials, and needs another way to get it. Obviously, only special processes (e.g. that run with vyconfapi GID) will be allowed to set OnBehalfOf.

EnterConfigurationMode

That's what the shell will use for the "configure" command. Takes two optional arguments: Exclusive and OverrideExclusive. Exclusive mode will be an advisory lock, to avoid having to kill sessions in situations when a user left session open and went home, or a transition script hung. In the interactive shell, configure will display a message like:

$ configure

Warning: configuration is locked, user dmbaturin entered exclusive mode on 2017 Jan 06 22:20

If you are certain that you will not interfere with their work, you can override the lock.

Enter configuration mode? [n/Y]

Set, Delete

These are obvious I guess. They take configuration path. Set optionally takes argument "ephemeral" that makes the node market ephemeral, which means it will not appear in the saved config (this is for HA transition scripts, IDS tools and similar applications).

Copy, Rename

These take the base path, source, and destination. This means, among other things, that there will be no "copy rule 10 to rule 20" for the user, just "edit ... rule; copy 10 to 20".

Commit, Rollback

Commit takes optional Confirmed and ConfirmTimeout arguments for commit-confirm, and also optional Comment arguments for adding commit descriptions. Rollback takes Revision argument.

Comment

Adds a comment to a node.

ShowConfig

Takes optional Path argument for displaying parts of the config, and also optional Format argument that right now can be Curly or JSON. Instead of reading individual values through return_value(), config scripts (and web GUI) will be able to use the JSON output of ShowConfig to get the complete configuration of a node in machine-readable format, it can also be useful for external tools that work with the config (including migration scripts).

Exists, GetValue, GetValues, ListChildren

These will hardly be used by interactive tools, they are meant for config scripts, just like the exists(), returnValue(), returnValues(), and listNodes() functions we have now.

RunOpMode

Takes a path, runs an op mode command.

What's next

Next these operations will be wrapped into a higher level client library. At the same time I'll add some message routing, and then we will be able to start VyConf and send some commands to it through a non-interactive client, and have some fun with it. A runnable application will make it easier to test the features we are going to develop next, such as config rendering, building and displaying config diffs, and running actual config scripts at commit time.

VyOS 2.0 development digest #7: Python coding guidelines for config scripts in 1.2.0, and config parser for VyConf

daniil@sentrium.io (Daniil Baturin) — Wed, 04 Jan 2017 19:27:32 GMT

Python coding guidelines for 1.2.0

In some previous post I was talking about the Python wrapper for the config reading library. However, simply switching to a language that is not Perl will not automatically make that code easy to move to 2.0 when the backend is ready, neither it will automatically improve the design and architecture. It will also improve the code in general, and help keeping it readable and maintainable.

You can find the document here: http://wiki.vyos.net/wiki/Python_config_script_policy

In short:

Logic for config validation, generating configs, and changing system settings/restarting services must be completely separated
For any configs that allow nesting (dhcpd.conf, ipsec.conf etc.) template processor must be used (as opposed to string concatenation)
Functions should not randomly output anything to stdout/stderr
Code must be unit-testable

Config parser for VyConf/VyOS 2.0

Today I pushed initial implementation of the new config lexer and parser. It already supports nodes and node comments, but doesn't support node metadata (that will be used to mark inactive and ephemeral nodes).

You can read the code (https://github.com/vyos/vyconf/blob/master/src/curly_lexer.mll and https://github.com/vyos/vyconf/blob/master/src/curly_parser.mly) and play with it by loading the .cma's into REPL. Next step is to add config renderer. Once the protobuf schema is ready we can wrap it all into a daemon and finally have something to really play with, rather than just run the unit tests.

Informally, here's what I changed in the config syntax.

Old config

interfaces {

  /* WAN interface */

  ethernet eth0 {

    address 192.0.2.1/24

    address 192.0.2.2/24

    duplex auto

  }

}

New config

interfaces {

  ethernet {

    /* WAN interface */

    eth0 {

      address [

        192.0.2.1/24;

        192.0.2.2/24;

      ];

      duplex auto;

      // This kind of comment is ignored by the parser

    }

  }

}

As you can see, the changes are:

Leaf nodes are now terminated by semicolons rather than newlines.
There is syntax for comments that are ignored by the parser
Multi nodes have the array of values in square brackets.
Tag nodes do not receive any special formatting.

I suppose the last change may be controversial, because it can lead to somewhat odd-looking constructs like:

interfaces {

  ethernet {

    eth0 {

      vif {

        21 {

          address 192.0.2.1/24

        }

      }

    }

  }

}

If you are really going to miss the old approach to tag nodes (that is "ethernet eth0 {" as opposed to "ethernet { eth0 { ...", let me know and I guess I can come up with something. The main difficulty is that, while this never occurs in configs VyOS config save produces, different tag nodes, e.g. "interfaces ethernet" and "interfaces tunnel" can be intermingled, so for parsing we have to track which ones were already created, and this will make the parser code a lot longer.

I'm pretty convinced that "address 192.0.2.1/24; address 192.0.2.2/24" is simply visual clutter and JunOS-like square bracket syntax will make it cleaner. It also solves the aforementioned problem with interleaved nodes tracking for leaf nodes.

Let me know what you think about the syntax.

VyOS 2.0 development digest #6: new beginner-friendly tasks, design questions, and the details of the config tree

daniil@sentrium.io (Daniil Baturin) — Sat, 31 Dec 2016 09:02:03 GMT

The tasks

Both tasks from the previous post have been taken up and implemented by Phil Summers (thanks, Phil!). New tasks await.

First task was very simple: the Reference_tree module needs functions for checking facts about nodes, analogous to is_multi. For config output, and for high level set/delete/commit operations we need easy ways to know if the node is tag or leaf, or valueless, what component is responsible for it etc. It can be done mostly by analogy with is_multi function and its relatives, so it's friendly to complete beginners. But Phil Summers implemented it before I could make the post (thanks again, Phil!).

Second task is a little bit more involved but still simple enough for anyone who started learning ML not long ago. It's about loading interface definitions from a directory. In VyOS, we may have a bunch of files in /usr/share/vyos/interfaces such as firewall.xml, system.xml, ospf.xml, and so on, and we need to load them into the reference tree that is used for path validation, completion etc.

Design questions

To give you some context, I'll remind you that the vyconf shell will not be bash-based, due to having to fork and modify bash (or any other UNIX shell) to get completion from the first word to begin with, and for variety of other reasons. So, first question: do you think we should use the vyconf shell where you can enter VyOS configuration commands as login shell, or we should go for JunOS-like approach when you login to a UNIX shell and then issue a command to enter the configuration shell? You can cast your vote here: https://phabricator.vyos.net/V2

Second question is more open-ended: we are going to printing the config as the normal VyOS config syntax, and as set commands, but what else should we support? Some considerations: since "show" will be a part of the config API, it can be used by e.g. web GUI to display the config. This means config output of XML or JSON can be a useful thing. But, which one, or perhaps both? And also we need to decide what the XML and/or JSON shouid look like, since we can go for a generic schema that keeps node names in attributes, or we can use custom tags such as <interfaces> (but then every component should provide a schema).

Now, to the "long-awaited" details of the config tree...

The tree

As I already said, VyOS config is essentially a multi-way tree: nodes have children and data associated with them. For instances, node "system" has children named "host-name", "name-server", and so on, and node "host-name" may have value "vyos" associated with it. However, the data is not limited to value alone: nodes may have comments, and if we implement long wished for activate/deactivate, it will also be a piece of data associated with the node, internally.

Config tree nodes have this kind of data attached to them:

type config_node_data = {
    values: string list;
    comment: string option; (* set by the "comment" command *)
    inactive: bool; (* set by "deactivate" command *)
    ephemeral: bool; (* set by scripts that create temporary nodes *)
}

Reference tree nodes have this kind of data:

type ref_node_data = {
    node_type: node_type; (* tag, leaf, or "normal" *)
    constraints: (Value_checker.value_constraint list); (* used for value validation *)
    help: string; (* displayed in tab completion *)
    value_help: (string * string) list; (* value format help in tab completion *)
    constraint_error_message: string; (* displayed if the value doesn't match constraints *)
    multi: bool; (* indicates that node can have more than one value *)
    valueless: bool; (* indicates that node can't have values (such as "disable") *)
    owner: string option; (* which component is called if node is changed in proposed config *)
    keep_order: bool; (* whether config output is allowed to auto-sort nodes or not *)
    hidden: bool; (* whether the node will show up in completion *)
    secret: bool; (* whether the value is sensitive data and should be obscured in output *)
}

Apart from the config tree that represents the running config and proposed configs from sessions, we also need a way to store information about available commands (really, allowed node names in the config tree) to validate paths (as in, "interfaces ethernet eth0" is valid while "interfaces foo bar0" is not), get help strings, get information needed to validate values and so on. The key observation here is that if we take a fully populated config tree (where every possible node is created) and attach value validation data instead of values to leaf nodes, we can validate config paths simply^* by checking if they exist in that tree, and validate values by retrieving validation data in the same way as we retrieve values from the config and doing something with that data. We'll call that a reference tree, because we use it for the reference when we need to check what's allowed.

Config tree:

interfaces:

  ethernet:

    eth0:

      address: data(values=[192.0.2.1/24, 192.0.2.2/24])
Reference tree:

interfaces:

  ethernet: data(type=tag)

    address: data(multi=true,value_constraint=ipv4|ipv6)

Ok, not quite that simple. Tag nodes (nodes whose children can have variable names, such as "ethernet" there) ruin the pretty picture: in the reference tree that path is "interfaces ethernet address", while in a config tree this path would be invalid. But, you get the idea.

At the top of every tree, there is a root node, and every other node is its child. These are the primitive operations on the tree nodes:

List its children
Update the data associated with it
Retrieve the data associated with it
Insert a child

There are some practical considerations that come into play, however. First, a lot of time in VyOS we don't insert nodes directly, we do something like "set interfaces tunnel tun0 parameters ip key 42", where of all nodes involved perhaps only "interfaces" already exists. This needs some workaround to make such inserts convenient: I went with a function that takes default data value and creates the missing nodes on the way, with default data attached to them. This approach works very well for config tree where only leaf nodes have any meaningful data, and for building the reference tree from interface definitions we can use the direct insertions sequentially.

But, how does this translate to high level set/delete operations? There are some tricky points.

Suppose we have this command: "set interfaces ethernet eth0 address 192.0.2.1/24". To add it to the config tree, we need to create a node at path "interfaces ethernet eth0 address" and put "192.0.2.1" into the "values" field of its data. But wait, how do we know which of those is the value? We cannot know without consulting the reference tree, so the cooperation between config and reference tree functions needs to be very close. For this reason, the function for validating paths doesn't simply return true or false, instead it returns the path and value parts, or raises an exception if the command is invalid.

Another tricky thing about set is where exactly you put the new child in the list of children. Most of the time order doesn't matter, but in route-maps, firewall rulesets, and other things that are read top down, changing the order changes the semantics! VyOS 1.x avoids this issue by using numeric names for such nodes, and doing numerical sort in the config scripts, but the downside is that node names have a hidden meaning (sorting in the output is also numeric, but it doesn't guarantee that config script really treats it this way, since it's just a convention), and it also makes reordering rules quite annoying since you have to rename them (try to insert a rule between say rules 5 and 6, you get to rename both, and probably some nodes before and after that too).
For this reason, while we are still to decide what the syntax for it will look like in the CLI, I already implemented the foundation for it in the tree module. The insertion function takes a "position" argument that can make it insert at the beginning, at the end, before child with certain name, or after child with certain name.

To learn more about the trees, you can read the vytree.ml, config_tree.ml, and reference_tree.ml modules. If you have any questions about them, feel free to ask.

What's next?

By the next post, I hope I'll have a draft of the protobuf schema and its implementation, so that's what I'll write about. In 1.2.0, we are working on packaging the Python library, so there may be some news about using Python in 1.x development soon too.

VyOS 2.0 development digest #5: doing 1.2.x and 2.0 development in parallel

daniil@sentrium.io (Daniil Baturin) — Thu, 22 Dec 2016 01:33:41 GMT

There was a rather heated discussion about the 1.2.0 situation on the channel, and valid points were definitely expressed: while 2.0 is being written, 1.2.0 can't benefit from any of that work, and it's sad. We do need a working VyOS in any case, and we can't just stop doing anything about it and only work on 2.0. My original plan was to put 1.2.0 in maintenance mode once it's stabilized, but it would mean no updates at all for anyone, other than bugfixes. To make things worse, some things do need if not rewrite, but at least very deep refactoring bordering on rewrite just to make them work again, due to deep changes in the configs of e.g. StrongSWAN.

There are three main issues with reusing the old code, as I already said: it's written in Perl, it mixes config reading and checking with logic, and it can't be tested outside VyOS. The fourth issue is that the logic for generating, writing, and applying configs is not separated in most scripts either so they don't fit the 2.0 model of more transactional commits. The question is if we can do anything about those issues to enable rewriting bits of 1.2.0 in a way that will allow reusing that code in 2.0 when the config backend and base system are ready, and what exactly should we do.

My conclusion so far is that we probably can, with some dirty hacks and extra care. Read on.

The language

I guess by now everyone agrees that Perl is a bad idea. There are few people who know it these days, and there is no justification for knowing it. The language is a minefield that lacks proper error reporting mechanism or means to convey the semantics.

If you are new to it, look at this examples:

All "error reporting" enabled, let's try to divide a string by an integer.

$ perl -e 'use strict; use warnings; print "foobar" / 42'

Argument "foobar" isn't numeric in division (/) at -e line 1.

0

A warning indeed... Didn't prevent program from producing a value though: garbage in, garbage out. And, my long time favorite: analogous issues bit me in real code a number of times!

$ perl -e 'print reverse "dog"'

dog

Even if you know that it has to do with "list context", good luck finding information about default context of this or that function in the docs. In short, if the language of VyOS 1.x wasn't Perl, a lot of bugs would be outright impossible.

Python looks like a good candidate for config scripts: it's strongly typed, the type and object system is fairly expressive, there are nice unit test libraries and template processors and other things, and it's reasonably fast. What I don't like about it and dynamically typed languages in general is that it needs a damn good test coverage because the set of errors it can detect at compile time is limited and a lot of errors make it to runtime, but there are always compromises.

But, we need bindings. VyConf will use sockets and protobuf messages for its API which makes writing bindings for pretty much any language trivial, but in 1.x.x it's more complicated. The C++/Perl library from VyOS backend is not really easy to follow, and not trivial to produce bindings for. However, we have cli-shell-api, which is already used in config scripts written in shell, and behaves as it should. It also produces fairly machine-friendly output, even though its error reporting is rudimantary (then again, error reporting of the C++ and Perl library isn't all that nice either). So, for a proof of concept, I decided to make a thin wrapper around cli-shell-api: later it can be rewritten as a real C++ binding if this approach shows its limitations. It will need some C++ library logic extraction and cleanup to replicate the behaviour (why the C++ library itself links against Perl interpreter library? Did you know it also links to specific version of the apt-pkg library that was never meant for end users and made no promise of API stability, for its version comparison function that it uses for soring names of nodes like eth0? That's another story though).

Anyway, I need to add the Python library to the vyatta-cfg package which I'll do soon, for the time being you can put the file to your VyOS (it works in 1.1.7 with python2.6) and play with it.

Upd: since then, the Python library is a part of VyOS 1.2.0+ images and is in the default PYTHONPATH. Check out scripts in https://github.com/vyos/vyos-1x

Right now it exposes just a handful of functions: exists(), return_value(), return_values(), and list_nodes(). It also has is_leaf/is_tag/is_multi functions that it uses internally to produce somewhat better error reporting, though they are unnecessary in config scripts, since you already know that about nodes from templates. Those four functions are enough to write a config script for something like squid, dnsmasq, openvpn, or anything else that can reload its config on its own. It's programs that need fancy update logic that really need exists_orig or return_effective_value. Incidentally, a lot of components that need that rewrite to repair or could seriously benefit from serious overhaul are like that: for example. iptables is handled by manipulating individual rules right now even though iptables-restore is atomic, likewise openvpn is now managed by passing it the config in command line options while it's perfectly capable of reloading its config and this would make tunnel restarts a lot less disruptive, and strongswan, the holder of the least maintainable config script, is indeed capable of live reload too.

Which brings us to the next part...

The conventions

To avoid having to do two rewrites of the same code instead of just one, we need to make sure that at least substantial part of the code from VyOS 1.2.x can be reused in 2.0. For this we need to setup a set of conventions. I suggest the following, and let's discuss it.

Language version

Python 3 SHALL be used.

Rationale: well, how much longer can we all keep 2.x alive if 3.0 is just a cleaner and nicer implementation?

Coding standard

No single function shall SHOULD be longer than 100 lines.

Rationale: https://github.com/vyos/vyatta-cfg-vpn/blob/current/scripts/vpn-config.pl#L449-L1134 ;)

Logic separation and testability

This is the most important part. To be able to reuse anything, we need to separate assumptions about the environment from the core logic. To be able to test it in isolation and make sure most of the bugs are caught on developer workstations rather than test routers, we need to avoid dependendies on the global state whenever possible. Also, to fit the transactional commit model of VyOS 2.0 later, we need to keep consistency checking, generating configs, and restarting services separated.

For this, I suggest that we config scripts follow this blueprint:

def get_config():

    foo = vyos.config.return_value("foo bar")

    bar = vyos.config.return_value("baz quux")

    return {"foo": foo, "bar": bar} # Could be an object depending on size and complexity...
def verify(config):

    result do_some_checks(config)

    if checks_succees(result):

        return None

    else:

        raise ScaryException("Some error")
def generate(config):

    write_config_files(config)
def apply(config):

    restart_some_services(config)
if __name__ == '__main__':

    try:

       c = get_config()

       verify(c)

       generate(c)

       apply(c)

    except ScaryException:

        exit_gracefully()

This way the function that process the config can be tested outside of VyOS by creating the same stucture as get_config() would create by hand (or from file) and passing it as an argument. Likewise, in 2.0 we can call its verify(), update(), and apply() functions separately.

Let me know what you think.

VyOS 2.0 development digest #4: simple tasks for beginners, and the reasons to learn (and use) OCaml

daniil@sentrium.io (Daniil Baturin) — Wed, 21 Dec 2016 14:09:03 GMT

Look, I lied again. This post is still not about the config and reference tree internals. People in the chat and elsewhere started showing some interest in learning OCaml and contributing to VyConf, and Hiroyuki Satou even already made a small pull request (it's regarding build docs rather than the code itself, but that's a good start and will help people with setting up the environment), so I decided to make a post to explain some details and address common concerns.

The easy tasks

There are a couple of tasks that can be done completely by analogy, so they are good for getting familiar with the code and making sure your build environment actually works.

The first one is about new properties of config tree nodes, "inactive" and "ephemeral", that will be used for JunOS-like activate/deactivate functionality, and for nodes that are meant to be temporary and won't make it to the saved config respectively.

The other one is about adding "hidden" and "secret" properties to the command definition schema and the reference tree, "hidden" is meant for hiding commands from completion (for feature toggles, or easter eggs ;), and "secret" is meant to hide sensitive data from unpriviliged users or for making public pastes.

Make sure you reference the task number in your commit description, as in "T225: add this and that" so that phabricator can automatically connect the commits with the task.

If you want to take them up and need any assistance, feel free to ask me in phabricator or elsewhere.

The reasons to learn (and use)

Main reasons are type system expressiveness and type safety. There is also convenience of the toolchain.

First fun things, is that OCaml is one of the few languages that can be compiled to native code and can be used interactively. It can also be compiled to bytecode. Interactive read-eval-print loop uses the bytecode implementation. Bytecode is also used to run OCaml on platforms that compiler doesn't target. It's also used by the js_of_ocaml tool that compiles OCaml to JS (and it's powerful enough to compile OCaml itself to JS, here you can try it out in your browser entirely on the client side: https://try.ocamlpro.com/

The default REPL (the "ocaml" executable) is pretty spartan (no history even), so I suggest that you install utop immediately, it makes a great REPL. First install opam, then run "opam install utop".

Now, some teasers about the type system. First, OCaml can umabiguously infer the types of expressions, including functions, through what's known as Hindley-Milner type inference, at compile time. You never have to enter type annotations (other than in module signatures), even though you can. It knows that 42 is an int, "true" is a bool, and "fun x → x mod 2 = 0" is a function of type "int → bool". It does come at cost, in particular, there is no ad-hoc polymorphism (aka function overloading). There are tools that let you view the inferred types of expressions in your editor: https://opam.ocaml.org/blog/turn-your-editor-into-an-ocaml-ide/

I'll give a seemingly trivial but demonstrative example, formatted output. We all know (or should know) that this C program segfaults:

#include <stdio.h>

int main(void) {

    printf("%d bottles of %s on the wall\n", "foo", 99);

}

In Python or Go this is a runtime error. OCaml can prevent this at compile time:

utop # Printf.printf "%d bottles of %s on the wall" "beer" 99;;

Error: This expression has type string but an expression was expected of type int

By partial application we can see the inferred type: from the format string it knows that the arguments should be int and string respectively (the mysterious "unit" type, you can think of it like "void" if you want, for now):

utop # Printf.printf "%d bottles of %s on the wall";;

- : int -> string -> unit

While the implementation of familiar-looking printf with format strings is quite involved, strange-looking but equally type safe printf could be implemented in just a few lines, it's not some compiler magic.

Another serious reason is the way it handles the data. By default, all data is immutable. This means that when you bind say a list to a new name and start modifying it, the compiler will rearrange the pointers or copy just enough data for you, without you having to worry about making sure to copy the right things. The guarantee that nothing can be modified by other parts of the program allows the compiler to do this safely and efficiently. Only if you declare something as mutable, then you can destroy its original state.

Learn more

There are a bunch of resources you can use to learn the language:

Tutorials on ocaml.org are informative, but the structure may be a bit confusing for a newcomer.

This book is great, but not free (DRM-free PDF for $15). This would be my "editor's choice" though.

Real World OCaml is also great, but with one caveat: it uses an alternative standard library written by its authors (Core.Std) throughout, and that library is not compatible with the standard library that comes with the compiler. It's a bit like C++ with Boost imported into the main namespace: you are reading a book about C++ with Boost in the main namespace, rather than a book about C++/STL itself. I'm not saying Core.Std is bad, but I do think that one should learn about the standard library before using non-standard ones, and then decide if they want to use it. VyConf doesn't use that library.

Possible concerns

Difficulty of imperative programming

There is none. If your introduction to functional programming was in Haskell, you may remember that it's not easy to say insert a debug print or logging in the middle of the program. Well, OCaml doesn't have this issue, as it doesn't try to limit side effects in any way: if you want to modify a mutable reference, or add log or debug print, you can.

While the code people normally write is functional when they can and imperative only when they have to, the imperative "escape hatch" is always there. Also, OCaml uses strict rather than non-strict evaluation as opposed to Haskell, so you never have to use monads just to make sure things are executed in order. While a lot of libraries are monads, you rarely need more than one monad at a time (let's admit it, IO is the most used monad in Haskell programs), so you hardly ever end up with a pile of monad transformers.

Performance

Performance is a common concern, especially when it comes to unfamiliar languages, so I made up a microbenchmark that hopefully addesses it. For something more realistic, as you already know if you used "opam switch", OCaml compiler can bootstrap itself in a few minutes, and the binary it uses for it isn't even native code.

We all know that microbenchmarks are not the best performance indicators, but no performance discussion is complete without one. I selected a really dumb task solved in a really dumb way for this one: build a 10000000 long list of integers and sum it up. In real life anyone would use a stream for it, but the point is to demonstrate how it can handle large datastructures, rather than to sum up a bunch of number effectively. While the task itself is contrived, VyConf processes datastructure in pretty similar ways, so it may be a valid approach here.

The languages compared are OCaml, C++, and Python (one person suggested that I also add Go, so I did). In OCaml I used the single linked list type from the standard library, but reimplemented list folding myself because late at night I thought the List.fold_list is not tail recursive and would cause a stack overflow (actually, it is and I've done extra work). In C++ I used the std::list template and iterator. In Python, I used the standard list class. In Go, I used the "container/list" package from the standard library.

You can view the source code and the testing session protocol here. Here are aggregated resulsts:

Wall time (average of three runs), measured with the usual UNIX time utility:

OCaml: 0.76s

C++ (gcc default): 1.45s
C++ (gcc -O2): 0.62s
Python: 2.18s

Go: 3.3s

Memory consumption measured with valgrind/massif (except for Go, which doesn't work with valgrind and required code modification to use its native pprof tool):

OCaml: 146MiB

C++: 230MiB

Python: 347MiB

Go: 190MiB

It should be noted that sum types are not objects, and a more "fair" comparison should have been done with a linked list implemented through OCaml objects, but they are expressive enough to let you have a linked list without objects, so...

Libraries

There aren't as many of them as for more popular languages, though not as bad as some make it to look. You can browse libraries available from the official OPAM repository here, the list is always growing.

You can also interface with C libraries without writing C code, through the ctypes library.

Limitations

There are some very genuine issues with OCaml (just like with any other language).

First, the standard library is quite... spartan. This is what spawned a few alternative "standard libraries" that haven't become really standard because they would break the old code. It's not like Python where you can find bindings for SQLite and a JSON parser right in the standard library. Installing libraries from OPAM is trivial so this isn't much of an issue, but you do need to make some decisions what to use.

Second, the garbage collector in mainline OCaml is single-threaded, and the runtime library has a global lock (Python users are familiar with GIL, this is the same thing). This means for effective parallel computations one needs to use subprocesses rather than threads (concurrency is a different thing). Runtimes without global locks do exist (for Python, OCaml, and Haskell alike), the problem is how to make them as fast as globcal lock runtimes for single-threaded applications, so they are not the default in any of those.

As I said, it also lacks ad-hoc polymorphism. It was sacrificed to make type inference decidable, but it can be annoying sometimes, for example, you cannot mix integers and floats in the same expression without explicit conversion. The alternative to is is "functors": modules that can be parameterized by other modules, which are powerful but not very concise or easy to write.

There is also no (easy) reflection, since it's normally compiled to native code and doesn't keep any type information in executables (apart from a special case called "polymorphic variants" that is used for implementing a form dynamic typing within a statically typed program). You can't address record fields or object members by names converted from strings, or iterate through them. Any metaprogramming can only happen at compile time (there are syntax extensions).

VyOS 2.0 development digest #3: questions for you, vyconf daemon config format, appliance directory structure, and external validators lookup

daniil@sentrium.io (Daniil Baturin) — Sat, 17 Dec 2016 14:08:57 GMT

Ok, I changed my mind: before we jump into the abyss of datastructures and look how the config and reference trees work, I'll describe the changes I made in the last few days.

Also, I'd like to make it clear that if you don't respond to design questions I state in these posts, I, or whoever takes up the task, will just do what they think is right. ;)

I guess I'll start with the questions this time. First, in the comments to the first post, the person who goes by kglkgvkvsd544 suggested two features: commit-confirm by default, and an alternative solution to the partial commit where instead of loading the failsafe config, the system loads a previous revision instead, in the same way as commit-confirm does. I don't think any of those should be the only available option, but having them as configurable options may be a good idea. Let me know what you think about it.

Another very important question: we need to decide on the wire protocol that vyconf daemon will use for communication with its clients (the CLI tool, the interactive shell, and the HTTP bridge). I created a task for it, https://phabricator.vyos.net/T216, let me know what you think about it. I'm inclined towards protobuf myself.

Now to the recent changes.

vyconfd config

As I already said, VyConf is supposed to run as a daemon and keep the running config tree, the reference tree, and the user sessions in memory. Obviously, it needs to know where to get that data. It also needs to know where to write logs, where the PID file and socket file should be, and other things daemons are supposed to know. Here's what the vyconf config for VyOS may look like:

[appliance]

name = "VyOS"
data_dir = "/usr/share/vyos/"

program_dir = "/usr/libexec/vyos"

config_dir = "/etc/vyos"
# paths relative to config_dir

primary_config = "config.boot"

fallback_config = "config.failsafe"
[vyconf]

socket = "/var/run/vyconfd.sock"

pid_file = "/var/run/vyconfd.pid"

log_file = "/var/log/vyconfd.log"

That INI-like language is called TOML, it's pretty well specified and capable of some fancy stuff like arrays and hashes, apart from simple (string, string) pairs. The best part is that there are libraries for parsing it for many languages, and the one for OCaml is particularly nice and idiomatic (it uses lenses and option types to access a nested and possibly underdefined datastructure in a handy and typesafe way), like:

let pid_file = TomlLenses.(get conf (key "vyconf" |-- table |-- key "pid_file" |-- string)) in

match pid_file with

| Some f -> f

| None -> "/var/run/vyconfd.pid"

The config format and this example reflects some decisions. First, the directory structure if more FHS-friendly. What do we need /opt/vyos for, if FHS already has directories meant for exactly what we need: architecture-independent data (/usr/share), programs called by other programs and not directly by users (/usr/libexec), and config files (/etc)?
Second, all important parameters are configurable. The primary motivation for this is making VyConf usable for every appliance developer (most appliances will not be called VyOS for certain), but for ourselves and for every contributor to VyOS it's a reminder to avoid hardcoded paths anywhere: if changing it is just one line edit away, a hardcoded path is an especially bad idea.

Here's the complete directory structure I envision:

$data_dir/

  interfaces/ # interface definitions

  components/ # Component definitions

$program_dir/

  components/  # Scripts/programs that verify the appliance config, generate actual configs, and apply them

  migration/       # Migration scripts (convert appliance config if syntax changes)

  validators/       # Value validators

$config_dir/

  scripts/              # User scripts

  post-config.d/   # Post-config hooks, like vyatta-postconfig-bootup.script

  pre-commit.d/   # Pre-commit hooks

  post-commit.d/ # Post-commit hooks

  archive/             # Commit archive

This is not an exhaustive list of directories an appliance can have of course, it's just directories that have any significance for VyConf. I'm also wondering if we should introduce post-save hooks for those who want to do something beyond the built-in commit archive functionality.

External validators

As you remember, my idea is to get rid of the inflexible system of built-in "types" and make regex the only built-in constraint type, and use external validators for everything else.

External validators will be stored in the $program_dir/validators. Since many packages use the same types of values (IP addresses is a common example), and in VyOS 1.x we already have quite a lot of templates that reference the same validation scripts, making it a separate entity will simplify reusing them, since it's easy to see what validators exist, and you can also be sure that they behave like you expect (or if they don't, it's a bug).
Validator executable take two arguments, first argument is constraint string, and the second is the value to be validated (e.g. range "1-99" "42").The expected behaviour is to return 0 if the value is valid and a non-zero exit code otherwise. Validators should not produce any output, instead the user will get the message defined in the constraintError tag of the interface definition (this approach is more flexible since different things may want to use different messages, e.g. to clarify why exactly the value is not valid).

That's all for today. Feel free to comment and ask questions. The next post really will be about the config tree and the way set commands work, stay tuned.

VyOS 2.0 development digest #2

daniil@sentrium.io (Daniil Baturin) — Mon, 12 Dec 2016 20:45:10 GMT

In the previous post we talked about the reasons for rewrite, design and implementation issues, and basic ideas. Now it's time to get to details. Today we'll mostly talk about command definitions, or rather interface definitions, since set commands is just one way to access the configuration interface.

Let's review the VyConf architecture (I included a few things in the diagram that we haven't discussed yet, ignore them for now):

At startup, VyConf will load the main config (or the fallback config, if that fails). But to know whether the config is valid, and to know what programs to call to actually configure the target applications, it needs additional data. We'll call that data "interface definitions" since it defines the configuration interface. Specifically, it defines:

What config nodes (paths) are allowed (e.g. "interfaces ethernet", or "protocols ospf")
What values are valid for that nodes (e.g. any IPv4 or IPv6 address for "system name-server")
What script/program should be called when this or that part of the config is changed

The old way

Before we get to the new way, let's review the old way, the way it's one in the current VyOS implementation. In the current VyOS, those definitions are called "templates", no one remembers why.

This is a typical template file:

vyos@vyos# cat /opt/vyatta/share/vyatta-cfg/templates/interfaces/ethernet/node.tag/speed/node.def

type: txt

help: Link speed

default: "auto"

syntax:expression: $VAR(@) in "auto", "10", "100", "1000", "2500", "10000"; "Speed must be auto, 10, 100, 1000, 2500, or 10000"

allowed: echo auto 10 100 1000 2500 10000
commit:expression: exec "\

	/opt/vyatta/sbin/vyatta-interfaces.pl --dev=$VAR(../@) \

	--check-speed $VAR(@) $VAR(../duplex/@)"
update: if [ ! -f /tmp/speed-duplex.$VAR(../@) ]; then

	   /opt/vyatta/sbin/vyatta-interfaces.pl --dev=$VAR(../@) \

	   	--speed-duplex $VAR(@) $VAR(../duplex/@)

	   touch /tmp/speed-duplex.$VAR(../@)

	fi
val_help: auto; Auto negotiation (default)

val_help: 10; 10 Mbit/sec

val_help: 100; 100 Mbit/sec

val_help: 1000; 1 Gbit/sec

val_help: 2500; 2.5 Gbit/sec

val_help: 10000; 10 Gbit/sec

We can spot a few issues with it already. First, the set of definitions is a huge directory tree where each directory represents a config node, e.g. "interfaces ethernet address" will be "interfaces/ethernet/node.tag/address/node.def". This makes them hard to navigate, you need to read a lot of small files to get the whole picture, and you need to do a lot of file/directory hopping to edit them. Now, try mass editing, or checking for mistakes before release...

Next, they use a custom syntax, which needs custom lexer and parser, and custom documentation (in practice, the source is your best bet, though I did write something of a syntax reference). No effortless parsing, no effortless analysis or transformation either.

Next, the value checking has its peculiarities. There is concept of type (it can be txt, u32, ipv4, ipv4net, ipv6, ipv6net, and macaddr), and there is "syntax:expression:" thing which partially duplicate each other. The types are hardcoded in the config backend and cannot be added without modifying it, even though they only define validation procedure and are not used in any other way. The "syntax:expression:" can be either "foo in bar baz quux", or "pattern $regex", or "exec $externalScript".

But, the "original sin" of those files is that they allow embedded shell scripts, as you can see. Mixing data with logic is rarely a good idea, and in this case it's especially annoying because a) you cannot test such code other than on a live system b) you have to read every single file in a package to get the complete picture, since any of them may have embedded shell.

Now to the new way.

The new way

Now when we discussed some issues with the old way, we are ready to state the requirements for the new way. It should:

Be purely declarative
Not be a custom format (easier to use and easier to learn for the contributors)
Be easily observable

The format

My idea is to use XML for those. I expected that after my previous post, people will come and object to OCaml. Either I was so persuasive, or no one got to that part of the post. Well, I bet XML will get objections, and I'll respond to it in advance. The reason XML has so much of a bad reputation is that it's often used in wrong places, or used improperly. The great thing about XML that justifies its bulkiness is that it enables the rest of the technology stack to work, namely XML Schema/RelaxNG, XSLT, XQuery, XPath... Unlike any other format, it has ready to use tools for formal description and verification, transformation, and querying.

XML Schema (in generic sense, either XSD or RelaxNG) allows me to formally define the valid grammar and share it with everyone so people don't need to guess or read the source to find out what exactly the grammar is. Examples are good but they invariably leave at leat some cases to guessing, and aren't machine verifiable. Schemas are, we can easily add schema check as a build step. I do something like:

$ rnv -q ./data/schemata/interface_definition.rnc data/examples/interface_definition_sample.xml && echo valid

valid

And I know that my file conforms to the schema. With XSLT, we can produce command reference right from those definitions without writing additional code, too. Ok, writing XSLT transforms can be tricky, but the rewards are great. There is no such tools for JSON or YAML, JSON Schema RFC draft never came to frutition. Just to make it clear, I don't suggest using XML for serialization format, though for the public API, XML RPC may be a good idea. Definitely not for internal communication though, no.

RelaxNG schema for the interface definitions was one of the first things I've made for the new backend, in its days as a prototype in Python, which I scraped for reasons already discussed in the previous post. I would feel bad for scraping it, but everything anyone else ever contributed to it was PEP8 fixes, so I don't feel bad for it really. We'll discuss what's in the schema now.

You can find the schema here: interface_definition.rnc

You can also find a simple example here: interface_definition_sample.xml

The taxonomy of config nodes

Well, before we get to describing the schema, we need to describe what the config is.

VyOS config, the multiway tree, consists of nodes with attached data. For the datastructure itself, there is no difference between nodes of course, but for the CLI and for scripts that handle the config, there is.

First, there are nodes that can have children (non-leaf nodes), and nodes that cannot (leaf nodes). For example, "protocols" node has children named "static", "bgp", "ospf" etc., which in terms has their own children. A typical leaf node is the "disable" node found in interfaces and many other places, it cannot have children.

Non-leaf nodes can have fixed or variable names. For example, the "protocols" node is always called "protocols". But we have nodes such as "interfaces ethernet" or "firewall name" that have children without predefined names, such as "interfaces ethernet eth0", or "firewall name Test". We call such nodes tag nodes. This is what they are called in current VyOS, I don't like this word much (what exactly is the tag here? "ethernet" or "eth0"?), but I don't see a better term. If you have any ideas, please share.

Leaf nodes are not all the same either. They differ in the number of values they can have. Node values (as in "system name-server 192.0.2.1") are not children, they are separate entities that are treated differently in value validation and config output and pretty much everywhere. We have valueless nodes that can't have any values at all, such as "disable", or "reboot-on-panic". We have nodes that can have only one value, such as "default-action" in firewall. We also have multivalue nodes, which we will call just multi nodes that can have many values, such as "address" under interfaces.

What do node types affect?

There are quite a few things that are affected by node types.

Tag nodes are rendered differently in the config output, "interfaces { ethernet eth0 { ..." rather than "interfaces {ethernet { eth0 { ...". While it's a trivial issue, I think it's easier to read and saves up some valuable vertical space.
Also, tag nodes, obviously, require child name validation (such as eth[0-9]+ for ethernet.

The number of allowed values in a leaf node affects the behaviour of the set operation. For normal nodes, set replaces the value, such as in "set system config-management commit-revisions 2000". For multi nodes, set appends a new value, as in "set system name-server 203.0.113.4".

New concept: order-preserving nodes

Things like firewall rulesets or route-maps are processed top down until the first match, and changing the order of rules can change the behaviour. This means there must be a way to preserve the order in the config output and in scripts.

Right now, the backend doesn't really care about the order, it's all in the output (which, due to bash limitations, sometimes sorts nodes in confusing order in completion), and in scripts. All nodes that must be ordered thus are forced to have numeric names to ensure unambiguous sorting.

There are a few annoying things about numbered rules. It's easy to configure yourself into a corner where you have to rename a lot of rules to insert a new one where it should be. If one used rules 1,2,3,4,5, they'll have to rename every single rule to create gaps. Myself, I normally leave gaps like 10,20,30, but even that runs out sometimes. Gaps like 100,200,300 would be future proof, but absurdly wide, combined with the current (completely arbitary!) limitation of 9999 rules in a single firewall. That limitation is not just in the validation code though, the assumption based on it is that the default action rule is internally numbered 10000, so changing it isn't exactly a one line fix.

JunOS doesn't use numbered rules, nodes such as firewall policies preserve child order, and there are commands to move nodes around. What I'm not very fond of though is that for a new node, you need to set it first, and then use "insert" command. I guess we could use a set option instead, e.g. a pseudo-pipe "set firewall name Foo rule block-ssh | before allow-all". Let's not focus on the command syntax yet, though.

What scripts to call when something changed: node owners

This one I have a mixed feeling about. While we already ruled out embedded logic in interface definitions, there's a question how granular node ownership can be.

Right now in the schema, any non-leaf nodes can have an "owner", a reference to separately defined "component" (such as "firewall", or "ospf"). I toyed with the idea to allow the owner attribute only for the top level node of each definition, but this may cause unnecessary fragmentation of definitions, e.g. inside "vpn", "vpn ipsec site-to-site" and "vpn ipsec profile" may legitimately have different owner, the site-to-site script and the DMVPN script respectively.

Components will be defined in separate XML files, see the schema. Component definition includes its name, scripts that must be run for checking if config is valid, generating application configs, and applying the changes to the system (I call them "verify", "update", and "apply" stages), and a list of dependencies.

The last part needs some elaboration. As you can easily see, there are things that must run in certain order, for example, IPv6 router advertisment should only be configured after the interfaces. Right now, VyOS uses hardcoded priorities for it, you can see them all by running "/opt/vyatta/sbin/priority.pl". They are taken from the priority: tag in templates.

There are two main issues with it. First, it's not very easy to track, you can only see them all on a live system, and it's not quite easy to see what really depends on what. Second, it absolutely rules out parallelized commits, even though most of components are independent and could easily be done in parallel.

The idea is to specify dependencies for each component instead, and use topological sort to determine the order at commit time, this way we can both easily see what the real dependencies are, and parallelize the commit.

Back in high school, we had a cool CS teacher. She was a part time teacher really, while her main job was image recognition research. She taught us a lot about algorithms and datastructures, and always emphasized the importance of learning the foundations of CS, rather than new hot things. I've had quite a geek crush on her. It's all irrelevant to the post of course, I'm just checking if you are still following. By the way, there's a a 15% discount to VyOS shirts.

Unsolved issue: pervasive components

By "pervasive" I mean things that exist in multiple places in the config. For example, we have "vrrp" subtree in ethernet, VLANs, bonding etc. We also have "ip ospf" subtree in literally every interface. And then, "firewall" is in every interface too. I still wonder how to handle this issue in the new backend.

Right now, we don't have real support for it. They are generated by scripts, such as this one. There are a few issues with this approach, first is that to add a new interface type, you need to modify a few unrelated packages, which makes life harder for contributors. The second issue is that it disperses the component logic over multiple places ("ip ospf bandwidth" under interfaces runs vtysh -c directly for instance), this is bad for system-wide consistency checks, and can make transactional rollback a lot harder to do.

I don't see an easy solution to this problem, however. If we use some kind of file inclusion, the primary scripts will not know where to look, since it may be hard to extract every such path, without knowing about every place that can include it (or we are back to making such scripts aware of every interface type etc.). A possible solution is to "notify" the right component when such a path changes, thus introducing a concept different from "ownership".

Lastly, the most radical solution that does have some appeal: not support it at all. It can as well be "protocols ospf interface eth0 bandwidth 10000", i.e. every subtree can be self-contained. It may also be easier to read. Though it's still radical, and I'm not sure if we have things that really would be harmed by this approach. Let me know what you think.

Value (and tag node name) validation

My idea is to abolish the concept of types, and instead use a single concept of a constraint instead. The only built-in constraint type I see useful is regex, due to its widespread use. The rest should be implemented with external validators. We can provide ready to use validation helpers for the common values, such as IPv4 and IPv6 hosts and networks, integer values etc.

The open question is if we should introduce "validator definition" files, or not. Right now my code assumes that they are loaded into a hashtable from somewhere, but I haven't implemented a loading mechanism yet.

Now I think that we should rather assume that validators are in say /usr/libexec/vyconf/validators and treat validator references as executable names. We cannot guarantee total absense of runtime errors anyway, if a validator is missing or crashes, we can simply assume the value is invalid to prevent invalid values from being entered (though for emergency cases when someone found a bug on a live system and can't commit, we should also introduce an option to ignore such errors, at user's own risk).

Conclusion

That's all for now. The open questions I want your feedback for:

How to handle pervasive definitions ("ip ospf" or "firewall" under interfaces) in the new system
How to load validators into the system, or if we should not introduce any indirection there and just use the names of helper executables and store them in a predefined location

In the next post we'll talk about the multiway tree and the implementation of set and delete commands, and open questions in that area.

VyOS 2.0 development digest #1

daniil@sentrium.io (Daniil Baturin) — Thu, 08 Dec 2016 01:49:38 GMT

I keep talking about the future VyOS 2.0 and how we all should be doing it, but I guess my biggest mistake is not being public enough, and not being structured enough.

In the early days of VyOS, I used to post development updates, which no one would read or comment upon, so I gave up on it. Now that I think of it, I shouldn't have expected much as the size of the community was very small at the time, and there were hardly many people to read it in the first place, even though it was a critical time for the project, and input from the readers would have been very valuable.

Well, this is a critical time for the project too, and we need your input and your contributions more than ever, so I need to get to fixing my mistakes and try to make it easy for everyone to see what's going on and what we need help with.

Getting a steady stream of contributions is a very important goal. While the commercial support thing we are doing may let the maintainers focus on VyOS and ensure that things like security fixes and release builds get guaranteed attention in time, without occasional contributors who add things they personally need (while maintainers may not, I think myself I'm using maybe 30% of all VyOS features any often) the project will never realize its full potential, and may go stale.

But to make the project easy to manage and easy to contribute to, we need to solve multiple hard problems. It can be hard to get oneself to do things that promise no immediate returns, but if you looks at it the other way, we have a chance to build a system of our dreams together. As of 1.1.x and 1.2.x (the jessie branch), we'll figure it out how to maintain it until we solve those problems, but that's for another post. Right now we are talking about VyOS 2.0, which gets to be a cleanroom rewrite.

Why VyOS isn't as good as it could be, and can't be improved

I considered using "Why VyOS sucks" to catch reader's attention. It's a harsh word, and it may not be all that true, given that VyOS in its current state is way ahead of many other systems that don't even have system-wide config consistency checks, or revisions, or safe upgrades, but there are multiple problems that are so fundamental that they are impossible to fix without rewriting at least a very large part of the code.

I'll state the design problems that cannot be fixed in the current system. They affect both end users and contributors, sometimes indirectly, but very seriously.

Design problem #1: partial commits

You've seen it. You commit, there's an error somewhere, and one part of the config is applied, while the other isn't. Most of the time it's just a nuisance, you fix the issue and commit again, but if you, say, change interface address and firewall rule that is supposed to allow SSH to it, you can get locked out of your system.

The worst case, however, is when commit fails at boot. While it's good to have SSH at least, debugging it can be very frustrating, when something doesn't work, and you have no idea why, until you inspect the running config and see that something is simply missing (if you run into it in VyOS 1.x, do "load /config/config.boot" and commit, this will either work or show you why it failed). It's made worse by lack of notifications about config load failure for remote users, you can only see that error on the console.

The feature that can't be implemented due to it is what goes by "commit check" in JunOS. You can't test if your configuration will apply cleanly without actually commiting it.

It's because in the scripts, the logic for consistency checking and generating real configs (and sometimes applying them too) is mixed together. Regardless of the backend issues, every script needs to be taken apart and rewritten to separate that logic. We'll talk more about it later.

Design problem #2: read and write operations disparity

Config reads and writes are implemented in completely different ways. There is no easy programmatic API for modifying the config, and it's very hard to implement because binaries that do it rely on specific environment setup. Not impossible, but very hard to do right, and to maintain afterwards.

This blocks many things: network API and thus an easy to implement GUI, modifying the config script scripts in sane ways (we do have the script-template which does the trick, kinda, but it could be a lot better).

Design problem #3: internal representation

Now we are getting to really bad stuff. The running config is represented as a directory tree in tmpfs. If you find it hard to believe, browse /opt/vyatta/config/active, e.g. /opt/vyatta/config/active/system/time-zone/node.val

Config levels are directories, and node values are in node.val files. For every config session, a copy of the active directory is made, and mounted together with the original directory in union mount through UnionFS.

There are lots of reasons why it's bad:

It relies on behaviour of UnionFS, OverlayFS or another filesystem won't do. We are at mercy of unionfs-fuse developers now, and if they stop maintaining it (and I can see why they may, OverlayFS has many advantages over it), things will get interesting for us
It requires watching file ownership and permissions. Scripts that modify the config need to run as vyattacfg group, and if you forget to sg, you end up with a system where no one but you (or root) can make any new commits, until you fix it by hand or reboot
It keeps us from implementing role-based access control, since config permissions are tied to UNIX permissions, and we'd have to map it to POSIX ACLs or SELinux and re-create those access rules at boot since the running config dir is populated by loading the config
For large configs, it creates a fair amount of system calls and context switches, which may make system run slower than it could

Design problem #3: rollback mechanism

Due to certain details (mostly handling of default values), and the way config scripts work too, rollback cannot be done without reboot. Same issue once made Vyatta developers revert activate/deactivate feature.

It makes confirmed commit a lot less useful than it should be, especially in telecom where routers cannot be rebooted at random even in maintenance windows.

Implementation problem #1: untestable logic

We already discussed it a bit. The logic for reading the config, validating it, and generating application configs is mixed in most of the scripts. It may not look like a big deal, but for the maintainers and contributors it is. It's also amplified by the fact that there is not way to create and manipulate configs separately, the only way you can test anything is to build a complete image, boot it, and painstakingly test everything by hand, or have expect-like tool emulate testing it by hand.

You never know if your changes may possibly work until you get them to a live system. This allows syntax errors in command definitions and compilation errors in scripts to make it into builds, and it make it into a release more than one time when it wasn't immediately apparent and only appread with certain combination of options.

This can be improved a lot by testing components in isolation, but this requires that the code is written in appropriate way. If you write a calculator and start with add(), sub(), mul() etc. functions, and use them in a GUI form, you can test the logic on its own automatically, e.g. does add(2,3) equal 5, and does mul(9, 0) equal 0, does sqrt(-3) raise an exception and so on. But if you embed that logic in button event handlers, you are out of luck. That's how VyOS is for the most part, even if you mock the config subsystem so that config read functions return the test data, you need to redo the script so that every function does exactly one thing testable in isolation.

This is one of the reasons 1.2.0 is taking so long, without tests, or even ability to add them, we don't even know what's not working until we stumble upon it in manual testing.

Implementation problem #2: command definitions

This is a design problem too, but it's not so fundamental. Now we use custom syntax for command definitions (aka "templates"), which have tags such as help: or type: and embedded shell scripts. There are multiple problem with it. For example, it's not so easy to automatically generate at least a command reference from them, and you need a complete live system for that, since part of the templates is autogenerated. The other issue is that right now some components feature very extensive use of embedded shell, and some things are implemented in embedded shell scripts inside templates entirely, which makes testing even harder than it already is.

We could talk about upgrade mechanism too, but I guess I'll leave it for another post. Right now I'd like to talk about proposed solutions, and what's being done already, and what kind of work you can join.

The plan

The plan to get out of this situation is the following:

Design and implement a new configuration backend
Decide on the new base distro and implement a new image build and upgrade mechanism on top of it
Rewrite integrations scripts

Briefly on the new distro, while Debian is a great distro and no one doubts it, there are new developments in package managers that can, potentially at least, give us safe and reversible upgrade for free. NiOS and Nix package manager looks like an interesting candidate. Right now we will focus on the backend, however.

The working title for the new backend is "vyconf", I'll use "vyconf" and "the backend" interchangeably from now on.

What the backend does?

If you are new to VyOS development, you need to know that there are two main parts: the config backend that loads and saves the config, handles set/delete/commit operations, checks if node values are valid, and runs the pre/post-commit hooks, and scripts that use the config read API of the backend to produce real configs for openvpn, iptables, and everything else that VyOS uses.

Therefore to get a working backend we need to design and implement the following pieces:

Config format grammar and a lexer/parser for it
The datastructure that represents the running config and proposed configs from sessions
Command definition syntax, representation of the command hierarchy, and value checking mechanism
set/delete/rename/copy operations logic
commit and rollback logic

Of course we also need frontends that could communicate to it. I'm planning the following frontend applications:

Non-interactive CLI client (like cli-shell-api, except with config modification functionality)
Interactive shell (replacement of vbash in current VyOS)
HTTP bridge (for both remote API and GUI)

They all need a common library for actual communication. Since non-interactive client needs the least amount of wrapping, it's sensible to start with it.

Is backend just for VyOS?

It's a big of a digression, but my observation is that hardly any of those functions are unique to VyOS, they could be used in any software appliance. Imagine if instead of writing everything by hand, people could take an existing, stable and robust config backend, and make their own software appliance with it. Rather than reimplement config handling logic, they could focus on their config scripts and custom frontend.

For us, it also means that we avoid unwarranted assumptions about the environment (in VyOS 1.2.0 jessie migration, we found enormous amounts of such assumptions deeply entrenched in the system!), and we also get more testers and contributors from outside the VyOS community.

Myself, I'm not going to try to account for hypothetical scenarios that may only occur outside VyOS, but I think we should keep the backend as OS/distro-independent and standalone as feasible and only make config scripts specific to VyOS.

What language I'm writing in?

This gets objections all the time, so I'm answering these questions here. I'm writing it in ML, specifically, the OCaml language of the ML family (the other notable members are StandardML and F#, though F# doesn't implement a number of important ML features such as its module system). This means you need to learn it too if you want to join the work on backend, but it's fairly easy to learn as it has fairly compact syntax and well-defined behaviour. The main stumbling block is the semantics which is very different from imperative languages.

While it's little known outside of the academia and investment banks, it has important properties that make it very well suited for this task. First, it compiles to native code and it's very fast (it can compile itself in less than 10 minutes, for instance). It offers a very expressive and safe type system that can find infinite loops in some cases (http://perl.plover.com/yak/typing/notes.html , the example is in slide 27) or allows you to make a typed printf where something like printf("%d", "foo") is a type error in a few lines (http://www.brics.dk/RS/98/12/).

ML family and Haskell are closely related, and they share a common trait: all data is immutable unless specified otherwise. This has two important effects. First, nothing can be modified accidentally. Unlike Haskell, OCaml does support mutable variables, but they have to be declared as mutable. Second, the guarantees that nothing is accidentally modified allows the language to implement structural sharing: no values are copied unless they really have to be copied.

My early prototypes of the underlying datastructures were in Python. While I like Python and use it for many tasks (and I think we can use it for config scripts too), pervasive mutability combined with destructiveness of functions from the standard library created need for copy or even deepcopy at virtually every step.

There are at least two examples of projects that migrated to OCaml at some stage: Unison file synchronizer and 0install package manager. The maintainer of 0install documented his search for new language better than I can possibly do, read his blog series on it if you are interested: http://roscidus.com/blog/blog/2013/06/09/choosing-a-python-replacement-for-0install/

The academic applications are more numerous and, for certain type of geeks, way more exciting (Coq proof assistant, CompCert the formally verified C compiler, FFTW the Fourier transform generator and many more), though yesterday academic techniques are, luckily, making it into everyday software development, and there are things like http://fbinfer.com/ and https://github.com/facebook/pfff by Facebook.

If you are new to it, you can get one of the books or read tutorials on ocaml.org.

Possible solutions to design problems

Partial commits

The obvious solution is to make commits all or nothing, but it has an unfortunate implication: if config load (which is a commit too) fails at boot, we are left with an unusable system. One way around it is to add the concept of a fallback config. People can make a minimal config that allows them access to the router at least and save it to the fallback config file, so that if bad things happen, they still can login and debug.

To support commit dry run and enforce good separation of concerns, at the backend level we can introduce three types of scripts: check, generate, and apply. When a user commits, first check scripts for all components are run, and if any of them fails, the commit fails. If checks pass, then generate scripts are run to produce real configs from the VyOS config, and then apply scripts are run to reload/restart daemons and do other things necessary to apply the changes to the underlying system.

Read/write disparity and testability

The new backend uses in-memory datastructures for everything. The config can be represented as a multi-way tree (a rose tree, if you prefer more flowery language, in literal sense). The main observation is that the config and the hierarchy of command definitions can be represented as the same data structure, with the only difference in data attached to the children. Thanks to support for parametric polymorphism in the ML type system, we don't even need to do anything special for this (in say C++ we'd make it a template).

Since it doesn't need any directories or files to work, the tree can be populated by hand, or initialized from a file and then manipulated in any way we want. This allows us to test its correctness in isolation, and I already have quite some unit tests for it.

I call the command hierarchy tree a "reference tree", since set/delete operations use it as a reference, to check which paths are allowed and what kind of values are valid for them.

Interprocess communication

A tricky point: we need the local processes to be able to communicate to the backend daemon without explicit authentication (having to enter password in the interactive shell after authenticating in SSH or local console would be very weird), but we also need to prevent operator level users from entering the configuration mode, and if we ever want RBAC, we need to know who the user is. Save for Kerberos or another SSO mechanism deployed locally, it leaves us with the only option, UNIX domain sockets, which, on Linux and FreeBSD at least, support options for UID and GID.

Conclusion

I guess that's all for now. If you want the code of the backend being written, look here: https://github.com/vyos/vyconf

To build it, you need a working OCaml setup, which is not hard to get. First, install OPAM (the OCaml package manager): http://opam.ocaml.org/doc/Install.html Then do "opam switch 4.03.0" and install the build tools and dependencies with "opam install oasis ppx_deriving_yojson lwt ounit xml-light pcre". When you have all that installed, do "oasis setup -setup-update dynamic" which will give you the setup.ml file and ./configure and Makefile wrappers. Then do he configure ("ocaml setup.ml -configure --enable tests"), and run "make", or "make test".

If you have any issues with it, feel free to ask me.

In the next post in this series we will talk about command definitions and validator definitions that are used for populating the reference tree, and the details of the multiway tree implementation and mapping of set/delete etc. to its primitive operations.