In short: a prototype of an HTTP API is now included in the nightly builds and available for testing. Right now it requires some manual configuration to get running, but a new “service https” CLI will also be available soon.
We need your testing and your feedback regarding the API design, so let us know about your experience with it!
Get the latest nightly build from https://downloads.vyos.io/?dir=rolling/current/amd64
Create a config file for the API server (/etc/vyos/http-api.conf), like this:
{
"listen_address": "192.168.1.1",
"port": 8080,
"debug": true,
"api_keys": [
{"id": "testapp", "key": "qwerty"}
]
}
Then start the service: "sudo systemctl start vyos-http-api".
The /configure endpoint takes a request serialized in JSON. The only HTTP method it uses is POST. Request data is passed in the data= field and the API key is passed in the key=field. Key identifiers from the config are purely informational and the application doesn't need to know them, they only appear in the server logs to avoid exposing keys in log files, you only need the key itself.
You can pass set, delete, or comment commands to it. The API will push the commands to the session and commit. Right now the API server is completely stateless and uses one shared session.
This is how you can create a dummy interface and assign an address to it for example:
curl -X POST -F data='{"op": "set", "path": ["interfaces", "dummy", "dum1", "address"], "value": "203.0.113.76/32"}' -F key=qwerty http://192.168.1.1:8080/configure
To compensate for the statelessness, you can send multiple commands at once:
[{"op": "set;", "path": ["system", "host-name"], "value": "vyos-router"},
{"op": "comment;", "path": ["system", "host-name"], "value": "Modified by the API"},
{"op": "delete;", "path": ["system", "options", "reboot-on-panic"]}]
Since internally there is no distinction between a path and a value, you can omit the value field and include the value in the path like it's done in the shell commands:
curl -X POST -F data='{"op": "set", "path": ["interfaces", "dummy", "dum10", "address", "203.0.113.99/32"]}' -Fkey=qwerty http://192.168.1.1:8080/configure
Separate value field make the semantics more clear though, and also makes it easier to create a command template once and update it with different values as needed.
While JSON-encoded requests may be unwieldy for humans to write, they are easy for machines to generate, and when the API stabilized, we will provide bindings and command line utilities for working with it.
There's also no HTTPS support yet: it will be provided by the reverse proxy once the new "service https" CLI and nginx configuration scripts are there.
Now to the background if you are interested...
Web API has been one of the most requested features. None of the open source Vy* systems ever had it in fact. The original Vyatta removed its GUI from the open source version and the new implementation that included a remote API was exclusive to the proprietary Subscription Edition.
I’ve always had objections to its design too. All the data passed in URLs, for example, “PUT /rest/conf/<token>/set/system/host-name/my-router”. For a web application developer, using that would be quite a headache, since there’s no easy way to just send it serialized request data.
It also had a peculiar selection of HTTP methods, for example, the DELETE method was used for ending configuration sessions, while PUT was used for set and delete commands, and POST was used for running op mode commands. You can find a copy of its documentation here if you are curious.
That’s all history of course and has nothing to do with VyOS.
The main issue with implementing it is that the original config backend was meant for use inside a shell session rather than programatically. Ubiquiti made an intermediate layer for it that uses the backend libraries directly and exposes their interface through a UNIX domain socket so that they could have proprietary code communicate with the backend without violating the GPL, but it’s a chunk of C++ code that is entirely undocumented and has no usage examples, so it would have to be essentially reverse engineered to start using it. Besides, we want to get rid of the old backend and its limitations eventually, not increase our reliance on it.
For a long time, it was believed that the most feasible way to use VyOS programatically was to emulate user sessions, and that’s what all older libraries including the Ansible modules did.
In 1.2.0, the highest priorities were fixing things that were obviously broken and establishing a path to the eventual backend replacement as well as making the system easier to contribute to—with new XML-based command definitions, Python APIs, and scripts rewritten in a structured and modular way. Lately we decided to get back to the HTTP API question, I spent quite some time researching the source code of the old backend and the ubnt-cfgd, and found a solution that now seems so obvious that I wonder why no one thought of it earlier.
The config backend provides a number of CLI utilities. The most well known one is cli-shell-api that is widely used for scripting. The commands behind the set/delete/commit shell functions are ${vyos_sbindir}/my_set, my_delete, my_commit etc. The running config is represented as a directory tree with directories representing config nodes and node.val files representing node values. When a user enters the “configure” command, a new directory is union-mounted with the running config.
Those commands take no arguments other than the path and the value, if any. How do they know where to find the session directories? Here’s what actually happens when you run “configure”: first the shell runs “cli-shell-api getSessionEnv <unique ID>” that generates unique environment variables for the session based on the ID (in practice, $PPID is always used for the ID to ensure global uniqueness). Then it runs “cli-shell-api setupSession” that uses those variables to actually setup a session. Then my_set and friends use the same variables to know what to modify.
When I was certain that environment variables is all it needs, the way forward was clear.
I guess one of the reasons no one thought of it before is that in C or Perl, working with environment variables is annoying. In Python, however, you can easily get the entire environment as a hash, insert values into it, and pass it to subprocess module functions as an argument.
The API server executable is /usr/libexec/vyos/vyos-http-api-server. In the source code it can be found at https://github.com/vyos/vyos-1x/blob/current/src/services/vyos-http-api-server
Since it's a security and reliability area, it can definitely benefit from a community review, so don't hesitate to contact us if you spot any issues.
The config write API is implemented as a module rather than baked into the HTTP server process, so that we can also use it in the future web GUI, and everyone can write their own code that modified the config.
The module is named vyos.configsession and its source is at https://github.com/vyos/vyos-1x/blob/current/python/vyos/configsession.py
Here's a simple usage example:
import vyos.configsession session = vyos.configsession.ConfigSession(os.getpid(), app="myscript") session.set(["system", "host-name"], value="vyos") session.commit()
You must make sure that the session id is globally unique (within the router of course), and the PID is probably the best way to get a globally unique integer.
Right now the API has a global lock to prevent more than one app or person from modifying the session at once. This solves the problem with garbage collecting dead sessions that may end up wasting quite some RAM over time. There is no protection against trying to commit while another commit is in progress, but it will be implemented soon—that's a bit harder to test and is not going to occur often so it wasn't the first priority.
From the backend's point of view, deleting a node that doesn't exist is not an error. After all, if there's nothing to delete, the "make sure the node is not in the config" goal is already achieved. It can be problematic when developing applications though since a typo in a path will not cause an error. For this reason the API supports a strict mode where this (and other similar conditions, in the future) is treated as an error. You can enable it by adding a "strict=true" field to your request.