Virtlyst 1.2.0 released

Virtlyst – a Web Interface to manage virtual machines build with Cutelyst/Qt/C++ got a new release.

This new release includes a bunch of bug fixes, most importantly probably being the ability to warn user before doing important actions to help avoid doing mistakes.

Most commits came from new contributor René Linder who is also working on a Bootstrap 4 theme and Lukas Steiner created a dockerfile for it. This is especially cool because Virtlyst repository now has 4 authors while Cutelyst which is way older has only 6.

For the next release I’ll also try to add user management (today you have a single admin account, and to add new users that need to be done via SQL) which wasn’t available on the original WebVirtMgr project but is surely the most important lacking feature.

Have Fun! https://github.com/cutelyst/Virtlyst/archive/v1.2.0.tar.gz

Advertisements
Virtlyst 1.2.0 released

Cutelyst 2.5.0 released

Cutelyst a C++ web framework based on Qt got a new release. This release has some important bug fixes so it’s really recommended to upgrade to it.

Most of this release fixes came form a side project I started called Cloudlyst, I did some work for the NextCloud client, and due that I became interested into how WebDAV protocol works, so Cloudlyst is a server implementation of WebDAV, it also passes all litmus tests. WebDAV protocol makes heavy use of REST concept, and although it uses XML instead of JSON it’s actually a good choice since XML can be parsed progressively which is important for large directories.

Since the path URL now has to deal with file paths it’s very important it can deal well with especial characters, and sadly it did not, I had tried to optimize percent encoding decoding using a single QString instead of going back and forth toLatin1() then fromUTF8() and this wasn’t working at all, in order to better fix this the URL is parsed a single time at once so the QString path() is fully decoded now, which will be a little faster and avoid allocations. And this is now unit tested 🙂

Besides that there was:

  • Fix for regression of auto-reloading apps in cutelyst-wsgi
  • Fix csrf token for multipart/form-data (Sebastian Held)
  • Allow compiling WSGI module when Qt was not built with SSL support

The last one and another commit were to fix some build issues I had with buildroot, which I also created a package so soon you will be able to select Cutelyst from buildroot menu.

Have fun https://github.com/cutelyst/cutelyst/releases/tag/v2.5.0

Cutelyst 2.5.0 released

Cutelyst on TechEmpower benchmarks round 16

Yesterday TechEmpower released the results for round 16 of their benchmarking tests, you can see their blog about it here. And like for round 15 I’d like add my commentary about it here.

Before you look into the results web site it’s important to be aware of a few things, first round 16 runs on a new hardware newer and more powerful than the previous rounds, they also did a Dockerization of the tests which allowed us to pull different distro images, cache package install and isolate from other frameworks. So don’t try to compare to round 15.

Having put Cutelyst under testing there has brought many benefits to it, in previous rounds we noticed that due testing on a server with many CPU cores it letting the operating system do the scheduling wouldn’t be a good idea, so we added CPU affinity feature to cutelyst-wsgi, while uWSGI also has it our logic does this core pinning for threads and not only process.

While testing at home on an AMD Phenon II x4 using pre-fork mode was much faster than threaded mode, but on my 5th Gen Intel laptop the results were closer but threading was much better, and thanks to TFB I found out that each process were being assigned to CPU core 0, which made 28 processes be bound to a single core. In this new round you can now see the difference between running threaded or in pre-fork mode is negligible in some tests pre-fork is even faster. Pre-fork does consume more RAM, running 100 threads of Virtlyst uses around 35MiB, while 100 process around 130 MiB but multiple process are better if your code happens to crash.

An important feature of Tech Empower Benchmarks are it’s filters, they allow you to filter stuff that doesn’t matter to you, due the above fix you call look at the results and see the close threads vs pre-fork match here.

Using an HTML templating engine is very important for real world apps, in round 16 I’ve added a test that uses Grantlee for rendering, they are around 46% slower than creating the HTML directly on C++, there might be room for improvements in Grantlee but the fortunes results aren’t bad.

Some people asked why there were some many occurrences of Cutelyst in the tests, the reason is that these benchmarks allow you to tests some features and see what performs better, in round 15 it became clear that using EPoll vs Qt’s glib based event loop was a clear win, so in 16 we don’t have Qt’s glib tested anymore and EPoll is now the default event dispatcher in Cutelyst 2 on Linux.

This round however I tried to make a reasoning about TCP_NODELAY and while results are closer it’s a bit clear that due the blocking nature of the SQL tests TCP_NODELAY will decrease latency and increase performance a bit due it not waiting to send a bigger TCP packet.

As I also mentioned on the release of Cutelyst 2.4.0 a fix was made to avoid stack overflow and you can see the results on the “Plain Text” test, before that fix cutelyst was crashing and respawning and that limited the results to 1,000,000 requests/second, after the fix it is 2,800,000 requests/second.

This round used Ubuntu 18.04 as base, Cutelyst 2.4.0 and Qt 5.9.5.

Last but not least if you still try to compare to round 15 🙂 you might notice Cutelyst went lower on the ranking, that’s in part due frameworks getting optimizations but mostly due new micro/platform frameworks, but you can enable the Fullstack classification filter if you want to compare frameworks that would provide similar features that what Cutelyst offers.

Cutelyst on TechEmpower benchmarks round 16

Cutelyst 2.4.0 released

Cutelyst, the C++/Qt web framework got another up.

This release includes:

  • Fix for our EPoll event loop spotted by our growing community
  • An Sql query conversion optimization (also from community), we have helper methods to create QVariantHash, QVariantMap, QVariantList… from QSqlQuery to be used by Grantlee templates
  • New Sql query conversion to JSON types (QJsonObject, QJsonArray..) making it easier and faster to build REST services as shown on our blog post about Creating RESTful applications with Qt and Cutelyst
  • New boolean methods to test for the HTTP method used (isPUT(), isDelete()…) easing the work on REST apps
  • Fix for our CPU affinity logic, thanks to the TechEmpower benchmarks I found out that when creating many workers process they were all being assigned to a single CPU core, I knew that pre-fork was usually slower than threads on modern CPUs but was always intrigued why the difference was so big. Hopefully it will be smaller now.

Since I’m currently building a REST service OAuth2 seems to be something that would be of good use, one user has started working on this, hopefully this will soon be ready.

Have fun https://github.com/cutelyst/cutelyst/releases/tag/v2.4.0

 

Cutelyst 2.4.0 released

Creating RESTful applications with Qt and Cutelyst

This mini tutorial aims to show you the fundamentals of creating a RESTful application with Qt, as a client and as a server with the help of Cutelyst.

Services with REST APIs have become very popular in recent years, and interacting with them may be necessary to integrate with services and keep your application relevant, as well as it may be interesting to replace your own protocol with a REST implementation.

REST is very associated with JSON, however, JSON is not required for a service to become RESTful, the way data is exchanged is chosen by the one who defines the API, ie it is possible to have REST exchanging messages in XML or another format. We will use JSON for its popularity, simplicity and due to the QJsonDocument class being present in the Qt Core module.

A REST service is mainly characterized by making use of the little-used HTTP headers and methods, browsers basically use GET to get data and POST to send form and file data, however REST clients will use other methods like DELETE, PUT and HEAD, concerning headers many APIs define headers for authentication, for example X-Application-Token can contain a key generated only for the application of a user X, so that if this header does not contain the correct data it will not have access to the data.

Let’s start by defining the server API:

  • /api/v1/users
    • GET – Gets the list of users
      • Answer: [“uuid1”, “uuid2”]
    • POST – Register new user
      • Send: {“name”: “someone”, “age”: 32}
      • Answer: {“status”: “ok / error”, “uuid”: “new user uuid”, “error”: “msg in case of error”}
  • /api/v1/users/ – where UUID should be replaced by the user’s UUID
    • GET – Gets user information
      • Answer: {“name”: “someone”, “age”: 32}
    • PUT – Update user information
      • Send: {“name”: “someone”, “age”: 57}
      • Answer: {“status”: “ok / error”, “error”: “msg in case of error”}
    • DELETE – Delete user
      • Answer: {“status”: “ok / error”, “error”: “msg in case of error”}

For the sake of simplicity we will store the data using QSettings, we do not recommend it for real applications, but Sql or something like that escapes from the scope of this tutorial. We also assume that you already have Qt and Cutelyst installed, the code is available at https://github.com/ceciletti/example-qt-cutelyst-rest

Part 1 – RESTful Server with C ++, Cutelyst and Qt

First we create the server application:

$ cutelyst2 --create-app ServerREST

And then we will create the Controller that will have the API methods:

$ cutelyst2 --controller ApiV1

Once the new class has been instantiated in serverrest.cpp, init() method with:

#include "apiv1.h"

bool ServerREST::init()
{
    new ApiV1 (this);
    ...

Add the following methods to the file “apiv1.h”

C_ATTR(users, :Local :AutoArgs :ActionClass(REST))
void users(Context *c);

C_ATTR(users_GET, :Private)
void users_GET(Context *c);

C_ATTR(users_POST, :Private)
void users_POST(Context *c);

C_ATTR(users_uuid, :Path('users') :AutoArgs :ActionClass(REST))
void users_uuid(Context *c, const QString &uuid);

C_ATTR(users_uuid_GET, :Private)
void users_uuid_GET(Context *c, const QString &uuid);

C_ATTR(users_uuid_PUT, :Private)
void users_uuid_PUT(Context *c, const QString &uuid);

C_ATTR(users_uuid_DELETE, :Private)
void users_uuid_DELETE(Context *c, const QString &uuid);

The C_ATTR macro is used to add metadata about the class that the MOC will generate, so Cutelyst knows how to map the URLs to those functions.

  • :Local – Map method name to URL by generating /api/v1/users
  • :AutoArgs – Automatically checks the number of arguments after the Context *, in users_uuid we have only one, so the method will be called if the URL is /api/v1/users/any-thing
  • :ActionClass(REST) ​​- Will load the REST plugin that will create an Action class to take care of this method, ActionREST will call the other methods depending on the called method
  • :Private – Registers the action as private in Cutelyst, so that it is not directly accessible via URL

This is enough to have an automatic mapping depending on the HTTP method for each function, it is important to note that the first function (without _METHOD) is always executed, for more information see the API of ActionREST

For brevity I will show only the GET code of users, the rest can be seen in GitHub:

void ApiV1::users_GET(Context *c)
{
    QSettings s;
    const QStringList uuids = s.childGroups();

    c->response()->setJsonArrayBody(QJsonArray::fromStringList(uuids));
}

After all the implemented methods start the server:

cutelyst2 -r --server --app-file path_to_it

To test the API you can test a POST with curl:

curl -H "Content-Type: application/json" -X POST -d '{"name": "someone", "age": 32}' http://localhost:3000/api/v1/users

Okay, now you have a REST server application, made with Qt, with one of the fastest answers in the old west 🙂

No, it’s serious, check out the benchmarks.

Now let’s go to part 2, which is to create the client application that will consume this API.

Part 2 – REST Client Application

First create a QWidgets project with a QMainWindow, the goal here is just to see how to create REST requests from Qt code, so we assume that you are already familiar with creating graphical interfaces with it.

Our interface will be composed of:

  • 1 – QComboBox where we will list users’ UUIDs
  • 1 – QLineEdit to enter and display the user name
  • 1 – QSpinBox to enter and view user age
  • 2 – QPushButton
    • To create or update a user’s registry
    • To delete the user record

Once designed the interface, our QMainWindow sub-class needs to have a pointer to QNetworkAccessManager, this is the class responsible for handling communication with network services such as HTTP and FTP. This class works asynchronously, it has the same operation as a browser that will create up to 6 simultaneous connections to the same server, if you have made more requests at the same time it will put them in a queue (or pipeline them if set).

Then create a QNetworkAccessManager *m_nam; as a member of your class so we can reuse it. Our request to obtain the list of users will be quite simple:

QNetworkRequest request(QUrl("http://localhost:3000/api/v1/users"));

QNetworkReply *reply = m_nam->get(request);
connect(reply, &QNetworkReply::finished, this, [this, reply] {
    reply->deleteLater();
    const QJsonDocument doc = QJsonDocument::fromJson(reply->readAll());
    const QJsonArray array = doc.array();

    for (const QJsonValue &value : array) {
        ui->uuidCB->addItem(value.toString());
    }
});

This fills with the data via GET from the server our QComboBox, now we will see the registration code which is a little more complex:

QNetworkRequest request(QUrl("http://localhost:3000/api/v1/users"));
request.setHeader(QNetworkRequest::ContentTypeHeader, "application/json");

QJsonObject obj {
    {"name", ui->nameLE->text()},
    ("age", ui->ageSP->value()}
};

QNetworkReply *reply = m_nam->post(request, QJsonDocument(obj).toJson());
connect(reply, &QNetworkReply::finished, this, [this, reply] {
    reply->deleteLater();
    const QJsonDocument doc = QJsonDocument::fromJson(reply->readAll());
    const QJsonObject obj = doc.object();

    if (obj.value("status").toString() == "ok") {
        ui->uuidCB->addItem(obj.value("uuid").toString());
    } else {
        qWarning() << "ERROR" << obj.value("error").toString();
    }
});

With the above code we send an HTTP request using the POST method, like PUT it accepts sending data to the server. It is important to inform the server with what kind of data it will be dealing with, so the “Content-Type” header is set to “application/json”, Qt issues a warning on the terminal if the content type has not been defined. As soon as the server responds we add the new UUID in the combobox so that it stays up to date without having to get all UUIDs again.

As demonstrated QNetworkAccessManager already has methods ready for the most common REST actions, however if you wanted to send a request of type OPTIONS for example will have to create a request of type CustomOperation:

m_nam->sendCustomRequest("OPTIONS", request);

Did you like the article? Help by giving a star to Cutelyst and/or supporting me on Patreon

https://github.com/cutelyst/cutelyst

Creating RESTful applications with Qt and Cutelyst

Cutelyst 2.3.0 released

Cutelyst – The C++ Web Framework built with Qt, has a new release.

In this release a behavior change was made, when asking for POST or URL query parameters and cookies that have multiple keys the last inserted one (closer to the right) is returned, previously QMap was filled in reverse order so that values() would have them in left to right order. However this is not desired and most other frameworks also return the last inserted value. To still have the ordered list Request::queryParameters(“key”) builds a list in the left to right order (while QMap::values() will have them reversed).

Some fixes on FastCGI implementation as well as properly getting values when uWSGI FastCGI protocol was in use.

Lastly a crash when performing TechEmpower benchmarks resulted in the removal of some nested event loops calls that were creating a huge stack, and made me realize that my proposed methods to make async apps easier were broken by design, basically you can’t use QEventLoop::exec() to wait for events. So Context::wait() and next() are deprecated and the code disabled, with the exception of WebSockets until I can make some changes to HTTP/1 and FastCGI parser both are to be considered synchronous.

Removing a QCoreApplication::processEvents() that was added in order to try to give other connections an execution slice almost doubled the speed in plain text when benchmarking locally, using a single core the requests per second have gone from 105 K to 190 K, after understanding the stupidity I did this isn’t of much surprise 🙂

https://github.com/cutelyst/cutelyst/releases/tag/v2.3.0

Cutelyst 2.3.0 released

Virtlyst 1.1.0 released

Virtlyst – a libvirt web interface to manage virtual machines has a new release.

This release finishes support to connect to TCP or TLS virtd servers, it also fixes creating new instances from the flavor panel. And a few other fixes.

Of course I forgot to add the mandatory screenshot last time so here it goes:

instances

console

Go get it! https://github.com/cutelyst/Virtlyst/releases/tag/v1.1.0

If you like this software you can also support my work via patreon.

Virtlyst 1.1.0 released