Cutelyst 2.13 and ASql 0.19 released

Cutelyst the C++/Qt Web Framework and ASql the ASync SQL library for Qt applications got new versions.

Thanks to the work on ASql Cutelyst got some significant performance improvements on async requests, as well as a new class called ASync, which automatically detaches the current request from the processing chain, and attaches later on when it goes out of scope.

With the ASync class you capture it on your ASql lambda and once the query result arrives and the lambda is freed and the ASync object gets out of scope and continues the processing action chain.

KDAB’s fix on a 10 year old bug raised my attention back to a note on QTcpSocket documentation:

Note: TCP sockets cannot be opened in QIODevice::Unbuffered mode.

Which is actually wrong, according to the source code, Cutelyst has been using buffered mode since always due that, so hopefully this new version will be a bit faster and consume less memory, it’s important to notice that once the Kernel tells it’s going to block QTcpSocket writes get buffered anyway.

Now talking about ASql you might notice the jump on release versions, this is because I’ve been experimenting some changes and didn’t want to write a post at each new feature.

ASql is now at the closest API I’d like it to be, unfortunately one of my goals that would be to be able to handle it’s AResult object to Grantlee/Cutelee and be able to iterate over it’s records just once, but the call QVariant::canConvert with QVariantList must succeed and the class be able to be casted to QAssociativeIterable or QSequentialIterable, which I didn’t managed to get it working in a way I like.

But AResult has hash() and hashes() methods that convert the data to a format that can be used in templating.

On the plus side I added iterators (if you have experience with iterators please review my code as this was the first time I wrote this kind of code) that also work on for ranged loops, and they also have faster type conversion methods, instead converting the data to a QVariant type, and them using QVariant to get the data out of it, one can just call toInt() which will get the int straight from the result set without checking each time if it’s an int.

Added AMigrations which is an awesome database maintenance class that is both simple and helpful to maintain database schemas.

  • ACache class to cache special queries
  • Support for data inside QJsonValue
  • Single Row mode (the lambda get’s called once per result)
  • Prepared Queries
  • Scoped Transaction class
  • Notifications – this is my favorite PostgreSQL feature, it’s hard to image some other big databases lack such an useful feature.

Oh, and Cutelyst results on TechEmpower already got better thanks to ASql, hoping to see even better results when I update the tests to 2.13 that has Unbuffered and faster async handlying.

https://github.com/cutelyst/asql/releases/tag/v0.19.0

https://github.com/cutelyst/cutelyst/releases/tag/v2.13.0

Cutelyst 2.13 and ASql 0.19 released

Cutelyst 2.12 released

Cutelyst a Qt web framework got a new version, this version has one important fix for async apps using FastCGI or HTTP2, also fixes for SO_REUSEPORT, and as a new feature is has support for listen queue backlog on UNIX, sadly QTcpServer doesn’t support both of these (I need to find time to write some patches…).

Cutelyst is available for many Linux distributions but doesn’t for Debian/Kubuntu, so I’m also adding the CPack deb built will all but uWSGI support for 20.04 amd64.

Have fun https://github.com/cutelyst/cutelyst/releases/tag/v2.12.0

Cutelyst 2.12 released

Announcing ASql – async Sql for Qt

When developing Qt applications be it Desktop, Mobile or Web that need to talk directly with a database the use of QtSql is usually the best choice, it has many database drivers, comes ready on their installer, but has a blocking API, which means a query will block you GUI thread.

My Cutelyst Web projects also used QtSql, and for low traffic this isn’t a big issue because you are not freezing users GUI, but you are freezing the request queue instead.

One of the Cutelyst apps I developed this year has a very high traffic, and a side effect of blocking came in play, I’ve about 3k TVs connected via websockets, once a TV connects it does an authentication query, it takes ~17ms, now when there is an event PostgreSQL notifies the application which does some 20 other queries a little more expensive (~30ms) and send them to the TVs, this event also blocks the Cutelyst process.

Every TV does a websocket ping each 15 seconds to check if the connection is stalled, when that event happens plus some latency added 15s timeout and the TV disconnect, this of course happens with lots of TVs, so 1k TVs reconnecting will take around 17 seconds, this means new TVs disconnecting and the snow ball is done, system only recovers if I increase the DB hardware on AWS.

How can async fix this, you might be wondering, well the problem is that the 15s timeout happened to do a DOS on my own system, I can increase the value but when there are 10k TVs this will still be a problem, there are of course other ways to deal with this, but if the SQL query doesn’t block the Cutelyst process it will be able to timely reply the Websocket’s pongs.

ASql only supports PostgreSQL at time (PR are welcome), and depends on QtCore and libpq (PostgreSQL library), it can be used on Desktop apps and is 100% async from connection to queries, there’s even a class to help with DB versioning and migration.

Check it out: https://github.com/cutelyst/asql

Announcing ASql – async Sql for Qt

Cutelyst 2.11 and SimpleMail 2.1 released and a fork called Cutelee

Cutelyst the web framework based on Qt and SimpleMail the SMTP library got another update.

The current scenario has made me a lot less productive, after all 4 kids at home all the time it’s really hard to concentrate. Still many fixes and a few features have landed both.

Simple mail had issues building on Windows, and also got some docs fixes.

Cutelyst got some Session parameters available on config, an WebSocket improvement to avoid setting a null body to skip RenderView processing, and a new view called Cutelee.

This year I got a big project, one that is pushing scalability on Cutelyst further, but it also requires that things progresses faster, and this means Grantlee was starting to become an issue. A while ago it’s maintainer proposed to move it’s code to KDE, got my hopes high, but that got postponed to Qt6, I couldn’t wait for that and I also had an years old Pull Request waiting there, making in fact Cutelyst pages less safer as an escaping code wasn’t escaping one special character. A Cutelyst contributor also had patches pending there, I tried my best to avoid this, the author is a great programmer, did an awesome job but we needed something that can move a little faster.

Cutelee was born, and already have many commits, it also got some performance improvements, like not converting QDate(Time) to an string and converting back to use the :date “yy/dd” formater, another important change was that it can inject filters without the hassle of creating a Qt Plugin, putting it into the right and error prone directory hierarchy, %with% tag also supports the new Django syntax, and I hope we can keep improving it at a faster rate.

I didn’t do a new release of it yet, as some installation issues raised at Grantlee weren’t fixed yet on Cutelee.

Have fun

https://github.com/cutelyst/cutelyst/releases/tag/v2.11.0

https://github.com/cutelyst/simple-mail/releases/tag/v2.1.0

https://github.com/cutelyst/cutelee

Cutelyst 2.11 and SimpleMail 2.1 released and a fork called Cutelee

Cutelyst 2.10.0 and SimpleMail v2 released!

Cutelyst the C++/Qt Web framework and SimpleMailQt just got new releases.

Cutelyst received many important bugfixes and if you are compiling it with View::Email it also requires SimpleMail 2, the latter got an Async API which is on production for a few months, allowing for a non-blocking send mail experience.

Check them out!

https://github.com/cutelyst/cutelyst/releases/tag/v2.10.0

https://github.com/cutelyst/simple-mail/releases/tag/v2.0.0

Cutelyst 2.10.0 and SimpleMail v2 released!

SimpleMailQt v2.0.0-beta1

On my last post I talked about the new async simplemail-qt API that I wanted to add, yesterday I finished the work required to have that.

SMTP (Simple Mail Transfer Protocol) as the name says it’s a very simple but strict protocol, you send a command and MUST wait for the reply, which is rather inefficient but it’s the way it is, having it async means I don’t need an extra thread (+some locking) just to send an email, no more GUI freezes or an HTTP server that is stalled.

The new Server class has a state machine that knows what reply we are waiting, and which status code is the successful one. Modern SMTP servers have PIPELING support, but it’s rather different from HTTP PIPELING, because you still have to wait for several commands before you send another command, in fact it only allows you to send the FROM the RECIPIENTS email list and DATA commands at once, parse each reply and then send the mail data, if you send the email data before you are allowed by the DATA command the server will just close the connection.

So in short we only save a few (but important) round trips, this new version also includes support for the RESET command, this is useful for example when you have a long list of emails, and some of them have malformed RECIPIENTS email, you will get an error for the command but instead of closing the connections and start the long SMTP handshake you just RESET and try to send the next email.

Cutelyst master is already using this new class if you set ViewEmail::setAsync(true);. With that set you won’t get the SMTP mail queued reply but will still be able to process more HTTP requests while waiting for the SMTP reply.

I plan to do a 2.0.0 stable release in the following weeks so if you use this library please test and suggest API changes that you may like.

Have fun!

https://github.com/cutelyst/simple-mail/releases/tag/v2.0.0-beta1

 

SimpleMailQt v2.0.0-beta1

Cutelyst 2.9.0 and simple-mail-qt 1.4.0 released!

Today I’m rolling out two releases, Cutelyst, which is a Qt Web Framework and SimpleMailQt which is a SMTP client library.

Cutelyst has got many bug fixes, a few API additions, some docs fixes, and most importantly it fixed a memory leak introduced in 2.8.0 that makes applications using chained actions leak.

I was planning for a v3 due some changes I was planning but changed my mind and I think the current version can be kept for a little longer, my current plan is to add SNI support for the WSGI module so that a sort of Virtual Hosts is possiblem.

The ideia for Virtual Hosts is that you can get rid of Nginx (or any front end server) if you want and can, the front servers are great, but they do a work that could be avoided. When a request arives the server it has to parse, identify to which application it has to redispatch and create a new request (in HTTP/FastCGI/whatever), on single core servers the OS will put it to sleep, wake cutelyst, which then parses the request again, creates a reply, which is then parsed by the front and recreated for the client.

Of course they are fast and all, but surely if it could be avoided would be great, on an IPv6 only world I’d just put a different IP for each application and be done, but not the reality now do I’ll try to see if a new Cutelyst application could load all your applications in a single process. It’s just an idea atm.

SimpleMailQt also got quite a few bug fixes, most importantly it got it’s CMake modernized so it builds fine on Windows now.

For those that don’t know SimpleMailQt, it was based on a SmptClientForQt with a port to CMake and many API changes to be more Qt style. But it’s a blocking library which also doesn’t do pipelining, so for v2 I’ll be creating a new async engine that will be async and do pipelining if the server supports.

Have fun:

https://github.com/cutelyst/cutelyst/releases/tag/v2.9.0

https://github.com/cutelyst/simple-mail/releases/tag/v1.4.0

Cutelyst 2.9.0 and simple-mail-qt 1.4.0 released!

Cutelyst 2.8.0 released

Cutelyst a Qt/C++ Web framework got a new release!

This release took a while to be out because I wanted to fix some important stuff, but time is short, I’ve been working on polishing my UPnpQt library and on a yet to be released FirebaseQt and FirebaseQtAdmin (that’s been used on a mobile app and REST/WebApp used with Cutelyst), the latter is working quite well although it depends ATM on a Python script to get the Google token, luckly it’s a temporary waste of 25MB of RAM each 45 minutes.

Back to the release, thanks to Alexander Yudaev it has cpack support now and 顏子鳴 also fixed some bugs and added a deflate feature to RenderView, FastCGI and H2.

I’m also very happy we now have more than 500 stars on GitHub 🙂

Have fun https://github.com/cutelyst/cutelyst/releases/tag/v2.8.0

Cutelyst 2.8.0 released

Cutelyst 2.7.0 released, async is back!

Cutelyst a Qt/C++ Web Framework just got a new version. This time bringing back proper async support.

Perl Catalyst Framework was conceived as a sync/blocking framework, due that the Mojolicious framework was created, NodeJS and a few other web frameworks have pushed the async programming to the web. Performance wise being async doesn’t mean you get faster response times, rather the opposite, the need to unroll stack and make extra calls makes a CPU bound application slower.

But depending on the problem to solve it allows you to serve more users at the same time, using the same CPU cores. A typical modern application might receive a request from a Phone App then do a REST API call to an external API which might take 2ms or 200ms for the reply. While waiting for the response, typical sync/blocking applications can’t process more requests, or have to spawn more threads increasing RAM consumption and leveraging concurrency to the OS scheduler. On an Async web application, you can process another request while you wait for the previous request, thus possibly making the first request reply to be sent back to the user at a later time than if there was a dedicated process just waiting for his reply.

So, both ways have pros and cons and IMHO I’d like to support them both. When I started Cutelyst I thought that if I ever need async I could have a local QEventLoop to wait for the reply and would be able to deal with async requests, not recently I found out that supporting QEventLoop was causing stack overflow due it being used in pipelined scenarios, after that I removed it’s usage and performance improved in PlainText tests of TechEmpower, so I advised against using it and marked Cutelyst as not async.

Recently I got the need to call a few APIs from a REST API I did with Cutelyst, QNeworkAccessManager is an async only API, so either I create a thread for it to mimic sync behavior or allow Cutelyst to be used in async mode, of course I did the latter. The Context class has now detachAsync() and attachAsync() methods.

When you c->detachAsync(), you tell the engine to not send the headers and body immediately to the client, this will also break the action chain, meaning an end() method won’t be called, then when you get your async reply you call c->attachAsync().

Once c->attachAsync() is called the action chain is resumed so the end() method that possibly would render the HTML can perform it’s job.

It’s important to always be child ( QObject->setParent(…) ) of the Context pointer as if the client closes the connection it might be deleted.

This release was not only made of async:

  • A helper UserAgent singleton was added so that you have a thread_local QNetworkAccessManager with static methods to do more common REST tasks like sending a QJsonObject (yes, eventually I find time to write a patch for QNAM)
  • A new Sql::Transaction class gives you a scoped SQL transaction, preventing the mistake of not rolling back when you leave the scope in case of an error.
  • A few Headers helpers like ETag and If-Match comparison
  • Sebastian Held added support for dealing with Front-End Proxy Headers
  • Improved Stats precision using QElapsedTimer::nsecsElapsed()

And finally Cutelyst is not “half modern” CMake anymore, if that can even exist. The include directories of the exported Targets are now properly exported, so that you can remove:

include_directories(
    ${Cutelyst2Qt5_INCLUDE_DIR}
)

Have fun https://github.com/cutelyst/cutelyst/archive/v2.7.0.tar.gz

Cutelyst 2.7.0 released, async is back!

Facebook AccountKit with Qt/C++ on Android

Facebook’s AccountKit is an authentication service that can use your email or phone number to login to your services, it doesn’t require that the user has a Facebook account, just a valid email or phone.

The cool thing about it is that it sends SMS for free, and although sending SMSs is cheap being free of charge is something you might want to look when creating a new App, in fact here in Brazil some big Apps do make use of it.

So long story short story I wanted to add this to my Qt Android App.

Thanks to the great help of Kai Uwe Broulik who had give some tips on the past on how to call Java code I started with this same approach:

  • Create a Java file
  • Put the Java code of AccountKit there
  • Call the static function of Java from C++

And… it did work BUT I was unable to get the result, this is because the Java code starts a new Activity and returns the result on a method that must be implemented on the code that Qt creates for starting your activity, so this means I’d need to change the XML manifest to call my subclass instead of what Qt provides, and would need to do a bunch of other stuff myself, basically would get more maintainance work to do.

So another option would be to do all of this from C++, which Kai said it would be better at first, but doing a bunch of JNI scared me a bit, also I hadn’t found the reference page to their API (which he found after the work was done lol). There’s a tool called javap that can dump the Java signature of what is inside the package you got, so with the help of this we went on porting all the calls to AccountKit API to C++/Qt, this way we can call QtAndroid::startActivity() passing a pointer to a class that will handle the result of the new Activity, all in C++ 🙂 .

There were some initial issues with the Enums used (aparently some new Java7 stuff), but the code now doesn’t require a single line of Java code, which is great when integrating in Qt Android Apps.

The result is on GitHub, so if you need this just copy the integration class, it has IFDEFs so if you are testing on !Android it will compile fine. (You still need to follow the Gradle and manifest integration that’s on their dev site), feel free to make PR for fixes and features 🙂

https://github.com/ceciletti/third-party-qt-utils

Facebook AccountKit with Qt/C++ on Android