Litteras, the yet another! And Microsoft Exchange situation.

Before you jump to the comments section and start a flame war, please let me give you a little ground.

Two years ago when I got back to Brazil I went to the same job but hired by a different company, the new company uses Microsoft Exchange as the mail solution. No matter how hard I could try it’d almost impossible to think they would ever move away from it, I’d need to convince people that I don’t know personally which are also in another continent…

I believe I’m not alone in this, so, instead of dreaming with the IMAP day I decided to take a look at what email clients could talk to Exchange, at that time there was ZERO Linux email clients supporting it, and to be fair I’m not talking about MAPI, which is another protocol used to communicate with Exchange but it’s disabled/blocked on my company and in many others.

There is some program that takes the OWA (Outlook Web Access) and convert it to IMAP, but I didn’t like the idea nor had the will to setup it as it looked complex. So for some time I just gave up and used the webmail, but it’s really bad, especially in  2007 version it doesn’t auto refresh the page, and even if it did it wouldn’t notify me about new emails.

So when my boss told me he could easily setup it on his OSX Mail I got intrigued, how could it talk to exchange if MAPI was not an option? After some research I became aware of EWS (Exchange Web Services), which is a SOAP specification to talk to Exchange using HTTP. I then tried to use gSOAP library to auto generate the interfaces and code to talk to it (as I have used it before for another SOAP project), but as soon as the code was also linked to any KDE library I got some DSO weird error from the linker… I tried to find how to fix this linker issue but couldn’t get help or find a solution.

I then sort of gave up and time passed again, and Evolution Mail got EWS support, but it also didn’t work with my company setup, no idea why, but still it didn’t work. Recently a new version of it started to work with my company’s server, and I started using it, but well besides the fact that it’s a GTK app it doesn’t work well for me, it’s slow, the address completion is quite useless…

So time to give Litteras another try, but wait! Why not give KMail EWS support instead of the yet another email client?

UPDATE EDIT:

To put it simple: I don’t like MySQL on my servers (you can imagine how I fell about it on a desktop), even if it was PostgreSQL I simply think it’s wrong to store my mails on a SQL database. Granted KMail works great for lots of users, but I myself don’t like the underground tech, it’s probably much more a matter of taste.

So, KMail developer are telling me the emails are stored as regular files which is something I do know. But then there is a dependency with akonadi, which is the one that can use MySQL, sqlite… So to not spread FUD I’ll try to put it another way, I myself while trying to use it didn’t find a way to avoid akonadi, and I saw lots of other people not being able to do it too. Every place I look for information it says that Akonadi will cache the email information to be easily retrieved in other places, so it’s not the same as storing the emails on the data base. Still I myself (as mentioned earlier) don’t like this idea much. If one can still disable akonadi I actually find hard to believe it’s possible since all information I have found is that right now Akonadi has the resources to fetch the emails. So basically: continue using what you have been using and is working for you, in my taste I just don’t like, and that’t the reason why not everyone uses GTK or Qt or PostgreSQL, there are sometimes technical reasons, but it’s not entirely the case here, MySQL might perform well for this use case but past experience with it on servers gave me a trauma.

PS: I hope I clarified this part, as I knew it would be hard to explain it’s a personal matter (tho I ended the line stating this), and I needed to state my reasons for not willing to go the KMail/Akonadi way right now as that would be 1st question.

—————-

BUT If you like KMail and would like EWS support be happy. I’m building an EWS Qt library so that this will benefit any KDE/Qt developer willing to write a yet another mail client, and adding this kind of support for KMail should be much easier when the library is in place, I could even try to do it myself.

So what about Litteras?

Litteras already is a KDE application (as it uses KDE libraries), and I want it to feature EWS, POP3 and IMAP support. Locally I plan to store the emails in MailDir format, and index them with Xapian. I also want it to feature a clean user interface and most importantly do lot’s of magic when setting up an email account.

litteras

Right now you can just type your email and password and it will find the EWS server if it was deployed following Microsoft specification, it’s actually even better than EWS than Evolution in this regard right now as they didn’t do DNS SRV search to find the right server (which is the case needed for my company). It also download your folders and messages but doesn’t get the body yet, nor store then locally.

Email composer...
Email composer…

Now let’s do business :D I want your support:

  • Do you want EWS support using Qt technologies?
  • Want a library to make it easier for KMail use it?
  • A different KDE/Qt mail client?
  • Even Plasma Active/Tizen could benefit from this Exchange support as not having Exchange support might be a no go for many users.

Here is the indiegogo campaign to support two months of development, an amount of 6500 USD. Details on what I’ll do can be found there.

indiegogo
http://igg.me/at/litteras/x/525675

Go grab the code and hack it if you want: https://gitorious.org/litteras

Thank you.

Litteras, the yet another! And Microsoft Exchange situation.

58 thoughts on “Litteras, the yet another! And Microsoft Exchange situation.

  1. Nassos Kourentas says:

    – Do you want EWS support using Qt technologies? Yes, PLEASE!
    – Want a library to make it easier for KMail use it? Yes, PLEASE!

  2. TheBlackCat says:

    Nice!

    One correction, though: KMail does not store mail in a MySQL database, are any other sort of database for that matter. This is a common myth but it simply isn’t true.

      1. No one is stopping you from developing what you want (it’s FOSS, and the lib part is good), but would you please at least avoid spreading further misinformation? I’m having already a hard time with it when doing support for KMail and Kontact in the KDE Forums (also given the fact that only seldomly the KDE PIM developers let out what they’re doing).

        The data are kept in a data-dependent manner. In IMAP disconnected mode, they are files (one file per mail), no need to keep them in the database. Same for contacts (they’re kept as vcards). So, not quite “the same thing”.

      2. dantti says:

        Ok, did a quick update on the text, I just hope this clarifies a bit the case. I tried my best to express it’s my personal taste, but sometimes expressing it is hard.

    1. If you look at the context the misinformation is used to jusitfy a decision, so it wasn’t used by mistake but on purpose.
      We don’t know why it was deemed necessary to justify the decision, but using false claims regarding alternative approaches is a common technique. Sad, but expected

  3. Hi!

    I just want to say that “Literas” (one ‘t’) in spanish means “Bunk beds” xD

    Good luck with your project, I’m sure it’ll be useful!

    1. dantti says:

      Ah the word Litteras is from latin which means letters (following the Caligra hype :P), I bet there might be even more weird meaning than “Bunk beds”, just hoping for not seeing some dirt word meaning…

  4. We already have two EMail applications in KDE. I really don’t understand why you want to write yet another Mail Client UI. You have your reasons against KMail but what is wrong with expanding Trojita?

    How are you planing to maintain Litteras after the two month? There is little worth in an email client that only gets little attention by a single developer. This does not sound like a sustainable project to me.

    That said I’d really like to see that library.

    1. dantti says:

      Well I had a talk with Trojita developer some time ago, and he explained to me that if I’d like to use his IMAP implementation I’d need to use Qt models, which to me isn’t something the can be shared between different projects, a model is just too personal to a project, and I’d want to change too much stuff in it that I don’t feel would be welcome all the time.
      After the two months if it’s good enough for messages/contacts I’ll evaluate POP3 and IMAP, if too much work I might consider another campaign. And when that’s all done, I’ll just put into a maintenance mode where I care most about bugs, since I’m personally interested in having an email client that works with the protocols I need it will have it’s maintenance granted, as I do with my other ~10 projects I maintain… This also doesn’t mean someone willing to improve couldn’t come in and implement features, in the old KMail 1 days I’d probably try to push the feature to it instead.

      And as said the library is there for other projects to use, right now it already features AutoDiscover, SyncFolders and SyncFolderItems, now I want to add rename/delete/create folder and then try to get the message body.
      Best.

      1. Trojita developer here. I spoke with Daniel a couple of weeks ago and his opinion was that the Qt’s MVC API is not the way to go. Our opinions differ in this matter; I find it a reasonable abstraction — definitely better than trying to bake one’s own. It turns out that IMAP is in its nature very asynchronous, so it makes a lot of sense to use an asyncrhonous API like QAbstractItemModel for that.

        I don’t understand the QML bit in the comments — in Trojita, we support both QtWidgets and QML based GUI and both are powered by the same model under the hood. Yes, there are GUI-specific and QML-specific proxies, but that’s really *the* way to use the Qt’s models.

        That said, good luck to Daniel with his project. Trojita’s IMAP code is open source and patches for making it more usable as a library are very welcome.

      2. dantti says:

        It’s not QML in itself that is a problem, I remember you saying you have QML and QtWidgets GUIs, the thing to me is that whenever I do a model, I expose the information I want to expose, and sometimes depending on how huge the model is exposing all information might slow things down. So I better stick to the argument I don’t feel comfortable to using an API that is a model.

        libews-qt is totally async and still not a model, IMHO you find it easier to design it based on models, I don’t. I have a hard time designing models since I never know if the way I’m doing is going to work well. An example is Apper package model, I have changed it several times to find the best performance I could as it’s often a list with more than 1k entries, and I guess the same would happen with emails, maybe your models already perform well, I’m not personally comfortable with it.

        That said it’s really hard to state that sometimes the reasons are a matter of taste😛

      3. Well, I use mailboxes with tens of thousands of messages in them on a daily basis. The Qt’s MVC API is not a bottleneck — you can perform lazy loading without any problems.

        Be sure to use whatever fits your purpose — I simply disagree with your “models cannot be shared between projects”, which is not true, technically, and I wanted to point that out.

        I’ll be first to agree that writing an efficient and huge model is not trivial, btw.

      4. dantti says:

        Yes, technically it is not true indeed. Anyways nothing is written in stone maybe in future I might use your IMAP implementation or maybe we can get to some agreement on where it makes sense to merge the two things. Right now I find easier to write everything from ground up, it’s not just about the effort but the flexibility it gives to try different approaches without breaking user’s workflow.

    1. dantti says:

      Yes, I’m talking to them to evaluate what could be reused as right now they only started working on the calendar part. So we can have a win-win situation..

  5. jstaniek says:

    I noticed the lib is GPL. Could you use at least LGPL? As the rest of KDE and Qt? Thanks.

    Also the Qt models is definitely a thing to share between projects. And please do not mix UI and the lib is not needed…

    1. dantti says:

      Part of the code is GPL but I’m updating the files as I can.
      And well a model is definitely not the thing to share. How can I mix a model with another model, yes it’s possible but much more work than just calling functions and do what I want. It also represents how one expects data, so if I want a model for a QML thing for instance I’d need to write a model over another model.
      The lib doesn’t even link with GUI code.

    1. dantti says:

      Sure, I’m very inclined to OSX Mail, but there are some stuff I don’t like and want to do differently…

  6. I’ve given up and actually used OWA directly. In fact I’ve also given up on using Mac Mail or Outlook on Mac as well and just OWA. Firefox seems to support OWA very well on Linux for some reason, but Safari on Mac is great. I’ve given up using mail clients for Exchange (except Outlook on Windows).

    1. dantti says:

      Well, the problem with any web access is the web access itself, as the internet might fail, servers might fail, so it’s really handy to have a local copy. Not to mention the speed while searching…

    1. dantti says:

      Hmm I’ll take a look, I really looked at a lot of libs, there is even a QtSOAP in QtSolutions, but that has so many limitations that I ended up sort of building my own.

      1. Weird, Services.wsdl sounds like it would contain service tags.
        Can you mail the file to me or David Faure?

  7. I also would like to see Exchange Support in KMail. Until now I connect via IMAP to my company’s exchange which works quite well, but without calendar support which is sad.

    For me, calendar support would be more important than email.

    So long

    1. dantti says:

      Ok, I believe the doing the calendar part of the library will be easy, and there is an akonadi resource working on that, so if we figure out that sharing this lib would work well would be nice to see it as a resouce there. Surely it will be easier/faster to see this in KMail first…
      I for myself don’t use the calendar but I know many people do.

  8. TimDrub says:

    Although not reslly obvious when going to the site DavMail also supports EWS and I am using it quite succesfully with Kontact for Calendring and Contacts. I use iMAP for emails and I’d say focussing on non IMAP stuff (OoO config, server side rules, calendar sharing, etc.) is much more important to get Exchange users away from Outlook.
    Anyway, my dream is a native EWS connector for Kontact, so yes, please work on this.

    1. dantti says:

      Yes, but as my server doesn’t allow IMAP for emails, DavMail (which is the one I looked and thought it was too complex :P) doesn’t seem to be an option for me. Or does it grab mails too? anyway even if it does, the architecture is quite complex to setup for a regular user IMO.

      1. Tim Drub says:

        Yes, it does. It supports POP, IMAP and even SMTP and translates it into EWS. I found the setup straight forward although I am probably not a good reference.
        In the end its a local “proxy” kinda thing that does protocol translation.

      2. dantti says:

        Nice, they should probably make the website sell it as an easier thing. Of course that would never be as simple as just entering email/password on just one application.

      3. It could probably be made easy if they add a way for normal users to add configuration, e.g. a D-Bus interface.

  9. Alex Bremer says:

    I can really understand why you do not like KMail/Akonadi, it is far from being perfect. However, KMail finally got usable again after all those years and I can not see how one single developer will be able to code a usable e-mail application within a decent time frame. No matter how good you are in application development.

    If you should ever change your mind and implement an Exchange backend for Akonadi, I will be the first to back your project on Indiegogo. However I can’t support you current goal, sorry. There are already two mail clients out there and both are far from being perfect. Starting another one will not improve the e-mail situation on KDE in any way.

    1. dantti says:

      That is why I decided to make a library together with the client, once the library is complete writing the Akonadi resource shouldn’t be hard, actually there is already an effort to talk with EWS in playground but the developers where having some issues where I hope they can benefit from libews-qt so both clients can have EWS support.
      And as I said I might even do an Exchange backend for Akonadi but I’d only do that when the lib is done and if nobody does this first. I have used KMail for quite some time, changing it to my needs would mean I’d need to go into code I don’t know and dealing with Akonadi which is something I don’t like, the same way I don’t feel willing to write a GTK app, tho I have to read the code of lots of them all the time.
      Best.

      1. Alex Bremer says:

        I can see your motivations as a developer, but a library is not something which is usable for me as an end user. A library first has to be integrated into a project, which again needs someone willing to do it. As an end user I’m more than willing to donate money, but with your current goal I see no benefits for me. A library is just an abstract thing which might or might not help me in the future with my workflow.

        Maybe your work will be the foundation for many great things to come. However as an end user I’m more focused on solutions I will benefit directly from. A library is just too abstract, there are so many libraries out there for so many things, but not enough developers who are willing to integrate them into the programs I actually use as an end user.🙂

  10. Mark says:

    I like the general idea of having a fundraiser to solve problems which affect users and which just no one has the time to fix. It is better than just helplessly waiting and waiting. For this problem I do not have a need, but if there would be a fundraiser for fixing some open Kmail bugs (which still make it unusable for me) or my favourite KDE Printer wish, I would be willing to donate.

    1. dantti says:

      Hmmm, what is your favorite KDE Printer wish? If you are talking about print-manager I managed to narrow the bugs down to 2, if you are talking about KDE print dialog maybe I could take a look and maybe give a hand but I’m not responsible for that part.

      1. Mark says:

        It collects print jobs, allows to remove pages, rearrange them and creates a new print job out them. This is very useful, when I do research for university or a project on the web. Printed web pages often contain overhead, also it makes often sense after collecting to rearrange content in way that it makes more sense or to add other documents which I have locally. Finally I can merge them by printing into a pdf or directly print them as a handout. At the moment I either fireup Windows (which is a hassle, I am so used to KDE/Kubuntu/Linux) or do it all a bit more manually with PDFchain, which takes much more time.

    1. dantti says:

      Pretty hard, the problem is that the library contains code to create and parse EWS SOAP messages and it also has the means to send/receive them, one could make the creating/parsing toolkit agnostic (notice that I don’t use the GUI part of the toolkit) but using Qt DOM classes make it much easier.
      You could also use gSOAP which has nothing to do with GLIB, but that doesn’t like KDE code and didn’t want to link with my stuff…

  11. Hey,

    I’m glad you focus on the library part – honestly, writing a new mail client & infrastructure, well, you might have read my blog about that subject: http://blog.jospoortvliet.com/2013/05/on-innovation-free-software-nih-geary.html

    The argument I make is that any decent mail client infrastructure will end up looking like Akonadi. You can’t store a mailbox with 20.000 mails as separate files on disk and expect that folder to open in a few miliseconds so you will create some kind of cache. And you want to be able to search the files – you already said you’ll use Xapian, Akonadi uses Nepomuk for that.

    Of course, building your own stuff is more fun, and as long as it is meant for a limited usecase, it will probably do fine. But if you want it to be usable for serious use, you WILL end up writing Akonadi again. And to paraphrase what Aaron said: are we trying to make better Free Software mail clients or are we just creating loads of them?

    Anyway, have fun with it in any case – it’s good to get EWS support in a lib, and eventually, in KDE PIM.

    1. dantti says:

      Yes, I saw you blog post back then and I did a small rant about it on G+😛
      Thing is the idea of Akonadi is great but I strongly disagree with it’s design.

      I don’t expect to open 20k files and have performance, but that’s what Xapian is
      for, I index the files and use the index to present the information to the user,
      this will be blazing fast and have an indexed search for free.

      If Akonadi was about apps providing plugins so I could expose the information to
      it using the storage I like the most then I’d like it. I don’t like idea
      of not being able to try something different for my use case. MySQL is no
      good for all use cases, it might be good for some but not all, which is
      the reason I want a yet another mail client Akonadi free.

      If everything goes well with the lib and nobody get’s interest in adding an
      akonadi resource I might do it myself as I understand there are several users
      that like KMail and could benefit from it, right now I find easier to understand
      the EWS protocol building my own toy.

      1. “If Akonadi was about apps providing plugins so I could expose the information to
        it using the storage I like the most then I’d like it.”

        Just to clarify for other readers: Akonadi is about providing access to data in whatever storage one would like to use, but it is not using in-process plugins but out-of-process helper programs called agents. The main reason is robustness, one faulty “plugin” shouldn’t be able to take all others down.

      2. dantti says:

        Right, I always miss a point😛
        The issue I have is that instead of directly accessing the data, the application uses the cache of it. By having plugins I meant that if some app wants to retrieve information it could go to the source of the data instead of a cache of it, by using the a plugin which can manage the data in what ever format it in.
        So say my information is on Xapian, instead of caching it on MySQL (or any other backend), the plugin would retrieve the information from there, so there wouldn’t be a need for syncing the cache, and the embedded mysql (or whatever other option there is), but this would mean changing the whole akonadi design and I’m sure there will be people disagreeing with this approach.
        Whether it’s better or not will probably depend on how one looks at the issue. I’m having hard time to express myself on this…

      3. No offense, and I might be mistaken, but I always have a strong impression that invariably, when people don’t like Akonadi, they don’t understand it. Certainly it is complicated and perhaps not explained terribly well – and maybe you DO really understand how it works and why it was designed this way.

        If so, sorry for saying this, but I really think that if you never talked to one or more of the Akonadi folks in real life about Akonadi and it’s design, I think it is fair to assume you don’t really understand it.

        As Kevin just said – Akonadi is independent of storage (does not have to be mysql, as you know it supports postgreSQL too) and could probably use anything…

        Usually, people consider Akonadi to complex. Ye, as the saying goes, one should keep things as simple as possible- but no simpler… I think that describes Akonadi quite well: simple, but not too simple.

        Last thing: I am not a developer, so I can only rely on what other people tell me. But I never met somebody smart who really looked properly at the Akonadi design and said anything else than that it was a good design. Now, you might be the first – provided you can convince me you had a good look at it😀

      4. dantti says:

        Yes, I might be completely wrong. But I’ve read a bunch of docs about it, and see the flows it has, and how things are handled.
        A big problem for me when it comes to a discussion is to put into an understandable way what I want to point out.
        I do understand the need for akonadi, and I do understand the complexity, but I disagree with having a cache of information, and especially in SQL. I like SQL but I feel it’s not the best storage for this information.
        Imagine for a moment if Nepomuk had plugins that indexed your appointments on it’s index based on a plugin that goes and read a bunch of task files on the disk. If you search for an appointments tag you then get all results you want, and still didn’t need akonadi. Now you want to open the appointment, and IMO nothing better that the right application for the task which is the one which created that.
        Then you can say that akonadi will fetch my mails when KMail is not open, true a good thing, still it could be achieved by daemon that spawns a new process for your account plugin (which is what akonadi does), but instead of caching it warns nepomuk.
        I just hope you understand my points, again I’m not trying to question if my design is better or not, but my personal opinion on what is the design I alone (or not) would like.

      5. Well, clarifying a couple of more things🙂
        Caching in SQL is an option, by default limited to data items of 4KB or less, threshold configurable of course.

        Whether to cache or not is also configurable, on a “per-folder” basis (every Akonadi collection has its own cache policy).

        Direct vs. indirect access: direct access does not work reliably if multiple clients access the data potentially simultaniously.
        The KResource framework tried that. File locking is a mess (highly depending on filesystem capabilities), means a lot of complexity in the data access plugins which should only be doing data access, means testing a lot of combinations, etc.

        There is a reason why GNOME has already been using indirect access for years (EDS), it is the more scalable solution.

        Retrieving data through the index: most index systems don’t preserv the original content. Features like cryptography need bit-exact data.

  12. Emre Erenoglu says:

    Keep up the good work, I’m willing to see one day a usable backend to kmail/kontact that uses your library! I’m adding myself to this interesting discussion.

  13. blue says:

    I would be a happy man, if I could give up my solution to use the Exchange of our Company in Kmail via davmail.
    Wish you much success in your project!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s