13 comments

  • vamega 1 hour ago
    I really hope Fastmail implements the JMAP spec for calendars and contacts soon. They’ve had the mail part of the spec implemented for a while, but it still requires CardDAV/CalDAV for contacts and calendar access.
  • 9dev 3 hours ago
    While JMAP seems to scratch every itch of a sucker for proper web API design, I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP. Is there really any new-ish binary protocol these days? Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO. Then again, a lot of solid thinking went into these things, so there probably are a lot of good reasons that I’m not aware of.

    Still, it’s an interesting space, I think.

    • SchemaLoad 0 minutes ago
      How much overhead do you think fetching emails over http takes? Fetching text documents with HTTP seems like the prime usecase for it. And it's the only way you could have something that works in web clients.

      I'm struggling to think of any real benefits to not using HTTP other than it would be more interesting.

    • WorldMaker 2 hours ago
      > binary protocol

      Email was never a binary protocol. Notoriously so, it's why MIME types and MIME encodings get so complicated.

      Most of the "old internet" protocols (email, FTP, even HTTP itself) were bootstrapped on top of built-mostly-for-plaintext Telnet. HTTP as the new telnet has a bunch of improvements when it comes to binary data, request/response-based data flows, and some other considerations. HTTP/3 is even inherently a binary protocol, it's lack of "telnet-compatibility" one of the concerns about switching the majority of the web to it.

      vCard/vCal/iCard/iCal were also deeply "plaintext formats". JSON is an improvement because it is more structured, even more efficient, than those predecessors. JSON may not look efficient, but it compresses extremely well and can be quite efficient in gzip and Brotli streams.

      I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet", even if it doesn't sound like "binary protocol efficiency" at first glance. Especially as HTTP/3 pushes HTTP more efficient and more "binary", and arguably "more fundamental/basic" with HTTP/3 even taking over more roles in the TCP/UDP layer of the internet stack. (Telnet would never try to replace TCP.) HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of. Sure, it would be neat to see more variety and experiments outside of the HTTP stack, too, but HTTP is too useful at this point not to build a bunch of things on top of it instead of as their own from-scratch protocol.

      • zzo38computer 25 minutes ago
        > Notoriously so, it's why MIME types and MIME encodings get so complicated.

        I made up ULFI because I thought MIME has some problems.

        > JSON may not look efficient

        Efficiency is not the only issue; there is also the consideration of e.g. what data types you want to use. JSON does not have a proper integer type, does not have a proper binary data type (you must encode it as hex or base64 instead), and is limited about what character sets can be used.

        (Also, like other text formats, escaping will be needed.)

        > I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet"

        I think it can be, depending on the specific use; sometimes it isn't, and will make it worse. (HTTP does have the advantage of having URLs and virtual hosting, although I think it adds too much complexity more than should be needed.) However, I still think that DER is generally better than JSON.

        > HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of.

        I think it depends on the specific application. However, even then, I think there are better ways than using HTTP with the complexity that it involves, most of which should not be necessary (even though a few parts are helpful, such as virtual hosting).

      • 9dev 38 minutes ago
        It’s not that I cannot appreciate the improvements in the space, I’m just wondering if there might be a big part of the design space for widely used protocols that ends up unexplored because the default for almost anything now is HTTP. It has basically become OSI layer 8 at this point.
      • p_l 2 hours ago
        A lot of the textual nature of older IETF protocols, including the CR LF line endigns, can be probably traced to how easy it was to bang out a bad implementation full of subtle problems that could be debugged by sitting an undergrad student at a teletype instead of spending time on having some binary serializer (that telecom companies definitely had money for)
        • JoshTriplett 2 hours ago
          Yeah, a fair bit of email protocol reeks of "is this tolerant of `telnet mailserver 25` and whatever garbage that might produce".
          • p_l 2 hours ago
            A lot of old RFCs explicitly mention running on top of TELNET.

            Additionally, as much people like to harp about "telcos focusing on connection-oriented protocols while we ran loops around them with packets", the reality is that NCP and later TCP pretty much focused on emulating serial lines around, and one of the earliest ways to access ARPAnet outside of machines directly on it was through calling into a TIP which set up bidirectional stream from your modem to a port on some host.

        • cyberax 2 hours ago
          It's also a reflection of the state-of-the-art at the time. Binary protocols were an unmitigated disaster, with standards bodies thinking that applying to the ISO for your organizational ID is a perfectly fine step that anyone does anyway.
      • p_l 2 hours ago
        Another point is that the use of HTTP for everything, outside of the issue of middle boxes breaking protocols for everyone, is that it's essentially capitulation to the wisdom of OSI multi-layered protocols - we replicate their feature sets by reusing bits and pieces of HTTP spec all the time.
      • frumplestlatz 51 minutes ago
        HTTP is also a large, complex stack for any server/client that isn’t already a web server or a browser.
        • zzo38computer 22 minutes ago
          Yes, this is the other issue with using HTTP.
    • zzo38computer 26 minutes ago
      > I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP

      It shouldn't. For some cases it helps, but other times it doesn't. Sometimes it helps but there would be better ways to do it, making it on a simpler protocol or making an entirely new protocol (which might or might not use TCP; sometimes it is better to use TCP and sometimes not) depending on the specific case.

      > Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO

      I dislike JSON. I think it has many problems, and that DER is a better format.

      (There are also the "small web" protocols such as Gemini and Scorpion and Spartan and Titan, which avoids some of the complexity of HTTP; I had considered using DER-over-Scorpion rather than JSON-over-HTTP. It is also possible to use SSH, although SSH does not have virtual hosting.)

    • wmf 2 hours ago
      I wonder if you could transparently upgrade to CBOR over HTTP/2.
  • sylens 2 hours ago
    We need better client support for JMAP. Apple Mail, Thunderbird, Outlook (as if), and so on. I'm surprised some of the smaller ones like Canary or Spark don't implement it as a product differentiator.
    • woodruffw 2 hours ago
      Serious question: what’s the differentiator if major email providers don’t support it?

      (This should not be interpreted as a defense of IMAP.)

      • mjrpes 28 minutes ago
        One big differentiator is JMAP allows one network connection to track new emails that may get delivered across different folders. With IMAP you need a connection open for each folder.
        • woodruffw 24 minutes ago
          Okay, that's a great feature! But guess I'm asking what the differentiator is if major email providers don't use it.
      • sylens 2 hours ago
        Fastmail supports it, and it sounds like Thundermail will too
      • dijit 2 hours ago
        I'd make a reasonable guess that it enables much better Javascript clients, either via Electron or the Web Browser.

        You don't need major providers to support it, they support SMTP and that's how messages are relayed. JMAP is just so you: the client, can fetch your mail from wherever you host your mail.

    • Eric_WVGG 1 hour ago
      We need better server support first.

      I’ve got a friend who’s been pitching me on building a new email client for years. “I’ll do it if we exclusively use JMAP.” “okay does that include Gmail and Apple/iCloud accounts?” “Nope.”

      I could sort of see dual-supporting Gmail's proprietary API and JMAP, but unless the #2-5 competitors support it… what’s the point? (sorry to put on the pessimism hat)

    • frumplestlatz 53 minutes ago
      There isn’t really a great motivating feature or use-case driving client or server adoption.

      To be honest, I’m not sure why end-users would want JMAP for e-mail access.

      It would be interesting if they do successfully roll out all of these additional RFC proposals providing a cohesive “groupware” protocol covering calendering, contacts, file shares, etc, we see notable server implementations, and interest is enough to drive client support.

      That’s a lot of “ifs”.

      • tjoff 50 minutes ago
        Because IMAP is horrible, it is another driving reason why we are moving towards the dystopian world of webmail.
        • frumplestlatz 48 minutes ago
          Horrible how, exactly?

          People say things like that, and I wonder if I’ve just been living in a gilded tower of using Apple Mail with decent IMAP server implementations.

          I’m also pretty familiar with the wire protocol and its implementation — it’s never struck me as particularly horrible.

          A new protocol isn’t likely to solve the problem of poorly implemented clients and servers — e.g. Google doesn’t really care about good IMAP support, so they’re unlikely to care much about JMAP, either. They just want you to use their webapp.

  • NoboruWataya 1 hour ago
    > They are robust, widely adopted, and battle-tested. Yet, their XML-based design is notoriously verbose, inconsistent, and difficult to implement correctly. Information is scattered across HTTP headers, XML payloads, and even embedded iCalendar data, creating endless compatibility and interoperability challenges between clients and servers.

    Can others confirm if these problems are widespread? I get that these protocols are probably a pain to develop for but given they are "robust, widely adopted and battle-tested" it seems that is probably a solved problem. It's better to have one standard that is used everywhere than to have to choose between two standards.

  • kridsdale1 44 minutes ago
    I worked on the iCalendar, CalDAV, and CardDAV parsers at Apple in 2010 or so, and I see no reason to believe that today’s Macs, iPhones, and Watches, are using anything more modern.

    I haven’t been there in more than a decade. I really am curious what the response in Apple (and Google) is to this spec.

    • frumplestlatz 36 minutes ago
      Of all of their proposals, this is the most interesting part to me.

      I researched what it would take to implement a full calendaring server once, and after reading all the RFCs, just backed away slowly from the whole idea and never thought about it again.

      • kridsdale1 18 minutes ago
        Yeah. It’s a super messy domain. You never know when the parliament of Brazil will pass a law changing the time zone definition.
  • btown 3 hours ago
    For those needing to deal with customers/clients/internal teams with Google Workspace/Outlook and wanting JMAP-style (though not JMAP) modern JSON APIs, Nylas might be a viable option: https://www.nylas.com/

    Nylas pricing has gotten better recently, but is still quite high though - at $1.50/connected account/month at scale, it's likely material to your per-user margin if it's part of your SaaS offering.

    But if you have a use case where this is a no-brainer (like capturing/analyzing/building custom real-time UI around your internal sales team's emails) then it's remarkably powerful.

  • lifty 3 hours ago
    It’s such a breeze to self-host your own email server using Stalwart. It has been a new era for email self-hosters like myself since these kind of fully integrated email servers like Stalwart appeared. Another good one but not as actively maintained is Maddy.
    • drdaeman 2 hours ago
      I'm setting up Stalwart right now, migrating from my current Maddy+Postfix+Dovecot+Rspamd setup. Not exactly my experience.

      The documentation is not great - I'd say it's just about barely enough to get an overall idea, but there's no one proper single definitive overview of what options exist, what are their possible values, what are the defaults, and how they relate to each other. Maddy docs, despite looking a bit sloppy, were a lot easier to get through. IMHO Stalwart makes it unnecessarily difficult to write a non-minimal static configuration file, hooking everything up correctly.

      To be fair, maybe there is a page like that but I haven't found it, despite trying.

      I know the Web UI allows to do the configuration by clicking through the forms, but this approach conflicts with declarative deployment practices. In my case it's giving me nondescript 500 errors in the UI with "Failed to write local configuration" in the logs because the .toml file is read-only.

      • audelair 2 hours ago
        Not sure if yours is setup different, but there are several key fields that need to be written to the config.toml file, and I've seen my file get updated when I make changes to the listeners or stores settings.

        But in general, I agree that it has not been a very smooth experience. Having messed around with maddy and mox, Stalwart has had quite a few gotchas. Despite being a single binary promising simplicity, I'm finding it to be a real challenge figuring out how it all fits together, and I'm mostly learning by trial and error since the documentation is often outdated.

        My biggest gripe is that it doesn't use the config.toml for every setting, or at least doesn't seem to have the option to do so. I broke my installation and had to find the posgresql key-value pairs for the settings, which was made harder by the fact that everything was stored as binary, which also made me have to edit it as binary as well. These were very simple settings that would have been a breeze in a flat configuration file. I absolutely do not like how necessary the WebAdmin is to manage simple things.

        That said, the integration with calendar/contacts is nice even without JMAP... Getting Thunderbird and Roundcube setup with plugins and proper settings made it so easy to get several users setup with calendars, contacts, and shared email-boxes and shared contacts right upon first login.

        The S3 storage is also working great (Hetzner Frankfurt VPS paired with AWS eu-central-1), and AWS downtime a few days ago notwithstanding, I'm feeling good about the reliability that gives me, leaving me mainly with the PosgresQL data store the main thing to keep backed up.

        This is a hugely ambitious software and as such, there will be many things that I will have a hard time getting used to as a hobbyist, but also a lot to be gained. I'm sticking around for now and waiting for version 1, improved documentation, and more clarity on how it all works.

        • 0x073 35 minutes ago
          Is there a reason you use aws s3 vs the hetzner object storage?
          • audelair 22 minutes ago
            I heard some second-hand accounts that Hetzner's object storage was pretty slow even in the same location. For email, that sounded not too ideal.

            Also, I only have 5 mailboxes right now holding less than 15GB of data total... S3 is still cheaper than the minimum at Hetzner since I don't need anything close to a TB.

    • zenmac 3 hours ago
      Yes, are there any decent JMAP web mail client that we can use?

      I have asked sooo many times since Stalwart first was introduced, but not got a straight answer. It is just FastMail or Topicbox. I want something like roudcoube or wildduck that can be used over https that I can self-host!

      • realityfactchex 2 hours ago
        It looks like Cypht [0] is the most actively maintained JMAP webmail client listed at [1], assuming that works.

          [0] https://github.com/cypht-org/cypht
          [1] https://jmap.io/software.html
        • audelair 2 hours ago
          I tried very hard to get it to work, but I simply couldn't get it to connect with my Stalwart instance over JMAP. I do have the permissive CORS and end-points and proxy-protocol seemingly working with my test HTTPS requests, and I also successfully got JMAP to work with the Mailtemi app, but no luck yet with Cypht[0].
          • realityfactchex 2 hours ago
            Darn. This looks like the open issue: https://github.com/cypht-org/cypht/issues/931
            • zenmac 52 minutes ago
              Yeah that is kind of issue had me flintch when thinking of using stalwart. As much as it is so nice to install it as a server and ideologies behind it. Looks like gonna just stick with wildduck for now. Just don't like to hedge our email bets on mongodb community edition.
              • audelair 3 minutes ago
                I do have to hand it to the developer though. This is some serious longterm commitment to an open standard that has simply never taken off beyond one company (Fastmail). Current JMAP implementation is pretty much nonexistent, and I am back to using IMAP/WebDAV with Roundcube and plugins with Stalwart. To me, this is an exercise in patience and waiting for an eventual payoff that may or may not come in the next two years. Having followed the project closely for over a year and gone through a few upgrades and followed the community, I'm still optimistic and happy to be along for the ride.
    • drnick1 1 hour ago
      But is there any real benefit over Postfix + Dovecot other than "it's new and written in Rust?" Postfix and Dovecot have been around for decades and respect the Unix philosophy of doing one thing and doing it well.
      • heavyset_go 1 hour ago
        It's one tiny binary that does everything you could possibly need for hosting a mail server, including an admin UI, and you get a bunch of modern and convenient features for free.

        For example, it automatically handles Let's Encrypt certs for you. You get JMAP, CalDAV, WebDAV, CardDAV, IMAP4rev2, DKIM/SPF/DMARC, MTA-STS, DANE, spam filtering, SQL+blob+object storage backends, search, clustering, OpenTelemetry, etc all in one tiny binary.

        Downsides: some features are gated behind an enterprise version and I think the dev team is one guy, or at least it was a while ago.

        Having ran both for a long time, I'm sticking with Stalwart from now on as long as development continues.

  • yyyk 2 hours ago
    I can understand why JMAP instead of IMAP given the latter's antiquated design. I don't see the advantage to clients in replacing WebDAV though, and the others are a bit iffy too. They'll need to make a way better sales pitch than 'JSON vs XML' (serialization ain't tough, XML is supported everywhere).

    I guess contacts/calendar follows JMAP naturally when the clients already implement it, but that only applies in the 'already wrote a JMAP email client' case. Virtually any other case would rather stay with widely supported protocols?

    • sureglymop 1 hour ago
      Yeah because everything already supports WebDAV. It works well with iOS and Android which is imo a big advantage.

      However, doesn't stalwart already also support WebDAV though?

  • woleium 2 hours ago
    I wish there was an easy auto-update process for Stalwart. is anyone hosting an apt repo for it?

    edit: we use it on very resource constrained environments, the container version is too much overhead.

    • reachableceo 19 minutes ago
      What resource constraints allow you to apt-get but not docker pull? It’s the same resulting stack isn’t it?

      I am most curious

    • evrflx 1 hour ago
      Where is the overhead in a container? It is just a regular process. (Ok plus a container runtime process, but that is negligible)
      • woleium 1 hour ago
        negligible for you, perhaps ;)
    • dijit 2 hours ago
      isn't it a static binary? Can't you do it the old-school sysadmin way and pull down a binary from github releases and update a symbolic link?
      • heavyset_go 1 hour ago
        They make breaking changes to settings (and possibly data stores, but I forget) between versions, so to go from, for example, x.y.1 to x.z.5 might involve doing migrations between x.y.2 through x.z.5 just to use the latest version.

        This is not the case for all versions, but I've found it to be common enough that I have to read all of the release notes between point versions when upgrading.

        It can definitely be improved.

      • woleium 1 hour ago
        yes, but that’s not as simple as apt automatic upgrades
        • dijit 1 hour ago
          want me to write the script out for you?

          1. systemd timer

          2. curl github api

          3. if new release, fetch, verify checksum

          4. update symlink

          5. restart service

          i don’t think repackaging is actually easier here, for main services of a system is ok to skip the package manager.

          • heavyset_go 1 hour ago
            You will eventually break your email setup by doing this, see my other comment.
  • matesz 2 hours ago
    Running Stalwart in production for ~20 heavily used accounts for some company and no problems so far! The simplicity for such a complex stack and flexibility of deployments is off the charts!
    • jdalsgaard 2 hours ago
      I second that; only running it for personal use on a few domains, but handles all the complexity _extremely_ easily.
  • apitman 1 hour ago
    I didn't realize JMAP had a file system protocol. I'd be very interested to learn how it compares to Solid.
  • pluto_modadic 4 hours ago
    if they pull out the AI stuff that'll be soooo cool :D
    • jasonriddle 4 hours ago
      What AI stuff are you referring to? I just learned about this project from this blog post, so I don't have the full context on their AI work.
      • antx 2 hours ago
        From the site [0]:

        > Stalwart Enterprise leverages AI technology to provide unparalleled email security and management. With AI-powered features, Stalwart Enterprise excels in accurately classifying spam, detecting sophisticated phishing attempts, and blocking various types of network attacks. This intelligent approach ensures that your email environment remains secure and reliable. Stalwart Enterprise comes equipped with a pre-trained large language model (LLM), offering robust out-of-the-box protection. Additionally, it supports integration with leading AI providers such as OpenAI, Anthropic, and other cutting-edge platforms, allowing you to enhance and customize your security measures. By utilizing AI, Stalwart Enterprise delivers a smarter, more efficient email solution that proactively safeguards your communications and data.

        [0]: https://stalw.art/enterprise/

      • doublerabbit 4 hours ago
        It seems the enterprise edition has AI features and the community version doesn't. So if you don't want AI, use the community version.

        https://stalw.art/compare/

    • lifty 3 hours ago
      Why does the optional (supported only in the enterprise version) feature bother you?
    • batisteo 4 hours ago
      Do you mean the spam detection algorythm or something else?
  • refulgentis 3 hours ago
    Anyone got a link to a better sales job on JMAP & friends?

    It sounds awesome but the way it is intro'd here:

      Over the past few years, the IETF has been redefining how email, calendars, and contacts are synchronized and shared. Building upon the success of JMAP for Mail, several new protocol extensions have been introduced:
    
      JMAP for Calendars - A modern replacement for CalDAV and CalDAV Scheduling.
      JMAP for Contacts – A powerful alternative to CardDAV.
      JMAP for File Storage – A replacement for WebDAV-based file storage.
      JMAP Sharing – A modern successor to WebDAV ACL.
      JSCalendar - A clean, JSON-based evolution of iCalendar.
      JSContact – A modernized, JSON-native successor to vCard.
    
    ...gave me pause. A protocol I've never heard even though I hang out here for an hour a day, was so successful, that it launched 6 new projects?

    Sounds more like the parts of the web dev that give me ick (new and shiny; rush to copy new and shiny in other contexts; give it a year; and all of a sudden only 1 of the 6 actually was successful)

    • WorldMaker 2 hours ago
      The big pitch for JMAP is for a modern web-tech-only approach to email/calendar/"groupware" servers. One reason to do that would be to make it easier to also build email/calendar/"groupware" clients entirely out of modern web-tech. Today most "web email clients" are bespoke to specific stacks/email servers. A dream of JMAP is that with the right CORS policy a single web client could interact with multiple JMAP servers, using only fetch/XHR.

      The modernization efforts of JMAP are interesting, too. Most of the old protocols are a mess of bespoke plaintext formats full of quirks evolved over decades in a giant mess of different software. Even the stuff that was already web tech like WebDAV and its extensions CalDAV and CardDAV were full of quirks, violated some REST "rules", and originally intended for a different purpose (file shares/FTP replacement). JMAP is much closer to "plain REST" than WebDAV's complex HTTP protocol extensions/changes.

    • ggm 2 hours ago
      You may only just have heard of them, but the WG goes back to 2017.

      https://datatracker.ietf.org/wg/jmap/history/

      Bron is the principal of fastmail, who now own pobox. This is a serious activity.

      • anoncareer0212 2 hours ago
        Counterpoint: I Google'd "jmap gmail" and a top result is a comment from HN in 2019 saying Gmail will never implement JMAP (it has not)

        That's a really cruel response, because this is important work. I don't want my kids beholden to bigco.

        I think it's real & important.

        I also wanna make sure people like me, who have to keep tabs on the intersection of "how can I help liberate from BigCo" and "how can I make a livable wage doing so"

        It is, quite literally, real, but also something you shouldn't waste time on if you're already busy. (c.f. https://jmap.io/software.html)

    • JadedBlueEyes 2 hours ago
      If you look it up, you'll see that JMAP is 6 years old now. It's a protocol for doing email (and now other things) over HTTP, without many of the legacy issues from IMAP and SMTP. https://en.wikipedia.org/wiki/JSON_Meta_Application_Protocol / https://jmap.io/index.html
    • candiddevmike 2 hours ago
      JMAP and friends are very niche, none of the "mainstream" email clients (that ship with most computers/phones) support it. So this feature being available is unlikely to grow the userbase, IMO.

      Now JMAP is quite a bit nicer to use than IMAP's API, but IMAP's gravitational field is too strong to be supplanted. IMAP is also becoming somewhat of a niche protocol, as the majority of users use vendor proprietary protocols for accessing their emails on Gmail, Outlook/Hotmail, etc. So why invest the time to add a niche replacement for IMAP when the entire protocol is a second class citizen to mainstream email clients.

    • SomaticPirate 2 hours ago
      Agreed, also not clear what this or why it matters. This is a new self-hostable email server basically?