Pegasus Mail and Mercury Developer News Archives
April 2017 - "How many WaPs is that?"
I'm sure many people must be wondering if I'm still alive... I have to apologize for being so uncommunicative for so long - as I've remarked in other posts, for someone who has put most of his adult life into trying to help people to communicate, I'm pretty poor at doing it myself. I'd love to tell you why I find it so hard to do regular updates, but I can't, because I'm not really sure. It just seems to be a task I find unusually difficult: a lot of the time, it feels like I only have dry, technical things to report, and I guess I just don't imagine my community being too interested in that stuff. Still, there's no doubt I need to work on my communication skills.
My community of users have to be the most patient people in the world: each week I get a few messages asking me how Pegasus Mail v5 and Mercury v5 are going, but they're never angry or impatient, always encouraging - and I have to say I find that amazing... I've been saying that both programs were "near" for almost as long as I can remember, yet my community seems to be willing to accommodate delay after delay with a steady good grace I find quite humbling. I hope, in this posting, to give you an idea of why these releases are taking so very long.
To start with, you'll need some background.
The first version of Pegasus Mail for Windows to send a message appeared back in 1993 - that's twenty-four years ago. Since that time, I calculate there have been fifty-six releases in various forms, each release building on the ones before it in some way or other. That's a lot of releases, each one representing many, many changes to the source code I write to produce the program. Over the years, that source code has become ever larger and more complex; furthermore, because I'm human, my style has grown and evolved over the years, and the way I wrote things back in 1993 is almost invariably not the way I would do them now, so a lot of the code exists in a mixture of styles that makes maintenance very difficult.
By 2010, it had become absolutely clear to me that I could no longer reliably maintain the Pegasus Mail source code as it stood - it had become too large and had too much history and compromise in it... Add that to the fact that Mercury had also grown large and unwieldy, and I realized that I was going to have to find some way to reorganize and unify the programs - come up with common internal components so that I was maintaining less code, and the code I was having to maintain was more modern, more consistent, more streamlined.
At the time, I thought I could do this as a gradual process - replace a piece here, a piece there, and the programs would progressively converge on a new, maintainable standard... But I was wrong - or at best, naive. I would start working on modernizing one part of the program, only to find that if I were to do the work properly, I could salvage almost none of the old code - I would have to write it anew. Then I'd find that rewriting that part of the program couldn't proceed until I'd rewritten another part of the program on which it depended, so the process would start once more - and so on and on, over and over again.
Although it certainly wasn't my original intention, what I actually ended up doing was rewriting almost the entire program from scratch: this wasn't a process of modernization, as I had initially planned, but a process of writing a completely new program using an older one as a model.
It can be hard to get perspective on this process - on how large the work really is... But over time, I've come to look at what I do in terms of units I call "WaPs" - short for "War and Peace": Tolstoy's "War and Peace" (Oxford World Classics edition) has about 71,500 lines of text, so I defined 71,500 lines as 1 WaP, and use that to measure the size of the work I do. At the moment, the totals are roughly as follows:
That means that the code I am working on is currently roughly the size of nine copies of War and Peace... That's a lot of lines. I further estimate that in the last four years, I have updated, added or removed around 200,000 of those lines of code (2.797 WaP), and although that's an estimate, it's on the low side if anything. The cold reality of these numbers is that the combined bulk of Pegasus Mail and Mercury is now beyond the ability of a single programmer to maintain and improve unless ways can be found to streamline the process as much as possible, and it's mainly this that I have been focusing on for the last few years. I have no intention of giving up on these programs that have been part of my life for so long, but I simply have to find ways of making them manageable.
So what have I actually done so far? Here's a potted summary, by no means complete:
The last two are so important that they deserve some elaboration: the object framework (which I call "OIF" internally) is the basis for all the new code, and will allow third-party add-ons to be first-class citizens, with the same access to the internals of Pegasus Mail and Mercury as the programs themselves: this desire to open up the "guts" of the programs in a consistent, effective manner that made it easy for people other than me to create powerful new features for them was a major goal of the redevelopment, and required enormous effort, although I believe the results are really good.
The new Message Store is an example of how difficult some of this stuff can end up being; I have been working on it for several years and in that time have ended up scrapping what I had done and restarting twice. The Message Store is the low-level code that implements the folders and messages you work with on a day-to-day basis, and I wanted something that provided a clear way forward to future capabilities without sacrificing compatibility with existing data; getting the right balance between past and future makes this process unusually difficult - had I been able to scrap everything and simply start from scratch, it would have been a far simpler process, but I value data compatibility above almost everything else: it is a matter of pride to me that a folder created by the DOS version of Pegasus Mail in 1990 is still readable by the latest Windows version in 2017, and will remain readable by v5, even though newer and vastly better folder formats will also be available.
Hindsight, as the old saying goes, has 20-20 vision, and in hindsight, I would have to say that I could have handled the process of updating and modernizing my programs considerably better than I have. Had I realized when I started what I was getting myself into, I would certainly have tried to find a different strategy, some way of making the process cleaner, more compartmentalized, and more inclusive of my user community... But in the end, the code would still have had to be written, and it was always going to take a long time no matter how I approached it.
I hope this rather over-wordy update (roughly 0.002 WaP in size) offers at least some explanation why the process of bringing out the next generation versions of Pegasus Mail and Mercury is taking such a long time: in offering it, I would also like to offer my thanks to my community of users, testers and supporters for their considerable patience and encouragement, and my hope that they will stick with me for just a little longer. I really do feel like there's light at the end now.
All my very best to you all,
-- David --
My, my! Time flies when you're having fun, and sometimes even when you're not, it seems.
The last time I updated this page, I was struggling with a nasty deadlock issue in Mercury; since that time, I have managed to fix that issue and bring out a public beta of Mercury v4.80, but the struggles didn't end there. The public beta exposed a number of other unrelated issues that it has taken time to fix, but we're finally there now, and Mercury/32 v4.80 is an official release, with a far greater new feature set than I ever intended it to have. I also believe it will be one of the best and most stable releases I have ever done: I am quite proud of it, and of the effort my test team has put in to it.
You might ask, why has this taken so long? Why should it be such a struggle simply to bring out a new version of a well-established, reliable program? The answer is mainly 'history'. Mercury was originally written in 1993 as a special kind of program called an 'NLM', for a now largely forgotten network operating system called 'Novell NetWare'. I produced the first Windows version of Mercury in 1995, using quite a bit of code from the original NLM program. Since 1995, Mercury has grown and changed continually, as living software does, and has had a profusion of capabilities and features added to it that are far in excess of anything ever contemplated in the original code.
The problem is that this accretion of features has been done on an underlying code platform that is now more than twenty years old, and the result has been an almost chaotic blend of code styles and practices, with workarounds for various versions of Windows thrown in for good measure. This means that it gets more and more difficult to maintain the program with each new version of Windows and each new feature that is added. About five years ago, the difficulty of this process reached an apex, where the time taken to add any new feature, even quite a small one, had to be offset by a disproportionately huge investment in time and effort in testing and validation. Since that time, I have ploughed enormous ongoing effort in rewriting and modernizing the code to remove complications, improve robustness and reliability, and to make it much more maintainable, but this is a huge job - more than 100,000 lines of code already, and still counting.
Then there's the problem of backwards compatibility: this is the idea that a data file you created back in 1993 should continue to be usable by versions of the program released in 2015. I have always tried to make backwards compatibility a hallmark of the work I do - for example, a folder created in the original DOS version of Pegasus Mail in 1990 can still be read by both Pegasus Mail for Windows and the Mercury IMAP server in 2015. But these levels of backwards compatibility come at a price, and the older the program gets, the greater that price becomes. There is a huge amount of code and effort in both programs devoted to supporting data files and formats that many people probably no longer use.
And of course, there's the size of these programs: they are now very large - each on its own is a full-sized commercial-quality software package, yet there's only me writing, maintaining and documenting them. To give you a point of reference, I have a friend who runs a successful company making software not much larger than mine, and he employs more than fifty programmers, graphic artists, documentation specialists, web developers and support staff. Of necessity, this imbalance of size versus capacity means that I can simply no longer work as fast or produce as many updates as I could when the programs were smaller and more manageable; it also means that I have to choose carefully the things on which I focus my efforts - and I have to be pragmatic about that: I will always choose to focus on things like reliability and real-world features over pretty interface skins because I believe those are the things that most benefit my users, even though neglecting the eye-candy aspect of the programs comes at a cost, since for many people that's all they really see when they're evaluating them. To me, it's one of the sadder facts of the modern world that many people are much more interested in how something looks than in how well it will serve their needs, but I guess that's just the way it is, and grizzling about it, while possibly therapeutic, is largely pointless. <grin>
There is some good news though... Mercury v4.80 is much closer to version 5 than it is to version 4.7 - a lot of stuff is in there but not currently exposed, and as a result, I can assure you that you won't be seeing an untoward delay in getting the first v5 builds out once the dust has settled on this release. It's a slow process, and it has been very arduous, but the results are near, and I believe they will be good.
Each year, the Queen of England makes a seasonal broadcast to her people around Christmastime, in which she gives the folk a good old pep-talk. A few years ago (I think just after Lady Diana had been killed), she used a latin phrase to describe the year past - she referred to it as an "Annus Horribilis", or "Horrible year". Well, this year has been one of those for me. In a case of life imitating Monty Python, every absurd thing that could go wrong seems to have gone wrong this year, and I will not deny that at times I have been skirting the abyss of desperation here in Dunedin.
The most lingering and destructive issue has been a nasty problem that somehow got introduced into Mercury at some point during the significant code rewrite process that was needed leading up to version 5. This problem manifests itself as occasional, sporadic deadlocks - situations where Mercury simply stops responding. It's not a crash as such, so neither the Mercury Loader nor the Windows Service Manager will restart it, and it just sits there doing nothing until someone restarts it manually. Of all the problems you could have in a piece of mission-critical software, this is just about the worst, so it was obvious that no release could occur until the problem was fixed.
Unfortunately, the problem is intermittent - it might take anywhere from twelve hours to a week to manifest itself, which made tracking it down extremely difficult. In the end, to simplify the task of debugging, I moved forward a process I had hoped to defer for one more release - I ported the code from the ancient Borland compiler I was using to a much more modern version of Microsoft Visual Studio. Although this took some time, it has given me much better tools to deal with the problem and will, I hope, allow me to bring out the v4.80 release I had hoped to bring out in May before the end of the year.
On the positive side, the delay caused by the deadlock problem has given plenty of time to test the completely rewritten TCP/IP code based on OpenSSL, and it's now working very well. A much-requested change, this will make Mercury's SSL support much more interoperable with other systems, and will allow you to use real SSL certificates issued by CAs. During the port to Visual C, I've also done a lot of work tidying up and modernizing the Mercury code - you may not realize it, but Mercury has been around for over twenty years now, and just as was the case with Pegasus Mail a couple of years ago, there was a lot of accumulated cleanup that really had to be done during the port.
For Pegasus Mail: The recent revelations about the unprecedented intrusion by security services into material that most normal people would rightly believe to be private has affected me enormously - I have found the whole "breach of faith" aspect of the saga incredibly upsetting. As a result, I have made a conscious decision to redirect my current work efforts in Pegasus Mail towards increased security and privacy. I am now adding strong encryption facilities to the program, with the particular aim of making them so easy to use that there will be no reason not to do so. The new Contact Manager, already a pivotal feature for version 5, becomes even more important when seen in this light: I have now found and implemented a fully-encrypted version of the SQL database I use for the Contact Manager, meaning that even if someone steals your contact file, it will be useless to them. I have also heavily beefed-up the provisions for storing encryption-related settings and certificates in the Contact Manager. In time, I will also support encrypted folder formats in the program, and we will finally have a truly secure mail client. Other work in Pegasus Mail has involved completely replacing the spelling checker with a full multi-lingual version based on the same spelling checker as used in OpenOffice and Libre Office, completely rewriting the program's TCP/IP module to use OpenSSL and fixing a host of small bugs. We hope to have an early edition of Pegasus Mail v5 available at least for public preview early in the New Year.
Finally, on a personal note, I feel I have to apologize for communicating poorly. It's kind of ironic that I got into this field because I was passionate about making it possible and easy for people to communicate with each other, but that I'm not actually that good at doing it myself. For the most part, it's simply a case of burn-out - I've been handling too much mail for too many years (nearly twenty-five years, in the case of Pegasus Mail), and I've reached a point where I rapidly become overwhelmed by even quite modest volume. This doesn't mean that my commitment to my community is any the less, though: I remain totally and utterly devoted to providing tools that fit the needs of my people, and to ensuring that they remain as relevant and capable as it is in my power to make them.
Well, it's been a long time between drinks, as they say. 2012 was a frustrating year in which I did a lot of work, but ended up without having a lot to show by way of releasable products. Things are rolling on, though, and I expect 2013 to be a year where many of the fruits of those labours reach the public view.
Pegasus Mail The main focus for Pegasus Mail during 2012 was to replace the aging addressbook subsystem with something much, much better. The existing Pegasus Mail addressbook code is now very old - some of it dates from the time of the DOS version of the program - and it has reached the point where not only is it practically useless in terms of real functionality, but it is very difficult to maintain. The replacement I have developed for it is a fully-fledged Contact Manager, with an incredibly rich and extensive range of data attributes and capabilities. It will tie deeply into Pegasus Mail and will provide new functionality only dimly imagined in past versions of the program. It also showcases a lot of development I have done on user interface, both in terms of my philosophy towards UI, and the tools I have written to create it. I believe it has the best user interface experience I have ever produced, and that it will significantly expand the capabilities of the program as a whole. As a teaser, here's a simple screenshot of the current state of the Contact Manager UI in action.
Other developments include replacing the entire spelling checker subsystem with a completely internationalized version using standard dictionaries (over 80 languages available at last count), and improving the performance of network-related operations (such as POP3 and SMTP), including migrating to secure networking code based on OpenSSL.
Mercury/32 Two things in particular have occupied my attention in Mercury development over 2012: the first is a migration to using the OpenSSL libraries in the program to handle secure communication. Moving to OpenSSL promises much higher levels of interoperability with other systems and makes it much easier to use "real" certificates (i.e, those issued by Certification Authorities). As part of this process, I have put a lot of time into making my TCP/IP code in general more robust and improving performance where possible. The second focus in Mercury/32 is the complete decoupling of the Mercury user interface from the worker code that actually handles mail. This has become increasingly important with more modern versions of Windows that do not allow service processes to expose any kind of UI, but was always going to be something I would have to address in any case. While a very large and complicated job, this work is progressing well, and several of the Mercury protocol modules are now converted to the split UI/worker process model. A side-effect of this work will be that tools people have been requesting for a long time, such as easy ways of managing mailing lists remotely, will become practical to write.
Interfaces Probably one of the areas where the most work has happened in the last year is one that no user will ever directly see - the internal interfaces Pegasus Mail and Mercury use to communicate and manage data. I've been working towards a very highly-integrated object-based interface for a number of years, and the last year has seen this work finally come to fruition. The Mercury inter-process communication abilities required to separate the user interface from the worker processes are directly supplied by this new object-based interface, which is now a fixture in both programs. While my user community won't normally be aware of all the work that has gone into this, it was a crucial step in making the programs maintainable for the future, so it was pretty important stuff.
MailStore MailStore, the replacement foldering management layer I've been working on for a long time now, has taken somewhat of a hiatus in 2012. The problem, as it often has been in the past, is the IMAP code for accessing remote mailboxes: this code has required a major rewrite, but is unfortunately based on internet standards that are very difficult, time-consuming and (dare I say it) soul-destroying to implement properly. I decided that other, more visible additions to the programs, such as the Contact Manager and the Mercury UI separation were more important in the short-to-medium term, so I have put MailStore on the back-burner while I work on those. Mailstore does, however, remain a key development goal for both programs, and will eventually find its way into production.
There's a lot on the horizon, and a lot to look forward to this year. I'd like to take a moment to thank those of you who have stayed with me and have generously donated in support of Pegasus Mail. I'd also like to break with the reserved, almost shy lack of self-promotional sense that makes it very difficult for me to ask for support and say that donations from my users are increasingly important in times as hard as these; I appreciate any financial help you feel able to provide and hope as many of you as possible will find a way to offer your assistance.
30 November 2011:
I realize that it's been a long time since the last entry on this page, and I apologize for that: the delay has come about through a combination of work overload and system migration. The work overload centres around the vast, long-term, ongoing overhaul of now-ancient code in both Pegasus Mail and Mercury, a process that will secure the future maintainability of both programs. In the course of this year, I have rewritten large swathes of the underlying code and have produced more than 50,000 lines of new code, including completely new IMAP client code for the new back-end message store, a full object interface that will be the underpinning of all future development, a comprehensive dynamic string library, and new code to use the industry-standard OpenSSL libraries for secured communications. I have also begun the laborious process of moving the Mercury code from the old Borland compiler I have used for so long to Microsoft Visual C: fortunately, having already done this in the larger and more difficult Pegasus Mail codebase, I expect to find the Mercury port easier and quicker. Clearly, then, there's a lot of work going on "behind the scenes": much of the time, this process feels like walking through treacle - the progress is slow, and it's frustrating having so little to show for the hundreds of hours of work I've done. The only consolation here is that the real benefits will start to flow quickly and clearly once I move to full development of Pegasus Mail v5.0 and Mercury v5.0.
In the non-development area, I have now moved all work on Pegasus Mail and Mercury to Windows 7, and have migrated the development and maintenance of this web site to Adobe DreamWeaver, which should make it considerably easier to produce updates to the site and to this page more frequently. Moving development to Windows 7 allows me to consider innovations such as 64-bit versions of the programs, and installation options that more closely follow the Microsoft-mandated directory layouts for software. I have not yet personally tried the Windows 8 betas that Microsoft has released, but members of the test team who have tried them have not currently noticed any problems with either program in the desktop interface of the new version, and should any issues arise during the beta period for Windows 8, I am now in a good position to be able to address them.
On the release front, Pegasus Mail v4.63 and Mercury/32 v4.74 are now in final testing. Both programs are maintenance releases that fix numerous small problems (Pegasus Mail has over 60 such fixes, Mercury around 40). They should be released around the first week of December.
20 February 2011: Pegasus Mail v4.61 has now been released. This is really an interim release – it falls between v4.52 and v5.0, but we felt it was important to bring it out to deal with problems in the way other programs are now insisting on formatting their HTML mail (essentially, sending the entire HTML message on a single unwrapped line... <grrr>). That said, there are also some powerful new capabilities in there, and the new look-and-feel gives an idea of what we are trying to develop for the program in the future. On the developer front, things have slowed down a little in the last month as we got v4.61 ready to release (releasing a new version is an extraordinarily difficult and time-consuming process, which is one reason why there are fewer releases these days than there once were), but the new contact manager code is now in its late development stages and should be going into testing shortly: I expect an initial public release containing the new contact manager and further UI improvements sometime in the next couple of months. Work on integrating OpenSSL is proceeding well, hampered mainly by the relative lack of documentation available for OpenSSL, but we expect to have it integrated into both Mercury and Pegasus Mail in the very near future. Finally, the new back-end message store for both programs (see below under the August 2010 entry for more on this) has taken a back seat to releases and other development for the last month or so, but will be back into full development now that v4.61 has been released.
2 November 2010: Progress continues on the new contacts manager for Pegasus Mail version 5. The new contact manager uses an SQL-based back-end for storage, and offers an incredibly rich range of attributes and options: it will exist alongside the now aging Pegasus Mail addressbook format, and while existing addressbooks will continue to work, there will also be options to import their contents into the new contact manager database. The new code has been designed heavily with external synchronization in mind, but at least initially, we will not be including any built-in synchronization clients: there is an extensive API that other developers can use to perform synchronization (down to the attribute level) though, and as real-world usage solidifies after release, we will evaluate what synchronization clients might be most appropriate to provide as standard options. We hope to release an initial version of Pegasus Mail version 5 with the new contact manager in place either late in December this year, or early in 2011. A subsequent version with the entirely-new message store I have been working on for over a year will follow at a later stage. I'm personally very excited about the new contact manager - I believe it could have an enormous impact on the way many people work in a world of e-mail overload.
On the Mercury front, we have begun the process of integrating support for OpenSSL into all the Mercury protocol modules. This should provide more robust SSL support, which is becoming increasingly important as the Internet grows. We are also investigating what will be necessary for Mercury to exist in the IPV6 world - we do not expect this to be a large issue, and believe we can have IPV6 support in place well before most sites have any real need for it. Once the OpenSSL transition is complete, there will be a new release of Mercury with that code in place, probably in December 2010 or January 2011.
31 August 2010: Welcome to the new developer news page! I'm conscious that there can often be quite long gaps between releases for both Pegasus Mail and Mercury, so this page is aimed at keeping you informed of what we're working on during those gaps. The intention is to update the page once a month or so, or whenever there's a significant development in the programs.
I have spent much of the last year working on the single biggest change to the programs since 1993 - specifically, a completely new message store that will be common to both programs. The Message Store is the "back end" of the program, responsible for organizing and controlling access to mailboxes, folders, messages and other mail-related data. It constitutes about a third of all the programming in Pegasus Mail, and about a quarter of the programming in Mercury, so it's easily the single biggest discrete chunk in both pieces of software. Having an industrial-strength message store that is common to both programs will hugely improve the reliability and maintainability of the whole system, and is a crucial part of the process of taking the programs into the future.
As well as the mail store, Pegasus Mail now has an entirely new suite of beautiful user interface graphics by Sven Henze, the development of which was funded entirely by the Pegasus Mail Thousand - thank you all! I'm also working on a new contact manager to replace the aging address book format - the intention is that the contact manager will be as good as or better than anything else in the industry. Finally, Michael in der Wiesche has developed a new HTML renderer for Pegasus Mail that uses the rendering power of Internet Explorer to produce top-class HTML displays. The new renderer will be shipped with the next release as a built-in option - it will still be possible to use the existing renderer, but for higher-grade rendering, the IERenderer can be enabled with a single click. We expect to release an initial Pegasus Mail v5 version with these features before the end of the year, and another version with the new message store early in 2011.
Mercury is undergoing extensive internal redevelopment, with the primary initial aim of separating the user interface from the core transports. The aim is to make the server run as a true Windows service all the time, and have configuration and console monitoring done either over HTTP or using a dedicated remote app. We're also redeveloping the way Mercury handles domains, to allow true multi-domain and MX domain support, and the most important long-term goal is to have comprehensive, well-integrated webmail as a core part of the program (this, in fact was the primary driver behind the message store redevelopment).
That's all for now - watch out for more updates over the next few months.
-- David Harris --