… And then, everything got faster. Analysis to follow the changelog.

daemon 0.9.2

• Inoreader Sync. The sync-inoreader.py plugin landed, allowing you to synchronize with Inoreader. It requires the python3-requests package to be installed (most distros have a package for this). It also requires a real Inoreader account (not an OAuth Google/Facebook login). The details are given at the top of the plugin file. I enumerated some of the trade offs in the last post, but in short the plugin tries to give you access to the most items, so when items show up only in Inoreader data or the data canto gathered but not both, they’ll be displayed even though they’re not synchronized. This also means that if you’re synchronizing multiple canto-daemons with Inoreader, some items won’t be synchronized. For “perfect” synchronization of multiple cantos, sync-rsync.py is the better option.
• XDG support. The default location of canto files is now $XDG_CONFIG_HOME/canto/ (which is usually ~/.config/canto). This is only relevant for fresh copies, if ~/.canto-ng exists, it will continue to be used.
• Feed file format. This has been converted to a gzipped JSON dump. The reasoning for this is twofold. First, the daemon uses basically no DB features, except the caching (which really amounts to the database code just holding everything in memory for our usecase) and yet suffered from having to manage database code (like requiring reorganize() on GDBM) and deal with incompatibilities between distros. Second, a gzipped JSON file not only takes far less disk space, but is also platform agnostic – versus the Python shelf that uses Python-only serialization techniques. Old feed files will be migrated on the first use of the new version.
• Protocol changes. The protocol to communicate between daemon and client has changed. Instead of using fragments of data and searching for a message terminator, a leading 8-byte header has been added with a size in bytes. This was a relatively minor change, but it means that we don’t suffer from messages getting stuck in the buffer waiting on a read to timeout and, as a bonus, don’t need to worry about fragmented messages. In addition, ITEMS responses are now always in single blocks rather than 100 item pages. ITEMSDONE is still sent, although obsolete.
• Fetching is thread limited. By default, the daemon will only spawn a fetch thread per processor core. In the end, this turned out to be more of a memory issue (since the Python heap will expand and basically never contract) than anything, but obviously having a thousand threads waiting on two cores is a waste of time.
• canto-remote status This remote command can be used to query item counts, as you might use in a status bar. See canto-remote help status for more info.
• filter_read is now default.
• Performance. Various changes were made to increase performance. Chiefly, the feed index function was refactored such that global transforms are applied on tag changes instead of “on-the-fly” when responding to an ITEMS response.

curses 0.9.4

• Theming. The appearance of canto-curses is the same, but it’s defined in Python directly now, instead of a thousand characters of ternaries and escapes. It’s much clearer now, and as such there is a theme-default.py that functions as a plugin and can be modified to your tastes.
• Color system. The color config has been shaken up a bit. Now, all available colors (1-8 or 1-256 depending on your terminal) will be initialized to that color on a black background. You can still change color codes directly, however :color can now be used to change specific colors by element name. (i.e. :color unread green) instead of messing with gibberish color codes. See :help color for a list of elements you can change. On first run, canto-curses will attempt to migrate your old color scheme and it should work well for simple changes. If it butchers your colors, use :reset-config color to restore to default.
• Style system. Similar to :color, :style has been implemented that allows you to change the curses style (bold, dim, reverse, standout, underline) of a specific element. Also note that how these styles appear is entirely up to your terminal, so results may vary between them. See :help style for details.
• xdg-open is now the default browser.
• cleantitle.py is a plugin that allows you to strip content out of story titles. From annoying newlines to HTML fragments, this can help cleanup content from feeds that are poorly defined.
• Tab completion tweaks. Works more like Bash now.
• Update style: prepend. You can now use “:set update.style prepend” to get new items to be added to the top of the feed on update. The other options are “maintain” for sorting, and “append” for adding to the end.
• :help set will now list various common options and their settings / uses. It’s entirely static (i.e. won’t show current values), but should help to familiarize the user with some of the lower level options.
• Performance improvements. Various changes have been made to speed up many parts of the code. It still isn’t perfect, but a lot of larger operations are broken into smaller pieces, functions have been tweaked to run faster, and commands modified to be smarter. In my experience, after everything is loaded, canto runs very well, so I’ve put in a lot of effort to minimize the annoying pauses caused by loading feeds with massive amounts of post-filtered items. The initialization process has also been significantly reworked to make more sense.

All in all, a healthy changelog for a dot release. Subjectively, the performance has improved quite a lot, and the feature list isn’t too shabby either. I am a bit concerned with the invasiveness of some of the performance changes, but they’ve passed the (admittedly anemic) test suites and seem to be rock solid from my personal use. In other words, it’s time to push it out there and see how you guys break it.

I’m particularly surprised at how simple some of the performance changes were. A lot of the algorithms I used were naively implemented, particularly when trying to sort items into new, current, and old. That has to be done with linear complexity, because once you break into O(n^2) you’re fucked as soon as you get more than a handful of items. Other places it was the classic “cheap operations are expensive”, for example I was trying to cleanup the entire hook stack every time a hook callee was unregistered. This mean that a single story receiving die() and unregistering itself could cause a search of 1000s of possible hook callees. Yikes. Consider killing whole tags at once (say because of a config change filtering most of them) and you get into some seriously computational load for what should be almost instantaneous (like it is now).

On the other hand, I’m pleased that I was able to get a lot of the lead out of the system without causing any incompatibilities or other headaches.

Anyway, it’s not perfect, but there were too many changes pending release to delay any longer. If there are serious bugs that need to be ironed out, well, 0.9.5 and 0.9.3 aren’t exactly big deals.

Debian repos will be updated shortly.

Have fun!

Submit bugs!