canto-remote status double counting items in user/category tags
script.py plugin to support executing scripts to generate feeds (details at the top of the plugin)
- Add option to
sync-rsync.py plugin to defer initial sync.
I’m happy to say that this is the first release for the daemon in more than a year, but the oldest new commit in this release is only about two weeks old, so it hasn’t been lingering.
The fix, and the new plugin are self-explanatory. The new option to
sync-rsync.py is geared toward getting items to the client faster on startup (particularly noticeable when
canto-daemon is started by
canto-curses rather than on system/WM start). Rather than immediately trying to sync on startup before loading content from disk, allow the items on disk to be loaded and handed out to the client first. Technically this means “old” items can be seen again before the sync, but considering the alternative is seeing no items before that sync completes, I consider this better behavior.
Repos should already be updated.
Still need to write a post about future direction.
This will be the first release to charge patrons. Thank you!
Become a Patron!
Just an FYI that I was able to get the Debian/Ubuntu repos updated. Latest 0.9.9 curses build, and a rebuild of the already existing 0.9.6 daemon.
NOTE: I have dropped support for Ubuntus
wily since they’re officially unsupported and the vast majority of the repo traffic has been for
I have to give a shoutout to freight which greatly simplified the complicated task of taking already signed .debs and collecting them into a working repo. Converting to freight seems to have resolved the whole ‘weak digest’ problem with the repo, and I’m thankful that all of the GPG signing machinery is now abstracted from my build script.
Become a Patron!
Quick fix for losing symbolic keys outside of inputs after resize.
There will be a forthcoming post about how much curses/readline and Python are a volatile mix. I can’t seem to get Python/curses to detect further window size changes if you resize while the command line (i.e. readline) is open. Nothing crashes, but the window will be stuck at a certain set of dimensions.
Still haven’t looked at updating repos.
Won’t cost patrons a thing either. “Quick fix” should be synonymous with “oops, my bad”
Become a Patron!
- Fixed nonsense pending update numbers (gray numbers now reflect reality) (#40)
- Fixed some issues with exceptions / lockups during rapid resizing
- Fixed cursor being invisible and certain keys being uninterpreted (HOME, END) after resize
- Fixed some moronic selection fallbacks (#42, #45)
Once again, mostly pending commits from almost a year past.
Unfortunately I lost a disk with my script/chroots for building .debs, so it will take me a bit to upgrade them. I’ll take another stab at getting apt to shutup about weak digests too, but I’m hoping that creating signed .debs isn’t still such a pain in the ass.
I also want to point out a new header tab: Support Me, which links to my new Patreon account. I’m not 100% certain this is the right platform for donating to an open source project, but I have some other things in the works that might fit it a bit better and figured I may as well link this. My wife seems to think that people actually use and enjoy Canto, but I’m pretty sure that can’t be right. I fully expect to have no patrons, however I have spent the last 10 years hacking text RSS readers and haven’t received anything but the rare kind email as support. Not to mention this site isn’t crawling with ads, even unobtrusive ones.
Prove me wrong. If Canto has been useful to you, consider buying me a cup of coffee each time I make a release. Seriously, it’s like once a year…
Become a Patron!
Before you know it, it’s been almost a year since the last release and there are a handful of bugfixes that need to be pushed out.
- Fixed hanging file descriptors eventually killing the daemon (which fixes canto-remote misbehaving sometimes among other things)
- Fixed occasional hang from writing to closed sockets
- Better sync-inoreader behavior: killing unusable “???” items (me) and pushing changes back to Inoreader (thanks Fraterius)
- SIGINT and SIGTERM paths are now identical, where before SIGTERM attempted to exit ASAP without regard to running threads
- Daemon now dumps feed data into a temporary file and then moves it to replace the original data, to help prevent corruption if the daemon is killed in the middle of writing to the disk.
- Daemon now marches on if it can’t use the feed data because of corruption, instead of requiring the user to delete it.
- Minor Python 3.2 compatibility change (thanks Sathors)
- Fix tab completing empty command lines (closes issue 37)
I’ll get the Debian repos updated soon.
Administering packages for a distro you don’t use is a clusterfuck. Anyway, someone kindly pointed out that my Debian sid packages were still built for Python 3.4, and they’ve sometime since moved up to Python 3.5. Of course I’m oblivious to this on Arch.
That’s not the problem though. It seems like the Python package to .deb path is constantly having little tweaks made to it such that it actually takes effort to support new versions of a distro for no other reason than whatever tool you’ve decided to use has fallen out of favor. Ugh. Give me a PKGBUILD already so I can escape the million stupid Debian only binaries I need to put this shit together. Could this process be any more baroque? I literally have to use a special utility just to update the fucking changelog because its format is so locked down.
Then, of course, the version of GnuPG that Debian variants are currently using is different than the version Arch has so suddenly my actual
~.gnupg breaks the entire signing process. Which is just as well because I lost the key I used to sign the old packages when my last laptop drive went south, but regardless now I have a separate directory of keys that are only for Debian that I have to keep track of.
Anyway, the repos have been updated. They now include the latest Ubuntu variants (wily and xenial) but they’ve been obviously very lightly tested.
You will have to re-import the repo public key:
curl http://codezen.org/static/canto-pub.gpg | sudo apt-key add -
Yet more maintenance type fixes.
- Minor cleanup to excessive debug info and error paths in sync-inoreader
ItemLimit transform is now included by default, so you can, for example do “:set tag.transform ItemLimit(10)” to only show the first 10 items of the selected feed.
- Fix some old usage of tag_updater that caused weirdness like syncs happening when they shouldn’t and making your cursor jump.
- The above also fixed direct usage of :transform/:filter/:sort
- Fix waiting on pending configs (again) – hooks need to be called after the changes have been made.
- More fixes for sync-inoreader’s bad behavior with missing items (if you see ??? items that won’t go away, this is the fix)
- Socket changes to avoid deadlocks when both server and client sockets are full.
- Quiet exceptions on disconnects. These were mostly harmless, but scary in the log.
- Minor changes to lock returns
- Fix some breakage and deadlocking caused by forcing threads to wait for written config changes to be processed (e.g. tag config changes like collapsed were broken in 0.9.4). Ironically my test suite actually caught some of the more obvious breakage, if only I’d actually run the fucking thing.
- h/left and l/right are now bound, by default, to setting items read and unread respectively.
Short and sweet.
Thanks to everyone that submitted bugs.
Quick bugfix release for the daemon sync plugins.
- sync-inoreader: stop leaving dead items
- sync-rsync: update to new database format
Debian repos will be updated shortly.
… And then, everything got faster. Analysis to follow the changelog.
- Inoreader Sync. The
sync-inoreader.py plugin landed, allowing you to synchronize with Inoreader. It requires the python3-requests package to be installed (most distros have a package for this). It also requires a real Inoreader account (not an OAuth Google/Facebook login). The details are given at the top of the plugin file. I enumerated some of the trade offs in the last post, but in short the plugin tries to give you access to the most items, so when items show up only in Inoreader data or the data canto gathered but not both, they’ll be displayed even though they’re not synchronized. This also means that if you’re synchronizing multiple canto-daemons with Inoreader, some items won’t be synchronized. For “perfect” synchronization of multiple cantos,
sync-rsync.py is the better option.
- XDG support. The default location of canto files is now
$XDG_CONFIG_HOME/canto/ (which is usually
~/.config/canto). This is only relevant for fresh copies, if
~/.canto-ng exists, it will continue to be used.
- Feed file format. This has been converted to a gzipped JSON dump. The reasoning for this is twofold. First, the daemon uses basically no DB features, except the caching (which really amounts to the database code just holding everything in memory for our usecase) and yet suffered from having to manage database code (like requiring
reorganize() on GDBM) and deal with incompatibilities between distros. Second, a gzipped JSON file not only takes far less disk space, but is also platform agnostic – versus the Python shelf that uses Python-only serialization techniques. Old feed files will be migrated on the first use of the new version.
- Protocol changes. The protocol to communicate between daemon and client has changed. Instead of using fragments of data and searching for a message terminator, a leading 8-byte header has been added with a size in bytes. This was a relatively minor change, but it means that we don’t suffer from messages getting stuck in the buffer waiting on a read to timeout and, as a bonus, don’t need to worry about fragmented messages. In addition,
ITEMS responses are now always in single blocks rather than 100 item pages.
ITEMSDONE is still sent, although obsolete.
- Fetching is thread limited. By default, the daemon will only spawn a fetch thread per processor core. In the end, this turned out to be more of a memory issue (since the Python heap will expand and basically never contract) than anything, but obviously having a thousand threads waiting on two cores is a waste of time.
- canto-remote status This remote command can be used to query item counts, as you might use in a status bar. See
canto-remote help status for more info.
- filter_read is now default.
- Performance. Various changes were made to increase performance. Chiefly, the feed index function was refactored such that global transforms are applied on tag changes instead of “on-the-fly” when responding to an
- Theming. The appearance of canto-curses is the same, but it’s defined in Python directly now, instead of a thousand characters of ternaries and escapes. It’s much clearer now, and as such there is a theme-default.py that functions as a plugin and can be modified to your tastes.
- Color system. The color config has been shaken up a bit. Now, all available colors (
1-256 depending on your terminal) will be initialized to that color on a black background. You can still change color codes directly, however
:color can now be used to change specific colors by element name. (i.e.
:color unread green) instead of messing with gibberish color codes. See
:help color for a list of elements you can change. On first run, canto-curses will attempt to migrate your old color scheme and it should work well for simple changes. If it butchers your colors, use
:reset-config color to restore to default.
- Style system. Similar to
:style has been implemented that allows you to change the curses style (bold, dim, reverse, standout, underline) of a specific element. Also note that how these styles appear is entirely up to your terminal, so results may vary between them. See
:help style for details.
- xdg-open is now the default browser.
- cleantitle.py is a plugin that allows you to strip content out of story titles. From annoying newlines to HTML fragments, this can help cleanup content from feeds that are poorly defined.
- Tab completion tweaks. Works more like Bash now.
- Update style: prepend. You can now use “
:set update.style prepend” to get new items to be added to the top of the feed on update. The other options are “maintain” for sorting, and “append” for adding to the end.
- :help set will now list various common options and their settings / uses. It’s entirely static (i.e. won’t show current values), but should help to familiarize the user with some of the lower level options.
- Performance improvements. Various changes have been made to speed up many parts of the code. It still isn’t perfect, but a lot of larger operations are broken into smaller pieces, functions have been tweaked to run faster, and commands modified to be smarter. In my experience, after everything is loaded, canto runs very well, so I’ve put in a lot of effort to minimize the annoying pauses caused by loading feeds with massive amounts of post-filtered items. The initialization process has also been significantly reworked to make more sense.
All in all, a healthy changelog for a dot release. Subjectively, the performance has improved quite a lot, and the feature list isn’t too shabby either. I am a bit concerned with the invasiveness of some of the performance changes, but they’ve passed the (admittedly anemic) test suites and seem to be rock solid from my personal use. In other words, it’s time to push it out there and see how you guys break it.
I’m particularly surprised at how simple some of the performance changes were. A lot of the algorithms I used were naively implemented, particularly when trying to sort items into new, current, and old. That has to be done with linear complexity, because once you break into
O(n^2) you’re fucked as soon as you get more than a handful of items. Other places it was the classic “cheap operations are expensive”, for example I was trying to cleanup the entire hook stack every time a hook callee was unregistered. This mean that a single story receiving
die() and unregistering itself could cause a search of 1000s of possible hook callees. Yikes. Consider killing whole tags at once (say because of a config change filtering most of them) and you get into some seriously computational load for what should be almost instantaneous (like it is now).
On the other hand, I’m pleased that I was able to get a lot of the lead out of the system without causing any incompatibilities or other headaches.
Anyway, it’s not perfect, but there were too many changes pending release to delay any longer. If there are serious bugs that need to be ironed out, well, 0.9.5 and 0.9.3 aren’t exactly big deals.
Debian repos will be updated shortly.