Canto

I think 0.9.0 is basically done.

Over the last couple of weeks, I wrote the smartlink plugin for canto-curses that lets you define handlers for different content based on either the URL or the output of file when downloaded to /tmp.

I also wrote some fixes that should let canto-curses not contend on the sync_lock as much (as well as not occasionally deadlocking the logging module) as well as some nice improvements to logging output (pretty printing and reporting plugin-fatal errors).

In addition, I stopped using the readline completer, which was painfully slow. Completion still works with almost no changes infrastructure anyway. Readline is also still used for line editing.

The other bits I mentioned in the previous post I handled. I finally got around to the plugin audit, some stability improvements and dead code stripping. I did polish up :help a tiny bit, but I don’t feel the need to alias the command description. If anything, they are examples (he said, rationalizing away more work).

The only other noteworthy change is the canto-curses now defaults to ‘manual’ refresh, like a webpage. This is to prevent poor timing interactions with filters. If you want the old behavior you can do :set update.auto True, but this should only affect people that don’t already have configs.

So, I’m going to let the code kick around a bit longer, I’ll probably think of some more minor changes to be made, but if it’s settled down 0.9.0 will be forthcoming.

I’ve been taking on a lot more pay-the-bills work as of late, having the pleasure of eviscerating a crusty Perl codebase and replacing it with Python. It’s rewarding and cathartic, but it’s also exhausting so I haven’t been as active as last month.

I pushed a handful of tweaks to how canto-curses deals with readline, so stuff like line-editing and history (arrow keys) work more like you’d expect.

I’ve punted on a couple of things. Disconnect / reconnect, for example. I realized that :reconnect will only do something intelligent if the disconnect happened because of poor network and network socket connections are unsupported (in that they work, but would require more effort to be secure or SSH tricks). Locally running, a disconnect means that the daemon is toast which will either take user intervention (hung) or not, at which point restarting c-c will do basically the same thing. I cleaned up the messages and I’m going to call that good for 0.9.0.

I also nixed :edit which was the only remaining old command I was planning on porting. I realized that it was really just a stopgap I inserted to make editing huge theme strings less painful, but now that sort of thing is better left to plugins.

My list is pretty short now with those taken care of. The abbreviated version is thus:

• Stability. Commands shouldn’t hang c-c on errors and every other bug fix that crops up.
• Better support for running without installation. Stuff like canto-remote shouldn’t be assumed in PATH
• Docs. :help is almost there, but needs yet more polish.
• Plugin and hook audit still needs to be done, this will probably be the very last thing I do.
• Stripping remaining dead command code.
• Fetch + MIME

It’s going to be harder to collect the willpower needed to get started than it will be to bash this stuff out =P.

A few things crossed off the list. Hopefully this will be the last alpha. I wouldn’t have even put it out if it wasn’t for the taglist change.

• :help now lists keybinds, :help commands now gives the command list
• ? is now the default help bind (old users should do :bind ? help to see this behavior since it won’t be overridden from the old search bind)
• Some log output has been downgraded so it doesn’t create info windows
• Search commands will now properly prompt for terms if not given (only relevant if it’s tied to a keybind)
• Cleaned up some alias work
• Tag list enumeration (for stuff like :tag-state and :del) now properly matches up when not all tags are visible
• All the manpages have been brushed up
• Fixed some of the generic daemon config error notification
• Config variables are now changed with :set
• :tag-config has now been eliminated in favor of :set tag.[setting]

The :set change is probably the biggest change. It allows you to set defaults and feed / tag options more easily. feed.setting and tag.setting will now work as expected based on selection.

Old config options like global_transform, browser and txt_browser are now converted to

:set global_transform [setting]
:set browser [browser]
:set browser.text [True|False]

A few fixes have landed in git, a little polish. Some fixes for c-c startup deadlocks that I don’t consider common enough to push another alpha release.

The good news is that with current git I haven’t experienced any crashes or bad behavior.

The bad news is that my TODO has been reduced to stuff that’s going to be tedious, but required by due diligence. Here’s my scrub list for a release:

• Documentation (mostly man pages now that :help is supported, but also polish there getting keybind output to be more natural)
• Along with documentation, plugins and hooks need to be audited to make sure they come from appropriate threads and hold appropriate locks
• Tab completion polish (path completions for browser, arrow keys for history instead of C-u or whatever the weird readline default bind is)
• c-c disconnect and (possibly) reconnect support
• c-c feed setting query/set (this is the only bullet that will require daemon changes)
• c-c fix or remove remaining kwarg commands (old system – I think “edit” is the only one that needs to be “ported”, the others can mostly be removed or hidden)
• c-c fix unfiltered filtered items jumping (i.e. with filter_read set an item as read, then unread again and it may jump to the end of the feed since to the backend it’s “new”)
• c-c hide or polish raw config settings (i.e. all of the aliases for canto-remote one-config) a lot of these are too low level and would be better used in a plugin context now that that’s a little more mature
• c-c fetch, possibly with MIME support in plugin form
• c-c option to shutup log.info output (mostly for autocmd plugin)

I won’t pretend this list is complete, and obviously any more serious bugs will be handled, but when this list is down to nothing, it’ll be time for 0.9.0.

After I package up an 0.9.0 release and make a post about it here, I’ll probably go on a spree of filing bugs with various distros. 0.7.x packages need to be dropped and, if there are 0.8.0 packages they need to be upgraded (so AUR packages will be marked out of date etc.). I won’t badger distros to include 0.9.x, but I will continue to automatically generate releases for Debian/Ubuntu until they’re not useful anymore.

Looking beyond 0.9.0, I imagine there will be 0.9.x maintenance releases for bugs, and when that’s quieted down for a few months I’ll either have a slew of smaller improvements and plugins for 1.0.0 or I’ll tag that code 1.0.0 and let it sit for bug releases or until there’s some reason to make changes again.

Another bump after a weekend of work.

To be clear, this is all still alpha. The only reason I’m packaging and tagging this stuff up is that it’s a ton better than the previous versions even if it’s still incomplete.

• Locking improvements in the daemon, should be more robust on failure, less likely to hang, more likely to report problems to clients
• Locking simplified in canto-curses
• Daemon is more conservative about parsing the config
• Sync plugin locking fix on conf sync
• Canto-curses now compensates for the removal of protection in the daemon (cause of selection disappearing)
• Canto-curses now honors update configuration settings instead of a hard-coded five second sync
• Plugins for curses and next are now installed to [prefix]/lib/canto/plugins
• Reader improvements (better at finding enclosed media, content toggles work again) and plugin enablement
• reader-extras.py plugin to add extra content to reader output, in addition to providing a way to examine item content.
• Various bug fixes…

The list of items I want to tackle is growing smaller. The next things on my plate are more documentation improvements like automatically listing related keybinds and sorting of the command groups in the help overview. Blowing the dust off the manpages is going to be interesting. I’m considering generating a markdown file and then using ronn or another tool to convert it into manpages and also the online manual page. We’ll see.

There are a bunch of interface things that need work too. Some things need to be hidden (more plumbing than porcelain type stuff), others need to be better exposed.

I have some more ideas for plugins implementing some throwbacks to 0.7.x like [ / ] cycles, as well as social integration, but I don’t know if those are better left for a future real release.

Of course, bugs are always on the table, but my hand-written TODO that keeps me awake at night is now under a page long =P.

I just tagged a new release and updated the repos.

The big thing was a fix to canto-daemon blowing up the feeds file. Fortunately, the update will losslessly correct the situation.

Another important thing to get out there was that canto-curses now supports the :help command and :help [command] will provide detailed help on that command, its functions and its arguments.

In addition, :categories, :categorize, :show-category and :remove-category were added to organize tags. These are rough, for the moment but allow you to group tags up and control what’s displayed.

I pushed some interesting features to git last night. The daemon takes some options for caching (memory v. performance tradeoffs), and I trashed a couple of hundred lines of code that just weren’t worth it. You can check the git log if you’re interested because this post is about the meta-game.

I tagged releases last night for v0.9.0-alpha2. There were no alpha1 tags, by the way, even if the code reported its version as such.

It’s time to start thinking about real releases again. I’m hoping to have 0.9.0 out this month, and if all I had to do was code that wouldn’t be an issue, but the documentation for this project has always been subpar and it’s going to take a bit of a rethink to make it useful. The fact that there are still people that hand edit the config is evidence of the fact that they don’t know any better or they couldn’t figure out how to get the configuration they want via a client and that’s obviously no good.

If you’ve ever been here before, you’ll note that the 0.8.x manual, plugin, and theming information has been gutted. Information about 0.7.x is non-existent. codezen.org/canto and codezen.org/canto-ng are now crudely linked to the same content (this stuff). I consider it a clean break. However, to compensate for all the poor bastards out there using ancient cantos from Debian and Ubuntu, I haven’t left you out in the cold. The Get Canto page includes new information about a self-hosted apt repo for sid, trusty, and utopic that you can use to upgrade. Trust me, you want to. The number of serious bugs that have been tackled in the last year and a half is staggering.

In addition, the push to make canto easier to use and configure includes revamping the manpages, and giving canto-curses internal help (for which, I am pleased to say, the information is already there but unexposed).

tl;dr – old information gutted, new information forthcoming, apt repo setup, and 0.9.0 this month.

A few interesting things done in the daemon.

First, the daemon should be a lot more memory friendly. With proper threading a lot of cleanup work can go on in the background, so the –nowb flag is now obsolete and the database is constantly synchronized and trimmed when work is completed. My practical experiments (8-10 hour long canto-daemon instances) showed about a 50% decrease from older versions with no plugins running.

Second, file sync plugins were added to allow you to synchronize your database between multiple daemon instances. WAN and LAN facing canto-daemons have been operational since day one (with SSH port forwarding for any semblance of security), but now you can sync to any place accessible via rsync – which includes remote SSH servers and also local filesystems so you could pair it with sshfs/NFS/Dropbox you’ve setup previously. The plugins still require more testing, but so far they’ve been pretty well behaved here. They do need additions for canto-remote sync to work, but for now it syncs on start, stop, and periodically in between.

To try it out, copy or link canto-next/plugins/sync* to ~/.canto-ng/plugins/ and then tweak sync-rsync.py to set a SYNC_LOCATION (read the top of the file) and restart the daemon. Do the same (or copy over) on another machine to sync between the two.

Some plugin control switches were added, mostly for debugging, and some other minor cleanups too like closing the dangling configuration file handle.

All in all, I’m pleased with the progress of both memory consumption and a heavy duty use of the plugin architecture.

My next bit is to round out the canto-remote part of the file sync plugins and then I’ll be back to canto-curses to work on more polish.

The streak of commits is still alive at 16 days, although I’ve slowed down since Saturday because I’m taking a vacation to visit my parents in St. Louis, which isn’t the riot stricken hellhole it’s made out to be lately. I won’t say much about that other than racism and looting both suck and I hope we pass legislation for police accountability that would make this situation an objective no-brainer. Anyway, I’ve been sneaking in commits whenever I can.

The big change is that I finally fixed some long standing bugs with the daemon. Items should no longer be lost. The daemon also comes up with items faster on startup, and supports multiple serial socket level transforms (which will make some client features easier). I’ve also mitigated a problem with filtering that was causing crashes. I wasn’t able to solve it outright, but I have made it harmless and can collect debug info on it. There were also some nasty bugs exposed by fixing these bugs, like socket cleanup being broken, so I’d highly suggest a pull if you’re running git.

Canto-curses has had a few improvements, mostly thanks to IRC users in #canto complaining (seriously, that works). So adding new feeds and deleting them no longer crashes the latest code. I also solved some busy loop behavior that kept the client and daemon up eating CPU. I implemented the ability to do deeper completions (i.e. specific completions for :remote addfeed aka :add). This will provide better help text for any of the :remote commands, which includes all the config settings etc. Sometime after I’m back in Austin, I’ll spend an evening whirlwinding through and setting up most of these.

I’ve also added a couple of proof-of-concept plugins. One that implements a favorites system with a token change to item display. One that sets the title for xterm compatible terminals. I also updated the yank plugin to work with the new stuff.

My short term goal is to just keep moving, keep the streak up. I still want to make tags easier to use. The favorites plugin is definitely the first step because it allows custom user tags and items to change based on them. Now there needs to be (better) support for changing the tags that are viewed. I’d also like to setup old 0.7.x style cycles (like the old [ and ] binds) in plugins as well.

Still a lot of stuff to do, but it’s all less than major.

I’m hoping that 0.9.0 for the daemon and curses will be out sometime in September but, as always, don’t quote me on that. School’s starting up again for my daughter, and my work comes in waves so it’s hard to count on having the freetime and willpower to continue moving so fast.