Canto

The streak continues.

I made a bit of detour into the Story/Tag drawing code and made a few changes to clear it up.

Surprisingly, what started as a simple cleanup inadvertently turned into optimization. Better separating the prep and drawing logic, which includes locking operations that can take longer than you expect (and certainly much longer than when the drawing code was initially written in the single-threaded days) seems to have made it snappier than expected. So much that, in addition to writing this post, I’m wondering if there wasn’t a subtle cache bug that I now fixed unknowingly.

The changes also made it pretty easy to test whether the cache memory usage was worth it. Long story short, it absolutely is. Without the caches, the memory dips a small amount, but the interface crawls. With the caches on, I can still have a few hundred items knocking around smoothly and memory usage is still like a third of an empty instance of gedit based on htop RES.

Bit of a change for the update titles, as I realized that it was going to be confusing going on for years =P.

Anyway, if you look at my GitHub you’ll see that it’s been quite a productive five days since I returned from the beach.

I implemented the custom tag stuff. I made completions work better. I hooked up more commands, got aliases to work, fixed some readline bugs (which unfortunately required more C because the Python standard library is making some… poor assumptions). I also updated the yank plugin for canto-curses that’s in git.

The input line isn’t quite rock solid, but with the extra readline work it seems to be pretty close. I’ve copped out on some completions, but I consider that pretty low priority compared to some of the other stuff.

My last few have patches been getting plugins up to a more acceptable level. In particular, allowing plugins (like yank) to “suggest” binds, i.e. non-destructively update a key bind. So, for example, dropping the yank plugin into the .canto-ng/plugins directory will automatically bind ‘y’ and ‘Y’ the next time you run canto-curses, unless those keys already have bindings.

Another change I just committed is similarly intended to make bracketing displayed content from plugins a lot easier. For example, a “favorites” plugin that adds the user tag “favorite” is able to trivially add different markers (i.e. a star character or a different color). This would also make something that, say, colorized items based on domain, or based on a Bayesian heat map a lot easier (more on that in the future).

These two changes have made canto-curses plugins way more useful and, as a bonus, they work flawlessly with the new command infrastructure so plugins get completion basically for free. In addition, the new command format makes it a thousand times easier to use the command interfaces for objects because they take natural args. For example, tagging an item from a “favorites” plugin is as easy as doing taglist.cmd_tag_item('favorites', [ items ]) instead of having to decode a bunch of bullshit in an abused key word argument dict.

I’m pleased that things seem to be coming together so well.

My next immediate push is to complete the custom individual item tag work, in particular making it easy to shift between custom tags as part of completing the “favorites” plugin. I think the best way to work on that is two-fold. First, a command that will filter on tag, most likely by using the InTags filter (like from the example in the manual). The second is to make it easy to operate on tagged items inline by automatically supporting domains for the item-list argument type. So, for example, you could go through and “favorite” items, and then do something like goto favorite,* and all of the other item-list taking commands work for free.

I’m trying to make a conscious effort to separate some of the uglier workings from the user and push that into the plugin space. This was already sort of done by adding the config variables to the format string so you could change stuff like the selected and marked formats without changing the whole string. But stuff like temp-transform just isn’t useful enough, and I don’t believe that I ever want users manually twiddling configs themselves unless they’re into that level of control so stuff like toggle and set will be gone in favor of more specific commands like toggle-enumerated or the aliases like browser.

Just got back from a great anniversary weekend in South Padre =).

In the meantime, most of what I’ve done since the last update has been hooking up commands in the new infrastructure. In git, most of the commands work now, including remote stuff. I’ve started work on the aliases, and it’s pretty straightforward, but there’s some rough edges that need to be handled before I put it up.

One thing that I’ve tacked on to this (eventual) release is that I’d really like it if tags would stop being useless and awkward.

In 0.8.x you can use :add-tag to effectively alias or categorize a current full tag. So it’s useful for saying “Slashdot = news” and “Hacker News = news” and then either seeing those items in a single tag or using transform voodoo to be able to view those items sorted by maintags (aka feed default tags). That transform voodoo sucks.

In addition, there’s currently no way to tag individual items with anything, as would be useful if you were going to manually sort or tag items (i.e. star/favorite/delete items), or even if you were going to, say, teach a Bayesian classifier to tag individual items interesting for you. This requires a bit more thought, but is something that should’ve been possible before but hasn’t ever had support added and it needs some support from the database format (or abuse of currently existing state stuff).

So, my current thought is to add the capability to tag individual items to the daemon, which should be trivial because it already deals with items in multiple tags it just doesn’t have the protocol to do it from a client instead of the config. Then, to change the client such that when you change tags, they’ll still be grouped by maintag (feed), but internally filtered to only show items in the selected tags. Basically it will no longer be

=== News (user tag) ===
- slashdot item
- reddit item
- hacker news item

Instead, it will default to

=== Slashdot ==
- news item

=== Reddit ==
- news item

== HN ==
- news item

The original view will still be possible, but won’t have any sugar behind it.

My reasoning here is that, in the first example, the fact that the items are in a ‘News’ tag is actually redundant. You’ve chosen the tag to display, you already know what you’re looking at (although it should probably be displayed in the tag header anyway). So, you don’t lose anything by grouping by maintag (except a small amount of vertical screen real estate) and you gain context (knowing where the story is coming from) and I gain simplicity. A win-win.

Just a quick check in. The daemon had a couple of more little tweaks, but overall has been solid. I bumped the version number to 0.9.0-rc1 because it will break released 0.8.x canto-curses thanks to removal of some dead code that it used to rely on. I’d keep it around for backwards compatibility, but it had a (small) performance effect and the version number starts with a 0 for a reason.

I also pushed more to the completion branch of canto-curses and bumped its version to 0.9.0-alpha1.

Included in this are some readline fixes, in particular the laggy scroll that got reintroduced when I implemented a custom readline_getc.

Completion information is actually used and displayed now out of line as well as inline. I fixed some bugs in the argument level completion stuff, and a couple of exceptions from stuff like completing an empty line that used to be non-empty.

I also got the reader up and running and rewired into the new command infrastructure. That means most of the Info/ErrorBox stuff works now too, although they need the “destroy” command hooked up.

Next Steps

The next thing on my plate is a weird bug in the completion stuff where double tapping Tab on an already complete command/argument causes what looks like a newline to be printed in the input box, but I believe is actually screen corruption. I’m not clear if readline is doing something wrong (like this would be good behavior outside of curses) or what, but I know that causing a refresh fixes it and displays what you’d expect in the input box so I don’t think it’s canto state that’s wrong.

Another annoyance is that startup is a bit slow, it waits for a five second refresh. I want initial tag population to trigger refreshes themselves. I’ve seen a few threading bugs too when trying to fork a daemon and connect to it too quickly, so there’s work there probably on both ends.

A lot of the other work is stuff I mentioned last time. Hooking up the remaining commands (mostly stuff that’s usually not important, like color but also a bunch of commands that are implemented via remote and all of its aliases).

Definitely want to get to the plugin/hook analysis. I’m thinking that I should probably implement something a little heavier, like a new reader type interface for Reddit comments. If I can do that in a couple of plugins, that’d be pretty convincing.

Config snippets look a little bit like feature creep at the moment, so I’m not sure if 0.9.0 will see that or not.

I haven’t had as much time the last couple of weeks, but I did make important progress. As of right now, pushed to the completion branches of github, I have a working prototype of the new command stuff. The important difference from last time is that completion information for root commands and their arguments is working and the same infrastructure is being used to validate and dispatch commands. The caveat being that only one command has been changed to use it, rel-set-cursor which allows scrolling.

I also pushed an actual verified fix for the daemon problem I thought I may have fixed last time. I haven’t seen the daemon fail since.

What’s Next

The curses interface is almost ready to be pushed into the master git branch. The only thing remaining is mechanically changing each remaining command to use the new system. It won’t be too bad to get the interface usable, but there are a lot of commands and a lot of argument types and it’s going to be rather rote.

After that, there’s a lot of logical cleanup stuff, although some of that is going to come up in the command refactoring.

Finally, some of the more minor features, like config snippet and taking a critical look at the plugin and hooks that are built in to the codebase. This will have to include the documentation I never wrote for plugins (omg!).

The daemon, including the fix patch, is okay until further notice. The only thing I can think that might be necessary, would be minor additions to the protocol to, say, advertise premade transforms to provide nice completions for those commands.

Oh yeah, and soon I’ll bump the versions to 0.9.0 =).

Okay, so in the spirit of the… 10 day update, here’s what’s been going down.

• Found one more daemon bug, potential fix in git
• Stripped about 550 lines of command infrastructure out of canto-curses, some of which will return.
• Wrote a new style of register / unregister for commands and argument types
• History works! Don’t think I should bother having it remember between sessions though…
• Got root and test completions working both inline and in infoboxes

Most of my time was spent jockeying the readline infrastructure. It kinda sucks when you’re dealing with a standard library that’s only half-implemented. I kept trying to figure out how I could accomplish my goals with what’s already there, but in the end it just wasn’t enough.

As a plus, the readline C infrastructure I’ve added would be trivial to turn into a Python patch to the official module, so maybe I can get it mainlined one day.

Okay, so I achieved all of what I set out in the TODO of the last post and I put it up in the threading branch in git.

It’s still missing commands, but it doesn’t require poking to see output and scrolling, using the browser, quitting etc. works. Now, actually using the command line isn’t so hot because stuff like aliases still don’t work so DON’T USE THE CODE unless you’re curious and probably already have a working config. There’s a reason that it’s tucked into a separate branch.

Threading still isn’t 100% done in that it’s not tested enough and I’m sure I’ll still find some issues in both the client and the daemon, but it’s close enough that I’m willing to move on to completion, which is where I’ll restore all of the commands that I broke and probably redo most of the command infrastructure anyway. There are also some things I’ve left in there for the sake of testing, like hardcoding the sync rate to every five seconds, but that should stay in place until release because it should help provoke any remaining threading problems.

The Memory Leak that Wasn’t

I spent a day or two tracking down an ethereal memory leak. On one of three machines I have laying around (all Intel/x86_64/Arch with the same base software) I discovered a massive jump in top‘s RES stat every time an update cycle occurs. This happens with the old code (July of last year) as well as the latest git, but only on one machine – my work laptop.

I’ve taken a look with gc, objgraph, tracemalloc, pymple and nothing seems to register any changes in the internal Python environment on the broken machine and meanwhile, the RES size will leap up by 4-5M everytime the fetching threads begin. Before and after the updates the number of objects of each type and their respective sizes are identical (barring a few legitimate additional stories every once in awhile).

top is obviously a bit of a blunt instrument here, but even taking into consideration that Python will maintain its own heap and allocations (which means RES might be closer to a high water mark instead of a current usage stat) I can’t see a reason for RES to jump so drastically every single time and I definitely can’t think of a reason that this one particular machine would be a problem.

Anyway, I’m considering this a bit of a non-issue when it comes to the daemon code itself, but I am curious about the cause and plan on doing some digging to at least isolate the important factor (i.e. Python versions, etc.).

tl;dr If your canto-daemon memory is spiralling out of control, let me know (comment/IRC/email).

If you follow any of my repos on GitHub you’ll know that my commit style is sparse. I like a handful of well ordered, well documented commits without a whole bunch of churn. Unfortunately, that isn’t very good for communicating progress when you’re in the middle of basically detonating the code and stitching it back together as I am now so I’m here to give you a two week update on my last set of posts.

The Daemon is Probably Good

I’ve pushed a couple of commits to the daemon as I’ve been using it with the old client code. Fixed a more subtle deadlock, and finally got the database trimming code to shut the fuck up on Arch by disabling gdbm file locks (which sorta sucks, but it’s protected from other daemons by the pid file lock and if some other program kills the database in the meantime I consider it fair to have undefined behavior).

All in all, for a change as large as threading I think it’s gone well and I haven’t had any trouble with it recently. Sometime between now and when I release it (likely as 0.9.0) I’m going to give it a few days of constant uptime to coax out any issues.

Curses Client

The client is where I’m focused in the immediate term. I’ve successfully split the curses interface into a number of threads. The old code looked like

signal thread -> gui thread <- input thread

by necessity. They communicated with each other using Queues which I have now sworn off. The new code is now

signal thread |-> config thread
              |-> item thread
              |-> command thread (input)
              |-> gui thread

The config and item thread are entirely independent of the rest of the code which is nice because it means that they can be developed and debugged independently too. These were actually pretty easy to do as well since all locking is internal and as such the callback code makes it a drop in replacement.

The item thread now updates a TagCore object that mirrors the content from the daemon. Later the signal thread performs a "sync" that updates the Tag object (the graphical representation of a tag) for the next screen update. This means that the client will always have the latest tag information, but we still have control over when it's displayed.

The command thread now receives input and resolves it into commands and issues them at which point it will release the GUI thread to update the screen. The nice thing here is that gui thread can be released elsewhere too so you fun stuff like getting new items still works while the user is in an input box (which needs to be locked down), but more importantly stuff like info boxes work so help, error, or completion information can be provided while the user is in an input box.

Client Performance

I've been floored with how well the revamped client code performs. It's made the client start up snappier, it's made the occasional lagging input disappear. Surprisingly, it also fixed the Cairo terminal sluggishness, which I wasn't expecting but now understand. Basically, when the input thread is running as fast a possible grabbing keys and putting them on a queue, you end up with a whole bunch of keys that were issued before the interface was ready for them. Now that there is no input thread, only a command thread that takes input (semantics, semantics), keys between commands are ignored instead of queued. Cairo came into play with this because on the unthreaded code, Cairo made the redraw take longer which meant that commands took longer, which meant that more input got queued instead of handled and suddenly your cursor is moving seconds after you stopped giving it input. Xterm et. al., which do basically nothing but throw characters on the screen (no post-processing), meant that our commands were being done (nearly?) as fast as input. So that's been resolved now and even though Cairo terminals are still noticeably slower, they still feel responsive.

In terms of memory performance, I can guarantee that it hasn't gotten worse but I can't say if it's gotten better either. When I release 0.9.0 I'm going to do some rough memory and CPU performance metrics to quantify this.

Still TODO

Threading is almost done. I'm using it as my daily client now. There are a few caveats. I've dropped a couple of commands because they no longer make sense in their current context. There's also a bunch of little due diligence type things that need to be done, like resolving the fact that hooks are being called from the config and item threads that are manipulating objects without locking. The initial interface doesn't show up without being poked, same with the reader. I also have some HACK comments in the code for stuff that I put in just to get threading bootstrapped and now need attention. I'll most likely get this stuff cleaned up this week.

What's next

After I squash all of these new incompatibilities, I'll release the threading code to git. After that, it's going to be all about completion. I'm going to blow up the command structure. I've already routed the input through readline in the threading code, the rest is just providing information to the library and the user.

A few days ago I talked about what was currently getting my attention in the daemon. tl;dr multi-threading.

Part of the logic there was that the daemon was pretty stable and after a few days of tweaking around on the threading implementation I’m convinced that that’s true. I had one annoying thread bug squashed but, if anything, I take that as proof that the threading is actually working =P.

I mentioned that the curses interface was in much rougher shape. I took a year off from developing canto, but I still used it pretty much every day in the meantime and there are quite a few annoyances.

So here’s the current priority list:

• Threading. As I mentioned before, multi-threading the daemon was done mostly to allow better client behavior. This will require a bit of a redesign of the curses interface’s structure, but it shouldn’t be too bad. Problems I’d like to address with threading:
  • Unresponsive during item parsing
  • “Waiting on content…” after the initial load
  • Interface ceases updating while getting line input
• Command line overhaul with tab completion. I finally got down to brass tacks and hacked up a working interface to readline that will provide completions and line-editing in a standard fashion. The work to display test completions and route input through readline is already done, now generating deep completions and actually using them is TODO. This is actually made pretty easy thanks to how the command line worked before. I’m hoping that I can write this such that plugin commands could use built in completions or define their own without much overhead.
• Better plugins. The plugin system is extremely powerful and, in my opinion, already works pretty well with the daemon because most of the time you want to use it to modify feed content. However, for the curses interface, there are common tasks that are difficult to implement without looking at the code and that sucks. I want to fix this by adding more interesting hooks and more places where the default code will look for and use plugin attributes.
• Local config snippets. This is one that’s aimed squarely at a future where connecting to the daemon over LAN is fully supported. The interface will read settings from a local file so that you can have local configuration (especially useful for stuff like browser). This would also be a nice way to move around complex theme configurations.
• Polish. A lot of other little nits, like not mentioning how to close windows, or awkward configuration (although that will hopefully be addressed mostly with the tab completion).
• Defaults are going to change, because some of them don’t make sense. Auto-updating, for example, just doesn’t work. I mean, it’s functional, but it’s almost always bad behavior because it shifts the list around. It’s still a useful feature if you were going to, say, display the items on a screen without interacting, but that’s not your usual usecase.

In truth, I view making the command line more robust as priority one because it will make a lot of the dark corners of canto get a lot brighter, but it will be so much easier to do that after threading the interface (because I’ll be able to use the standard canto info boxes that show up to provide info instead of hacking up a expanded buffer that works while the rest of the interface is frozen) that it had to be bumped down.

The end goal is to get `canto-curses` to 0.9.0 in shape such that it can be used easily, predictably, and performantly.

It’s been almost a year since I’ve had the energy to seriously look at canto. Between a job that was soul-crushing with layoffs of a lot of people I consider friends, subsequent unsuccessful job hunting, a five-day two-man startup, and the usual stress of being a husband and father, I needed a hiatus.

… Until a week or two ago …

What’s in tree now

Over the past few days I’ve pushed a couple of nice commits to the canto trees.

• Much faster Reddit plugin for the daemon
• Much faster config stuff, so doing stuff like :browser or any other configuration from canto-curses or canto-remote is instantaneous instead of lagging for a second or two.
• A coarse grain multi-threading implementation for the daemon.

Multi-threading

Multi-threading is something that I’ve avoided for a long time in the daemon. Here’s why:

• The daemon was designed to be a logical abstraction of the feed database that could function over a socket. That is, it’s meant to be a server like screen, not Apache.
• The daemon already took advantage of threading for fetching feeds to avoid long IO timeouts by necessity, and that’s easily separated.
• Multi-threading introduces subtle bugs

You can see why I was reticent. However, here’s why I’ve started to implement it.

• It makes the daemon marginally more responsive.
• It will allow better and faster client behavior.
• The daemon is relatively mature.

The big thing here is that it will allow clients to have better behavior. For example, currently in canto-curses if you, say, start it up and then immediately switch a setting (like :filter), you will clear the items… and then proceed to get a bunch more that don’t follow that filter setting. That’s shit, but there are only three ways to handle this. Either you teach the client to ignore the subsequent responses from the daemon (which with a single thread means the daemon is still generating output, and the client is still reading it, just not acting on it), you close the socket which discards the information but also all of your connection state (like events you’re watching), or you bite the bullet and make multiple daemon connections not suck which lets you keep your state on one connection and start a fresh connection to get data with your new settings.

The yields other benefits, like being able to get data out of line. For example, opening the reader requires information that 99% of the time you don’t need hanging around (i.e. 8k of text when all you care about is the 100 byte title — over 10k items that adds up quite quickly). In the current system, the reader has to wait in line until it gets the information it needs. Crappy. If the client and daemon are threaded, the reader makes its request and, even if the daemon’s in the middle of fulfilling a big request for another connection, the reader will get it’s information immediately.

I also mentioned maturity. Canto, as a whole, isn’t even close to perfect. However, the daemon itself is pretty stable and polished and ready for an undertaking like threading.

The interface part that you actually use? Not so much. The command line sucks, there’s no completion, bad or non-existent error handling. There’s also an ultra-powerful text rendering system built into that’s basically impossible to use if you’re not me or at least decent at Python.

This multi-threading change is going to lead to a revamp of the curses interface. I’ll get into that a little later (after I’ve actually coded a lot of it), but the threaded daemon is an important part of it.