Bookmarks

There are three reasons why I don’t like using web browsers for storing my bookmarks. First, browsers make it hard for me to access my bookmarks from my phone or a friend’s computer; second, I find the browsers’ interface for managing bookmarks unwieldy and confusing; and third, I switch between browsers very often, which means I inadvertently have my bookmarks spread out between Chrome, Firefox, Safari, Opera Mobile and the Android Browser. Web-based bookmarking services are accessible via any Internet-connected computer, have user interfaces that are in line with my personal preferences and are browser agnostic. They alleviate pretty much all the issues I have with browser-based bookmarking.

After thinking about it for longer than I should have, I built my own little bookmarking tool yesterday evening and deployed it to my VPS. It’s a part of Can ‘O Beans and lives behind this humble bookmarks page. It lets me add bookmarks to my collection, assign descriptions and tags to them, and search through my collection by tag or hostname. Next up on my TODO list is a Chrome extension. Also in the works is a better search, an RSS feed, an import/export feature and several UI tweaks.

Building this little tool must have taken me about three hours. Time well spent, I’d say.

Scripting tmux

tmux ranks highly on the list of programs that I cannot live without. I consider its split-screen and terminal multiplexing capabilities absolutely essential for day-to-day hacking. It belongs to that rare breed of software that has measurably improved my development productivity, software that makes me genuinely happy.

This is what my tmux pane layout usually looks like when I’m working on a project:

 ---------------------------------------------------------------
|                               |                               |
|                               |                               |
|                               |                               |
|                               |             repl              |
|                               |                               |
|       directory operations    |                               |
|       git interactions        |-------------------------------|
|       etc.                    |                               |
|                               |        directory watcher      |
|                               |        server                 |
|                               |        logger                 |
|                               |        etc.                   |
|                               |                               |
 ---------------------------------------------------------------

Since I use this layout for nearly every project that I work on, it makes sense to have tmux automatically set it up for me so that I don’t have to type a bunch of keyboard shortcuts every time I start working on a new project. Luckily, tmux is highly scriptable. Here’s a BASH function that automatically sets up the three-pane layout from above:

setup_tmux_layout() {
    # Create a new window.
    tmux new-window -a -n “$1” -c “$2”
    # Now split it twice, first horizontally and then vertically.
    tmux split-window -h
    tmux split-window -v
}

Now I can run the following from my shell:

$ setup_tmux_layout <window_name> <starting directory>

(Note that at least one tmux session must be active for this to work. This function affects the most recently active tmux session.)

This is good, but we can go a step further. I have several Django projects, and whenever I start working on one of them there are a number of additional Python/Django-specific actions that I take.

  • Switch to the project directory in all three panes.
  • Activate the project’s virtualenv in all three panes.
  • Run a git status in the large pane on the left.
  • Start a Django shell (python manage.py shell) in the top right pane.
  • Run the Django development server (python manage.py runserver) in the bottom right pane.

Once again, it makes sense to automate these actions. Here is the actual code I have in my ~/.bash_profile to do that.

workon_project() {
    if [ $# -lt 2 ]
        then
            echo “Usage:”
            echo -e “tworkon_project <project directory> <virtualenv name>”
            return 1
    fi
    # Create a new window.
    tmux new-window -a -n “$2” -c “$1”
    # Send keys to the large pane on the left.
    tmux send-keys “workon $2” C-m
    tmux send-keys “git status” C-m
    # Split the window horizontally.
    tmux split-window -h -c “$1”
    # Send keys to the top right pane.
    tmux send-keys “workon $2” C-m
    tmux send-keys “python manage.py shell” C-m
    # Split the window again, this time vertically.
    tmux split-window -v -c “$1”
    # Send keys to the bottom right pane.
    tmux send-keys “workon $2” C-m
    tmux send-keys “python manage.py runserver” C-m
}

Brilliant! Now I can perform all those tedious actions in one fell swoop by running this in my shell:

$ workon_project <project directory> <virtualenv name>

(In case you are wondering, the workon command above comes from virtualenvwrapper.)

This little function has saved me hundreds of keystrokes in the last few months, and it only scratches the surface of what tmux is capable of. If you are a regular tmux user, I highly recommend skimming the tmux manpage at least once.

A Django Admin Wishlist

It is okay to skim this post and only read the parts that you find interesting.

When I was just learning how to use Django, I dismissed django.contrib.admin (“the admin” from now on) as a nice-to-have extra, a marginally useful demo of framework functionality, but not much more. I didn’t even enable it for most of the apps I wrote as a Django novice. The simple, honest django.forms.Form and the darkly magical django.forms.ModelForm together did everything I needed. Learning to customize the admin was a waste of my time. Or so I thought.

As the scope of my projects grew, my attitude towards the admin thawed a little. I had heavily leveraged it for an app that I had written for my personal use and found that it was more customizable than I had initially believed. I was surprised to discover that I could make it mostly work the way I wanted by making only minor adjustments to my Python code. While experimenting with some third-party Django apps, I found that all of them use the admin in one way or another. Mezzanine, Cartridge, django-cms, Zinnia, Satchmo, django-filebrowser … all of them hook into and extend the admin framework instead of using something home-grown. Of course, sometimes the admin framework doesn’t have the functionality these apps need, and the developers end up building the missing functionality from scratch, but it’s the existing functionality that gets them 70–75% of the way.

Consequently, when I was starting work on an internal webapp for my company, I decided to not waste my time writing a custom management UI for it and instead use the admin from day one. This was about a month ago. Since then I have learned a lot about how to customize and extend the admin to my app’s needs. I have also, in the course of customizing the admin, come across some annoying limitations in the admin framework. Some of these limitations are easily overcome using third-party apps, others require a bit of extra work on the developer’s part, and still others need to be looked at by the core Django developers. This post details the limitations I ran into, and — where possible — ways to overcome them.

Limitation #1: The admin cannot display nested inlines

Let’s say you have three models called Group, Person and EmailAddress. A Group can have multiple Persons, and a Person can have multiple EmailAddresses. You might be tempted to do this in your admin.py:

# In admin.py.

class EmailAddressInline(admin.TabularInline):
    model = models.EmailAddress
    extra = 1

class PersonInline(admin.TabularInline):
    model = models.Person
    extra = 3
    inlines = [
        EmailAddressInline,
    ]

class GroupAdmin(admin.ModelAdmin):
    inlines = [
        PersonInline,
    ]

admin.site.register(models.Group, GroupAdmin)

This will not work. The admin framework doesn’t expect to ever find an InlineModelAdmin nested inside another InlineModelAdmin, so the inlines list defined on PersonInline makes no sense to it and it will happily ignore it. When you try to edit or add a Group in your admin, you will see a list fields for editing or adding related Persons at the bottom of the page, and that’s it. The list of fields for editing or adding EmailAddresses that you expected to see attached to each Person will not appear in the admin UI.

If you want nested inlines, you will have to extend the admin template for your model to handle them. You will also have to create a custom ModelForm subclass to deal with the data that you get from your new template. See this StackOverflow question for some more details.

Support for displaying nested inlines in the admin is an oft-requested feature that even has a working patch, but it hasn’t made its way into Django yet because it doesn’t meet the community’s quality standards. See ticket #9025.

Limitation #2: All staff users can access all AdminSite instances

If you have multiple admin sites in your project, then all Users who have their is_staff flag set to True have access to all those admin sites. Often, this is not desirable. For example, if you’re building a forum application, you might want to have separate dashboards for forum administrators and forum moderators. Or, if you’re building an app for managing your company’s payroll, you might want separate dashboards for accountants and employees. You don’t want someone from one of these groups to accidentally stumble across the other group’s admin interface.

The solution I use to get around this limitation is a little obtuse, but it works. First, I create a new model to serve as my AUTH_PROFILE_MODULE:

# In models.py.

class ForumUserProfile:
    pass

# In settings.py.

AUTH_PROFILE_MODULE = ‘appname.ForumUserProfile’

On this model I define one permission for each admin site in my project:

class ForumUserProfile:
    class Meta:
        permissions = (
            (‘access_user_admin’, ‘Can access user admin.’),
            (‘access_mod_admin’, ‘Can access moderator admin.’),
            (‘access_admin_admin’, ‘Can access main admin.’),
        )

Then, I create a middleware that returns an HTTP 403 error if a user tries to access an admin he doesn’t have the permission to access:

# In middleware.py.

class RestrictAdminMiddleware(object):
    RESTRICTED_URLS = (
        (‘/user/’, ‘access_user_admin’),
        (‘/mod/’, ‘access_mod_admin’),
        (‘/admin/’, ‘access_admin_admin’), 
    )

def process_request(self, request):
    user = request.user

    # If user is not authenticated, we’ll let the admin
    # framework deal with him. We just care about staff users.
    if not user.authenticated: return None

    # Superusers are allowed to access everything, so we’ll
    # let them through.
    if user.is_superuser(): return None

    if user.is_staff():
        for url in self.RESTRICTED_URLS:
            if re.findall(url[0], request.path):
                if not user.has_perm(url[1]):
                    raise PermissionDenied()

    return None
# In settings.py.

MIDDLEWARE_CLASSES = (
    ‘django.middleware.common.CommonMiddleware’,
    ‘django.contrib.sessions.middleware.SessionMiddleware’,
    ‘django.middleware.csrf.CsrfViewMiddleware’,
    ‘django.contrib.auth.middleware.AuthenticationMiddleware’,
    ‘django.contrib.messages.middleware.MessageMiddleware’,
    # Our shiny new middleware!
    ‘appname.middleware.RestrictAdminMiddleware’,
)

Finally, I set up the appropriate permissions for my users in the admin and I’m good to go.

Limitation #3: It is not possible to reorder, group or hide models on the admin index page without extending the index template

When the number of models you want to manage using the admin becomes large, it becomes necessary to organize them into logical groups for easy navigation. The admin currently groups models by app, which is fine when your apps have a small number of models. This organization scheme quickly becomes unwieldy when the number of models in a single app grows large. It becomes even worse when you have two related models in two separate apps that you think should logically be displayed under one group.

While the admin app itself doesn’t currently have this functionality, it is easy to add. You can:

  • Extend the admin index template (admin/index.html) and create groups yourself.
  • Use a third party app that allows you to customize the admin using a straightforward Python API. Two such apps are django-admin-tools and django-grappelli.

I personally use django-grappelli on all my websites.

Limitation #4: ImageFields are not displayed as thumbnails

If you have an ImageField on any of your models, nine times out of ten you want to see a thumbnail of the image when you’re editing an existing instance of that model. There are several third party apps that add this feature to the admin, and the Django wiki lists a variety of solutions.

Limitation #5: Select boxes cannot be “chained”

This is a use-case that occurs very often. You have two select boxes on a page, and the list of choices in the second select box depends on the choice the user has made in the first select box. For example, the first select box could contain a list of Indian states and the second one a list of cities. You want to dynamically change the list of cities in the second box depending on which state is selected in the first box. The Django admin does not allow you to do that out of the box.

Fortunately, the solution is very simple: use django-smart-selects.

Limitation #6: “Save and continue editing” reloads the page

I think the purpose of the “Save and continue editing” button is lost if it reloads the page every time you click it. If you’re editing a complex model that has many fields, reloading the page means you lose your position on the page and have to scroll around to find the field you were editing. While composing long-form text in a <textarea> — like I’m doing right now — clicking “Save and continue editing” means losing your position within the five-hundred or so words you just wrote. This is not an issue if you use the admin for one-off edits, but it becomes a major usability problem if you spend most of your time in the admin.

A full solution involves quite a bit of JavaScript, but you can get halfway there with a simple asynchronous POST request. Here’s an example:

# In admin.py.

class JournalAdmin(admin.ModelAdmin):
    class Media:
    js = (‘ajax_submit.js’,)
// In ajax_submit.js.

// NOTE: I’m not very good at writing idiomatic JavaScript.
// If you think this snippet can be improved, do
// let me know.

// WARNING: this code is meant for demonstration purposes. 
// Do not use it in production. It fails on several edge cases
// and, in the process, destroys your data, empties your bank
// account, kidnaps your children and takes down reddit for a
// month.

“use strict;”;

var AJAXSubmit = function () {
  if (!$) {
    var $ = django.jQuery;
  }

  function ajax_submit(e) {
    e.preventDefault();

    var data = {
      // Don’t forget the CSRF middleware token!
      “csrfmiddlewaretoken”:
      $(“input[name=’csrfmiddlewaretoken’]”).val(),
      “title”: $(“textarea#id_title”).val(),
      “slug”: $(“input#id_slug”).val(),
      “author”: $(“select#id_author”).val(),
      “published_on_0”: $(“input#id_published_on_0”).val(),
      “published_on_1”: $(“input#id_published_on_1”).val(),
      “content”: $(“textarea#id_content”).val()
    };

    if ($(“input#id_published”).is(“:checked”)) {
      data[“id_published”] = “on”;
    }

    $.ajax({
      type: “POST”,
      url: “”,
      data: data,
      success: function() {
        alert(“Saved!”);
      }
    });

    return false;
  };

  $(document).ready(function() {
    var btn = $(“div.submit-row input[name=’_continue’]”);
    btn.click(ajax_submit);
  });
}();

Warning: don’t use this code in production. I’ve only tested it on Firefox and Chrome, where it fails on several edge-cases. If you use this in production and lose data, don’t blame me.

Limitation #7: Generic relationships are displayed poorly

Currently, the admin treats generic relationships just like any other data. Consider this:

# In models.py.

class TaggedItem(models.Model):
    tag = models.SlugField()
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey()

In the admin for TaggedItem, content_type will be displayed like any other foreign key: a select box containing a list of all content types in your project. object_id will not get any special treatment either: it will be displayed as a simple <input> box. This kind of treatment makes it impossible to figure out at a glance which object the generic foreign key actually refers to.

What I’d like the admin to do here is display the URL returned by the content_object’s get_absolute_url() in a read-only field right after the object_id field. Alternatively, the read-only URL field could simply contain a link to the content_object’s admin page.

Limitation #8: While editing a model instance with a relationship to another model, it is possible to add a new instance of the other model to participate in the relationship, but impossible to edit an existing related instance

Let’s say you’ve got a BlogPost model with a ManyToMany relationship to a Tag model. While editing your BlogPost in the admin, you see a nice multi-select box where you can choose the Tags you want to associate with your BlogPost. There’s a small “+” button next to the multi-select box. Clicking this button opens a pop-up window where you can add new Tag instances. Now, what happens when you create a new tag using this feature but accidentally give it an incorrect name? You’ll find that there’s no edit button next to the multi-select box. To fix the name of your newly-created tag, you will have to navigate to the admin page for that tag and fix the name from there. This is another minor inconvenience that can become a major usability issue if you use admin a lot.

I haven’t sorted this issue out yet, but I can think of a couple of solutions that could work. I’ll post them here in a separate blog post after I’ve implemented them in my own apps.

Closing Remarks

The Django admin app is incredibly useful. With a little tweaking, it makes writing custom management interfaces for your webapps completely unnecessary. Yes, it has some shortcomings, but most of them can be overcome with a little extra code. Besides, Django development happens at a very fast pace, so you can expect some of these shortcomings to be fixed in future releases of the framework.

Cache all the Things!

Yesterday night I installed memcached on the server that runs AnkurSethi.in and hooked it up to Django’s caching framework. Since then, page load times are significantly lower and so is the server load. This little machine is now ready to handle whatever the Internet throws at it.

According to the Chrome developer tools, the slowest loading resource on AnkurSethi.in is now PT Sans, the font that I use for all the text on this website. Since I set up caching, resources from my own server take less than 700ms to reach me in New Delhi. On the other hand, PT Sans as served from the Google CDN takes up to 1 second to get here. This doesn’t bother me much, though, since the browser caches the font after it is downloaded once, which means further page loads are blazing fast. Besides, web fonts are loaded asynchronously, so you can still read the page content while the font is loading in the background. Optimizing font loading time is simply not a productive endeavor.

Setting up caching in Django was easy enough, but I did run into one major issue: cache expiry. When I finish writing a journal entry, I often go back and edit it multiple times in order to fix lingering typos and grammatical errors. Sometimes I even do this after I’ve published it. I never want to serve up a journal entry with incorrect spelling or grammar. However, I had set a one hour expiry limit on the items in my cache. This meant that, even after I had made corrections to an existing journal entry, the server would continue to serve the incorrect entry from the cache for some time. In the worst case, “some time” could mean an hour, which was unacceptable.

The solution I used to fix this issue was simple, if inelegant. I hooked up the post_save signal that is defined on every Django model to a function that simply nukes the entire cache. It may sound like a terrible solution, but it works great in practice. Sure, rebuilding the cache is expensive, but not expensive enough to warrant a more complex solution. Having to rebuild the entire cache is a huge issue for large, distributed, high-traffic webapps where running one database query may take tens of seconds. It is, however, not an issue for my humble personal blog, which contains a few kilobytes of data, runs off a single machine and is lucky to see a few hundred visitors per day. On the whole, I’m very happy with my current setup.

One more item crossed off the TODO list for this website. Next item: support for syntax highlighting.

A Whole New Can of Beans

I’m envious of people who have consistently posted to their blogs for years. I, too, have blogged semi-regularly over the years, but I’ve never managed to stick to a single blog for long.

All I remember about my first blog is that it was mostly a link dump, but I do remember my second one. Titled Infuriated, it was a rant blog where the 15-year-old me wrote angry, vitriol-laden posts about things that annoyed me. A few months later, just when I was running out of swear words, someone introduced me to the world of Free Software. I scrapped Infuriated and started a short-lived blog called The Free Speech Blog, where I wrote exactly one holier-than-thou post about something disgusting that Sony had done. Then I proceeded to neglect all my blogs for a couple of months until one day I got angry at something that had happened at school and created yet another rant blog on WordPress.com (you see a pattern here?) This one was called Badger Alert!, and ended up being very different from Infuriated. While Infuriated was just pure vitriolic fun, Badger Alert! had content that was more spontaneous and sincere than anything I had written before or have written since. You see, by this time I was well into my rebellious teenage years, and the smallest things would tick me off. Badger Alert! was my “personal space” on the internet, somewhere I could talk about my feelings and occasionally vent my anger. It got very pretentious sometimes, but on the whole it was the most fun I ever had writing.

The last blog I wrote before I dropped off the grid for close to two years was titled A Series of Uncool Events, hosted at Uncool.in (that URL redirects to AnkurSethi.in now). I didn’t have much fun with it, and I wasn’t particularly proud of what I posted there, yet it ended up being my most popular blog. The most popular post on Uncool.in, titled Computer Science FAIL: Higher Education in India, was a list of dangerously incorrect information that a professor at my college had fed my class during our first Introduction to Computer Science lecture. The post stayed on the Hacker News front page for a little time and sparked a handful of comments. It was more popular on reddit, where it spawned a long comment thread complete with flamewars and bad puns. It was was blogged and re-blogged across the Internet, leading to over a hundred thousand page views in the space of a week or two. At one point, it even showed up on the Cincom Smalltalk community blog. The resulting increase in PageRank ensured at least a few hundred hits every day. For the first time ever, my blog showed up as the number one result when you ran a search for my name on Google. Eventually I got rid of Uncool.in too, because I have a medical condition commonly known as being a complete idiot.

After Uncool.in, I became apathetic about blogging. I didn’t feel I had enough to say, or that what I had to say mattered, or that blogging was worth it at all. I’d purchased a nice beefy VPS for hosting my personal website and blog, but I didn’t set it up with a blogging tool. For many months both AnkurSethi.in and Uncool.in pointed at a plain HTML page with my email address and twitter handle on it.

If you’re part of the tech community, a blog or a personal webpage is a wonderful thing to have. I wouldn’t say it’s essential, because some of the best engineers out there don’t care for or need a web presence, but it’s definitely a huge plus. Blogging leads to good things. For me, it was one of the ways I found people who shared my interests. Every time one of my posts got more than a few hundred hits, I’d get emails from people who wanted to talk or just say hi. Blogging was also a way to get involved in discussions about topics I really cared about. Instead of merely participating in discussions that were trending on Hacker News or reddit, I could start discussions that I wanted to have. Now, it wasn’t as if I wrote insightful posts that sparked deeply nested comment threads across the Internet. No. The best discussions I had via my blog were simply the ones I had with people I already knew, and most of these took place over IRC, IM or beer. Still, there was the occasional post that would get attention, and that was nice.

As soon as I stopped blogging, I started feeling the negative effects. I noticed a drop in communication sent my way, and participated in fewer online discussions on the whole (IRC flamewars are not discussions). As my long-form writing output decreased, my thinking started to get muddled. Plus, there were so many things I was mulling over but was unable to discuss with anyone. I wanted to talk about how great Racket was, about the state of programming documentation, about things I was doing at college, about my little side projects and hacks, about Linux and free software, about privacy, about the books I was reading and the music I was listening to. I tried writing in real, physical, paper-and-pencil journal for a while, but it didn’t feel the same as writing publicly on a blog. Twitter and Facebook were useless for the kind of conversations I wanted to have, and reddit and Hacker News had their own issues. Eventually, I installed WordPress on my new VPS so I could start blogging again.

Despite having spent several hours setting up the new WordPress blog and getting it to behave just right, I never really posted to it because my frustrations with WordPress had grown to a point where I couldn’t even stand to look at the interface (a blog post for some other day). The blog just sat there, attracting only bots and spammers.

But then something wonderful happened. Earlier this year, I started getting into web development and, as a way of getting to know how to build, deploy and maintain a Django app, I decided to replace WordPress with a custom blogging tool of my own creation. Can ‘O Beans was born.

If you go look at the Can ‘O Beans code on GitHub, you’ll notice two things. One, some of the code — especially the template code — is atrocious. Once you get over that, the second thing you’ll notice is that Can ‘O Beans is as simple as a blogging tool can possibly be. It consist of exactly one database model, plus some templates and CSS. I didn’t even build a publishing interface, choosing instead to use the Django admin for writing new posts.

Can ‘O Beans as I’m using it now was “finished” a few weeks after I started building it, but I didn’t deploy it for months because I couldn’t bring myself to dive into the big bad world of WSGI servers and reverse proxies and databases and what have you. Then last month, I found that I needed to deploy several Django apps for a project I’m working on. This forced me to sit down for three days and absorb everything I could about using Django in production. I wanted to experiment with what I had learned on my own server before I went live with it on the server my project is hosted on, so Can ‘O Beans finally got a chance to shine.

And kids, that is why AnkurSethi.in once again has the pleasure of hosting real content instead of an unfriendly white HTML page.