Archive

Uncategorized

When we recently set up a new Linux web host, I had to configure two 4TB disks into a RAID1 (mirrored) array.

I created whole-disk partitions on the two disks using fdisk, created the array with mdadm, created an ext filesystem on it, and mounted it. Job done, right?

Unfortunately, some time later we noticed the filesystem was only 2TB, not 4TB as we’d expected. This turned out to be caused by a limit on the size of the old-school PCDOS partitions fdisk can create. The fix was to use parted and GPT partitions instead.

I wondered whether it was possible to do this without destroying the array… long story short: it is. (I did back up the data first as a precaution.)

The plan:

  • remove one disk from the array
  • repartition it with parted
  • re-add it to the array
  • sync the array
  • remove the other disk from the array
  • repartition it with parted
  • re-add it to the array
  • sync the array
  • grow the ext filesystem to use the expanded space

Here are the commands, in this case for an md device /dev/md3 made up of /dev/sdc1 and /dev/sdd1:

Remove one disk from the array:

mdadm /dev/md3 --fail /dev/sdd1 --remove /dev/sdd1

Create a gpt partition using most of the disk (the page I copied this from said it would leave 1MB unused at either end):

parted --align optimal /dev/sdd
mklabel gpt
mkpart primary 1 -1

Re-add to the array:

mdadm --add /dev/md3 /dev/sdd1

(This caused a re-sync but it was very quick, which I found surprising; shouldn’t it have had to copy 2TB of stuff?)

Repeat the above for the other disk.

Grow the RAID array:

mdadm --grow /dev/md3 --size=3815318M

(I got to that figure by repeated grows in a binary-chopish fashion. parted reported the partition at 4001G, but passing this to the grow command yielded ‘no space left on device’.)

Finally, resize the filesystem:

resize2fs /dev/md3

It seems that with Yosemite, Apple have decided to change the system font from Lucida Grande to Helvetica Neue.

Perhaps on retina screens it looks lovely, but on my 2009 MacBook Pro it’s a step backwards in terms of legibility. (For example, the ‘i’s and ‘l’s in “legibility” are a bit of a blur.

Alas, it seems you can’t revert to Lucida Grande. However, you can improve the appearance of text in applications that use the default system font. (For me, that’s primarily Firefox and Thunderbird.)

  • Download TinkerTool
  • In the Fonts section, change the Application font to Lucida Grande 12pt
  • Restart your applications (or your system)

This doesn’t seem to fix the Finder, but suddenly my Firefox tabs are a lot more readable.

Edit: I’ve since found https://github.com/jenskutilek/FiraSystemFontReplacement which lets you install a new font in such a way that it replaces Helvetica Neue as the system font. How does it work?

These Fira fonts have a special name table with names identical to those of the system fonts. Because the font folder /Library/Fonts takes precedence over the fonts which are in /System/Library/Fonts, these specially crafted fonts are used for the user interface instead of the real system fonts. The original system fonts are not deleted or modified in any way.

The font is rather pleasant too. Perhaps there’s a way to copy Lucida Grande and modify it to replace the system font in the same way?

I used to back up my DVDs with RipIt, but found that it couldn’t handle some DVDs that MDRP could, so I switched. Whereas RipIt extracts the DVD contents to a .dvdmedia folder (playable by Apple’s DVD player application), MDRP has an option to rip the DVD to a single .iso file, which is playable in VLC and XBMC.

I then wanted to convert my .dvdmedia folders into .iso files. In theory a .iso is just a file container, so my first attempt was to use Disk Utility to make a CD .img of the folder and then rename it to .iso. This kind of worked, in that VLC did begin to play something when pointed at the .iso, but there were weird issues – no DVD menu, no audio, or the wrong language.

It turns out that the structure of the .iso is important, and just dumping the DVD files into it isn’t good enough. I found a forum post out there (which unfortunately I can’t find again) which pointed me at a command line tool called mkisofs, which I was familiar with from work when mastering multimedia CDs for replication.

Unfortunately mkisofs isn’t available by default, but you can get it from the Homebrew package manager.

Once installed, the command you need is

mkisofs -dvd-video -o output.iso input.dvdmedia

It didn’t work for all my .dvdmedia rips, but did for most.

If you’ve used jquery, you’ll be used to attaching click events etc to items:

// for <a class="delete" href="{url}">, get user confirmation
// before navigating off to {url}

$('a.delete').click(function () {
    return confirm('Really?');
});

Sometimes though, the items you want to attach behaviours to don’t exist yet (e.g. because they’re created and destroyed dynamically in response to UI interaction).

You could attach the behaviour at the point you create the new element, but if you’re doing that in several places, you’ll have the same code in various places. Plus, your implementation of the behaviour for one kind of element will be buried inside the implementation of the behaviour of some other kind of element (or refactored into a function in some more public scope than it ought to be).

It turns out that jquery has a method .on() which allows you to attach event handlers to elements that may or may not yet exist:

$('body').on('click', 'a.delete', function () {    return confirm('Really?');});

The documentation explains how this works (attaching the handler to some parent element — the body element in the above example — and using browser event bubbling to catch events originating on the selected children).

It’s quite nice, you end up with quite a clean pattern whereby you create elements with a CSS class for which you have previously defined the behaviour using .on().

I was processing a UTF-8 dataset where the performance copyright symbol (Unicode code point 0x2117) had been entered as ASCII “(P)”. I wanted to replace this sequence with the actual symbol, but I didn’t know how to create an arbitrary UTF-8 character. After a bit of googling (more than 5 mins, otherwise I wouldn’t be posting!) here’s the answer:

txt = re.sub(r'[(]P[)]', unichr(0x2117).encode('utf-8'), txt)

This is probably one of those things that Microsoft admins just know, but I didn’t, and couldn’t find the answer by Googling…

If you want to share a folder on Windows 2003 Server but the usual Sharing options just aren’t there, either in the right-click menu or the Properties dailog, then open up the Services control panel and check that the “Server” service is running.

Service

For a while I’ve been using IETester for testing websites in older versions of IE.

Recently I had an issue with HTML inputs of type “file” not rendering correctly:

Screenshot

Suspecting that it might be IETester, I downloaded the IE8 virtual machine from Microsoft. Virtual PC is a free download for Vista Pro, and built in to Win7 I believe. The link’s at the bottom of the page. The only thing I need to do to allow the VM to see the website was to set Virtual PC’s networking mode to ‘NAT’.

So if you’re having the same issue, and you’re not using a stock IE8, try the VM before wasting too much more time.

Incidentally – although VirtualBox can boot from .vhd files, I didn’t have much luck in this case, I had to use VirtualPC.

Python’s ctypes library lets you load a dynamically-loaded C library (.so, .dll, .dylib depending on platform) and access the symbols within (call functions, etc).

The documentation is quite good, but here’s something that I didn’t immediately find an answer to.

Suppose you have a C structure, and a function that returns a pointer to such a structure:

struct foo {
    int bar;
};

struct foo *newfoo () {...}

In Python, you want to access the members of the struct foo returned by a function. How do you do it?

First, you need to define a Python class that represents struct foo:

class Foo (ctypes.Structure):
    _fields_ = [ ("bar", c_int) ]

Next, you need to tell ctypes that newfoo() returns a pointer to such a thing:

mylib.newfoo.restype = POINTER(Foo)

Finally, after invoking newfoo(), you need to use the pointer’s contents member to dereference it:

p = mylib.newfoo()
p.contents.bar = 17

I’ve been using django-taggit to provide a tagging model for content items in my app. However, I wanted to arrange the tags into a hierarchy/taxonomy. It’s simple enough to use a custom through model to define a custom tag model with a parent pointer, which lets you arrange your tags into a tree:

from taggit.models import TagBase, ItemBase
from taggit.managers import TaggableManager

...

# the custom tag model
class HierarchicalTag (TagBase):
    parent = models.ForeignKey('self', null=True, blank=True)

# the through model
class TaggedContentItem (ItemBase):
    content_object = models.ForeignKey('ContentItem')
    tag = models.ForeignKey('HierarchicalTag', related_name='tags')

# the content item
class ContentItem (ItemBase):
    tags = TaggableManager(through=TaggedContentItem, blank=True)

However, suppose you have a tree of tags like this:

Vehicle
    Car
        BMW
            Z4
        Ford
            Fiesta
        Chevrolet
            Volt

and you have content items tagged with leaves (Z4, Fiesta, Volt), but you want to search for all items tagged with anything from the ‘Car’ branch of the tree. Chances are you’ll end up writing a recursive function to gather up all the descendants of ‘Car’, which doesn’t scale because it involves many SQL queries, or using esoteric SQL syntax available only in the big database engines (and certainly not sqlite3).

At work, where we use a non-relational database engine, we long ago overcame the same issue (efficient manipulation and querying of hierarchical models), so I already had an idea of what I needed to do. But, as is the way with Python and Django, I figured there would probably already be packages that implement efficient hierarchical data — and there are.

The two main contenders seem to be django-mptt and django-treebeard. I tried mptt first, mainly because the consensus seemed to be that it was smaller and easier to use, but also because it purported to allow you to add hierarchical structure to existing models by configuration, which in my case would mean I didn’t have to define a custom tag model and could attach hierarchy directly to taggit’s Tag model.

However, my experience of mptt was poor – the documentation appeared to be out of date with respect to both the version of mptt I got from pip and the latest git trunk. Also, when I tried to use mptt’s admin classes for Django, I got exceptions (I admit I didn’t try very hard to overcome them).

So I gave treebeard a go, and had a much smoother time. Treebeard implements a number of hierarchy techniques with different performance characteristics (e.g. cheap querying but expensive insertion), allowing you to choose which one suits your application’s use of the trees. In my case I went for ‘Materialised Path Trees’ because it’s the relational equivalent of the technique I’m already familiar with. Implementing hierarchical tags was a straightforward case of having my custom tag model extend treebeard’s MP_Node model which, as the name suggests, implements a node in a Materialised Path Tree:

from treebeard.mp_tree import MP_Node

...

class HierarchicalTag (TagBase, MP_Node):
    node_order_by = [ 'name' ]

class TaggedContentItem (ItemBase):
    content_object = models.ForeignKey('ContentItem')
    tag = models.ForeignKey('HierarchicalTag', related_name='tags')

class ContentItem (ItemBase):
    tags = TaggableManager(through=TaggedContentItem, blank=True)

(The node_order_by is what treebeard uses to order siblings when a new node is added to the tree.) That was literally all that was needed. Going back to the ‘Car’ example, the code to find all ContentItems tagged with any of the descendants of ‘Car’:

# look up the Car term
car = HierarchicalTag.objects.get(name='Car')

# get a queryset of all its descendants: with treebeard this is 1 SQL statement
# use HierarchicalTag.get_tree(car) if you want to include 'Car'

treeqs = car.get_descendants()

# now find the ContentItems using an inner queryset
qs = ContentItem.objects.filter(tags__in=treeqs)

I found myself needing to write a Django view to serve a file. The path to the file is held in the database, and is not in the URL space, so using Django’s static file stuff isn’t appropriate.

It’s easy once you twig that an HttpResponse object is a file-like object; just open the source file and copy it to the response object. This was my first attempt:

with open(path, 'rb') as f:    response = HttpResponse(mimetype=file.mime_type)    copyfileobj(f, response)    return response

(copyfileobj is from the python shutil package.)

It works, though I was mildly concerned that the copy presumably causes the whole file to be read into memory before being served, rather than being written directly down the socket to the client. There is a more elegant way to do it:

f = open(path, 'rb')return HttpResponse(content=f, mimetype=file.mime_type)

Since a Python File object is an iterable, and content can be an iterable or a string, this works. What I’m not sure about, though, is when or how the file f gets closed.