Python ctypes: efficiently writing binary data

This is one of those ones where you only realise the answer is one line in the documentation after an hour or two of fruitless googling…

I was using ctypes to access a C function which returns a pointer to a buffer and the size of the buffer, along the lines of:

char *get_data (size_t *len);

Mapping this into Python wasn’t hard:

get_data = lib["get_data"]
get_data.restype = POINTER(c_char)
get_data.argtypes = [ POINTER(c_size_t) ]

...

len = c_size_t()
data = get_data(byref(len))

(Note that you don’t use c_char_p as the return type, because that’s used for NULL-terminated strings, whereas our function returns binary data which may include NULLs.)

I then wanted to write the data to a file. However, a Python File object’s .write() method takes a string (or bytes object), so I can’t pass it ‘data’ as is (because ctypes doesn’t know how to cast it to a string, which isn’t unreasonable given that it also doesn’t know how long it is).

So the question is: how do you cast a pointer to data to a Python string?

The answer is to use the ctypes string_at() function:

len = c_size_t()
data = get_data(byref(len))

outfile.write(string_at(data, len))

I’m not 100% sure whether this copies the buffer or simply shares the pointer, but my understanding from various forum posts out there is that it is the latter, and so much more efficient than converting the data to a string by iterating over the bytes and constructing the string.

Advertisements
1 comment
  1. Torkel Bjørnson-Langen said:

    I’ve just check the source code: ctypes.string_at() copies the data!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: