Python: celery, subprocess and stdin

I had the following Python code to invoke ffmpeg and extract a frame from a movie:

from subprocess import Popen, PIPE
args = [
    '-i', filein,
    '-r', '1',
    '-vframes', '1',
    '-f', 'image2',
    '-t', '00:00:01',
    '-ss', '20',
p = Popen(args, stdout=PIPE, stderr=PIPE)
( out, err ) = p.communicate()

(At the end of this, out contains the bytes of the frame.)

This was working fine from a standalone python test program, but when invoked as a celery task, ffmpeg would fail to decode the video. In the celery log (running celeryd in verbose mode, python celeryd -l info) the errors looked like this:

[mpeg4 @ 0x464d90]illegal MB_type
[mpeg4 @ 0x464d90]Error at MB: 1338
[mpeg4 @ 0x464d90]concealing 1344 DC, 1344 AC, 1344 MV errors...

At first I thought perhaps the environment that the task ran in was different, so I tried a few things:

  • hard-wiring the path to ffmpeg in the code (in case it was picking up a different ffmpeg from somewhere)
  • examining the ffmpeg output, which is quite verbose about the details of how it was built, to establish the right ffmpeg was being run
  • capturing the output of the env shell command and comparing it to the working environment, to look for salient differences

None of this led anywhere.

Since the code worked standalone, I figured it was something about how the subprocess was being invoked. Most likely, I thought, seeing as the working version was being run from an interactive shell, it was something to do with file descriptors and standard input/output.

My code wasn’t dealing with stdin explicitly, so I made it do so by opening a pipe and sending an empty string (the argument to communicate()):

p = Popen(args, stdin=PIPE, stdout=PIPE, stderr=PIPE)
( out, err ) = p.communicate('')

Turns out my hunch was right, and this worked just fine. I’m not sure why it’s necessary, and at some point I’ll return and experiment a bit more.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: