Categories
python subprocess

How to redirect output with subprocess in Python?

118

What I do in the command line:

cat file1 file2 file3 > myfile

What I want to do with python:

import subprocess, shlex
my_cmd = 'cat file1 file2 file3 > myfile'
args = shlex.split(my_cmd)
subprocess.call(args) # spits the output in the window i call my python program

4

  • Executing such a command in subprocess would not give you any output. May be you want to run it without > myfile redirecting output from cat file1 file2 file3 into python?

    – PoltoS

    Feb 11, 2011 at 2:52

  • @PoltoS I want to join some files and then process the resulting file. I thought using cat was the easiest alternative. Is there a better/pythonic way to do it?

    Feb 11, 2011 at 2:55


  • os.sendfile()-based solution is possible, see Reproduce the unix cat command in python

    – jfs

    May 12, 2015 at 18:12

  • 1

    I think that output redirection (‘>’ or ‘>>’) doesn’t work in subprocess.Popen (at least in Python 2.7) (in shell=True mode) In this example, as others point out, you can work around this by not using redirection, but in other cases redirection is useful. If redirection or piping is not supported in subprocess.Popen is should be documented (and/or os.system() should be not be deprecated until this is fixed)

    – Ribo

    Jul 19, 2016 at 12:50

16

UPDATE: os.system is discouraged, albeit still available in Python 3.


Use os.system:

os.system(my_cmd)

If you really want to use subprocess, here’s the solution (mostly lifted from the documentation for subprocess):

p = subprocess.Popen(my_cmd, shell=True)
os.waitpid(p.pid, 0)

OTOH, you can avoid system calls entirely:

import shutil

with open('myfile', 'w') as outfile:
    for infile in ('file1', 'file2', 'file3'):
        shutil.copyfileobj(open(infile), outfile)

7

  • 1

    It works, but let me ask you then: What’s the point of the subprocess library if os.system already gets the job done? I get the feeling I should’ve been using subprocess instead since it’s a library dedicated to this task, although since I’m doing this just for myself I’ll be fine using os.system this time.

    Feb 11, 2011 at 2:53

  • The subprocess library is much more flexible than os.system, and can model os.system precisely, but it is also more complex to work with.

    Feb 11, 2011 at 2:56

  • 13

    os.system came before subprocess. The former is a legacy API that the latter intends to replace.

    – Santa

    Feb 11, 2011 at 3:27

  • 5

    @catatemypythoncode: you should not use os.system() or shell=True. To redirect output of a subprocess, use stdout parameter as shown in Ryan Thompson’s answer. Though you don’t need a subprocess (cat) in your case, you could concatenate files using pure Python.

    – jfs

    May 25, 2015 at 7:05

  • 5

    OTOH = On the other hand

    – Cephlin

    Mar 9, 2016 at 17:49

16

UPDATE: os.system is discouraged, albeit still available in Python 3.


Use os.system:

os.system(my_cmd)

If you really want to use subprocess, here’s the solution (mostly lifted from the documentation for subprocess):

p = subprocess.Popen(my_cmd, shell=True)
os.waitpid(p.pid, 0)

OTOH, you can avoid system calls entirely:

import shutil

with open('myfile', 'w') as outfile:
    for infile in ('file1', 'file2', 'file3'):
        shutil.copyfileobj(open(infile), outfile)

7

  • 1

    It works, but let me ask you then: What’s the point of the subprocess library if os.system already gets the job done? I get the feeling I should’ve been using subprocess instead since it’s a library dedicated to this task, although since I’m doing this just for myself I’ll be fine using os.system this time.

    Feb 11, 2011 at 2:53

  • The subprocess library is much more flexible than os.system, and can model os.system precisely, but it is also more complex to work with.

    Feb 11, 2011 at 2:56

  • 13

    os.system came before subprocess. The former is a legacy API that the latter intends to replace.

    – Santa

    Feb 11, 2011 at 3:27

  • 5

    @catatemypythoncode: you should not use os.system() or shell=True. To redirect output of a subprocess, use stdout parameter as shown in Ryan Thompson’s answer. Though you don’t need a subprocess (cat) in your case, you could concatenate files using pure Python.

    – jfs

    May 25, 2015 at 7:05

  • 5

    OTOH = On the other hand

    – Cephlin

    Mar 9, 2016 at 17:49

5

@PoltoS I want to join some files and then process the resulting file. I thought using cat was the easiest alternative. Is there a better/pythonic way to do it?

Of course:

with open('myfile', 'w') as outfile:
    for infilename in ['file1', 'file2', 'file3']:
        with open(infilename) as infile:
            outfile.write(infile.read())