How to Execute Shell Commands in Python
Sometimes one needs to run a shell command in Python. There are many reasons for this, usually, it is to access the functionality of a command-line based third-party program though it can be for other useful tasks such as running maintenance on the server.
In this tutorial, we will learn how to execute shell commands in Python.
Import the os package
The easiest way to run a shell command is with the
os native Python package.
os provides a collection of methods for working with the file system including running shell commands. Put the following at the top of your program to import the
Running a Command using os.system()
os package has been imported we can use the
system() method to execute a command. We will store the response from the command in a variable and print it. The example below is executing an
ls command to list files and directories in the current working directory.
import os output = os.system("ls") print(output)
If you are running your program inside Jupyter notebook the
ls command will return
0 for success or
-1 for an error; the actual output of the command will be shown in the Jupyter terminal window.
system() method works great if we just need to run a command and know if it was successful – to get output data we will need to open a pipe.
Open a Pipe to get the Output from a Terminal Command
To get the output from a terminal command in Python using
os we can open a pipe using the
popen() method. This will create a stream that can be read with the
read() method. Let's run the same command (
ls) as we did in the first example and get the output inside the Python program.
stream = os.popen("ls") output = stream.read() print(output)
enumerate.ipynb example.json fruit.json new_dir sentence.txt shell commands.ipynb
Get the Output of a Terminal Command as an Array
read() method will collect the whole output and return it as a string. To return each line as an element of an array use the
stream = os.popen("ls") output = stream.readlines() print(output)
['enumerate.ipynb\n', 'example.json\n', 'fruit.json\n', 'new_dir\n','sentence.txt\n', 'shell commands.ipynb\n']
To remove the
\n (newlines) and extra empty spaces iterate through the lines using a
for loop and use the
.strip() method before appending the line to a new array.
stream = os.popen("ls") temp = stream.readlines() output =  for l in temp: output.append(l.strip()) print(output)
['enumerate.ipynb', 'example.json', 'fruit.json', 'new_dir','sentence.txt', 'shell commands.ipynb']
Using the subprocess Package
The most flexible way of running commands in Python is by using the subprocess package, which can be used to chain inputs/outputs among various other functionalities. To use it import it at the top of your program.
Run a Command with subprocess
The easiest way to use subprocess is with the
run() method. It will return an object containing the command that was run and the return code.
success = subprocess.run(["ls", "-l"]) print(success) print(success.returncode)
CompletedProcess(args=['ls', '-l'], returncode=0) 0
note - to run multiple commands or use command arguments, they must be passed into subprocess as an array of strings.
Get the Output of a Command from subprocess
To get the output of a command with subprocess pass
stdout=subprocess.PIPE as an argument of
subprocess.run(). This essentially creates a property on the output object containing the returned data from the terminal command. We can get this property on the output of subprocess to get the output of the command.
success = subprocess.run(["ls"], stdout=subprocess.PIPE) print(success.stdout)
Plain Text Output from subprocess
subprocess.pipe() returns data in a byte format, which in most cases isn't very useful. To get a formatted plain text output pass
text=True as an argument of
success = subprocess.run(["ls"], stdout=subprocess.PIPE, text=True) print(success.stdout)
enumerate.ipynb example.json fruit.json new_dir shell commands.ipynb
Provide an Input to the Command with subprocess
It is possible to send data to use in the terminal command by passing
input="" as an argument
success = subprocess.run(["ls"], input="some data")
Don't Display Any Output in the Console
To not display any output in the console pass
stdout=subprocess.DEVNULL as an argument of
response = subprocess.run(["ls", "-l"], stdout=subprocess.DEVNULL)
Using Popen in subprocess
Let's try the
Popen() method to get the output from a command, which provides more shell interaction options. To run open a pipe and get an output in Python we will have to
subprocess.pipe() as the second and third arguments.
process = subprocess.Popen("ls", stdout=subprocess.PIPE, stderr=subprocess.PIPE) output = process.communicate() print(output)
(b'enumerate.ipynb\nexample.json\nfruit.json\nnew_dir\nsentence.txt\nshell commands.ipynb\n', b'')
To get the output from the pipe opened in
communicate() method must be called.
You will see that the output is in the bytes format. To fix that pass
universal_newlines=True as an argument in
process = subprocess.Popen("ls", stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
Read more about the Popen constructor for more details about the extra functionality it has.
You now know how to run shell commands in Python using two different packages and get the output from the command if needed.