Learn to write command line utilities in Python

shebang and docstring

Next we’re going to look at each section in turn to look at the functionality.

#!/usr/bin/env python
"""
A sorting hat you can run on the command line
"""

That very first line is referred to as a ‘shebang’ and it tells your command line shell (of which there are many, but ‘bash’ is the most common) which program to use to execute everything that follows. In this case we’re using a command called env to tell bash where to find python.

Note: I’m using python 3 for this example. On some systems that have both python 2 and 3, 3 is referred to as python3, not just python. If that’s the case for you, you’ll need to modify this script to reflect that.

After the shebang is a standard python docstring, just telling you what the app is all about.

import

import argparse
import hashlib

Next we import the external modules we’re going to use. Lucky for us, python has an extensive and varied standard library of modules that ship with it, so we don’t need to install anything extra.

‘argparse’ will parse command line arguments for us. If you think of a command line tool like ls, arguments are things you can put after it to modify its behaviour. For example ls -l has -l as the argument and causes ls to print ‘longer’ output with more information than the standard output. For ls *.wav, the *.wav argument is a pattern which causes ls to only emit files that match that pattern.

‘hashlib’ is a module that implements various hash and message digest algorithms, which we’ll need later on for the sorting part of the utility.

Handling arguments

PARSER = argparse.ArgumentParser()
# add a positional argument
PARSER.add_argument("name", help="name of the person to sort")
# Add a debug flag
PARSER.add_argument("-d", "--debug", help="enable debug mode",
                    action="store_true")
# Add a short output flag
PARSER.add_argument("-s", "--short", help="output only the house",
                    action="store_true")
ARGV = PARSER.parse_args()

This block sets up a new argument parser for us and adds some arguments to it. Arguments that don’t start with -- are ‘positional’, which basically means that it’s a mandatory argument. If you define multiple positional arguments they must be specified at run-time in the order they are defined.

In our case, if we don’t specify the ‘name’, then we’ll get an error:

$ ./sortinghat.py
usage: sortinghat.py [-h] [-d] [-s] name
sortinghat.py: error: the following arguments are required: name

We didn’t have to create this error message, argparse did that for us because it knows that ‘name’ is a required argument.

The other arguments are ‘flags’, which means we can turn things on and off with them. Flags are specified with -- for the long form and - for the short form, you don’t have to have both but this has developed into something of a convention over the years. Specifying them separately like this is also useful as it gives you full control over how the short options relate to the longer ones.

If, for instance, you wanted two arguments in your application called --force and --file, the convention would be to use -f as the short form, but you can’t use it for both. Explicitly assigning the short form version allows you to decide what you want to use instead. Maybe you’d go for -i for an input file or -o for an output file or something like that.

These arguments are flags because we set action="store_true" in them, which stores True if they’re set and False if they’re not.

If you omit the action="store_true", you get an optional argument. This could be something like --file /path/to/file, where you must specify something immediately after the argument. You can use these for specifying additional parameters for your scripts. We’re not really covering that in this script though, so here are a few quick examples to get you thinking:

• --config /path/to/config_file – specify an alternate config file to use instead of the default
• --environment production – run against product data rather than test data
• --algo algorithm_name – use a different algorithm instead of the default
• --period weekly – change the default calculation period of your utility
• --options /path/to/options/file – provide options for an analysis from an external file

Another freebie we get from argparse is -h and --help. These are built-in and print nicely formatted help output for your users or future-self!

$ ./sortinghat.py -h
usage: sortinghat.py [-h] [-d] [-s] name

positional arguments:
  name         name of the person to sort

optional arguments:
  -h, --help   show this help message and exit
  -d, --debug  enable debug mode
  -s, --short  output only the house

Lastly for this section, we use parse_args() to assign the arguments that have been constructed to a new namespace called ARGV so we can use them later. Arguments stored in ARGV are retrievable using the long version of the argument name, so in this example: ARGV.name, ARGV.debug and ARGV.short.

Everything from this point onward is largely to do with the functionality of the utility, not the command line execution of it, so we’ll go through it quite quickly.

Printing debug messages

I didn’t want to get bogged down using a proper logging library for this small tool, so this function takes care of our very basic needs for us.

def debug_msg(*args):
    """prints the message if the debug option is set"""
    if ARGV.debug:
        print("DEBUG: {}".format("".join(args)))

Essentially, it will only print a message if ARGV.debug is True and that will only be true if we set the -d flag when we run the tool on the command line.

We can then put messages like debug_msg("Debug option is set") in our code and they’ll do nothing unless that -d flag is set. If it is set, you’ll get output like:

$ ./sortinghat.py -d Mark
DEBUG: Debug option is set
DEBUG: Your name is - Mark
DEBUG: The name_hash is - f1b5a91d4d6ad523f2610114591c007e75d15084
DEBUG: The house_key is - f
Hello Mark, you can join Slytherin!

Using a technique like this – or perhaps a --verbose flag – can help to provide additional information about what’s going on inside your utility at run time that could be helpful to others or your future-self if they encounter any difficulties with it.

The debug_msg() function is used in this way throughout the rest of the program.

Figuring out the house

To figure out what house to assign someone to we use the same approach that we did for the R version. We calculate the hash of the input name and store the hexadecimal representation. Since hex uses the numbers 0-9 and the characters a-f, we can assign the four Hogwarts houses to these 16 symbols evenly in a Python dictionary.

We can then use the first character of the input name hash as the key when retrieving the value from the dictionary

HOUSES = {"0" : "Hufflepuff",
          "1" : "Gryffindor",
          "2" : "Ravenclaw",
          "3" : "Slytherin",
          "4" : "Hufflepuff",
          "5" : "Gryffindor",
          "6" : "Ravenclaw",
          "7" : "Slytherin",
          "8" : "Hufflepuff",
          "9" : "Gryffindor",
          "a" : "Ravenclaw",
          "b" : "Slytherin",
          "c" : "Hufflepuff",
          "d" : "Gryffindor",
          "e" : "Ravenclaw",
          "f" : "Slytherin"
         }
NAME_HASH = hashlib.sha1(ARGV.name.lower().encode('utf-8')).hexdigest()
HOUSE_KEY = NAME_HASH[0]
HOUSE = HOUSES[HOUSE_KEY]

We also make sure that the input name is converted to lower case first to prevent us from running into any discrepancies between, for example, ‘Mark’ and ‘mark’.

Printing output

Here in the final section, we use the value of ARGV.short to decide whether to print the long output or the short output. Flags are False by default with argparse, so we can test if it’s been set to True (by specifying the -s flag on the command line) and print accordingly.

if ARGV.short:
    print(HOUSE)
else:
    print("Hello {}, you can join {}!".format(ARGV.name, HOUSE))

Using the -s flag on the command line results in the following short output:

$ ./sortinghat.py -s Mark
Slytherin

Since the flags are optional you can combine them if you need to, so something like ./sortinghat.py -s -d Mark will produce the expected output – debug info with the short version of the final message.