The 'with' Statement in Python

Jun 5, 2021

Coming from C++, one of the more intimidating features of base Python was the with keyword. There's really no equivalent in C++, besides possibly creating something within a local scope and defining a class destructor.

To make matters worse, most Python tutorials just explain that with is the accepted way of opening files, but don't describe the mechanisms behind this powerful feature. This led me to believe that there was an entire keyword dedicated to just opening (and closing) files, and there was some magic behind the open function that was hidden to developers.

Not until I did a little more digging did I realize that with could be used for a wide assortment of applications, and my own classes could implement their own code to execute when using the with statement.

First, what does the with statement actually do? It enters a context in which an object can be considered active. When entering the context, it performs some action on the object. In the context of the open function, it opens a file handle. Upon exiting the context (whether that be through standard code execution, an exception being raised, the program exiting, etc), the with statement performs an additional action on the object. I'd assume this is typically the reverse of what was done to enter the context, but in theory, it can actually be anything. In the case of the open function, the file handle is closed.

Let's explore the standard example of opening a file:

1
2
3

with open('city_names.txt') as file_handle:
    for line in file_handle:
        print(line, end='')

This outputs the content of the file 'city_names.txt', shown below:

Boston
New York
Newark
Scranton
Toronto
...

So what's happening here?

When open is called on city_names.txt, the with statement enters a context where city_names.txt is open for reading. It establishes a contract with the application that it will call close on the file handle when the context is exited, no matter how it is exited. It also returns the file handle, which the application can access with the as keyword. Theoretically, the application can skip the as clause and it can still be completely valid code. However, I'm not sure how useful it would be in this case.

Once the context is entered, the open file handle is looped through line-by-line, printing each. After the loop exits, the with block is exited, closing the file handle with it.

It's important to note that if the below code is executed, the file handle will still be closed, even though the program completes with an error:

1
2
3
4

with open('city_names.txt') as file_handle:
    for line in file_handle:
        print(line, end='')
    raise Exception

This is the major benefit of using with - the application developer can always be sure that code will be executed (in this case, the close function) when a context is exited, even if the exit is not performed cleanly.

To compare, the alternative would be overly verbose and error prone:

1
2
3
4
5
6
7
8
9

try:
    file_handle = open('city_names.txt')
    for line in file_handle:
        print(line, end='')
    raise Exception
except:
    if not file_handle.closed:
        file_handle.close()
    raise

Using the with statements with your own objects is fairly straightforward and requires defining two dunder methods within a class:

• The __enter__ method does whatever it needs to enter the context and returns an object to manipulate in the context (or None if appropriate).
• The __exit__ method presumably cleans up what __enter__ has created and handles any exceptions which are raised during the processing of the context.

The function signature for __enter__ is very simple and takes self. The return value can be bound to a variable using the as keyword in the with statement.

The function signature for __exit__ is a little bit more complicated since it needs to gather information about any exceptions which are raised during the processing of the context. At its simplest, the __exit__ method would look like this:

1
2

def __exit__(self, exc_type, exc_value, traceback):
    return True

• The exc_type argument contains the exception type. If a ValueError was raised during processing of the context, exc_type would equal ValueError.
• The exc_value argument contains the exception instance. This may contain important information about the exception for handling, such as the error message.
• The traceback contains the traceback object.

The return value of the __exit__ method tells the interpreter whether an exception was handled successfully. If an exception was raised during the context and __exit__ returns None or anything but True, the exception will be propagated forward after __exit__ completes.

If a context completes without any exceptions, exc_type, exc_value, and traceback are all set to None by the interpreter.

Here's a very simple example which wraps the open function to show how __enter__ and __exit__ are called by the interpreter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

class OpenWrapper:
    ''' A class which wraps the open function '''
    
    def __init__(self, file_name):
        ''' Class constructor

            :param file_name:  Path to file
            :return:  New object
            '''
        self._file_name = file_name
        self._file_handle = None

    def __enter__(self):
        ''' Enters context by opening file and returning handle

            :return: 
            '''
        print('__enter__(self)')
        self._file_handle = open(self._file_name)
        return self._file_handle

    def __exit__(self, exc_type, exc_value, traceback):
        ''' Exits context by closing file handle

            :return: None
            '''
        print(f'__exit__(self, {exc_type}, {exc_value}, {traceback})')
        self._file_handle.close()
        return None

# Main function
if __name__ == '__main__':
    print('begin program')

    # Construct OpenWrapper
    wrapper = OpenWrapper('city_names.txt')
    print('OpenWrapper constructed')

    # Call __enter__ on wrapper, which binds the return value
    # (OpenWrapper._file_handle) to fh using the 'as' keyword.
    # 
    # Note that wrapper could be constructed on this line as
    # well. I've broken the construction and the 'with' keywords
    # into two lines for verbosity.
    with wrapper as fh:
        for line in fh:
            print(line, end='')

    # After context exits, OpenWrapper.__exit__() is called
    print('end program')

Execution of this script results in:

begin program
OpenWrapper constructed
__enter__(self)
Boston
New York
Newark
Scranton
Toronto
...
__exit__(self, None, None, None)
end program

Since the context completed without any exceptions, the __exit__ method was passed self, None, None, None and no exception was propagated forward. If this program raised an EOFError after the for loop continued, the __exit__ method could be updated to quietly handle this error and still do its due diligence in closing the file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Previous file contents omitted for brevity...
    def __exit__(self, exc_type, exc_value, traceback):
        ''' Exits context by closing file handle

            :return: None
            '''
        print(f'__exit__(self, {exc_type}, {exc_value}, {traceback})')

        # Close file regardless of error type
        self._file_handle.close()

        # If EOFError was encountered, notify interpreter that it has
        # been successfully handled. Other errors will be propaged
        # forward. 
        if exc_type is EOFError:
            return True

# Main function
if __name__ == '__main__':
    with OpenWrapper('city_names.txt') as fh:
        for line in fh:
            print(line, end='')
        raise EOFError('EOF encountered')

This would result in:

__enter__(self)
Boston
New York
Newark
Scranton
Toronto
Manchester
Augusta
Denver
Miami
__exit__(self, <class 'EOFError'>, EOF encountered, <traceback object at 0x7f4cede049c0>)

Now that an actual error was encountered, the last three parameters of __exit__ are populated. Since the exc_type is defined to be EOFError, __exit__ does not propagate the exception forward and the program ends without a traceback. If the exception encountered was actually a ValueError, the output would have looked like this:

__enter__(self)
Boston
New York
Newark
Scranton
Toronto
...
__exit__(self, <class 'ValueError'>, ValueError encountered, <traceback object at 0x7fd284704a00>)
Traceback (most recent call last):
  File 'test.py', line 43, in 
    raise ValueError('ValueError encountered')
ValueError: ValueError encountered

Although it doesn't have a parallel in other languages I've worked with, I've quickly come to love using the with keyword, and I find it's an incredibly important tool to make robust code that's easy to interpret and maintain. Though initially confusing, once I had a visual on the internal workings, I find it was an easy concept to grasp and I find myself wishing it was available in my C++ and MATLAB work.

Time to implement a workaround, I guess...