Home Grown Extensions

However, more than this, mitmproxy also accepts DIY Python scripts as extensions. As an example, Listing 2 shows a script in Python 3, as required by mitmproxy, which picks up the URLs of all incoming responses and stores them together with the length of the corresponding web content in bytes in a newly created dump.log file.

01 #!/usr/bin/env python3
02 from mitmproxy import ctx
03 import re
04
05 class URLDumper:
06   def __init__(self):
07     ctx.log.warn( "URLDumper ready" )
08
09   def request(self, flow):
10     ctx.log.warn( "URLDumper request" )
11     flow.request.anticache()
12
13   def append_to_dump(self, url, content):
14     with open("dump.log", "a") as f:
15       f.write("%s (%d bytes)\n" %
16               (url, len(content)))
17
18   def response(self, flow):
19     url = flow.request.url
20     self.append_to_dump(url,
21             flow.response.content)
22
23 addons = [
24     URLDumper()
25 ]

To do this, it imports the ctx context object from the mitmproxy package in line 2, to be used later for calls to the console logging function. The package doesn't have to be installed anywhere; it's just that mitmproxy magically finds it when it imports the script, because it manipulates the Python interpreter's package search paths to its own installation.

By design, the proxy jumps to the request() method in line 9 before each request. This method uses the flow variable to field an object for the current web request and, for consistency, calls the request object's anticache() method. This simply discards headers like If-modified-since on every request to make sure the browser fetches items anew every time. Nevertheless, browsers sometimes still use a cache regardless and don't even bother to ask the server if an image has changed. If you run into that situation, hit Shift+Reload to force it to reload everything on the current page.

In addition, the proxy jumps to the response() method in Line 18 for each incoming response. There Listing 2 first extracts the URL of the request, then retrieves the content of the retrieved web resource as a string with a content attribute, and passes both to the append_to_dump() method from line 13 for logging. The function opens the dump.log file in append mode and appends the URL as well as the length of the content string in readable format. To see the script in action, run

$ mitmproxy --mode regular -s URLDumper.py\ --set console_eventlog_verbosity="warn"

which passes the name of the new Python script to the proxy and shows a message reading URLDumper ready in the proxy window footer shortly after launching. This is just a brief confirmation that the add-on's initialization function completed successfully.

For convenience, the proxy will be monitoring the Python script. If it changes at run time, the proxy notices this and reinitializes the script immediately. Figure 9 shows the results of the script in action: After a browser session that had localhost:8080 set as proxy and which visited the Linux Magazine website, the entries shown in Figure 9 were found in dump.log.

Figure 9: Using the extension in Listing 2, the proxy logged the URLs the browser called when visiting the Linux Magazine website in dump.log.

Looking for Trouble …

If an error occurs in the Python script, either during compilation or at run time, the console window displays an ominous message, namely that details are "in the event log." Some searching on the Internet solved the mystery: To view the log, you need to type

:console.view.eventlog

in the console window. The window then switches to the error messages log for which you were looking. Usually, it's easy to discover what was causing Python to complain.

Another killer mitmproxy application is the ability to log batteries of request sequences, in order to replay them later and expose the server to a preconfigured test case. The man pages – for operating both the console and the API – can be found on mitmproxy.org. The content looks a bit unkempt, but patient searching will usually reveal the information you desire.

Oh, and before I forget: If you do not want to leave a gaping vulnerability on your system after testing with mitmproxy, you need to remove the CA certificate you added for test purposes after completing the tests. Better safe than sorry!

The Author

Mike Schilli works as a software engineer in the San Francisco Bay area, California. Each month in his column, which has been running since 1997, he researches practical applications of various programming languages. If you email him at mailto:mschilli@perlmeister.com he will gladly answer any questions.