If things just don't work when you are developing a web application, the question immediately arises as to what data the browser and web server are actually exchanging. Tools for snooping on the network such as Wireshark (as well as proxies that sit between the client and the server) leave both requests and responses untouched, while logging them for inspection.

mitmproxy, which stands for man in the middle (MITM) proxy, is the king of the hill in this category; it makes the impossible possible by logging encrypted HTTPS requests. But first, let's look at the simplest, unencrypted case, for which my aging website perlmeister.com that still uses plain old unencrypted HTTP is a great choice.

Figure 1 shows how the browser retrieves the requested page's HTML text, along with some images and JavaScript snippets from the server. The mitmproxy tool, which is available for download as binary from [1], sports a terminal user interface (UI), which displays a double arrow to the left of the current request, called Flow in mitmproxy parlance. When you press the Enter key, the detailed request data come up, as shown in Figure 2.

Figure 1: The proxy server logs the browser's HTTP requests while pages are being retrieved.

Figure 2: At the push of the Enter key, the proxy shows details of the request/response data.

Using the cursor keys (or, in typical Vi-style: h, j, k, l), you can move to the left/down/up/right and jump between Request, Response, and Details on the request detail page. You can always go back to the next level up with Q. On the main screen listing the flows, use the cursor keys (or J for down and K for up) to select a specific one. Press D to clear the current flow (marked >>), and hammer D repeatedly to clear the whole screen.

Command-Line Option

There are several options to place mitmproxy between the client and the server in order to start snooping on the data exchanged. The simplest approach is configuring it on the client side, where applications typically provide options to configure HTTP or SOCKS proxies. To start the proxy, call mitmproxy from the command line, which will render the currently running terminal black, and log intercepted data, while waiting for user commands from the keyboard.

If you try to persuade Chromium on Ubuntu to use a proxy, officially located under Settings | Advanced/Security | Open Proxy Configuration, you are in for a surprise, because the desktop application won't allow it and outputs an error message (Figure 3). Fortunately, the program accepts the command-line option

$ chromium-browser --proxy-server="localhost:8080"

Figure 3: On Ubuntu, Chromium rejects the Proxy configuration in the Settings menu.

telling it to use a proxy server on the host (localhost) and port (8080), which is where mitmproxy listens by default. The important thing is to run Chromium as a single instance on the Ubuntu desktop. If you already have another instance running, you might find yourself tearing your hair out in frustration because no requests show up in the logs. If it's configured correctly, however, when you access URLs in the browser, the data will appear in the proxy UI for further analysis.

Breaking Encryption

So far, so good. But what happens if the browser sends an HTTPS request? By design, the protocol will prevent sniffing attacks from men in the middle. And, lo and behold, a Chromium browser configured to use mitmproxy reports that an attack is in progress (Figure 4).

Figure 4: If mitmproxy is listening in on an HTTPS connection, the browser spots the spoofed certificate.

But if you then go to Advanced and click on the Proceed (unsafe) link, Chromium lifts the barrier and lets you continue at your own risk, while mitmproxy in turn will eagerly log the connection data. This may seem like a silly question, but how does it do that if the client and server are using an encrypted connection?

By using a revolutionary trick, mitmproxy identifies itself to the client as the server, and to the server as the client (Figure 5). Instead of sending back the actual server response, it sends the client a fake certificate. This leads to the client – assuming it doesn't check or validate the certificate – opening an encrypted connection to the proxy instead of the server, if the user continues despite the warning.

Figure 5: mitmproxy sits between the client and the server and pretends to be the server for the client, and the client for the server.

By being in possession of the encryption keys, the proxy can now easily decrypt and log the data going over the wire (Figure 6) that it then encrypts again for forwarding to the server, which thinks it is talking to the client. In other words, a classic MITM attack.

Figure 6: Doing the impossible: mitmproxy sniffing an encrypted HTTPS connection.

Hard Cases

If the client does not allow communication over obviously hacked channels even if the user tries to force it, the sys admin can get in on the data heist and install a spoofed Certificate Authority (CA) certificate on the client, which mitmproxy conveniently provides. However, many clients (e.g., mobile phones) do not allow proxy configurations at all or simply ignore them. For these cases, mitmproxy offers the option of a transparent proxy mode, in which the data are routed to the proxy instead of to the server using lower-level network tools such as iptables. The documentation on [2] illustrates solutions for these trickier cases.

Home-grown programs written in Go rely on the system's CA store for encrypted communications. On Ubuntu, it comes with the ca-certificates package. Out of the box, it doesn't contain the counterfeit mitmproxy CA certificate, of course, but the sys admin can add it. As an example, Listing 1 [3] submits an HTTPS request to get the TLS-protected website of a well-known online giant. If mitmproxy is running, the program aborts the request with the following error message if the client is rerouted via the proxy server by setting the HTTP_PROXY environment variable:

01 package main
02
03 import (
04   "fmt"
05   "io/ioutil"
06   "net/http"
07 )
08
09 func main() {
10   resp, err := http.Get("https://amazon.com")
11   if err != nil {
12     panic(err)
13   }
14   defer resp.Body.Close()
15   body, err := ioutil.ReadAll(resp.Body)
16   fmt.Println(string(body))
17 }

$ go build client.go
$ HTTP_PROXY=localhost:8080 ./client
panic: Get https://usarundbrief.com: x509:
certificate signed by unknown authority

What happened? Instead of Amazon's certificate, the client received a self-signed certificate, injected by mitmproxy, and the client rightfully complains about it. Now, to convince this pesky client to accept the certificate anyway (for testing purposes only, of course), the sys admin copies it into the Ubuntu system collection using the command sequence in Figure 7. And, sure enough, the following call to

$ HTTP_PROXY=localhost:8080 ./client

Figure 7: This command sequence installs the counterfeit CA certificate on the Ubuntu system.

displays the TLS-protected Amazon website without any complaints, while mitmproxy logs the transaction as an MITM.

By the way, Chromium on Ubuntu does not use the system store for trusted CAs; instead, it comes with its own collection. In order to add the counterfeit certificate to this collection, a few tricks are required, which mitmproxy explains when the user accesses the magic mitm.it domain in the browser with the proxy configured, after clicking the button with the desired operating system (Figure 8). It works similarly on Firefox, which offers a configuration menu for adding a new certificate below Options | Privacy & Security | View Certificates | Authorities | Import.

Figure 8: For instructions on palming off a mitmproxy-compliant CA certificate on Linux, you need to press Other here.