Troubleshooting HTTPS connections with mitmproxy
Home Grown Extensions
However, more than this, mitmproxy
also accepts DIY Python scripts as extensions. As an example, Listing 2 shows a script in Python 3, as required by mitmproxy
, which picks up the URLs of all incoming responses and stores them together with the length of the corresponding web content in bytes in a newly created dump.log
file.
Listing 2
URLDumper.py
01 #!/usr/bin/env python3 02 from mitmproxy import ctx 03 import re 04 05 class URLDumper: 06 def __init__(self): 07 ctx.log.warn( "URLDumper ready" ) 08 09 def request(self, flow): 10 ctx.log.warn( "URLDumper request" ) 11 flow.request.anticache() 12 13 def append_to_dump(self, url, content): 14 with open("dump.log", "a") as f: 15 f.write("%s (%d bytes)\n" % 16 (url, len(content))) 17 18 def response(self, flow): 19 url = flow.request.url 20 self.append_to_dump(url, 21 flow.response.content) 22 23 addons = [ 24 URLDumper() 25 ]
To do this, it imports the ctx
context object from the mitmproxy
package in line 2, to be used later for calls to the console logging function. The package doesn't have to be installed anywhere; it's just that mitmproxy
magically finds it when it imports the script, because it manipulates the Python interpreter's package search paths to its own installation.
By design, the proxy jumps to the request()
method in line 9 before each request. This method uses the flow
variable to field an object for the current web request and, for consistency, calls the request object's anticache()
method. This simply discards headers like If-modified-since
on every request to make sure the browser fetches items anew every time. Nevertheless, browsers sometimes still use a cache regardless and don't even bother to ask the server if an image has changed. If you run into that situation, hit Shift+Reload to force it to reload everything on the current page.
In addition, the proxy jumps to the response()
method in Line 18 for each incoming response. There Listing 2 first extracts the URL of the request, then retrieves the content of the retrieved web resource as a string with a content
attribute, and passes both to the append_to_dump()
method from line 13 for logging. The function opens the dump.log
file in append mode and appends the URL as well as the length of the content string in readable format. To see the script in action, run
$ mitmproxy --mode regular -s URLDumper.py\ --set console_eventlog_verbosity="warn"
which passes the name of the new Python script to the proxy and shows a message reading URLDumper ready in the proxy window footer shortly after launching. This is just a brief confirmation that the add-on's initialization function completed successfully.
For convenience, the proxy will be monitoring the Python script. If it changes at run time, the proxy notices this and reinitializes the script immediately. Figure 9 shows the results of the script in action: After a browser session that had localhost:8080
set as proxy and which visited the Linux Magazine website, the entries shown in Figure 9 were found in dump.log
.
Looking for Trouble …
If an error occurs in the Python script, either during compilation or at run time, the console window displays an ominous message, namely that details are "in the event log." Some searching on the Internet solved the mystery: To view the log, you need to type
:console.view.eventlog
in the console window. The window then switches to the error messages log for which you were looking. Usually, it's easy to discover what was causing Python to complain.
Another killer mitmproxy
application is the ability to log batteries of request sequences, in order to replay them later and expose the server to a preconfigured test case. The man pages – for operating both the console and the API – can be found on mitmproxy.org. The content looks a bit unkempt, but patient searching will usually reveal the information you desire.
Oh, and before I forget: If you do not want to leave a gaping vulnerability on your system after testing with mitmproxy
, you need to remove the CA certificate you added for test purposes after completing the tests. Better safe than sorry!
Infos
- Linux binaries for mitmproxy: https://mitmproxy.org/downloads/#4.0.4/
- mitmproxy how-to: https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/
- Listings for this article: ftp://ftp.linux-magazine.com/pub/listings/linux-magazine.com/224/
« Previous 1 2
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
So Long Neofetch and Thanks for the Info
Today is a day that every Linux user who enjoys bragging about their system(s) will mourn, as Neofetch has come to an end.
-
Ubuntu 24.04 Comes with a “Flaw"
If you're thinking you might want to upgrade from your current Ubuntu release to the latest, there's something you might want to consider before doing so.
-
Canonical Releases Ubuntu 24.04
After a brief pause because of the XZ vulnerability, Ubuntu 24.04 is now available for install.
-
Linux Servers Targeted by Akira Ransomware
A group of bad actors who have already extorted $42 million have their sights set on the Linux platform.
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.
-
XZ Gets the All-Clear
The back door xz vulnerability has been officially reverted for Fedora 40 and versions 38 and 39 were never affected.
-
Canonical Collaborates with Qualcomm on New Venture
This new joint effort is geared toward bringing Ubuntu and Ubuntu Core to Qualcomm-powered devices.
-
Kodi 21.0 Open-Source Entertainment Hub Released
After a year of development, the award-winning Kodi cross-platform, media center software is now available with many new additions and improvements.
-
Linux Usage Increases in Two Key Areas
If market share is your thing, you'll be happy to know that Linux is on the rise in two areas that, if they keep climbing, could have serious meaning for Linux's future.
-
Vulnerability Discovered in xz Libraries
An urgent alert for Fedora 40 has been posted and users should pay attention.