Analyse requests sent from your mobile device while using any app
Read more about the project at timsh.org
Fill in the form if you found something interesting in the requests - let's analyse all apps from the list together!
https://forms.gle/CE6y7XkpRNJkGqEeA
Follow this process
-
Install mitmproxy https://docs.mitmproxy.org/stable/overview-installation/ on mac, simply use
brew install mitmproxybe aware that mitmproxy is recognised as malware by some antiviruses - looks scary but it's not.
-
Turn on developer mode on iPhone / Android device I did the whole experiment on iOS so I will not include any specific instructions for Android (though they should be even easier). https://developer.apple.com/documentation/xcode/enabling-developer-mode-on-a-device
You will need to have developer mode turned on for the next step.
-
Configure mitmproxy and iphone to work together
-
Launch mitmproxy in terminal. I prefer to use mitmweb because the interface is actually very helpful for initial discovery (and understanding the scale of the RTB requests).
Use this command to launch the proxy that by default won't listen to the traffic on your computer (or any other device) without manually connecting to a proxy.
mitmweb --listen-host 0.0.0.0 --listen-port 8080 -
Now
ipconfig getifaddr en0to find your computer local IP address. By the way, your iphone and computer must be in the same wifi network for all of this to work. -
Next, open the settings on iphone and setup manual proxy with:
server= the ip address you just foundport= 8080 -
On iphone, open browser and go to mitm.it further instructions are described here, TLDR: you need to install the certificate and enable full trust to be able to decrypt TLS-encrypted traffic.
https://jasdev.me/intercepting-ios-traffic https://support.apple.com/en-us/102390
-
-
We're all set! Now you're able to intercept and decrypt all traffic going through iPhone. If you only want to record traffic coming from a specific app, close all apps, "Clear flows" in MitmWeb and then open the desired app.
-
Take any app from the list (or just any app)
In order to download it from App Store, you might have to turn off proxy on iphone, download the app and then turn it on again and clear the flows.
-
Open the app and wait / click / play - you'll immediately see hundreds of requests flowing in mitmweb.
-
When you feel like there's enough (you could even leave it open or play for an hour or so to collect more data), close the app and switch off the phone, then in mitmweb press File → Save all.
This will give you a
flowsfile - rename it as "appname.flow" -
Open the mitm_test.ipynb - either in local Jupyter Notebook or in Google Colab, both work fine. Further instructions are included in the file itself.
-
Repeat steps 5-7 for as much apps as you need, just don't forget to clear the flows before each recording. When you're done, press
Ctrl+Cin terminal to stop mitmproxy and turn off proxy on iphone. If that's your main device, you also MUST turn off the certificate trust setting that you enabled before.
Check the instructions in visualise_domains.ipynb to create a visualisation of domain and subdomain frequency in the data - just like this one: