🔎 Internet-connected Devices Census Python Framework
- Description
- Grinder Workflow
- Grinder Map
- Requirements
- Current Features
- Setup and Configure Environment
- Build in Docker
- Usage
- Examples
- Add Your Own Queries
The Grinder framework was created to automatically enumerate and fingerprint different hosts on the Internet using different back-end systems: search engines, such as Shodan or Censys, for discovering hosts and NMAP engine for fingerprinting and specific checks. The Grinder framework can be used in many different areas of researches, as a connected Python module in your own project or as an independent ready-to-use from the box tool.
Grinder Framework can easily build an interactive map with found hosts in your browser:
Also, Grinder can show you some basic information:
- Python 3.6+
- Shodan and Censys accounts
Required to collect hosts, both free and full accounts are suitable. Also, it's possible to use only one account (Censys or Shodan, Shodan is preferable). - Nmap Security Scanner 7.60+
Version 7.60 and newer has been tested with currently used in Grinder scripts (ssl-cert.nse, vulners.nse, etc.). - TLS-Attacker
Required only for TLS scanning. Version 3.0 has been tested. - TLS-Scanner
Required only for TLS scanning.
- Collecting hosts and additional information using Shodan and Censys search engines
- Scanning ports and services with boosted multiprocessed Nmap Scanner wrapper
- Scanning vulnerabilities and additional information about them with Vulners database and Shodan CVEs database
- Retrieving information about SSL certificates
- Scanning for SSL/TLS configuration and supported ciphersuites
- Scanning for SSL/TLS bugs, vulnerabilities and attacks
- Building an interactive map with information about the hosts found
- Creating plots and tables based on the collected results
- Custom scanning scripts support (in LUA or Python3)
- Confidence filtering system support
- Special vendors scanning and filtering support
The Grinder framework is still in progress and got features to improve, so all the tasks and other features will always be described in this project. If you got some awesome ideas or any other interesting things for Grinder, you can always open a pull request or some issues in this repository.
- Install Nmap Security Scanner, if not installed.
- Clone the repository
git clone https://github.com/sdnewhop/grinder
- Clone and install TLS-Attacker.
- Clone TLS-Scanner in directory with Grinder and install it.
- Create virtual environment
cd grinder
python3 -m venv grindervenv
source grindervenv/bin/activate
- Check if virtual environment successfully loaded
which python
which pip
- Install project requirements in virtual environment
pip3 install -r requirements.txt
- Run the script
./grinder.py -h
- Set your Shodan and Censys keys via a command line argument
./grinder.py -sk YOUR_SHODAN_KEY -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET
or via an environment variable
export SHODAN_API_KEY=YOUR_SHODAN_API_KEY_HERE
export CENSYS_API_ID=YOUR_CENSYS_API_ID_HERE
export CENSYS_API_SECRET=YOUR_CENSYS_API_SECRET_HERE
- Deactivate virtual environment after use and restore default python interpreter
deactivate
- First, complete all steps from the "Setup and Configure Environment/Grinder Installing" section.
- After the scan is completed, go to the "map" folder
cd map/
- Run Flask (use
--helpkey for more information)
flask run
- Open in your browser
http://localhost:5000/
Also, Grinder map provides simple API methods such as /api/viewall/, /api/viewraw/<host_id>, you can learn more from list of application routes
flask routes
To build the Grinder framework as a docker image you can use the script docker-build.sh, and to run this image you can use the script docker-run.sh:
docker run -it --rm --volume $(pwd)/results:/code/results --volume $(pwd)/map:/code/map grinder-framework -husage: grinder.py [-h] [-r] [-u] [-q QUERIES_FILE] [-sk SHODAN_KEY] [-cu]
[-cp] [-ci CENSYS_ID] [-cs CENSYS_SECRET] [-cm CENSYS_MAX]
[-sm SHODAN_MAX] [-nm] [-nw NMAP_WORKERS] [-vs]
[-vw VULNERS_WORKERS] [-sc] [-vc VENDOR_CONFIDENCE]
[-qc QUERY_CONFIDENCE] [-v [VENDORS [VENDORS ...]]]
[-ml MAX_LIMIT] [-d]
The Grinder framework was created to automatically enumerate and fingerprint
different hosts on the Internet using different back-end systems
optional arguments:
-h, --help show this help message and exit
-r, --run Run scanning
-u, --update-markers Update map markers
-q QUERIES_FILE, --queries-file QUERIES_FILE
JSON File with Shodan queries
-sk SHODAN_KEY, --shodan-key SHODAN_KEY
Shodan API key
-cu, --count-unique Count unique entities
-cp, --create-plots Create graphic plots
-ci CENSYS_ID, --censys-id CENSYS_ID
Censys API ID key
-cs CENSYS_SECRET, --censys-secret CENSYS_SECRET
Censys API SECRET key
-cm CENSYS_MAX, --censys-max CENSYS_MAX
Censys default maximum results quantity
-sm SHODAN_MAX, --shodan-max SHODAN_MAX
Shodan default maximum results quantity.
-nm, --nmap-scan Initiate Nmap scanning
-nw NMAP_WORKERS, --nmap-workers NMAP_WORKERS
Number of Nmap workers to scan
-vs, --vulners-scan Initiate Vulners API scanning
-vw VULNERS_WORKERS, --vulners-workers VULNERS_WORKERS
Number of Vulners workers to scan
-sc, --script-check Initiate custom scripts additional checks
-vc VENDOR_CONFIDENCE, --vendor-confidence VENDOR_CONFIDENCE
Set confidence level for vendors
-qc QUERY_CONFIDENCE, --query-confidence QUERY_CONFIDENCE
Set confidence level for queries
-v [VENDORS [VENDORS ...]], --vendors [VENDORS [VENDORS ...]]
Set list of vendors to search from queries file
-ml MAX_LIMIT, --max-limit MAX_LIMIT
Maximum number of unique entities in plots and results
-d, --debug Show more information
Run the most basic enumeration with Shodan and Censys engines without map markers and plots (results will be saved in database and output JSON):
./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -rRun an enumeration with 10 Nmap scanning workers, where maximum Censys results is 555 hosts per query and maximum Shodan results is 1337 hosts per query, update map markers, count unique entities and create plots:
./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -u -q FILE_WITH_QUERIES.json -cu -cp -cm 555 -sm 1337 -nm -nw 10 -r Run an enumeration with filtering by vendors (only Nginx and Apache, for example) and confidence levels (only "Certain" level, for example) for queries and vendor:
./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -u -q FILE_WITH_QUERIES.json -v nginx apache -qc Certain -vc Certain -rRun an enumeration with 10 workers of Nmap Vulners API scanning:
./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -u -q FILE_WITH_QUERIES.json -vs -vw 10 -rRun an enumeration with custom scripts which are described in .json file with queries:
./grinder.py -sc -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -rRun Grinder with debug information about scanning:
./grinder.py -d -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -rFor more options and help use
./grinder.py -hTo add your own vendors and products with queries you can simply create a new .json file in the directory with queries and choose it while running Grinder in the "run" scan mode.
Format of file with queries:
[
{
"vendor": "YOUR OWN VENDOR HERE",
"product": "YOUR OWN PRODUCT HERE",
"shodan_queries": [
{
"query": "YOUR SHODAN QUERY HERE",
"query_confidence": "QUERY CONFIDENCE LEVEL {tentative|firm|certain}"
}
],
"censys_queries": [
{
"query": "YOUR CENSYS QUERY HERE",
"query_confidence": "QUERY CONFIDENCE LEVEL {tentative|firm|certain}"
}
],
"scripts": {
"py_script": "NAME OF PYTHON SCRIPT FROM /custom_scripts/py_scripts",
"nse_script": "NAME OF NSE SCRIPT FROM /custom_scripts/nse_scripts"
},
"vendor_confidence": "VENDOR CONFIDENCE LEVEL {tentative|firm|certain}"
},
{
}
]