A Python script that collects and stores cryptocurrency price data from the Bitget API into MongoDB time-series collections. This tool is designed to maintain a rolling window of the latest 3,000 or UNLIMITED records for specified trading pairs.
- Historical Data Fetching: Initial bulk fetch of up to 3,000 historical candlestick records per token.
- Time-Series Collections: Utilizes MongoDB time-series collections for efficient storage and querying.
- Gap Filling: Detects and fills missing data between the last recorded entry and the current time.
- Live Updates: Continuously fetches new data at the start of each minute.
- Duplicate Prevention: Checks for existing timestamps to avoid redundant data.
- Automatic Cleanup: Ensures only the latest 3,000 records are retained per token.
- Tokens:
ETHUSDT,ADAUSDT(configurable). - Interval: 1-minute candlesticks.
- Python 3.8+
- MongoDB Atlas cluster (or local MongoDB instance with time-series support).
- Python Libraries:
pip install requests pymongo python-dotenv
-
MongoDB Connection:
- Replace the
MONGODB_URIin the script with your MongoDB connection string. - Format:
mongodb+srv://<username>:<password>@cluster0.iittg.mongodb.net/alert3.
- Replace the
-
Script Parameters (Modify in Code):
TOKENS = ["ETHUSDT", "ADAUSDT"] # Add/remove trading pairs INTERVAL = "1min" # Supported: 1min, 5min, 15min, etc. (check Bitget API) LIMIT = 200 # Max records per API request (do not exceed API limits) TOTAL_RECORDS = 3000 # Records to retain per token SLEEP_TIME = 2 # Seconds between API requests to avoid rate limits
-
Clone the Repository:
git clone https://github.com/yourusername/crypto-data-collector.git cd crypto-data-collector -
Install Dependencies:
pip install -r requirements.txt # Create a requirements.txt with the libraries
python data_collector.py- MongoDB Connection Check: Verifies connectivity to your database.
- Create Time-Series Collections: If they don't exist.
- Fetch Initial Data: Populates up to 3,000 historical records per token.
- Fill Gaps: Ensures no missing data between the oldest record and the current time.
- Live Updates: Runs indefinitely, fetching new data every minute.
Each token has a dedicated time-series collection named <TOKEN>_timeseries (e.g., ETHUSDT_timeseries). Documents include:
{
"token": "ETHUSDT", // Trading pair
"timestamp": ISODate("2023-10-01T00:00:00Z"), // UTC time
"open": 1700.5, // Opening price
"high": 1712.3, // Highest price during the interval
"low": 1698.2, // Lowest price
"close": 1705.7, // Closing price
"base_volume": 450.2, // Volume in the base currency (e.g., ETH)
"quote_volume": 765432.1 // Volume in the quote currency (e.g., USDT)
}-
Connection Errors:
- Ensure the MongoDB URI is correct and whitelisted in Atlas.
- Check network connectivity.
-
No Data Fetched:
- Verify the token symbols match Bitget's supported pairs.
- Check Bitget API status for outages.
-
Gap Filling Issues:
- The loop condition in
fill_gaps()may contain a bug. Replace&withand:while fetched < total_minutes and total_minutes > 0:
- The loop condition in
-
Rate Limits:
- Increase
SLEEP_TIMEif encountering HTTP 429 errors.
- Increase
MIT License. Replace with your preferred license.
Note: Avoid committing sensitive data (e.g., MongoDB credentials). Use environment variables in production.