8000 ccxt.async_support · Issue #24207 · ccxt/ccxt · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccxt.async_support #24207

Open
DaniilSkyRable opened this issue Nov 8, 2024 · 18 comments · May be fixed by #24217
Open

ccxt.async_support #24207

DaniilSkyRable opened this issue Nov 8, 2024 · 18 comments · May be fixed by #24217
Assignees
@Dan-krm
Labels
question

Comments

@DaniilSkyRable
Copy link
DaniilSkyRable commented Nov 8, 2024 •
edited
Loading

Operating System

Windows 11

Programming Languages

Python

CCXT Version

4.4.27

Description

i have some problem,idk why code dont work.
Log -
Fetched data from bingx: 1039 tickers received
Fetched data from okx: 605 tickers received
Fetched data from phemex: 377 tickers received
Fetched data from bybit: 629 tickers received
Fetched data from bitget: 885 tickers received
Fetched data from gateio: 3796 tickers received
Error fetching data from mexc: mexc GET https://contract.mexc.com/api/v1/contract/detail
Fetched data from huobi: 778 tickers received
Fetched data from kucoin: 1200 tickers received
Fetched data from bitrue: 1644 tickers received
Fetched data from bitmart: 1153 tickers received
Error fetching data from binance: binance GET https://fapi.binance.com/fapi/v1/exchangeInfo
Error fetching data from xt: xt GET https://dapi.xt.com/future/market/v1/public/symbol/list
Fetched data from bitfinex: 447 tickers received
binance: 0 тикеров получено.
xt: 0 тикеров получено.
gateio: 3796 тикеров получено.
bitget: 885 тикеров получено.
bingx: 1039 тикеров получено.
bitmart: 1153 тикеров получено.
bitrue: 1644 тикеров получено.
huobi: 778 тикеров получено.
kucoin: 1200 тикеров получено.
bitfinex: 447 тикеров получено.
phemex: 377 тикеров получено.
bybit: 629 тикеров получено.
okx: 605 тикеров получено.
mexc: 0 тикеров получено.

Code

  import ccxt.async_support as ccxt
import asyncio
import sys
import json
import os

if sys.platform.startswith('win'):
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

with open(os.path.join(os.path.dirname(__file__), "exchanges_config.json"), "r") as file:
    exchanges_config = json.load(file)

exchanges = {
    name: getattr(ccxt, name)(config)
    for name, config in exchanges_config.items()
}

async def fetch_tickers(exchange_name, exchange):
    try:
        exchange.options['defaultType'] = 'spot'
        exchange.options['type'] = 'spot'
        exchange.options['marginMode'] = 'spot'

        tickers = await exchange.fetch_tickers(params={'type': 'spot'})
        print(f"Fetched data from {exchange_name}: {len(tickers)} tickers received")
        return tickers
    except Exception as e:
        print(f"Error fetching data from {exchange_name}: {e}")
        return {}
    finally:
        await exchange.close()

async def main():
    tasks = []

    for exchange_name, exchange in exchanges.items():
        tasks.append(fetch_tickers(exchange_name, exchange))

    results = await asyncio.gather(*tasks)

    for exchange_name, tickers in zip(exchanges.keys(), results):
        print(f"{exchange_name}: {len(tickers)} тикеров получено.")

asyncio.run(main())
@Dan-krm
Copy link
Contributor
Dan-krm commented Nov 9, 2024

Hi @DaniilSkyRable at first glance it looks like the marginMode is set incorrectly, the marginMode should be either 'cross' or 'isolated' and is only for spot-margin markets

Another thing is you don't need to set the type option, defaultType option and type param. Only one needs to be set since they all do the same thing.

@DaniilSkyRable
Copy link
Author

Hi @Dan-krm ,why do I need to set marginmode if it is spot trading, I want to get only spot pairs, but I noticed that it gives tokens not only in spot pairs, but also from exchangers, can you tell me how I can get only spot pairs and only, if you do not mind?

@Dan-krm
Copy link
Contributor
Dan-krm commented Nov 9, 2024 •
edited
Loading

Hi @Dan-krm ,why do I need to set marginmode if it is spot trading, I want to get only spot pairs, but I noticed that it gives tokens not only in spot pairs, but also from exchangers, can you tell me how I can get only spot pairs and only, if you do not mind?

In that case you don't need to set marginMode and can leave it blank, that could be why you're getting more than just spot tickers. Also the signature for fetchTickers is fetchTickers(symbols, params) and it looks like you're setting the params in the symbols argument.

I would change this code snippet:

    try:
        exchange.options['defaultType'] = 'spot'
        exchange.options['type'] = 'spot'
        exchange.options['marginMode'] = 'spot'

        tickers = await exchange.fetch_tickers(params={'type': 'spot'})
        print(f"Fetched data from {exchange_name}: {len(tickers)} tickers received")
        return tickers

To this:

    try:
        tickers = await exchange.fetch_tickers(None, params={'type': 'spot'})
        print(f"Fetched data from {exchange_name}: {len(tickers)} tickers received")
        return tickers

or this:

    try:
        exchange.options['type'] = 'spot'
        tickers = await exchange.fetch_tickers()
        print(f"Fetched data from {exchange_name}: {len(tickers)} tickers received")
        return tickers

If there is any exchanges returning more than just the spot pairs still, you can let me know which exchange and I can take a closer look and see if I can adjust it

@DaniilSkyRable
Copy link
Author
DaniilSkyRable commented Nov 9, 2024 •
edited
Loading

MEX,Binance,XT, dont work ,
https://dapi.xt.com/future/market/v1/public/symbol/list
binance GET https://fapi.binance.com/fapi/v1/exchangeInfo
https://api.mexc.com/api/v3/exchangeInfo
This code cant work in async, why?

@Dan-krm
Copy link
Contributor
Dan-krm commented Nov 9, 2024 •
edited
Loading

MEX,Binance,XT, dont work , https://dapi.xt.com/future/market/v1/public/symbol/list binance GET https://fapi.binance.com/fapi/v1/exchangeInfo https://api.mexc.com/api/v3/exchangeInfo This code cant work in async, why?

Those are swap/future endpoints for xt and binance, and the mexc endpoint you shared isn't for fetchTickers.

If you have the subType option set somewhere like when you initialize the exchange it could be defaulting to using the swap/future endpoints instead of spot on some exchanges, so I'd remove setting that if possible

@DaniilSkyRable
Copy link
Author
DaniilSkyRable commented Nov 9, 2024 •
edited
Loading

MEX,Binance,XT, dont work , https://dapi.xt.com/future/market/v1/public/symbol/list binance GET https://fapi.binance.com/fapi/v1/exchangeInfo https://api.mexc.com/api/v3/exchangeInfo This code cant work in async, why?

Those are swap/future endpoints for xt and binance, and the mexc endpoint you shared isn't for fetchTickers.

If you have the subType option set somewhere like when you initialize the exchange it could be defaulting to using the swap/future endpoints instead of spot on some exchanges, so I'd remove setting that if possible

This my config, check this

{
    "binance": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot",
            "type": "spot"
        },
        "urls": {
            "api": {
                "public": "https://api.binance.com/api/v3",
                "private": "https://api.binance.com/api/v3"
            }
        }
    },
    "xt": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        },
        "urls": {
            "api": {
                "public": "https://api.xt.com/v4",
                "private": "https://api.xt.com/v4"
            }
        }
    },
    "gateio": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "bitget": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "bingx": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "bitmart": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "bitrue": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "huobi": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "kucoin": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "bitfinex": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "phemex": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "bybit": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "okx": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot"
        }
    },
    "mexc": {
        "enableRateLimit": true,
        "options": {
            "defaultType": "spot",
            "type": "spot"
        },
        "urls": {
            "api": {
                "public": "https://www.mexc.com/open/api/v3",
                "private": "https://www.mexc.com/open/api/v3"
            }
        }
    }
}

@Dan-krm
Copy link
Contributor
Dan-krm commented Nov 10, 2024

@DaniilSkyRable I would remove the urls from your configuration and try again, those generally do not need to be set

@DaniilSkyRable
Copy link
Author

I've tried it without them, and it comes up exactly the same.

@Dan-krm
Copy link
Contributor
Dan-krm commented Nov 10, 2024

@DaniilSkyRable Could you share your verbose response for calling fetchTickers on mexc, xt, and binance by setting:
"options":{"verbose": true} and I'll see if I can identify the issue that you're having,

Here's some more info on sharing the verbose response:
https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md#how-to-submit-an-issue

@Dan-krm
Copy link
Contributor
Dan-krm commented Nov 10, 2024

I'm mostly interested in this info from the verbose response and if you're receiving any errors outside of the scope of ccxt:

binance.fetchTickers ()
fetch Request:
 binance GET https://api.binance.com/api/v3/ticker/24hr
RequestHeaders:
 {}
RequestBody:
 undefined

handleRestResponse:
 binance GET https://api.binance.com/api/v3/ticker/24hr 200 OK

@DaniilSkyRable
Copy link
Author
DaniilSkyRable commented Nov 10, 2024 •
edited
Loading

My code now

import ccxt.async_support as ccxt
import asyncio
import sys
import json
import os
import logging

if sys.platform.startswith('win'):
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

log_file = os.path.join(os.path.dirname(__file__), "verbose_output.log")
logging.basicConfig(
    filename=log_file,
    level=logging.DEBUG,
    format="%(asctime)s - %(levelname)s - %(message)s"
)

with open(os.path.join(os.path.dirname(__file__), "exchanges_config.json"), "r") as file:
    exchanges_config = json.load(file)

exchanges = {
    name: getattr(ccxt, name)(config)
    for name, config in exchanges_config.items()
}

all_tickers_data = {}

async def fetch_tickers(exchange_name, exchange):
    try:
        exchange.options['type'] = 'spot'

        if 'verbose' in exchange.options and exchange.options['verbose']:
            exchange.verbose = True
            logging.info(f"Verbose mode enabled for {exchange_name}")

        tickers = await exchange.fetch_tickers()
        logging.info(f"Fetched data from {exchange_name}: {len(tickers)} tickers received")

        all_tickers_data[exchange_name] = tickers
        return tickers
    except Exception as e:
        logging.error(f"Error fetching data from {exchange_name}: {e}")
        return {}
    finally:
        await exchange.close()

async def main():
    tasks = []

    for exchange_name, exchange in exchanges.items():
        tasks.append(fetch_tickers(exchange_name, exchange))

    await asyncio.gather(*tasks)

    output_path = os.path.join(os.path.dirname(__file__), "tickers_data.json")
    with open(output_path, "w") as file:
        json.dump(all_tickers_data, file, indent=4)

    print(f"Data saved to: {output_path}")
    print(f"Verbose output saved to: {log_file}")

asyncio.run(main())

verbose_output.log

this logs for MEXC,Binance
xt - work now
verbose_output.log

@Dan-krm
Copy link
Contributor
Dan-krm commented Nov 10, 2024

In one of your output logs it looks like mexc succeeded:
GET https://api.mexc.com/api/v3/ticker/24hr, Response: 200
2024-11-10 10:37:48,826 - INFO - Fetched data from mexc: 3068 tickers received

In your other output log mexc times out when calling fetchMarkets:
Error fetching data from mexc: mexc GET https://api.mexc.com/api/v3/exchangeInfo

On binance it looks like something might be timing out for you when loading the markets with fetchMarkets:
ERROR - Error fetching data from binance: binance GET https://fapi.binance.com/fapi/v1/exchangeInfo
ERROR - Error fetching data from binance: binance GET https://dapi.binance.com/dapi/v1/exchangeInfo

For Binance only, by default ccxt is loading the spot, linear and inverse markets. You can try loading only the spot markets by setting this in options:

exchange.options['fetchMarkets'] = [ 'spot' ]

Or set it in the config:

  "binance": {
      "options": {
          "fetchMarkets": [ "spot" ]
      }
  },

@DaniilSkyRable
Copy link
Author
DaniilSkyRable commented Nov 10, 2024 •
edited
Loading

With mexc is not clear situation, then it works, then does not work, I do not understand why ..., one of 20-30 runs can work, I do not understand why
Binance work,thx
can u get me link to doc ccxt.async_support ,i see only 2-3 examples in your doc

@Dan-krm
Copy link
Contributor
Dan-krm commented Nov 10, 2024

Yeah it's not clear to me why that is happening with mexc for you at the moment, the ratelimits look correct

This is all there is in the documentation for async support:
https://github.com/ccxt/ccxt/wiki/Manual#synchronous-vs-asynchronous-calls

You can also check some of the async examples here:
https://github.com/ccxt/ccxt/tree/master/examples/py

@DaniilSkyRable
Copy link
Author
DaniilSkyRable commented Nov 10, 2024 •
edited
Loading

Maybe there are some parameters for MEXC that we can try?

Yeah it's not clear to me why that is happening with mexc for you at the moment, the ratelimits look correct

This is all there is in the documentation for async support: https://github.com/ccxt/ccxt/wiki/Manual#synchronous-vs-asynchronous-calls

You can also check some of the async examples here: https://github.com/ccxt/ccxt/tree/master/examples/py

@DaniilSkyRable
Copy link
Author

I run the same code several times, sometimes xt works, and sometimes it writes that it cannot access futures, although the parameters are set to spot.

@DaniilSkyRable
Copy link
Author
DaniilSkyRable commented Nov 10, 2024 •
edited
Loading

I put 2 exchanges in the config, then it gives data on them and everything works (MEXC, XT), when I put all exchanges in the config (I have 14 in total), then those 2 do not work (MEXC, XT).
14 exchanges - file more 25 mb, i cant load this

I'll add this - when I put MEXC or XT first in the queue, then one of them will work, but dont work other exchange

@Dan-krm Dan-krm linked a pull request Nov 11, 2024 that will close this issue
Open
@Dan-krm
Copy link
Contributor
Dan-krm commented Nov 11, 2024

@DaniilSkyRable I've opened a pull request that will enable you to load only the spot markets for mexc and xt similar to how that can be controlled on binance.

Besides that I'm not sure if there is anything else that can be done on the CCXT side

I might start adding this functionality to other exchanges as well, so you can watch out for more exchanges with this supported feature in the changelog: https://docs.ccxt.com/#/CHANGELOG

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Dan-krm Dan-krm

Labels
question
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(mexc, xt): fetchMarkets enable the option to load only spot markets Dan-krm/ccxt
3 participants
@carlosmiei @Dan-krm @DaniilSkyRable
0