ccxt: Problems with repeated market symbols
Operating System
Linux Mint
Programming Languages
Python
CCXT Version
4.0.80
Description
fetch_markets sometimes returns markets that have the same ‘symbol’ and ‘type’. This is showcased by the first code below.
If I’m reading the code correcty, markets are set in set_markets method in exchange.py based on the fetch_markets list. They are sorted so that spot markets are first and then are indexed by the ‘symbol’ field in line 2050:
self.markets = self.index_by(values, 'symbol')
index_by method just happily keeps overwriting the markets when the same symbol keeps repeating (line 971-974):
for element in array:
if ((is_int_key and (key < len(element))) or (key in element)) and (element[key] is not None):
k = element[key]
result[k] = element
The last repeat is the one that is ultimately left into the exchange.markets dictionary.
When there are repeated symbols in fetch_markets list less markets are actually found in the end from exchange.markets. This is showcased by the second code below. According to issue #17718 it’s a feature, not a bug.
All the above and below rises several questions:
- Are there bugs in
fetch_marketsmethod in some of the exchanges mentioned in the code output? For instance, should Delta Exchange really have 150 markets with ‘BTC/USDT’ symbol? - Is sorting and choosing the markets working as planned in
set_markets? It looks like spot markets are sorted to be first, but the last repeated ‘symbol’ is selected. - Why is this sorting being done in the first place? Are the unified symbols not guaranteed to be different for different market types already in
fetch_markets? - Is it really the right choice to just pick one of the markets when there are repeated symbols and scratch the rest? Wouldn’t it be possible to force all the markets from
fetch_marketsto have a unique unified symbol? For example, one could add an underscore and a number to the end (this should obviously be the last resort and preferably not used at all). - Isn’t choosing one of the markets (by random basically) kind of dangerous? For example, if the ccxt user wants to buy from Binance with ‘BCH/BTC’ symbol using
create_order, they could end up buying something completely else than they actually meant, unless they look exchange.markets[‘BCH/BTC’] dictionary through very carefully. - Could there be tests added to the ccxt test suite to find repeated symbols automatically?
7. Bonus: Why isto_arrayin line 2038 in exchange.py, isn’tmarketsobject already an array/list?marketValues = self.sort_by(self.to_array(markets), 'spot', True)
Edit: Removed question 7, because set_markets is actually sometimes called with self.markets dictionary as an argument…
Code
exchange_ids = ccxt.exchanges
exchanges_with_repeated_symbols = []
for exchange_id in exchange_ids:
exchange = getattr(ccxt, exchange_id)({'enableRateLimit': True})
if exchange.has['fetchMarkets']:
try:
markets = exchange.fetch_markets()
except:
continue
market_types = {}
symbol_repeat_count = 0
for market in markets:
if market['symbol'] in market_types:
symbol_repeat_count += 1
print(exchange_id.ljust(18), 'repeated symbol:', market['symbol'].ljust(15), 'market type:', market['type'].ljust(15), ' earlier market type:', market_types[market['symbol']])
else:
market_types[market['symbol']] = market['type']
if symbol_repeat_count > 0:
exchanges_with_repeated_symbols.append(exchange_id)
print()
print('following exchanges had repeated market symbols:' + str(exchanges_with_repeated_symbols))
bequant repeated symbol: TRX/USDT market type: spot earlier market type: spot
bequant repeated symbol: ZRX/USDT market type: spot earlier market type: spot
bequant repeated symbol: ETH/USDT market type: spot earlier market type: spot
bequant repeated symbol: YFI/BTC market type: spot earlier market type: spot
bequant repeated symbol: EOS/USDT market type: spot earlier market type: spot
bequant repeated symbol: XLM/USDT market type: spot earlier market type: spot
bequant repeated symbol: HBAR/USDT market type: spot earlier market type: spot
bequant repeated symbol: ZEC/USDT market type: spot earlier market type: spot
bequant repeated symbol: EOS/BTC market type: spot earlier market type: spot
bequant repeated symbol: LINK/BTC market type: spot earlier market type: spot
bequant repeated symbol: ALGO/USDT market type: spot earlier market type: spot
bequant repeated symbol: HBAR/BTC market type: spot earlier market type: spot
bequant repeated symbol: USDC/USDT market type: spot earlier market type: spot
bequant repeated symbol: ALGO/BTC market type: spot earlier market type: spot
bequant repeated symbol: BCH/BTC market type: spot earlier market type: spot
bequant repeated symbol: XRP/USDT market type: spot earlier market type: spot
bequant repeated symbol: LINK/USDT market type: spot earlier market type: spot
bequant repeated symbol: LTC/BTC market type: spot earlier market type: spot
bequant repeated symbol: ZRX/BTC market type: spot earlier market type: spot
bequant repeated symbol: ZEC/USDT market type: spot earlier market type: spot
bequant repeated symbol: UNI/BTC market type: spot earlier market type: spot
bequant repeated symbol: TUSD/USDT market type: spot earlier market type: spot
bequant repeated symbol: YFI/USDT market type: spot earlier market type: spot
bequant repeated symbol: ETH/USDT market type: spot earlier market type: spot
bequant repeated symbol: MKR/USDT market type: spot earlier market type: spot
bequant repeated symbol: MKR/BTC market type: spot earlier market type: spot
bequant repeated symbol: BTC/USDT market type: spot earlier market type: spot
bequant repeated symbol: HBAR/USDT market type: spot earlier market type: spot
bequant repeated symbol: ZRX/USDT market type: spot earlier market type: spot
bequant repeated symbol: LINK/USDT market type: spot earlier market type: spot
bequant repeated symbol: HIFI/USDT market type: spot earlier market type: spot
bequant repeated symbol: LTC/USDT market type: spot earlier market type: spot
bequant repeated symbol: ETH/BTC market type: spot earlier market type: spot
bequant repeated symbol: BTC/USDT market type: spot earlier market type: spot
bequant repeated symbol: EOS/USDT market type: spot earlier market type: spot
bequant repeated symbol: BAT/USDT market type: spot earlier market type: spot
bequant repeated symbol: BAT/BTC market type: spot earlier market type: spot
bequant repeated symbol: ALGO/USDT market type: spot earlier market type: spot
bequant repeated symbol: XRP/BTC market type: spot earlier market type: spot
bequant repeated symbol: BCH/USDT market type: spot earlier market type: spot
bequant repeated symbol: BCH/USDT market type: spot earlier market type: spot
bequant repeated symbol: HIFI/USDT market type: spot earlier market type: spot
bequant repeated symbol: LTC/USDT market type: spot earlier market type: spot
bequant repeated symbol: UNI/USDT market type: spot earlier market type: spot
bequant repeated symbol: XRP/USDT market type: spot earlier market type: spot
bequant repeated symbol: XRP/USDT market type: spot earlier market type: spot
bequant repeated symbol: MKR/USDT market type: spot earlier market type: spot
bequant repeated symbol: XLM/BTC market type: spot earlier market type: spot
bequant repeated symbol: ZEC/BTC market type: spot earlier market type: spot
bequant repeated symbol: UNI/USDT market type: spot earlier market type: spot
bequant repeated symbol: BAT/USDT market type: spot earlier market type: spot
bequant repeated symbol: ETH/USDT market type: spot earlier market type: spot
bequant repeated symbol: OMG/BTC market type: spot earlier market type: spot
bequant repeated symbol: BTC/USDT market type: spot earlier market type: spot
bequant repeated symbol: TRX/USDT market type: spot earlier market type: spot
bequant repeated symbol: XLM/USDT market type: spot earlier market type: spot
bequant repeated symbol: OMG/USDT market type: spot earlier market type: spot
bequant repeated symbol: USDT/USDT market type: spot earlier market type: spot
bequant repeated symbol: OMG/USDT market type: spot earlier market type: spot
bequant repeated symbol: TRX/BTC market type: spot earlier market type: spot
binance repeated symbol: BCH/BTC market type: spot earlier market type: spot
binance repeated symbol: BCH/USDT market type: spot earlier market type: spot
binance repeated symbol: BCH/USDC market type: spot earlier market type: spot
binance repeated symbol: BCH/TUSD market type: spot earlier market type: spot
binance repeated symbol: BCH/PAX market type: spot earlier market type: spot
binance repeated symbol: BCH/BUSD market type: spot earlier market type: spot
bitbns repeated symbol: BCH/INR market type: spot earlier market type: spot
bitflyer repeated symbol: BTC/JPY market type: spot earlier market type: spot
bitflyer repeated symbol: ETH/BTC market type: spot earlier market type: spot
bitflyer repeated symbol: BTC/JPY market type: spot earlier market type: spot
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: options_combos earlier market type: options_combos
delta repeated symbol: BTC/USDT market type: spot earlier market type: options_combos
ndax repeated symbol: AAVE/CAD market type: spot earlier market type: spot
yobit repeated symbol: PRE/BTC market type: spot earlier market type: spot
yobit repeated symbol: PRE/ETH market type: spot earlier market type: spot
yobit repeated symbol: PRE/DOGE market type: spot earlier market type: spot
yobit repeated symbol: PRE/WAVES market type: spot earlier market type: spot
yobit repeated symbol: PRE/USD market type: spot earlier market type: spot
yobit repeated symbol: PRE/RUB market type: spot earlier market type: spot
following exchanges had repeated market symbols:['bequant', 'binance', 'bitbns', 'bitflyer', 'delta', 'ndax', 'yobit']
exchange_ids = ccxt.exchanges
for exchange_id in exchange_ids:
exchange = getattr(ccxt, exchange_id)({'enableRateLimit': True})
if exchange.has['fetchMarkets']:
try:
n_fetched_markets = len(exchange.fetch_markets())
exchange.load_markets()
n_loaded_markets = len(exchange.markets)
if n_fetched_markets != n_loaded_markets:
print(exchange_id.ljust(18), 'fetched markets:', str(n_fetched_markets).ljust(4), 'loaded markets:', str(n_loaded_markets).ljust(4))
except:
pass
bequant fetched markets: 299 loaded markets: 239
binance fetched markets: 2600 loaded markets: 2594
bitbns fetched markets: 222 loaded markets: 221
bitflyer fetched markets: 19 loaded markets: 16
delta fetched markets: 863 loaded markets: 713
ndax fetched markets: 46 loaded markets: 45
yobit fetched markets: 8964 loaded markets: 8958
About this issue
- Original URL
- State: closed
- Created 10 months ago
- Comments: 21
@JovianMoon yeah hopefully it’s fixed now 😅, thanks once again
@JovianMoon yes we will add a test to validate that we don’t have repeated markets with different ids (same symbol different id) so we can detect this immediately and automatically.
Will also be removed here: https://github.com/ccxt/ccxt/pull/19099
@JovianMoon Not sure why but one unrelated PR restored that change accidentally (probably the merge commit wenr wrong) , going to remove it again here: https://github.com/ccxt/ccxt/pull/19097
@JovianMoon but right now it is causing a lot of clashes because some exchanges treat them as different tokens (BCH and BCH ABC)
Hello @JovianMoon, Thanks for letting us know.
Sometimes, it might be “ok” to have duplicated markets at that point because some exchanges require different endpoint calls to get all markets and different calls might return the same market (hence the exact same market twice).
Not sure if this is the case, so we will dig into it, and we will let you know.