ReadTimeout - Exception: Max retries (12) on order ("") hit, raising.

[metaperl]

I encountered a read timeout after 12 retries. It appears that settings.py does not allow one to configure the number of retries or the delay between them.

How should we move forward with making the code robust in the face of errors such as this transcript shows:

2018-05-31 10:28:06,481 - INFO - ws_thread - Connecting to wss://www.bitmex.com/realtime?subscribe=quote:XBTUSD,trade:XBTUSD,instrument,order:XBTUSD,execution:XBTUSD,margin,position
2018-05-31 10:28:06,482 - INFO - ws_thread - Authenticating with API Key.
2018-05-31 10:28:06,484 - INFO - ws_thread - Started thread
2018-05-31 10:28:07,485 - INFO - ws_thread - Connected to WS. Waiting for data images, this may take a moment...
2018-05-31 10:28:07,486 - INFO - ws_thread - Got all market data. Starting.
2018-05-31 10:28:07,486 - INFO - market_maker - Using symbol XBTUSD.
2018-05-31 10:28:07,486 - INFO - market_maker - Order Manager initializing, connecting to BitMEX. Live run: executing real trades.
2018-05-31 10:28:07,486 - INFO - market_maker - Resetting current position. Canceling all existing orders.
2018-05-31 10:28:07,486 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:28:14,805 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:28:34,821 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:28:42,928 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:29:02,949 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:29:11,774 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:29:31,795 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:29:39,131 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:29:59,149 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:30:09,538 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:30:29,546 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:30:36,897 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:30:56,917 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:31:05,007 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:31:25,028 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:31:34,666 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:31:54,687 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:32:02,063 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:32:22,083 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:32:30,221 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:32:50,242 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:32:57,677 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:33:17,698 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:33:25,768 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:33:45,789 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:33:53,137 - WARNING - bitmex - Timed out on request: order (""), retrying...
Importing symbol settings for XBTUSD...
Unable to find settings-XBTUSD.py.
Traceback (most recent call last):
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 386, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 382, in _make_request
    httplib_response = conn.getresponse()
  File "/home/schemelab/install/miniconda3/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/home/schemelab/install/miniconda3/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/home/schemelab/install/miniconda3/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/home/schemelab/install/miniconda3/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 291, in recv_into
    raise timeout('The read operation timed out')
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/util/retry.py", line 347, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 388, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 308, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.bitmex.com', port=443): Read timed out. (read timeout=7)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 264, in _curl_bitmex
    response = self.session.send(prepped, timeout=timeout)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/adapters.py", line 499, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.bitmex.com', port=443): Read timed out. (read timeout=7)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./marketmaker", line 4, in <module>
    market_maker.run()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/market_maker.py", line 543, in run
    om = OrderManager()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/market_maker.py", line 220, in __init__
    self.reset()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/market_maker.py", line 223, in reset
    self.exchange.cancel_all_orders()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/market_maker.py", line 60, in cancel_all_orders
    orders = self.bitmex.http_open_orders()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 104, in wrapped
    return fn(self, *args, **kwargs)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 195, in http_open_orders
    verb="GET"
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 252, in retry
    raise Exception("Max retries (%d) on %s (%s) hit, raising." % (max_retries, path, json.dumps(postdict or '')))
Exception: Max retries (12) on order ("") hit, raising.
2018-05-31 10:33:53,147 - INFO - market_maker - Shutting down. All open orders will be cancelled.
2018-05-31 10:33:53,147 - INFO - market_maker - Resetting current position. Canceling all existing orders.
2018-05-31 10:33:53,147 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:34:00,429 - ERROR - bitmex - Unhandled Error: 400 Client Error: Bad Request for url: https://www.bitmex.com/api/v1/order?filter=%7B%22ordStatus.isTerminated%22%3A+false%2C+%22symbol%22%3A+%22XBTUSD%22%7D&count=500: {"error":{"message":"This request has expired - `expires` is in the past. Current time: 1527777240","name":"HTTPError"}}
2018-05-31 10:34:00,429 - ERROR - bitmex - Endpoint was: GET order: null

[ryanfox]

You can configure the timeout in settings.py. It defaults to

TIMEOUT = 7

[metaperl]

I have supplied a patch to make the timeout configurable as well as the number of retries and the delay between them - #119

[ryanfox]

The timeout is already configurable. Also, if the client can't make a successful connection in almost a minute and a half, that suggests issues with your network connection.

Based on your logs, you're trying to trade on the live platform. Keep in mind this from the README:

Develop on Testnet first! Testnet trading is completely free and is identical to the live market.

BitMEX is not responsible for any losses incurred when using this code. This code is intended for sample purposes ONLY - do not use this code for real trades unless you fully understand what it does and what its caveats are.
This is not a sophisticated market making program. It is intended to show the basics of market making while abstracting some of the rote work of interacting with the BitMEX API. It does not make smart decisions and will likely lose money.