Version/Environment (if relevant):
4.x, 5.x
Issue:
When deploying a Model API the build is very slow and the build logs list repeating network related errors like:
Jan 23 2023 18:14:08 +0100 #13 4.807 WARNING: Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f081026bd90>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/prometheus-client/
The model eventually deploys, but the build process is slow.
Root Cause:
Model API initialization code tries to install the python package prometheus-client 0.11.0 (among others), a client for metrics & monitoring tools. If your system is air-gapped or public/internet access is tightly controlled your system might not be able to fetch that python package from https://pypi.org, resulting in the error above.
Resolution:
Install prometheus-client
in your project's Compute Environment via a dockerfile instruction:
RUN pip install prometheus-client==0.11.0
If this isn't possible another idea is to modify pip's config files to make it not retry and "fail fast", like:
pip config --quiet --user set global.retries 0
pip config --quiet --user set global.timeout 1
pip config --quiet --user set global.disable-pip-version-check True
Notes/Information:
If the above isn't possible please reach out to Domino Technical Support referencing DOM #36045 and DOM #41738
Comments
0 comments
Please sign in to leave a comment.