CLI issue - Snapshot failed during domino upload-dataset

Version/Environment (if relevant):

Not relevant

Issue:

When uploading huge directories via domino upload-dataset if the network connection isn't fast or stable enough an error can occur during the CLI command:

+--------+

| Error! |

+--------+

Snapshot upload failed.  You can retry this upload with the following command:
domino upload-dataset testuser/testproject/testdata /user/ghud/mydirectory 301d95e2-00db-407d-8ebc-6055b68bac17 

Root Cause:

The underlying root can vary depending on total size, number of files and the size of the largest file in the directory plus the number of hops between hosts, network latency between hosts and other network or connection-specific nuances outside the sphere of Domino's control.

Resolution:

There are four possible solutions to try.

1. Work with your network administrations to ensure your system has speedy connections to your https://<your domino instance url>
   
   
2. Toggle two environment variables that can alter the network traffic sent via CLI.  This may require an iterative process of testing various values for improvement since network connections vary from customer to customer.
   a) Configurable Chunk Size
   
   1. Allows chunk size to be configured
   2. Environment Variable: DOMINO_UPLOAD_CHUNK_BYTES
   3. Default: 3145728 (3 MB)
   4. Benefit: Tuning based on average file size and network characteristics
   
   b) Threading
   
   1. Allows multi-threaded upload
   2. Environment Variable: DOMINO_UPLOAD_THREADS
   3. Default: 8
   4. Benefit: Allows parallel chunk uploads
      
      
3. Try zipping or tar'ing your directory.  Text will compress greatly, which can make the upload have less data and fewer files to move.  Once uploaded, open a workspace and use sudo apt-get install unzip and
   unzip file.zip -d destination_folder or tar to unarchive the data you uploaded.
   
   
4. Upload files into the dataset via the Domino UI.  On that page you can upload up to 50GB or 50,000 individual files.  https://docs.dominodatalab.com/en/5.1/user_guide/0efc5a/update-a-dataset/  and https://docs.dominodatalab.com/en/5.1/user_guide/d1a1ae/create-a-dataset/
   
   

Notes/Information:

On Windows use "export" to set environment variables, on Mac or Linux use "set"