Version/Environment (if relevant):
Current version release of Domino at time of writing this article is 5.5.0
Issue:
The Domino Data API has handy code for reading and extracting a resultset into a pandas dataframe described here
ds = DataSourceClient().get_datasource("MySnowFlake")
res = ds.query....
But it doesn't yet provide means for writing a dataframe to snowflake.
Root Cause:
In the current version this functionality has not been implemented directly in the Domino Data API.
Resolution:
You can use snowflake's Python connector, referenced here:
https://docs.dominodatalab.com/en/latest/user_guide/d4ef2b/connect-to-snowflake/#_create_a_snowflake_data_source
And then see flake's python API for writing dataframes here:
https://docs.snowflake.com/en/user-guide/python-connector-api.html#write_pandas
And simple example in Domino python script might be:
###Pull the OAuth token used to authenticate with the DataSourceClient and passed it to the snowflake connector via something like:
"""
Gets the Snowflake OAuth token to authenticate with the Python connector.
Returns
-------
str
Snowflake OAuth token string used to authenticate
"""
location = os.getenv("DOMINO_TOKEN_FILE", os.environ.get("DOMINO_TOKEN_FILE"))
with open(location, encoding="ascii") as token_file:
return token_file.readline().rstrip()
sf_username = os.environ.get("DOMINO_STARTING_USERNAME")\
.replace("_", ".") + "@yourcompany.com"
### Set the connection
ctx = snowflake.connector.connect(
user=sf_username,
authenticator="oauth",
token=get_sf_token(),
account=account,
warehouse=warehouse,
database=database,
schema=schema,
)
- Then you can pass this to the very handy write_pandas function mentioned above, which allows for inserts, overwrites, chunking, etc,
Notes/Information:
Comments
0 comments
Please sign in to leave a comment.