{"id":32246,"date":"2021-05-25T16:38:26","date_gmt":"2021-05-25T16:38:26","guid":{"rendered":"https:\/\/streamhub.co.uk\/?p=32246"},"modified":"2021-11-03T16:43:46","modified_gmt":"2021-11-03T16:43:46","slug":"an-approach-to-building-asynchronous-services-async-in-next-generation-cloud-data-warehouses","status":"publish","type":"post","link":"https:\/\/streamhub.co.uk\/an-approach-to-building-asynchronous-services-async-in-next-generation-cloud-data-warehouses\/","title":{"rendered":"How to build an Asynchronous layer over Snowflake"},"content":{"rendered":"Reading Time: <\/span> 5<\/span> minutes<\/span><\/span>

[et_pb_section fb_built=”1″ _builder_version=”4.0.6″ custom_padding=”1px||0px|||” da_disable_devices=”off|off|off” global_colors_info=”{}” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row _builder_version=”3.27.4″ min_height=”46px” custom_margin=”7px|auto||auto||” custom_padding=”0px||20px|||” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”3.27.4″ global_colors_info=”{}”][et_pb_blurb title=”@ET-DC@eyJkeW5hbWljIjp0cnVlLCJjb250ZW50IjoicG9zdF9kYXRlIiwic2V0dGluZ3MiOnsiYmVmb3JlIjoiIiwiYWZ0ZXIiOiIiLCJkYXRlX2Zvcm1hdCI6ImRlZmF1bHQiLCJjdXN0b21fZGF0ZV9mb3JtYXQiOiIifX0=@” use_icon=”on” font_icon=”%%128%%” icon_color=”#00ac69″ icon_placement=”left” content_max_width=”1100px” use_icon_font_size=”on” icon_font_size=”16px” _builder_version=”4.0.6″ _dynamic_attributes=”title” header_font=”Work Sans|600|||||||” header_text_align=”left” header_font_size=”14px” text_orientation=”center” min_height=”12px” custom_margin=”||0px|||” animation=”off” hover_enabled=”1″ locked=”off” global_colors_info=”{}” icon_color__hover=”#00ac69″ box_shadow_color__hover_enabled=”rgba(0,0,0,0.3)” box_shadow_color__hover=”rgba(0,0,0,0.3)” box_shadow_style__hover=”none” box_shadow_style__hover_enabled=”none” use_background_color_gradient__hover=”off” use_background_color_gradient__hover_enabled=”off” background_color_gradient_start__hover=”#2b87da” background_color_gradient_start__hover_enabled=”#2b87da” background_color_gradient_end__hover=”#29c4a9″ background_color_gradient_end__hover_enabled=”#29c4a9″ background_color_gradient_type__hover=”linear” background_color_gradient_type__hover_enabled=”linear” background_color_gradient_direction__hover=”180deg” background_color_gradient_direction__hover_enabled=”180deg” background_color_gradient_direction_radial__hover=”center” background_color_gradient_direction_radial__hover_enabled=”center” background_color_gradient_start_position__hover=”0%” background_color_gradient_start_position__hover_enabled=”0%” background_color_gradient_end_position__hover=”100%” background_color_gradient_end_position__hover_enabled=”100%” background_color_gradient_overlays_image__hover=”off” background_color_gradient_overlays_image__hover_enabled=”off” parallax__hover=”off” parallax__hover_enabled=”off” parallax_method__hover=”on” parallax_method__hover_enabled=”on” background_size__hover=”cover” background_size__hover_enabled=”cover” background_position__hover=”center” background_position__hover_enabled=”center” background_repeat__hover=”no-repeat” background_repeat__hover_enabled=”no-repeat” background_blend__hover=”normal” background_blend__hover_enabled=”normal” allow_player_pause__hover=”off” allow_player_pause__hover_enabled=”off” background_video_pause_outside_viewport__hover=”on” background_video_pause_outside_viewport__hover_enabled=”on” body_letter_spacing__hover=”0px” body_letter_spacing__hover_enabled=”0px” body_text_shadow_style__hover=”none” body_text_shadow_style__hover_enabled=”none” body_text_shadow_color__hover=”rgba(0,0,0,0.4)” body_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)”][\/et_pb_blurb][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ admin_label=”Services” _builder_version=”4.9.2″ max_width=”90%” custom_padding=”0px||0px|||” da_disable_devices=”off|off|off” locked=”off” global_colors_info=”{}” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row _builder_version=”4.0.6″ min_height=”58px” custom_padding=”4px|||||” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.0.6″ global_colors_info=”{}”][et_pb_divider color=”#003387″ divider_weight=”5px” _builder_version=”4.0.6″ global_colors_info=”{}”][\/et_pb_divider][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=”4.0.6″ custom_margin=”-28px|auto||auto||” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.0.6″ global_colors_info=”{}”][et_pb_text _builder_version=”4.9.2″ custom_margin=”|-24px||||” custom_padding=”||11px|||” global_colors_info=”{}”]<\/p>\n

Building an asynchronous solution for your analytic use-cases can be super powerful, especially building a deep-dive on-demand reporting solution. When we started building this at Streamhub, we found very little literature around this, so I am sharing this, hoping people pondering over this, can benefit from it. Our core analytical solution for such on-demand deep-dive use-cases is build upon Snowflake, based on design considerations listed in our previous <\/a>blog, and our asynchronous layer runs over this.<\/span><\/span><\/p>\n

<\/span><\/p>\n

What is an asynchronous service?<\/b><\/h4>\n

Asynchronous service is one of the common methods of interacting with backend systems for processes which takes a long time to compute. Our async approach is based on polling and involves the 3 simple steps:<\/span><\/p>\n

    \n
  1. Execute <\/b>the process asynchronously & get the processId immediately.<\/span><\/li>\n
  2. Poll <\/b>the status using the processId for every <\/span>x<\/span><\/i> seconds until the process completes\u00a0<\/span><\/li>\n
  3. Return the <\/b>response<\/span><\/li>\n<\/ol>\n

    <\/b><\/h4>\n

    <\/b><\/h4>\n

    How to achieve this?<\/b><\/h4>\n

    There are a number of ways to implement Async in Snowflake. Here is the method we found to be most effective. It is based on polling, and we made full use of Snowflake’s information_schema.query_history.\u00a0<\/span><\/p>\n

    <\/span><\/p>\n

    1. Executing an Async query<\/b><\/h4>\n

    Scala and Python both have different ways of submitting an async query, Let’s look at both examples<\/span><\/h4>\n

    <\/b><\/p>\n

    Scala:<\/b><\/p>\n

    Once you get the connection string, using the preparedStatement you can unwrap the <\/span>SnowflakePreparedStatement <\/span>which<\/span> allows you to execute an Async query using executeAsyncQuery() method. Return the queryId back to the service which triggered the sql.<\/span><\/p>\n\n\n\n
    \n
    def run(): (queryId, resultSet) = {\n val sqlString = \u201cSelect * from dual\u201d\n\u00a0val<\/span> conn<\/span> = getSnowflakeConnection()<\/span>\n\n\u00a0val<\/span> prep_statement = conn.prepareStatement(sqlString)\n\n val<\/span> (query_id, rs) = \n \/\/ run Async query<\/span>\n\u00a0if<\/span> (asyncRequest) {\n\u00a0 val<\/span> resultSet =\u00a0 prep_statement.unwrap(classOf[SnowflakePreparedStatement]<\/span>).executeAsyncQuery\n\u00a0 val<\/span> asyncQueryId = resultSet.unwrap(classOf[SnowflakeResultSet]<\/span>).getQueryID\n\u00a0 prep_statement.close()\n\u00a0 conn.close()\n\u00a0 (asyncQueryId, None<\/span>)\n } \n \/\/ run sync query<\/span>\n\u00a0else { \n\u00a0 val<\/span> resultSet = prep_statement.executeQuery\n\u00a0 val<\/span> queryId = resultSet.unwrap(classOf[SnowflakeResultSet]<\/span>).getQueryID\n\u00a0 (queryId, Some<\/span>(resultSet))\n }\n}<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Code: Submitting Async & Sync queries to SF(Scala)<\/i><\/b><\/p>\n
     <\/p>\n
    Python:<\/b><\/p>\n
    It is straightforward in python, just passing the param<\/span> _no_results=<\/span>True<\/span> will submit an async query<\/span><\/p>\n\n\n\n
    \n
    def run():<\/span>\n\u00a0\u00a0sql_string = <\/span>\"select * from dual\"<\/span>\n<\/span>\u00a0 cursor = get_snowflake_connection()<\/span>\n<\/span>\u00a0 <\/span>if<\/span> async_request:<\/span>\n<\/span> \u00a0 \u00a0 cursor.execute(sql_string, _no_results=<\/span>True<\/span>)<\/span>\n<\/span> \u00a0 \u00a0 query_id = cursor.sfqid<\/span>\n<\/span> \u00a0 \u00a0 <\/span>return<\/span> query_id, None<\/span>\n<\/span>\u00a0 <\/span>else<\/span>:<\/span>\n<\/span> \u00a0 \u00a0 cursor.execute(sql_string)<\/span>\n<\/span> \u00a0 \u00a0 result_set = cursor.fetchall()<\/span>\n<\/span> \u00a0 \u00a0 <\/span>return<\/span> query_id, result_set<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Code: Submitting Async & Sync queries to SF (Python)<\/i><\/b><\/p>\n
    <\/i><\/b><\/p>\n
    2. Polling for results\u00a0<\/b><\/h4>\n
    Let\u2019s poll the results, let’s say for every 5s using the queryId. Nice thing about Snowflake is that the history table provides the running status of the query in realtime and it is queryable as well as similar to any other table and it consumes a tiny amount of credits. This can be reduced by adding conditions in where clauses like warehouse, date etc to limit the search. I will leave that to you\u00a0<\/span><\/p>\n
    So for every 5 s the below query will be used to check the status.\u00a0<\/span><\/p>\n
    <\/span><\/p>\n\n\n\n
    \n
    \"\"\"select query_id, execution_status, error_message from table(information_schema.query_history()) <\/span>\n\u00a0\u00a0\u00a0where query_id = '{0}'\"\"\"<\/span>.format(query_id)<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Sql: to poll history table<\/i><\/b><\/p>\n
    <\/i><\/b><\/p>\n
    \"\"<\/i><\/b><\/p>\n
    Snowflake History UI<\/b><\/p>\n
    Keep polling until the status is in running<\/span><\/p>\n\n\n\n
    \n
    def<\/span> get_query_status<\/span>(self, query_id):<\/span>\n<\/span> \u00a0 sql = <\/span>\"\"\"select query_id, execution_status, error_message from table (information_schema.query_history())<\/span>\n<\/span> \u00a0 where query_id = '{0}'\"\"\"<\/span><\/span>.format(query_id)<\/span>\n<\/span> \u00a0 <\/span>try<\/span>:<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 self.cursor.execute(sql)<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 record = self.cursor.fetchone()<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 <\/span>if<\/span> record is not <\/span>None<\/span>:<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 print(record)<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <\/span>if<\/span> record[<\/span>1<\/span>] == <\/span>'FAILED_WITH_ERRO<\/span><\/span>R<\/span>':<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <\/span>return<\/span> 500<\/span>, record[<\/span>1<\/span>], record[<\/span>2<\/span>]<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <\/span>else<\/span>:<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <\/span>return<\/span> 200<\/span>, record[<\/span>1<\/span>], record[<\/span>2<\/span>]<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 <\/span>else<\/span>:<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <\/span>return<\/span> 404<\/span>, <\/span>\"FAILED_WITH_ERROR\"<\/span>, <\/span>\"query_id Not found\"<\/span>\n<\/span><\/span>\n<\/span> \u00a0 except snowflake.connector.errors.<\/span>ProgrammingError<\/span> as e:<\/span>\n<\/span> \u00a0 \u00a0 \u00a0 raise <\/span>Exception<\/span>(<\/span>'Error<\/span><\/span> {<\/span>0<\/span>} ({<\/span>1<\/span>}): {<\/span>2<\/span>} ({<\/span>3<\/span>})'.format(e.errno, e.sqlstate, e.msg, e.sfqid))<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Code: to get query status(Python)<\/i><\/b><\/p>\n
    <\/b><\/h4>\n
    3. Returning the response<\/b><\/h4>\n
    Once the status is changed from <\/span>running<\/b>, we can handle it in different ways based on the status. Here we are interested mostly in <\/span>Succeeded<\/span><\/i> or <\/span>Failed<\/span><\/i>.<\/span><\/p>\n