{"id":28376,"date":"2019-11-01T12:54:22","date_gmt":"2019-11-01T12:54:22","guid":{"rendered":"http:\/\/streamhub.co.uk\/?p=28376"},"modified":"2020-06-30T05:09:26","modified_gmt":"2020-06-30T05:09:26","slug":"apache-spark-tuning-manual","status":"publish","type":"post","link":"https:\/\/streamhub.co.uk\/apache-spark-tuning-manual\/","title":{"rendered":"Apache Spark Tuning Manual"},"content":{"rendered":"Reading Time: <\/span> 20<\/span> minutes<\/span><\/span>\n[et_pb_section fb_built=”1″ admin_label=”Header” _builder_version=”3.27.4″ custom_padding=”0px|0px|0|0px|false|false” locked=”off”][et_pb_row use_custom_gutter=”on” gutter_width=”2″ _builder_version=”3.27.4″ max_width=”1280px” use_custom_width=”on” custom_width_px=”1280px”][et_pb_column type=”4_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_text _builder_version=”3.27.4″ text_font=”Work Sans|700|||||||” text_text_color=”#000000″ text_font_size=”55px” text_line_height=”1.3em” ul_font=”||||||||” ol_font=”||||||||” header_font=”Work Sans|700|||||||” header_font_size=”55px” header_line_height=”1.5em” header_3_font=”||||||||” header_4_font=”||||||||” header_5_font=”||||||||” header_6_font=”||||||||” custom_padding=”||10px||false|false” text_font_size_tablet=”40px” text_font_size_phone=”30px” text_font_size_last_edited=”on|desktop” header_font_size_tablet=”40px” header_font_size_phone=”30px” header_font_size_last_edited=”on|desktop” border_color_all=”#000000″ border_width_bottom=”4px” locked=”off” inline_fonts=”Times New Roman” header_2_font_size__hover=”26px” header_2_font_size__hover_enabled=”26px” header_2_letter_spacing__hover=”0px” header_2_letter_spacing__hover_enabled=”0px” header_2_line_height__hover=”1em” header_2_line_height__hover_enabled=”1em” header_2_text_shadow_style__hover=”none” header_2_text_shadow_style__hover_enabled=”none” header_2_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_2_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)”]

Master Spark fundamentals & optimizations\u00a0<\/span><\/p>[\/et_pb_text][et_pb_text _builder_version=”3.27.4″ text_font=”Work Sans|600|||||||” text_line_height=”1.4em” link_font=”||||||||” link_text_color=”#1a0a38″ locked=”off” inline_fonts=”Times New Roman”]

Streamhub’s all-in-one notes on how to optimize & scale Spark jobs for something more than a POC<\/span><\/span><\/strong><\/h3>[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ use_custom_gutter=”on” gutter_width=”2″ specialty=”on” padding_1_tablet=”0px||” padding_1_last_edited=”off|desktop” admin_label=”Content” _builder_version=”3.22″ background_size=”contain” background_position=”top_center” inner_max_width=”1280px” custom_margin=”||-74px|||” custom_padding=”0|0px|54px|0px|false|false” use_custom_width=”on” custom_width_px=”1280px” background_size__hover=”cover” background_size__hover_enabled=”cover” background_position__hover=”center” background_position__hover_enabled=”center” use_background_color_gradient__hover=”off” use_background_color_gradient__hover_enabled=”off” background_color_gradient_start__hover=”#2b87da” background_color_gradient_start__hover_enabled=”#2b87da” background_color_gradient_end__hover=”#29c4a9″ background_color_gradient_end__hover_enabled=”#29c4a9″ background_color_gradient_type__hover=”linear” background_color_gradient_type__hover_enabled=”linear” background_color_gradient_direction__hover=”180deg” background_color_gradient_direction__hover_enabled=”180deg” background_color_gradient_direction_radial__hover=”center” background_color_gradient_direction_radial__hover_enabled=”center” background_color_gradient_start_position__hover=”0%” background_color_gradient_start_position__hover_enabled=”0%” background_color_gradient_end_position__hover=”100%” background_color_gradient_end_position__hover_enabled=”100%” background_color_gradient_overlays_image__hover=”off” background_color_gradient_overlays_image__hover_enabled=”off” parallax__hover=”off” parallax__hover_enabled=”off” parallax_method__hover=”on” parallax_method__hover_enabled=”on” background_repeat__hover=”no-repeat” background_repeat__hover_enabled=”no-repeat” background_blend__hover=”normal” background_blend__hover_enabled=”normal” allow_player_pause__hover=”off” allow_player_pause__hover_enabled=”off” background_video_pause_outside_viewport__hover=”on” background_video_pause_outside_viewport__hover_enabled=”on”][et_pb_column type=”3_4″ specialty_columns=”3″ _builder_version=”3.25″ custom_padding=”|||” padding_tablet__hover=”0px||” padding_last_edited__hover=”off|desktop” parallax__hover=”off” parallax_method__hover=”on” padding_tablet=”0px||” padding_last_edited=”off|desktop” custom_padding__hover=”|||”][et_pb_row_inner _builder_version=”3.27.4″ min_height=”3546px” locked=”off” parallax_1__hover=”off” parallax_1__hover_enabled=”off” parallax_2__hover=”off” parallax_2__hover_enabled=”off” parallax_3__hover=”off” parallax_3__hover_enabled=”off” parallax_4__hover=”off” parallax_4__hover_enabled=”off” parallax_method_1__hover=”on” parallax_method_1__hover_enabled=”on” parallax_method_2__hover=”on” parallax_method_2__hover_enabled=”on” parallax_method_3__hover=”on” parallax_method_3__hover_enabled=”on” parallax_method_4__hover=”on” parallax_method_4__hover_enabled=”on” use_background_color_gradient__hover=”off” use_background_color_gradient__hover_enabled=”off” background_color_gradient_start__hover=”#2b87da” background_color_gradient_start__hover_enabled=”#2b87da” background_color_gradient_end__hover=”#29c4a9″ background_color_gradient_end__hover_enabled=”#29c4a9″ background_color_gradient_type__hover=”linear” background_color_gradient_type__hover_enabled=”linear” background_color_gradient_direction__hover=”180deg” background_color_gradient_direction__hover_enabled=”180deg” background_color_gradient_direction_radial__hover=”center” background_color_gradient_direction_radial__hover_enabled=”center” background_color_gradient_start_position__hover=”0%” background_color_gradient_start_position__hover_enabled=”0%” background_color_gradient_end_position__hover=”100%” background_color_gradient_end_position__hover_enabled=”100%” background_color_gradient_overlays_image__hover=”off” background_color_gradient_overlays_image__hover_enabled=”off” parallax__hover=”off” parallax__hover_enabled=”off” parallax_method__hover=”on” parallax_method__hover_enabled=”on” background_size__hover=”cover” background_size__hover_enabled=”cover” background_position__hover=”center” background_position__hover_enabled=”center” background_repeat__hover=”no-repeat” background_repeat__hover_enabled=”no-repeat” background_blend__hover=”normal” background_blend__hover_enabled=”normal” allow_player_pause__hover=”off” allow_player_pause__hover_enabled=”off” background_video_pause_outside_viewport__hover=”on” background_video_pause_outside_viewport__hover_enabled=”on” use_custom_gutter__hover=”off” use_custom_gutter__hover_enabled=”off” gutter_width__hover=”3″ gutter_width__hover_enabled=”3″ make_equal__hover=”off” make_equal__hover_enabled=”off” border_radii__hover=”on||||” border_radii__hover_enabled=”on||||” box_shadow_style__hover=”none” box_shadow_style__hover_enabled=”none” box_shadow_color__hover=”rgba(0,0,0,0.3)” box_shadow_color__hover_enabled=”rgba(0,0,0,0.3)” max_width__hover=”100%” max_width__hover_enabled=”100%” filter_hue_rotate__hover=”0deg” filter_hue_rotate__hover_enabled=”0deg” filter_saturate__hover=”100%” filter_saturate__hover_enabled=”100%” filter_brightness__hover=”100%” filter_brightness__hover_enabled=”100%” filter_contrast__hover=”100%” filter_contrast__hover_enabled=”100%” filter_invert__hover=”0%” filter_invert__hover_enabled=”0%” filter_sepia__hover=”0%” filter_sepia__hover_enabled=”0%” filter_opacity__hover=”100%” filter_opacity__hover_enabled=”100%” filter_blur__hover=”0px” filter_blur__hover_enabled=”0px” mix_blend_mode__hover=”normal” mix_blend_mode__hover_enabled=”normal” animation_style__hover=”none” animation_style__hover_enabled=”none” animation_repeat__hover=”once” animation_repeat__hover_enabled=”once” animation_direction__hover=”center” animation_direction__hover_enabled=”center” animation_duration__hover=”1000ms” animation_duration__hover_enabled=”1000ms” animation_delay__hover=”0ms” animation_delay__hover_enabled=”0ms” animation_intensity_slide__hover=”50%” animation_intensity_slide__hover_enabled=”50%” animation_intensity_zoom__hover=”50%” animation_intensity_zoom__hover_enabled=”50%” animation_intensity_flip__hover=”50%” animation_intensity_flip__hover_enabled=”50%” animation_intensity_fold__hover=”50%” animation_intensity_fold__hover_enabled=”50%” animation_intensity_roll__hover=”50%” animation_intensity_roll__hover_enabled=”50%” animation_starting_opacity__hover=”0%” animation_starting_opacity__hover_enabled=”0%” animation_speed_curve__hover=”ease-in-out” animation_speed_curve__hover_enabled=”ease-in-out” hover_transition_duration__hover=”300ms” hover_transition_duration__hover_enabled=”300ms” hover_transition_delay__hover=”0ms” hover_transition_delay__hover_enabled=”0ms” hover_transition_speed_curve__hover=”ease” hover_transition_speed_curve__hover_enabled=”ease”][et_pb_column_inner saved_specialty_column_type=”3_4″ _builder_version=”3.25″ custom_padding=”|||” parallax__hover=”off” parallax__hover_enabled=”off” parallax_method__hover=”on” parallax_method__hover_enabled=”on” custom_padding__hover=”|||”][et_pb_image src=”https:\/\/streamhub.co.uk\/wp-content\/uploads\/2019\/11\/high-volume-analytics-resize.jpg” align=”center” align_tablet=”center” align_phone=”” align_last_edited=”on|desktop” _builder_version=”3.27.4″ width=”100%” max_width=”100%” min_height=”263px” height=”349px” max_height=”399px” custom_margin=”-29px||||false|false” custom_padding=”0px||||false|false” border_radii=”on|5px|5px|5px|5px” border_width_all=”5px” border_color_all=”#002f38″ locked=”off”][\/et_pb_image][et_pb_text _builder_version=”3.27.4″ text_font=”||||||||” text_line_height=”2em” header_font=”||||||||” header_2_font=”Work Sans|700|||||||” header_2_font_size=”25px” header_2_line_height=”1.5em” max_width_tablet=”” max_width_phone=”” max_width_last_edited=”on|tablet” header_2_font_size_tablet=”30px” header_2_font_size_phone=”15px” header_2_font_size_last_edited=”on|desktop” locked=”off” header_font_size__hover=”30px” header_font_size__hover_enabled=”30px” header_letter_spacing__hover=”0px” header_letter_spacing__hover_enabled=”0px” header_text_shadow_style__hover=”none” header_text_shadow_style__hover_enabled=”none” header_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_2_font_size__hover=”26px” header_2_font_size__hover_enabled=”26px” header_2_letter_spacing__hover=”0px” header_2_letter_spacing__hover_enabled=”0px” header_2_line_height__hover=”1em” header_2_line_height__hover_enabled=”1em” header_2_text_shadow_style__hover=”none” header_2_text_shadow_style__hover_enabled=”none” header_2_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_2_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_3_font_size__hover=”22px” header_3_font_size__hover_enabled=”22px” header_3_letter_spacing__hover=”0px” header_3_letter_spacing__hover_enabled=”0px” header_3_line_height__hover=”1em” header_3_line_height__hover_enabled=”1em” header_3_text_shadow_style__hover=”none” header_3_text_shadow_style__hover_enabled=”none” header_3_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_3_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_4_font_size__hover=”18px” header_4_font_size__hover_enabled=”18px” header_4_letter_spacing__hover=”0px” header_4_letter_spacing__hover_enabled=”0px” header_4_line_height__hover=”1em” header_4_line_height__hover_enabled=”1em” header_4_text_shadow_style__hover=”none” header_4_text_shadow_style__hover_enabled=”none” header_4_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_4_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_5_font_size__hover=”16px” header_5_font_size__hover_enabled=”16px” header_5_letter_spacing__hover=”0px” header_5_letter_spacing__hover_enabled=”0px” header_5_line_height__hover=”1em” header_5_line_height__hover_enabled=”1em” header_5_text_shadow_style__hover=”none” header_5_text_shadow_style__hover_enabled=”none” header_5_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_5_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_6_font_size__hover=”14px” header_6_font_size__hover_enabled=”14px” header_6_letter_spacing__hover=”0px” header_6_letter_spacing__hover_enabled=”0px” header_6_line_height__hover=”1em” header_6_line_height__hover_enabled=”1em” header_6_text_shadow_style__hover=”none” header_6_text_shadow_style__hover_enabled=”none” header_6_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_6_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” text_letter_spacing__hover=”0px” text_letter_spacing__hover_enabled=”0px” text_text_shadow_style__hover=”none” text_text_shadow_style__hover_enabled=”none” text_text_shadow_color__hover=”rgba(0,0,0,0.4)” text_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)”]

At Streamhub, we deal with many forms of data like user\u2019s (every minute) player viewership data, user\u2019s commerce data, user\u2019s subscription information, metadata feeds, metadata front-loaded, user panels, and several other third-party datasets. It is easy to imagine the amount of work required to unify these various forms of data from various sources and ingest them into our ecosystem and making them richer, standard and useful to our clients. At some point, we were running hundreds of jobs daily, crunching over terabytes of data to support our extensive use-cases. All this heavy lifting is done by Apache Spark running over AWS Elastic MapReduce, in the Yarn mode.<\/span><\/p>\n

In the last two years, we have learned massively about how to optimize Spark jobs over EMR. Writing your first job in Spark might take a day or two but learning to tune Spark takes months if not years. This article was initially created as internal team notes which I am now publishing hoping more people can benefit from it. This covers almost everything you should know to run your Spark jobs efficiently (on EMR\/Yarn but covers general aspects as well) which comprises of performance optimizations, resource allocations, cost reductions, understanding important concepts, common mistakes in coding\/configuring jobs, common exceptions and some must-reads\/watch about Spark we have found over time. Right, literally everything!<\/span><\/p>\n

<\/a><\/p>\n

This article will touch upon the following topics:\u00a0<\/span><\/p>[\/et_pb_text][et_pb_text quote_border_weight=”14px” quote_border_color=”#00ac69″ _builder_version=”3.27.4″ link_font=”||||||||” quote_font=”Work Sans|700|||||||” quote_text_color=”#000000″ quote_font_size=”18px” quote_line_height=”1.8em” header_font=”||||||||” header_2_font=”Work Sans|700|||||||” header_2_font_size=”25px” header_2_line_height=”1.5em” max_width=”700px” max_width_tablet=”” max_width_phone=”” max_width_last_edited=”on|tablet” header_2_font_size_tablet=”30px” header_2_font_size_phone=”15px” header_2_font_size_last_edited=”on|desktop” locked=”off” header_font_size__hover=”30px” header_font_size__hover_enabled=”30px” header_letter_spacing__hover=”0px” header_letter_spacing__hover_enabled=”0px” header_text_shadow_style__hover=”none” header_text_shadow_style__hover_enabled=”none” header_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_2_font_size__hover=”26px” header_2_font_size__hover_enabled=”26px” header_2_letter_spacing__hover=”0px” header_2_letter_spacing__hover_enabled=”0px” header_2_line_height__hover=”1em” header_2_line_height__hover_enabled=”1em” header_2_text_shadow_style__hover=”none” header_2_text_shadow_style__hover_enabled=”none” header_2_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_2_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_3_font_size__hover=”22px” header_3_font_size__hover_enabled=”22px” header_3_letter_spacing__hover=”0px” header_3_letter_spacing__hover_enabled=”0px” header_3_line_height__hover=”1em” header_3_line_height__hover_enabled=”1em” header_3_text_shadow_style__hover=”none” header_3_text_shadow_style__hover_enabled=”none” header_3_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_3_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_4_font_size__hover=”18px” header_4_font_size__hover_enabled=”18px” header_4_letter_spacing__hover=”0px” header_4_letter_spacing__hover_enabled=”0px” header_4_line_height__hover=”1em” header_4_line_height__hover_enabled=”1em” header_4_text_shadow_style__hover=”none” header_4_text_shadow_style__hover_enabled=”none” header_4_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_4_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_5_font_size__hover=”16px” header_5_font_size__hover_enabled=”16px” header_5_letter_spacing__hover=”0px” header_5_letter_spacing__hover_enabled=”0px” header_5_line_height__hover=”1em” header_5_line_height__hover_enabled=”1em” header_5_text_shadow_style__hover=”none” header_5_text_shadow_style__hover_enabled=”none” header_5_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_5_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_6_font_size__hover=”14px” header_6_font_size__hover_enabled=”14px” header_6_letter_spacing__hover=”0px” header_6_letter_spacing__hover_enabled=”0px” header_6_line_height__hover=”1em” header_6_line_height__hover_enabled=”1em” header_6_text_shadow_style__hover=”none” header_6_text_shadow_style__hover_enabled=”none” header_6_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_6_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” text_letter_spacing__hover=”0px” text_letter_spacing__hover_enabled=”0px” text_text_shadow_style__hover=”none” text_text_shadow_style__hover_enabled=”none” text_text_shadow_color__hover=”rgba(0,0,0,0.4)” text_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)”]

\n

EMR Instance-types: Understand your workload and use specialized instances<\/em>
EMR Purchasing options: choose the right option to optimize cost for your use-case: Spot or Spot Fleet or Spot blocking<\/em>
Resource utilization: Configure your spark cluster to fully use your resources<\/em>
\u2014 Static resource allocation<\/em>
\u2014 \u2014 Understanding Spark\u2019s memory usage is important<\/em>
\u2014 Dynamic resource allocation<\/em>
GC Tuning<\/em>
Bad code and related problems<\/em>
\u2014 Avoid shuffle<\/em>
\u2014 Some preferred methods: ReduceByKey over GroupByKey<\/em>
\u2014 Avoid serialization of the whole object<\/em>
\u2014 Operations reordering<\/em>
\u2014 Improve Joins<\/em>
\u2014 \u2014 Using Broadcast variable (Joining very large datasets with a relatively small dataset)<\/em>
\u2014 \u2014 Using filters pre-join (Joining very large datasets with a mid-sized dataset)<\/em>
Degree of parallelism<\/em>
\u2014 Repartition<\/em>
\u2014 Coalesce<\/em>
More on partitioners<\/em>
Choice of serializer<\/em>
Dedicated Spark local directories<\/em>
Writing to databases<\/em>
Data storage formats: Parquet, ORC, Databricks Delta & compression options<\/em>
Caching intermediate datasets<\/em>
Use higher-level APIs: Dataframes\/Datasets<\/em><\/p>\n<\/blockquote>[\/et_pb_text][et_pb_text _builder_version=”3.27.4″ text_font=”||||||||” text_line_height=”2em” header_font=”||||||||” header_2_font=”Work Sans|700|||||||” header_2_font_size=”25px” header_2_line_height=”1.5em” max_width_tablet=”” max_width_phone=”” max_width_last_edited=”on|tablet” custom_padding=”||20px|||” header_2_font_size_tablet=”30px” header_2_font_size_phone=”15px” header_2_font_size_last_edited=”on|desktop” locked=”off” header_font_size__hover=”30px” header_font_size__hover_enabled=”30px” header_letter_spacing__hover=”0px” header_letter_spacing__hover_enabled=”0px” header_text_shadow_style__hover=”none” header_text_shadow_style__hover_enabled=”none” header_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_2_font_size__hover=”26px” header_2_font_size__hover_enabled=”26px” header_2_letter_spacing__hover=”0px” header_2_letter_spacing__hover_enabled=”0px” header_2_line_height__hover=”1em” header_2_line_height__hover_enabled=”1em” header_2_text_shadow_style__hover=”none” header_2_text_shadow_style__hover_enabled=”none” header_2_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_2_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_3_font_size__hover=”22px” header_3_font_size__hover_enabled=”22px” header_3_letter_spacing__hover=”0px” header_3_letter_spacing__hover_enabled=”0px” header_3_line_height__hover=”1em” header_3_line_height__hover_enabled=”1em” header_3_text_shadow_style__hover=”none” header_3_text_shadow_style__hover_enabled=”none” header_3_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_3_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_4_font_size__hover=”18px” header_4_font_size__hover_enabled=”18px” header_4_letter_spacing__hover=”0px” header_4_letter_spacing__hover_enabled=”0px” header_4_line_height__hover=”1em” header_4_line_height__hover_enabled=”1em” header_4_text_shadow_style__hover=”none” header_4_text_shadow_style__hover_enabled=”none” header_4_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_4_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_5_font_size__hover=”16px” header_5_font_size__hover_enabled=”16px” header_5_letter_spacing__hover=”0px” header_5_letter_spacing__hover_enabled=”0px” header_5_line_height__hover=”1em” header_5_line_height__hover_enabled=”1em” header_5_text_shadow_style__hover=”none” header_5_text_shadow_style__hover_enabled=”none” header_5_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_5_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” header_6_font_size__hover=”14px” header_6_font_size__hover_enabled=”14px” header_6_letter_spacing__hover=”0px” header_6_letter_spacing__hover_enabled=”0px” header_6_line_height__hover=”1em” header_6_line_height__hover_enabled=”1em” header_6_text_shadow_style__hover=”none” header_6_text_shadow_style__hover_enabled=”none” header_6_text_shadow_color__hover=”rgba(0,0,0,0.4)” header_6_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)” text_letter_spacing__hover=”0px” text_letter_spacing__hover_enabled=”0px” text_text_shadow_style__hover=”none” text_text_shadow_style__hover_enabled=”none” text_text_shadow_color__hover=”rgba(0,0,0,0.4)” text_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)”]

This article was originally on Medium, in case you find this format is easier to read:\u00a0\u00a0https:\/\/medium.com\/@sambodhi_72782\/spark-tuning-manual-47b98ccb2b2c<\/a><\/p>\n

Reducing your processing cost is important, also because lower is the cost, better resources you can get for your spending and lower effort you have to make in improving the performance.<\/p>\n

<\/strong><\/span><\/h3>\n

<\/strong><\/span><\/h3>\n

I<\/strong><\/span>nstance-types: Understand your workload and use specialised instances<\/h3>\n