{"id":29384,"date":"2020-02-20T23:43:47","date_gmt":"2020-02-20T23:43:47","guid":{"rendered":"http:\/\/streamhub.co.uk\/?p=29384"},"modified":"2021-06-12T09:21:07","modified_gmt":"2021-06-12T09:21:07","slug":"how-to-design-a-modern-data-analytics-platform","status":"publish","type":"post","link":"https:\/\/streamhub.co.uk\/how-to-design-a-modern-data-analytics-platform\/","title":{"rendered":"How to design a modern data analytics platform in a rapidly changing world"},"content":{"rendered":"Reading Time: <\/span> 11<\/span> minutes<\/span><\/span>

[et_pb_section fb_built=”1″ _builder_version=”4.0.6″ custom_padding=”1px||0px|||” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on” da_disable_devices=”off|off|off”][et_pb_row _builder_version=”3.27.4″ custom_margin=”13px|auto||auto||” custom_padding=”0px|||||”][et_pb_column type=”4_4″ _builder_version=”3.27.4″][et_pb_blurb title=”@ET-DC@eyJkeW5hbWljIjp0cnVlLCJjb250ZW50IjoicG9zdF9kYXRlIiwic2V0dGluZ3MiOnsiYmVmb3JlIjoiIiwiYWZ0ZXIiOiIiLCJkYXRlX2Zvcm1hdCI6ImRlZmF1bHQiLCJjdXN0b21fZGF0ZV9mb3JtYXQiOiIifX0=@” use_icon=”on” font_icon=”%%128%%” icon_color=”#00ac69″ icon_placement=”left” content_max_width=”1100px” use_icon_font_size=”on” icon_font_size=”16px” _builder_version=”4.0.6″ _dynamic_attributes=”title” header_font=”Work Sans|600|||||||” header_text_align=”left” header_font_size=”14px” text_orientation=”center” custom_margin=”-7px||0px|||” animation=”off” hover_enabled=”1″ locked=”off” icon_color__hover=”#00ac69″ box_shadow_color__hover_enabled=”rgba(0,0,0,0.3)” box_shadow_color__hover=”rgba(0,0,0,0.3)” box_shadow_style__hover=”none” box_shadow_style__hover_enabled=”none” use_background_color_gradient__hover=”off” use_background_color_gradient__hover_enabled=”off” background_color_gradient_start__hover=”#2b87da” background_color_gradient_start__hover_enabled=”#2b87da” background_color_gradient_end__hover=”#29c4a9″ background_color_gradient_end__hover_enabled=”#29c4a9″ background_color_gradient_type__hover=”linear” background_color_gradient_type__hover_enabled=”linear” background_color_gradient_direction__hover=”180deg” background_color_gradient_direction__hover_enabled=”180deg” background_color_gradient_direction_radial__hover=”center” background_color_gradient_direction_radial__hover_enabled=”center” background_color_gradient_start_position__hover=”0%” background_color_gradient_start_position__hover_enabled=”0%” background_color_gradient_end_position__hover=”100%” background_color_gradient_end_position__hover_enabled=”100%” background_color_gradient_overlays_image__hover=”off” background_color_gradient_overlays_image__hover_enabled=”off” parallax__hover=”off” parallax__hover_enabled=”off” parallax_method__hover=”on” parallax_method__hover_enabled=”on” background_size__hover=”cover” background_size__hover_enabled=”cover” background_position__hover=”center” background_position__hover_enabled=”center” background_repeat__hover=”no-repeat” background_repeat__hover_enabled=”no-repeat” background_blend__hover=”normal” background_blend__hover_enabled=”normal” allow_player_pause__hover=”off” allow_player_pause__hover_enabled=”off” background_video_pause_outside_viewport__hover=”on” background_video_pause_outside_viewport__hover_enabled=”on” body_letter_spacing__hover=”0px” body_letter_spacing__hover_enabled=”0px” body_text_shadow_style__hover=”none” body_text_shadow_style__hover_enabled=”none” body_text_shadow_color__hover=”rgba(0,0,0,0.4)” body_text_shadow_color__hover_enabled=”rgba(0,0,0,0.4)”][\/et_pb_blurb][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ admin_label=”Services” _builder_version=”4.0.6″ max_width=”90%” custom_padding=”0px||0px|||” locked=”off” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on” da_disable_devices=”off|off|off”][et_pb_row _builder_version=”3.27.4″ custom_padding=”0px|||||”][et_pb_column type=”4_4″ _builder_version=”3.27.4″][et_pb_text _builder_version=”4.0.6″ custom_margin=”3px|||||”]<\/p>\n

Designing a successful data analytics platform for a rapidly changing world<\/h1>\n

Key design principles we learned while building Streamhub\u2019s video data analytics platform<\/h4>\n

[\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=”4.0.6″ min_height=”58px” custom_padding=”4px|||||”][et_pb_column type=”4_4″ _builder_version=”4.0.6″][et_pb_divider color=”#003387″ divider_weight=”5px” _builder_version=”4.0.6″][\/et_pb_divider][\/et_pb_column][\/et_pb_row][et_pb_row column_structure=”1_2,1_2″ _builder_version=”4.0.6″ custom_margin=”-28px|auto||auto||”][et_pb_column type=”1_2″ _builder_version=”4.0.6″][et_pb_text _builder_version=”4.0.6″ custom_margin=”|-24px||||” custom_padding=”||11px|||”]<\/p>\n

We are in the age of data where you can change the ending of a film if it doesn\u2019t match your taste! <\/span>Black Mirror<\/span><\/a>‘s ‘Bandersnatch’ is Netflix’s first major interactive film where its script has millions of permutations. <\/span><\/p>\n

\u201c<\/span>It\u2019s an ambitious film that saw Netflix making several innovations in its platform, such as reworking its cache memory to store multiple scenes or creating the \u2018Branch Manager\u2019 to help Charlie Brooker write the script<\/span><\/i>.\u201d <\/span><\/p>\n

Another significant milestone for data-driven content creation, following on from the story behind the blockbuster<\/span> Netflix remake of \u2018House of Cards\u2019.\u00a0<\/span><\/a><\/p>\n

From \u2018House of Cards\u2019 to \u2018Bandersnatch\u2019, what is that key ingredient of Netflix\u2019s success formula?<\/span> Innovation! <\/b><\/p>\n

Netflix has always pushed the boundaries of disruptive innovation, revolutionising how we consume content everyday!<\/span><\/p>\n

Simply put, innovation is the key to be ahead of the game for any company relying on data – innovating before your competitors, answering complex questions with confidence and rapidly adopting the latest powerful and sophisticated analytics tools. <\/span><\/p>\n

If you happen to be a start-up, cost and resource limitations make it that more challenging whilst having the upside of flexibility, rapid decision-making and higher risk-taking<\/span>.\u00a0<\/span><\/p>\n

<\/span><\/p>\n

[\/et_pb_text][\/et_pb_column][et_pb_column type=”1_2″ _builder_version=”4.0.6″][et_pb_image src=”https:\/\/streamhub.co.uk\/wp-content\/uploads\/2020\/02\/Black-Mirror-Bandersnatch-2018-movie-poster.jpg” _builder_version=”4.0.6″ min_height=”643px” custom_margin=”||-17px|||” custom_padding=”||0px|11px||”][\/et_pb_image][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=”3.27.4″ custom_padding=”0px|||||”][et_pb_column type=”4_4″ _builder_version=”3.27.4″][et_pb_text _builder_version=”4.0.6″ custom_margin=”3px|||||”]<\/p>\n

What we have learned<\/strong><\/h3>\n

At Streamhub, we have always been good at adopting the latest technologies. From RDBMS to HBase, Cassandra to Elasticsearch to Apache Spark to Redshift to Clickhouse to Apache Druid to Snowflake, we have seen it all! We have been spending a lot of time in switching between technologies to respond to changing business demands, adapting the platform for different customers, heavy integrations and so on. And as we all know, change is part of the game. For a long time however, we had little time to sink our thoughts into fundamental innovation to consider <\/span>how to build the foundation of our technology to embrace change more easily<\/b>.\u00a0<\/span><\/p>\n

There is also another side to it. The \u2018modern\u2019 wave of data analytics based on\u00a0 \u2018distributed computing\u2019 has rapidly evolved from Hadoop to Google\u2019s MapReduce followed by Apache Spark and \u2018analytics in the cloud\u2019 making it possible to analyse a large set of data in almost a couple of clicks without leaving the desk! This has made the space even more competitive and now more than ever it has become <\/span>vital for data businesses to focus on rapid development and innovation<\/b>.\u00a0<\/span><\/p>\n

This article talks about how you can design your big data system iteratively so you can continually respond to opportunities without putting too much cost from the get go and still make it flexible to adopt future business needs.\u00a0<\/span><\/p>\n

We will take you through five principles which are intended to introduce modularisation in your pipeline through tiering, the range of technologies that you can exploit, dimensions based on which we can separate these technologies, yet keep the core analytics narrow and defined, and keep yourself fast paced while keeping the costs in check. <\/span><\/p>\n

We highly recommend giving this a thought, no matter which stage your company is in. This has helped us not only set futuristic direction for the technology but also helped the product in understanding the reach of data, identifying low hanging features and getting a hint of time-consuming future features so we could plan; keeping development, product and sales strategy in sync. This article is meant for data technologists, architects and early data technology adopters or data-related business people.<\/span><\/p>\n

[\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=”3.27.4″ custom_margin=”-15px|auto||auto||” custom_padding=”0px|||||” min_height=”181px”][et_pb_column type=”4_4″ _builder_version=”3.27.4″][et_pb_text _builder_version=”4.0.6″ min_height=”126px” custom_margin=”30px|||||”]<\/p>\n

<\/span><\/h3>\n

1. As veterans of data say \u201cdesign in tiers\u201d, but also do think between the tiers.<\/strong><\/h3>\n

<\/span><\/p>\n

Data tiering is about moving data across a pool of storage to optimise cost and performance, which traditionally is between \u2018hot\u2019 or \u2018warm\u2019 servers or \u2018archived\u2019 storage. With more technologies involved in modern systems, this extends to cross technologies and cross cloud platforms. Data usually progresses through four broad tiers: In-transit, raw, cleansed & optimized, specialized-persistence. These tiers might break up further based on your business case. Our our reference, let\u2019s take this example:<\/span><\/p>\n

[\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=”3.27.4″ custom_margin=”-15px|auto||auto||” custom_padding=”0px|||||”][et_pb_column type=”4_4″ _builder_version=”3.27.4″][et_pb_image src=”https:\/\/streamhub.co.uk\/wp-content\/uploads\/2020\/02\/Image1.png” _builder_version=”4.0.6″ width=”95%” custom_padding=”|220px||||”][\/et_pb_image][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=”3.27.4″ custom_padding=”0px|||||”][et_pb_column type=”4_4″ _builder_version=”3.27.4″][et_pb_text _builder_version=”4.0.6″ custom_margin=”-17px|||||” custom_padding=”0px|||||”]<\/p>\n

<\/span><\/p>\n

Tiering helps you make your analytics system modular, which provides flexibility so that your organization can meet its ever-changing business requirements. Tiers could be based on:<\/span><\/p>\n