Tuesday, August 9, 2022
HomeBig DataHow Epos Now modernized their information platform by constructing an end-to-end information...

How Epos Now modernized their information platform by constructing an end-to-end information lake with the AWS Information Lab


Epos Now supplies level of sale and cost options to over 40,000 hospitality and retailers throughout 71 international locations. Their mission is to assist companies of all sizes attain their full potential by the ability of cloud know-how, with options which can be inexpensive, environment friendly, and accessible. Their options enable companies to leverage actionable insights, handle their enterprise from wherever, and attain prospects each in-store and on-line.

Epos Now at present supplies real-time and near-real-time studies and dashboards to their retailers on high of their operational database (Microsoft SQL Server). With a rising buyer base and new information wants, the crew began to see some points within the present platform.

First, they noticed efficiency degradation for serving the reporting necessities from the identical OLTP database with the present information mannequin. Just a few metrics that wanted to be delivered in actual time (seconds after a transaction was full) and some metrics that wanted to be mirrored within the dashboard in near-real-time (minutes) took a number of makes an attempt to load within the dashboard.

This began to trigger operational points for his or her retailers. The tip customers of studies couldn’t entry the dashboard in a well timed method.

Value and scalability additionally grew to become a significant downside as a result of one single database occasion was making an attempt to serve many alternative use instances.

Epos Now wanted a strategic answer to handle these points. Moreover, they didn’t have a devoted information platform for doing machine studying and superior analytics use instances, so that they selected two parallel methods to resolve their information issues and higher serve retailers:

  • The primary was to rearchitect the near-real-time reporting characteristic by transferring it to a devoted Amazon Aurora PostgreSQL-Suitable Version database, with a selected reporting information mannequin to serve to finish customers. This can enhance efficiency, uptime, and value.
  • The second was to construct out a brand new information platform for reporting, dashboards, and superior analytics. This can allow use instances for inner information analysts and information scientists to experiment and create a number of information merchandise, in the end exposing these insights to finish prospects.

On this submit, we talk about how Epos Now designed the general answer with help from the AWS Information Lab. Having developed a powerful strategic relationship with AWS during the last 3 years, Epos Now opted to reap the benefits of the AWS Information lab program to hurry up the method of constructing a dependable, performant, and cost-effective information platform. The AWS Information Lab program gives accelerated, joint-engineering engagements between prospects and AWS technical assets to create tangible deliverables that speed up information and analytics modernization initiatives.

Working with an AWS Information Lab Architect, Epos Now commenced weekly cadence calls to provide you with a high-level structure. After the target, success standards, and stretch targets have been clearly outlined, the ultimate step was to draft an in depth process listing for the upcoming 3-day construct part.

Overview of answer

As a part of the 3-day construct train, Epos Now constructed the next answer with the continued help of their AWS Information Lab Architect.

The platform consists of an end-to-end information pipeline with three most important parts:

  • Information lake – As a central supply of fact
  • Information warehouse – For analytics and reporting wants
  • Quick entry layer – To serve near-real-time studies to retailers

We selected three completely different storage options:

  • Amazon Easy Storage Service (Amazon S3) for uncooked information touchdown and a curated information layer to construct the inspiration of the information lake
  • Amazon Redshift to create a federated information warehouse with conformed dimensions and star schemas for consumption by Microsoft Energy BI, working on AWS
  • Aurora PostgreSQL to retailer all the information for near-real-time reporting as a quick entry layer

Within the following sections, we go into every element and supporting companies in additional element.

Information lake

The primary element of the information pipeline concerned ingesting the information from an Amazon Managed Streaming for Apache Kafka (Amazon MSK) subject utilizing Amazon MSK Join to land the information into an S3 bucket (touchdown zone). The Epos Now crew used the Confluent Amazon S3 sink connector to sink the information to Amazon S3. To make the sink course of extra resilient, Epos Now added the required configuration for dead-letter queues to redirect the unhealthy messages to a different subject. The next code is a pattern configuration for a dead-letter queue in Amazon MSK Join:

As a result of Epos Now was ingesting from a number of information sources, they used Airbyte to switch the information to a touchdown zone in batches. A subsequent AWS Glue job reads the information from the touchdown bucket , performs information transformation, and strikes the information to a curated zone of Amazon S3 in optimum format and format. This curated layer then grew to become the supply of fact for all different use instances. Then Epos Now used an AWS Glue crawler to replace the AWS Glue Information Catalog. This was augmented by means of Amazon Athena for doing information evaluation. To optimize for value, Epos Now outlined an optimum information retention coverage on completely different layers of the information lake to save cash in addition to hold the dataset related.

Information warehouse

After the information lake basis was established, Epos Now used a subsequent AWS Glue job to load the information from the S3 curated layer to Amazon Redshift. We used Amazon Redshift to make the information queryable in each Amazon Redshift (inner tables) and Amazon Redshift Spectrum. The crew then used dbt as an extract, load, and remodel (ELT) engine to create the goal information mannequin and retailer it in goal tables and views for inner enterprise intelligence reporting. The Epos Now crew wished to make use of their SQL information to do all ELT operations in Amazon Redshift, so that they selected dbt to carry out all of the joins, aggregations, and different transformations after the information was loaded into the staging tables in Amazon Redshift. Epos Now’s at present utilizing Energy BI for reporting, which was migrated to the AWS Cloud and related to Amazon Redshift clusters working inside Epos Now’s VPC.

Quick entry layer

To construct the quick entry layer to ship the metrics to Epos Now’s retail and hospitality retailers in near-real time, we determined to create a separate pipeline. This required growing a microservice working a Kafka client job to subscribe to the identical Kafka subject in an Amazon Elastic Kubernetes Service (Amazon EKS) cluster. The microservice acquired the messages, performed the transformations, and wrote the information to a goal information mannequin hosted on Aurora PostgreSQL. This information was delivered to the UI layer by an API additionally hosted on Amazon EKS, uncovered by Amazon API Gateway.

Consequence

The Epos Now crew is at present constructing each the quick entry layer and a centralized lakehouse architecture-based information platform on Amazon S3 and Amazon Redshift for superior analytics use instances. The brand new information platform is finest positioned to handle scalability points and help new use instances. The Epos Now crew has additionally began offloading a few of the real-time reporting necessities to the brand new goal information mannequin hosted in Aurora. The crew has a transparent technique across the selection of various storage options for the proper entry patterns: Amazon S3 shops all of the uncooked information, and Aurora hosts all of the metrics to serve real-time and near-real-time reporting necessities. The Epos Now crew may even improve the general answer by making use of information retention insurance policies in numerous layers of the information platform. This can tackle the platform value with out dropping any historic datasets. The info mannequin and construction (information partitioning, columnar file format) we designed significantly improved question efficiency and total platform stability.

Conclusion

Epos Now revolutionized their information analytics capabilities, profiting from the breadth and depth of the AWS Cloud. They’re now in a position to serve insights to inner enterprise customers, and scale their information platform in a dependable, performant, and cost-effective method.

The AWS Information Lab engagement enabled Epos Now to maneuver from concept to proof of idea in 3 days utilizing a number of beforehand unfamiliar AWS analytics companies, together with AWS Glue, Amazon MSK, Amazon Redshift, and Amazon API Gateway.

Epos Now’s at present within the technique of implementing the complete information lake structure, with a rollout to prospects deliberate for late 2022. As soon as stay, they are going to ship on their strategic aim to offer real-time transactional information and put insights instantly within the fingers of their retailers.


Concerning the Authors

Jason Downing is VP of Information and Insights at Epos Now. He’s chargeable for the Epos Now information platform and product path. He makes a speciality of product administration throughout a variety of industries, together with POS methods, cell cash, funds, and eWallets.

Debadatta Mohapatra is an AWS Information Lab Architect. He has in depth expertise throughout huge information, information science, and IoT, throughout consulting and industrials. He’s an advocate of cloud-native information platforms and the worth they will drive for patrons throughout industries.

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular