Monday, August 8, 2022
HomeBig DataTips on how to Collaborate Throughout Your AWS Knowledge Stack with Atlan...

Tips on how to Collaborate Throughout Your AWS Knowledge Stack with Atlan – Atlan


By Brendan Siebecker, Director of Alliances – Atlan
By Gaurav Malhotra, Options Architect – AWS


Extra individuals are utilizing knowledge as we speak than ever earlier than, but it surely’s getting more durable and more durable for everybody to collaborate on the identical knowledge. This contains knowledge engineers and analysts, product managers, entrepreneurs, researchers, and extra.

Atlan is an AWS Associate with the Amazon Redshift Prepared designation. It has pioneered a collaborative workspace serving to trendy knowledge groups work collectively higher. Atlan is a collaboration and orchestration layer—the glue that brings collectively your workforce, the instruments you like, and the information you want.

With deep integrations throughout the fashionable knowledge stack, Atlan helps groups create a single supply of reality for all of their knowledge belongings. Atlan is extending its suite of integrations via in depth collaboration with Amazon Net Providers (AWS), and now you can discover Atlan on AWS Market.

On this submit, we are going to share how firms use Atlan and AWS to democratize their knowledge, collaborate extra successfully, and unify all of their data and context in a single place. We’ll additionally present methods to combine Atlan and Amazon Redshift with a step-by-step walkthrough.

How Firms Use Atlan and AWS

Prospects within the AWS ecosystem can profit from Atlan’s seamless integration with a set of AWS companies—together with analytics tooling reminiscent of Amazon Redshift, Amazon Athena, and AWS Glue—in addition to in style instruments within the trendy knowledge stack reminiscent of Tableau, Apache Airflow, and dbt.

For instance, Postman (an API platform utilized by greater than 500,000 firms worldwide) makes use of AWS and Atlan to open up their knowledge, construct belief, and change into extra data-driven. That is vital as a result of Postman’s leaders staunchly imagine that everybody within the firm ought to be capable of entry knowledge and achieve insights from it. Nevertheless, earlier than Atlan, their knowledge was typically a thriller and context lived within the heads of early workforce members.

Prudhvi Vasa, Analytics Chief at Postman, defined the worth of democratizing and documenting knowledge with AWS and Atlan: “We’ve been in a position to catalog and doc all of our knowledge, which acts as a single supply of reality for our knowledge. The end result? Everybody is ready to discover the fitting knowledge for his or her use case, and the information is constant throughout the board for all accessing it,” says Vasa.

“Having a dependable knowledge basis, the place folks can discover and perceive all our knowledge opens the opportunity of having everybody take part in analyzing knowledge. This permits our total firm to change into extra data-aware and data-driven, which is the purpose for any main firm as we speak,” Vasa provides.

Atlan, AWS, and the Trendy Knowledge Stack

Atlan acts as a virtualized layer throughout quite a lot of instruments within the trendy knowledge stack. Its push- and pull-based metadata crawlers carry metadata from completely different instruments within the knowledge platform to construct a unified collaboration platform.

First, Atlan creates a strong search and discovery layer. It acts as a Google-like search engine for all your knowledge, the place you possibly can shortly uncover and entry any knowledge asset together with all of its related context and documentation. This search helps clever key phrase recognition, highly effective search filters, sorting by relevance or recognition, and even a Cmd+Okay shortcut.

This search doesn’t simply floor knowledge tables—it surfaces every little thing about a company’s knowledge. In as we speak’s day and age, knowledge belongings usually are not simply tables. That’s why Atlan lets folks search throughout each kind of information asset—enterprise intelligence (BI) dashboards, pipelines, code, fashions, queries, metrics, directed acrylic graphs (DAGs), and extra.

Second, Atlan unifies context from all of the completely different instruments in your knowledge stack in a single place. The place does this knowledge come from? Who makes use of it? Can I belief it? The “asset profile” in Atlan solutions questions like these with info like an information asset’s description, certification (verified, WIP, or deprecated), column previews, pattern knowledge, and Readme. This makes it simpler to grasp every knowledge asset (like lineage, documentation, and possession) in a single view.

Embedded Collaboration Integrations

Atlan is constructed on the premise of embedded collaboration, borrowing ideas from GitHub, Figma, Superhuman, and different trendy future-of-work instruments.

Embedded collaboration is about work occurring the place you might be, with the least quantity of friction. What for those who may request entry to an information asset whenever you get a hyperlink, and the proprietor may get the request on Slack and approve or reject it proper there?

What if, whenever you’re inspecting an information asset and must report a difficulty, you can instantly set off a help request that’s completely built-in along with your engineering workforce’s Jira workflow?

Embedded collaboration unifies these micro-workflows that waste time, trigger frustration, and result in device fatigue, turning time-consuming duties throughout a number of instruments into just a few clicks in whichever device you’re already utilizing.

Getting Began with Atlan and AWS

Atlan is constructed on high of AWS’s highly effective companies. By instantly integrating with AWS companies like Amazon Redshift, Atlan helps knowledge groups accomplish extra by making collaboration a seamless a part of their course of.

Atlan-Data-Stack-1
Determine 1 – Atlan helps knowledge groups share metadata and collaborate throughout the fashionable knowledge stack.

Within the following step-by-step information, we’ll present you methods to shortly combine Atlan with Redshift to open up a brand new world of collaboration, readability, and belief for contemporary knowledge groups.

Comply with the steps beneath to determine a connection and combine Atlan with a Redshift database.

Step 1: Choose the Supply

  • Log into your Atlan workspace.
  • Click on on the Workflow button within the left sidebar.
  • You’ll see the Market web page with the checklist of sources out there in your workspace. Click on on New Workflow on the high proper.
  • Choose Redshift from the checklist of choices within the integrations tab, and click on Setup Workflow.
Atlan-Data-Stack-2.2
Determine 2 – Choose Redshift from the checklist of choices within the Market.

Step 2: Present Credentials

  • To arrange a brand new connection, fill in your Redshift credentials on the Credential web page. Beneath is an instance:
    • Hostname: examplecluster.abc123xyz789.us-west-2.redshift.amazonaws.com
    • Port: 5439
    • Username: myusername
    • Password: xxxxxx
    • Default Database: dev
  • Choose the right authentication methodology (primary authentication, or an IAM person or IAM function).
  • Upon getting stuffed within the particulars, click on on Check Authentication after which Subsequent.
Atlan-Data-Stack-3.2
Determine 3 – Enter your Amazon Redshift credentials.

Step 3: Set Up Your Configuration

  • On the Connection web page, identify your connection and choose the customers or teams who ought to be capable of entry it.
Atlan-Data-Stack-4.2
Determine 4 – Specify the identify and entry guidelines to your Redshift connection.
  • On the Metadata web page, specify any metadata you wish to embody or exclude from crawling.
  • Click on Run to run the crawler as soon as, or click on on Schedule & Run to schedule it for a every day, weekly, or month-to-month run.
  • When you click on Run or set a schedule, the workflow will begin working.
Atlan-Data-Stack-8
Determine 5 – You will have now built-in Atlan with Amazon Redshift.

Atlan will crawl your Amazon Redshift occasion and ingest all the metadata into Atlan. This course of varies relying on the dimensions of the warehouse you’re seeking to crawl, but it surely typically takes lower than half-hour.

Step 4: Uncover Your Belongings

Now that you simply’ve efficiently related Atlan to Amazon Redshift and the Atlan crawler has ingested the metadata, you can begin discovering your belongings inside Atlan.

Within the instance beneath, an organization has 362 Redshift belongings inside Atlan. These are seen on the Discovery web page, filtered by the Amazon Redshift integration.

Atlan-Data-Stack-9
Determine 6 – Enriched desk belongings from Amazon Redshift in Atlan.

The information workforce can click on via to any asset to see related context, such because the column names, glossary phrases, classifications, standing, associated queries, and Readme.

Atlan-Data-Stack-10
Determine 7 – Knowledge asset profile for an asset from Amazon Redshift in Atlan.

They will additionally see the information asset’s lineage, which is auto-generated on the column stage for each Redshift asset. This helps knowledge groups see the place every asset comes from and which dashboards use it.

Atlan-Data-Stack-11
Determine 8 – Auto-generated, column-level lineage in Atlan for an Amazon Redshift asset.

Conclusion

Establishing Atlan is simple for analysts and enterprise customers alike. From connecting your tables and dashboards inside Amazon Redshift to enriching imported knowledge belongings, the method doesn’t require main knowledge engineering assets or time. Crawling occurs seamlessly and stays in sync along with your AWS situations.

To see it for your self, take a look at Atlan’s guided demo. Atlan may be downloaded from AWS Market, or you possibly can get began on the Atlan web site.

For extra info, take a look at these hyperlinks:


This text was initially printed on the AWS Associate Community (APN) weblog.

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular