Sublime

An inspiration engine for ideas

AllPeopleCollectionsArticlesAudioBooksFilesHighlightsImagesLinksNotesTextTweetsVideosSocial

Most commonly, ETL means moving data from some source system (e.g. a production database, Slack API) into an analytical data warehouse (e.g. Snowflake) where the data is easier to combine and analyze. Most data teams use a vendor like Fivetran or an orchestration platform like Airflow to do this.

Modal is a great solution for ETL if you are primaril... See more

Why you should move your ETL stack to Modal

For low throughput data, Grab uses Parquet with Copy on Write (CoW) .

Here's the main operations for Copy on Write:

Write Operations - Whenever there's a write, you create a new version of the file that includes the latest change. You can also keep the previous version for consistency and rollback purposes. This helps prevent data corruption, incon

The Architecture of Grab's Data Lake

The most likely organizational structures to initiate or accommodate big data technologies are either existing analytics groups (including groups with an “operations research” focus, as at UPS and Schneider National); innovation or architectural groups within IT organizations, as at USAA and Procter & Gamble; or R&D analytics groups, as at

Thomas H. Davenport • Big Data at Work: Dispelling the Myths, Uncovering the Opportunities

AI & ML

Luc Cheung • 2 cards

It requires considerable activity by data scientists to extract data from where it resides, put it into a structured format suitable for analysis, and model and analyze it.

Thomas H. Davenport • Big Data at Work: Dispelling the Myths, Uncovering the Opportunities

In today’s data world, there are so many options for an EL tool to avoid you developing your own extracting script and to help you gain a LOT of time.

Fivetran, Mage, and Airbyte to mention a few.

You don’t have to maintain custom scripts, these tools come with +300 connectors, basic scheduling, and error handling.

Jeremy • What I Learned After One Year of Building a Data Platform From Scratch

Pesto

pesto.tech

reverse engineering

Juan Orbea • 1 card

System design

Tekelala and • 35 cards