Reproducible data science over data lakes: replayable data p...

Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie.

Reproducible data science is enabled through Bauplan and Nessie, providing time-travel and branching semantics on data lakes, decoupling compute from data management.

arxiv.org

RelatedHighlights

Reflections on Palantir

nabeelqu.co nabeelqu.co

Data composability: what it is + why it matters

Danny Zuckerman dazuck.substack.com

So what abstractions do we have as of today? For example, let’s take the resource abstraction (Dagster, Prefect, referred to as an operator in Airflow). You abstract complex environments and connections away with a simple construct like that. You have the immediate benefits of defining that once and using it in every task or pipeline with context.r... See more

Data Engineering • Data Orchestration Trends: The Shift From Data Pipelines to Data Products

ByteByteGo-Big-Archive-System-Design-2023

Covering a wide array of technical topics from API testing to cloud services, this document shares insights on various programming, networking, and system design concepts for tech enthusiasts and professionals alike.

Link