Spark Declarative Pipelines

Apache Spark Declarative Pipelines (part of Databricks Lakeflow) simplifies batch and streaming ETL: engineers declare the data transformations they need while the platform automatically handles ingestion from any Apache Spark-supported source, dependency management, scaling and recovery, and data quality rules via Expectations. It offers unified batch and streaming processing, end-to-end incremental processing, an integrated pipeline development IDE, and is built on Unity Catalog and open table formats.