Prepare with Databricks : Certified-Data-Engineer-Professional exam braindumps as your best preparation materials

A data engineer is designing a pipeline in Databricks that processes records from a Kafka stream where late-arriving data is common. Which approach should the data engineer use?

• A. Use a watermark to specify the allowed lateness to accommodate records that arrive after their expected window, ensuring correct aggregation and state management.
• B. Use batch processing and overwrite the entire output table each time to ensure late data is incorporated correctly.
• C. Implement a custom solution using Databricks Jobs to periodically reprocess all historical data.
• D. Use an Auto CDC pipeline with batch tables to simplify late data handling.

A team of data engineer are adding tables to a DLT pipeline that contain repetitive expectations for many of the same data quality checks.
One member of the team suggests reusing these data quality rules across all tables defined for this pipeline.
What approach would allow them to do this?

• A. Add data quality constraints to tables in this pipeline using an external job with access to pipeline configuration files.
• B. Maintain data quality rules in a separate Databricks notebook that each DLT notebook of file.
• C. Use global Python variables to make expectations visible across DLT notebooks included in the same pipeline.
• D. Maintain data quality rules in a Delta table outside of this pipeline's target schema, providing the schema name as a pipeline parameter.

A data engineering workspace was automatically enabled for Unity Catalog, creating a workspace catalog. New team members report they can create tables in the default schema but cannot access table in other schemas within the same workspace catalog. Why are the new team members unable to access tables in other schemas?

• A. Workspace catalog permissions are not subject to inheritance rules.
• B. Tables in other schemas require additional BROWSEprivileges that new users don't receive automatically
• C. New users only receive CREATE TABLE privileges on the default schema.
• D. Workspace users receive USE CATALOG and specific privileges on default schema only.

A data pipeline uses Structured Streaming to ingest data from kafka to Delta Lake. Data is being stored in a bronze table, and includes the Kafka_generated timesamp, key, and value. Three months after the pipeline is deployed the data engineering team has noticed some latency issued during certain times of the day.
A senior data engineer updates the Delta Table's schema and ingestion logic to include the current timestamp (as recoded by Apache Spark) as well the Kafka topic and partition. The team plans to use the additional metadata fields to diagnose the transient processing delays.
Which limitation will the team face while diagnosing this problem?

• A. Updating the table schema requires a default value provided for each file added.
• B. New fields will not be computed for historic records.
• C. Spark cannot capture the topic partition fields from the kafka source.
• D. New fields cannot be added to a production Delta table.
• E. Updating the table schema will invalidate the Delta transaction log metadata.

A data team is implementing an append-only Delta Lake pipeline that processes both batch and streaming data. They want to ensure that schema changes in the source data are automatically incorporated without breaking the pipeline. Which configuration should the team use when writing data to the Delta table?

• A. overwriteSchema = true
• B. ignoreChanges = false
• C. validateSchema = false
• D. mergeSchema = true