Databricks Data Engineering Professional Practice Exam

To configure a grouped aggregation in a streaming data pipeline for average humidity and temperature, specifying a window function with a five-minute duration is crucial for effectively grouping the data over defined time intervals. This approach allows the streaming data to be processed in chunks, enabling the calculation of averages over each time window without the risk of processing too much data at once.

Using a five-minute window ensures that the aggregation computations are timely and relevant, capturing the most recent data points within that specific duration. Regularly updated averages can provide insights into trends and fluctuations in both humidity and temperature, which are valuable for monitoring purposes.

In contrast, relying on lag functions or directly referencing event times may lead to difficulties in aggregation over a continuous stream. Lag functions are typically used to access data from previous intervals, which does not directly facilitate grouped aggregations in real-time. Similarly, directly referencing event_time for grouping without a defined window may produce unpredictable results, as it could lead to each individual event being treated separately rather than as part of a cohesive time segment.

Aggregation over a ten-minute interval could also be useful, but it may not be the most efficient or timely approach for situations requiring near real-time analysis, which a five-minute window can better accommodate. In essence, the choice of a