What limitation will the data engineering team face when diagnosing latency issues?

The limitation faced by the data engineering team when diagnosing latency issues is that new fields will not be computed for historic records. This is crucial because diagnosing latency often involves understanding how data has flowed through the system and identifying bottlenecks. If new fields are added after historical data has been processed, the team will lack complete visibility into the dataset which may hinder their ability to fully analyze or understand the reasons for latency.

When working with streaming data or batch processes, any processing logic that is dependent on newly added fields will not backfill or reprocess historical data unless specifically configured to do so. As a result, the team will struggle to correlate new insights from recent data with older records, complicating the diagnosis of latency-related issues. This can lead to an incomplete view of data performance over time, making it challenging for data engineers to effectively troubleshoot and resolve latency problems.