What happens if an omitted critical field in a Kafka source is noticed after three months of production?

When a critical field is omitted in a Kafka source and this omission is recognized after a substantial amount of time—like three months in production—this situation highlights the limitations imposed by Kafka’s retention policies. Kafka stores messages for a configurable retention period, which by default could be set to a specific duration or size constraint. If the retention limit is reached, older messages, including any with the omitted critical field, are purged from the Kafka topic.

Thus, if only three months have passed and the retention policy is set to keep messages for a shorter duration than that, the data that includes the omitted field would no longer be available for reprocessing or correction. Consequently, corrective measures such as replaying data from the Kafka stream cannot occur, because the relevant data would no longer exist in the system.

This retention aspect underscores the importance of thoroughly validating that the data schema is complete and correct before production deployment, to avoid such critical omissions that cannot be rectified later due to data loss from retention policies.