Which configuration should be used when scheduling Structured Streaming jobs to recover from query failures efficiently?

When scheduling Structured Streaming jobs to recover from query failures efficiently, it's essential to configure the job with a new job cluster, allow unlimited retries, and limit maximum concurrent runs to one. This setup ensures that:

1. New Job Cluster: Using a new job cluster for each job execution provides isolation and prevents interference with other jobs. In the context of structured streaming, each job may require specific resources, configurations, or even dependencies that should not collide with those of other jobs. This strengthens reliability and consistency in resource allocation, especially for jobs that may be resource-intensive or sensitive to dependencies.

2. Retries: By setting the retries to unlimited, you provide the system with the capability to attempt to recover from transient issues or failures that may occur during job execution. In streaming applications, it is common to encounter situations where failures might arise due to temporary data-related issues, network problems, or even resource constraints. Unlimited retries enable the job to continue trying to process the streaming data until it succeeds, enhancing data resilience.

3. Maximum Concurrent Runs: Limiting the maximum concurrent runs to one ensures that only a single instance of the job runs at any given time. This is particularly important in structured streaming, where having multiple instances of the same