Inderjit, thank you for providing such a comprehensive analysis of the options. I would like to highlight an update to the Spark-monitoring Library that has been made since this publication has been published.

This library is not compatible with the most recent Databricks runtimes > 11, after Oct 22. Without it, we do not have an easy way to export compute and job execution metrics to Azure Log Analytics.