Sadly most data engineers never care going the extra mile and logging all strps to support any problem (we never know what problems can be, from bad files or changes in compute or passwords expiringl.. funny enough in my +20 years Oracle developer in different complex businesses most production ptoblems rarely were code problem. So i always add full detailed log ready to consume (and not just go and inspect 1000s log lines which non-sql guys (hadoop, spark, emr etc) they all miss not caring with production support. I have seen in a 120k lines of pyspark with around 6 levels of code depth (proc calls a proc calls ..) WITHOUT any exception blocks.. 3 years project it seems costing 2MM.. I think theres cultural differences in how data engineers care more about production support (Europeans are the best, but nowadays we have much work being outsourced to India.. and i stop here).
Sadly most data engineers never care going the extra mile and logging all strps to support any problem (we never know what problems can be, from bad files or changes in compute or passwords expiringl.. funny enough in my +20 years Oracle developer in different complex businesses most production ptoblems rarely were code problem. So i always add full detailed log ready to consume (and not just go and inspect 1000s log lines which non-sql guys (hadoop, spark, emr etc) they all miss not caring with production support. I have seen in a 120k lines of pyspark with around 6 levels of code depth (proc calls a proc calls ..) WITHOUT any exception blocks.. 3 years project it seems costing 2MM.. I think theres cultural differences in how data engineers care more about production support (Europeans are the best, but nowadays we have much work being outsourced to India.. and i stop here).
Sorry that your weekend was thrown for a loop by this! Thank goodness you kept the old account running, eh?! Solid lesson for everyone!