Data pipelines fails. Just like rebooting a system, a first fix if we don’t spot any obvious error message is to retry.
A secret weapon to make sure retry works is to make our data pipeline idempotent.
L Ma (2023). 'Idempotence', Datumorphism, 10 April. Available at: https://datumorphism.leima.is/cards/dev/idempotence/.