Idempotence

#Dev #Data #Data Pipeline

Data pipelines fails. Just like rebooting a system, a first fix if we don’t spot any obvious error message is to retry.

A secret weapon to make sure retry works is to make our data pipeline idempotent.

Planted: 2023-10-30 by L Ma;

References:

How to make data pipelines idempotent. 13 May 2021 [cited 1 Nov 2023]. Available: https://www.startdataengineering.com/post/why-how-idempotent-data-pipeline/

Similar Articles:

Bloom filter returns1 probably yes, or no. element in set element not in set bloom filter returns no …

As a person who is interacting with data, it is crucial to understand how the data is delivered. In …

Centered Kernel Alignment (CKA)

Centered Kernel Alignment (CKA) is a similarity metric designed to measure the similarity of between …

Hilbert-Schmidt Independence Criterion (HSIC)

Given two kernels of the feature representations $K=k(x,x)$ and $L=l(y,y)$, HSIC is defined as12 $$ …

L Ma (2023). 'Idempotence', Datumorphism, 10 April. Available at: https://datumorphism.leima.is/cards/dev/idempotence/.