PySpark: Compare Two Schemas

#PySpark #Python

To compare two dataframe schemas in [[PySpark]] Data Processing - (Py)Spark Processing Data using (Py)Spark , we can utilize the set operations in python.

def schema_diff(schema1, schema2):

    return {
        'fields_in_1_not_2': set(schema1) - set(schema2),
        'fields_in_2_not_1': set(schema2) - set(schema1)
    }

Planted: 2022-02-14 by L Ma;

Similar Articles:

Data Processing - (Py)Spark

Processing Data using (Py)Spark

Differential Learning Rates in PyTorch

Using different learning rates in different layers of our artificial neural network.

Documentation

Documenting my data science project using sphinx or mkdocs-material

Cookiecutter

Use cookiecutter to initialize a project

Additional Double Backet Links:

Data Processing - (Py)Spark

Processing Data using (Py)Spark

Lei Ma (2022). 'PySpark: Compare Two Schemas', Datumorphism, 02 April. Available at: https://datumorphism.leima.is/til/data/pyspark-schema-comparison/.