Data File Formats

GROWING This note is being developed. It is readible but changes are being applied.
cards/machine-learning/datatypes/data-file-formats.md Current Note ID: The unique ID of this note.
#data #data-engineering #data-science

Data storage is diverse. For data on smaller scales, we are mostly dealing with some data files.

work_with_data_files

Efficiencies and Compressions

Parquet

Parquet is fast. But

  1. Don’t use json or list of json as columns. Convert them to strings or binary objects if it is really needed.

Planted: by ;

References:
  1. Parquet Logical Type Definitions
  2. Working with Complex Data Formats with Structured Streaming in Apache Spark 2.1 Part 2 of Scalable Data @ Databricks
Dynamic Backlinks to cards/machine-learning/datatypes/data-file-formats:
Data File Formats
Data storage is diverse. For data on smaller scales, we are mostly dealing with some data files. …
cards/machine-learning/datatypes/data-file-formats Links to:
Data Types

LM (2021). 'Data File Formats', Datumorphism, 02 April. Available at: https://datumorphism.leima.is/cards/machine-learning/datatypes/data-file-formats/.