Phase 1: Foundations

File I/O with CSV, JSON, Parquet & Avro formats

Beginner ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine your computer is like a giant kitchen, and all the information it uses are like recipes. When you want to bake cookies, you need to read the recipe from a cookbook (that's "Input"). And if you invent a brand-new cookie, you'll want to write down your recipe so you don't forget it (that's "Output"). This whole process of reading and writing information to files is what computers call "File I/O." Just like you have different ways to store recipes – on a simple card, in a fancy binder, or a giant cookbook – computers also have different ways to store their information. Knowing which storage method, or "format," to use is super important, especially when dealing with lots and lots of "recipes" or data!

First, think about simple recipe cards. If you write "2 eggs, 1 cup flour, 1/2 cup sugar," where each line is an item, and commas separate the parts, that's a lot like a CSV (Comma Separated Values) file. It's straightforward, easy for computers and people to quickly read simple lists, like a shopping list or a table of scores. Now, imagine a more detailed recipe card that says: "Ingredients: {Dairy: [2 eggs, 1 cup milk], Dry Goods: [1 cup flour, 1/2 cup sugar]}." This is like a JSON (JavaScript Object Notation) file. It's more flexible because it can group related items together (like all dairy things) and describe complex data with many nested parts. It’s great for detailed instructions or information that might change often.

But what if you're running a giant bakery that makes thousands of cakes every day? You wouldn't want to flip through countless individual recipe cards! You'd need a super-organized system to quickly find, say, every recipe that uses chocolate chips, or calculate total flour needed for the day. That's where formats like Parquet and Avro come in. These are like super-efficient recipe books designed for massive-scale cooking. They store information in a way that’s not designed for you to read directly, but it’s incredibly fast for the computer to search, organize, and process huge amounts of data quickly. They’re built for speed and efficiency when dealing with truly enormous quantities of information.

So, as a "data engineer," you’re like the master chef and kitchen manager of the computer world. You learn about these different "recipe books" and "storage methods" so you can choose the best one for the task at hand. This means you can build smart systems that help computers quickly find the right ingredients, follow complex instructions, and efficiently store all the yummy data needed to make amazing things happen!

File I/O (Input/Output) is the fundamental process of reading data from files and writing data to files. As a Data Engineer, you'll constantly interact with various data sources and destinations, making robust file handling in Python crucial. Different data formats are optimized for different purposes – some for human readability, some for efficient storage and processing of massive datasets. Understanding when and why to use CSV, JSON, Parquet, and Avro is a core skill for building scalable data pipelines.

Let's start with the more common, human-readable formats. CSV (Comma Separated Values) files are simple text files where each line is a data record, and fields are separated by a delimiter (often a comma). They're straightforward, widely supported, and excellent for tabular, structured data. JSON (JavaScript Object Notation), on the other hand, is a flexible, semi-structured format that uses key-value pairs and can represent complex nested structures. It's often used for data exchange over APIs and logs due to its ability to handle schema evolution more gracefully than rigid tabular formats.

For big data scenarios, Parquet and Avro are preferred for their efficiency and advanced features. Parquet is a columnar storage format, meaning it stores data column by column. This is incredibly efficient for analytical queries, as you only read the columns you need, leading to faster query times and better compression. It's a staple in data lakes. Avro is a row-based binary format with a rich schema definition. It excels at data serialization, guaranteeing data compatibility even when schemas evolve, making it ideal for streaming data and data exchange between different systems. While not directly human-readable, both Parquet and Avro offer significant performance and storage benefits over text-based formats for large-scale data processing.

Key Takeaways

  • CSV: Simple, tabular data, human-readable, widely compatible.
  • JSON: Flexible, semi-structured, ideal for complex data, APIs, and logs.
  • Parquet: Columnar storage, optimized for analytical queries, compression, used in data lakes.
  • Avro: Row-based binary, strong schema definition, great for data serialization and schema evolution.
  • Python libraries (e.g., pandas, json, pyarrow, fastavro) simplify handling these formats.

Code Example

python
import pandas as pd
import json

# --- CSV Example ---
data_csv = {'name': ['Alice', 'Bob'], 'age': [30, 24]}
df = pd.DataFrame(data_csv)
df.to_csv('my_data.csv', index=False) # Write to CSV
print("CSV written.")

df_read = pd.read_csv('my_data.csv') # Read from CSV
print("CSV read:\n", df_read)

# --- JSON Example ---
data_json = {
    "users": [
        {"id": 1, "name": "Charlie", "email": "[email protected]"},
        {"id": 2, "name": "Dana", "email": "[email protected]"}
    ]
}
with open('my_data.json', 'w') as f: # Write to JSON
    json.dump(data_json, f, indent=4)
print("\nJSON written.")

with open('my_data.json', 'r') as f: # Read from JSON
    data_read = json.load(f)
print("JSON read:\n", data_read)

How this code works

This code demonstrates how to write and read data using two fundamental file formats in data engineering: CSV and JSON. It utilizes the pandas library for handling tabular CSV data and Python's built-in json module for structured JSON data.

The script first tackles CSV. A pandas.DataFrame is created from a dictionary data_csv, representing structured data. df.to_csv('my_data.csv', index=False) then saves this DataFrame to a file. The crucial index=False argument prevents pandas from writing the DataFrame's internal row index as an unwanted extra column in the CSV. Afterward, pd.read_csv('my_data.csv') efficiently reads the data back into a new DataFrame, confirming the write operation.

Next, the code handles JSON. A nested Python dictionary, data_json, is defined to mimic typical JSON structures. The with open('my_data.json', 'w') as f: statement safely opens the file for writing. Inside this block, json.dump(data_json, f, indent=4) writes the dictionary's contents to the file. The indent=4 argument is a helpful feature for beginners, as it formats the JSON output with four spaces for human readability. Finally, json.load(f) reads the JSON data back from the file into a Python dictionary, completing the cycle for JSON operations.

Related Articles

Hand-picked reads to go deeper on python for data engineering.

Back to Data Engineer Roadmap