CSV to Parquet: Convert Your Data to Columnar Format for Analytics
CSV is universal but slow for analytics — reading a 10GB CSV to compute a single sum requires parsing every row. Parquet's columnar layout lets analytics engines skip irrelevant columns and decompress only what they need. This tool parses your CSV, infers column types, and generates the exact Python + pandas code to produce a Parquet file you can drop into Spark, DuckDB, BigQuery, Snowflake, or S3.
DuckDB's one-liner is the fastest path for large files — it reads CSV and writes Parquet in a streaming fashion without loading the full dataset into memory.
Why Parquet Outperforms CSV for Analytics
Parquet's columnar layout means a query like SELECT SUM(revenue) FROM sales reads only the revenue column bytes — not the customer name, address, or timestamp bytes. On a 50-column dataset querying 3 columns, Parquet reads 6% of the data that a CSV query would read. Additionally, Parquet stores column statistics (min/max per row group) that allow Spark, DuckDB, and BigQuery to skip row groups entirely when a filter excludes them.
Type Inference and Why It Matters
CSV stores everything as strings. When you convert to Parquet with proper types (int64 for IDs, float64 for prices, datetime for timestamps), analytics engines can use native arithmetic without parsing overhead. A timestamp stored as a string requires a CAST on every row. A timestamp stored as Parquet's INT96 or TIMESTAMP_MILLIS is already numeric — no parsing required.
Parquet in the Modern Data Stack
Parquet is the native storage format for Apache Spark, Apache Hive, Trino (Presto), DuckDB, AWS Athena, Google BigQuery (external tables), Snowflake external stages, and Delta Lake / Iceberg table formats. If you're building a data lakehouse on S3, GCS, or ADLS, Parquet is the foundation. Converting CSVs to Parquet is typically the first step in a data ingestion pipeline.
Supported Types
- string (object in pandas)
- int64 — integer columns
- float64 — decimal columns
- float32 — lower precision, smaller file
- bool — true/false columns
- datetime64[ns] — date and timestamp columns
Frequently Asked Questions
Why can't this convert directly in the browser?
Parquet is a columnar binary format that requires a native encoder — pyarrow (C++ via Python bindings) or Java. There is no WebAssembly Parquet encoder stable enough for production use. This tool generates Python code using pandas + pyarrow that you can run locally in seconds.
What Python packages do I need?
pip install pandas pyarrow. pandas is the DataFrame library; pyarrow is the Parquet encoder. Install both with one command: pip install pandas pyarrow. Alternatively, use fastparquet: pip install pandas fastparquet.
What is Parquet good for?
Parquet is a columnar storage format designed for analytics queries. Because it stores data column-by-column rather than row-by-row, queries that read only a few columns (e.g., SELECT SUM(price) FROM orders) can skip irrelevant columns entirely. It also has excellent compression — Snappy, Gzip, or Zstd — because similar values in a column compress better than mixed row data.
Should I use Snappy or Gzip compression?
Snappy (the default) is fast to compress and decompress with moderate size reduction — good for interactive queries. Gzip gives 20-40% better compression but is slower to decompress — good for cold storage. Zstd is the best balance: near-Gzip compression with near-Snappy speed. Use Snappy for hot data, Zstd for archive, Gzip only if downstream tools require it.
How do I read the Parquet file back?
Python: pd.read_parquet('output.parquet'). Spark: spark.read.parquet('output.parquet'). DuckDB: SELECT * FROM read_parquet('output.parquet'). Snowflake: COPY INTO table FROM @stage/output.parquet FILE_FORMAT = (TYPE = PARQUET). Parquet is natively supported by all major data platforms.