Efficient analytics depend on how data is physically stored and read. Columnar storage organizes data by column rather than by row, which is highly efficient for analytics because queries typically read only a subset of columns. This layout also enables better compression ratios (similar values are stored together) and faster aggregation queries. Apache Parquet is an open-source columnar storage file format optimized for analytics workloads, supporting efficient compression and encoding schemes, nested data structures, and predicate pushdown for fast query filtering. Parquet is the dominant format in big data ecosystems with tools like Spark, Hive, Presto, Trino, and cloud data lakes. ORC (Optimized Row Columnar) provides similar capabilities with tight Hive integration and strong predicate pushdown via zone maps. Apache Avro, by contrast, is a row-based serialization format that stores data with its schema in the same file and supports schema evolution, making it well suited to Kafka message serialization and data exchange between systems. Avro, Parquet, and table formats like Iceberg and Delta Lake all support schema evolution, allowing pipelines to adapt to upstream changes gracefully.
Apache Arrow is a cross-language, columnar in-memory data format with zero-copy reads, eliminating serialization overhead when moving data between systems like Spark, Pandas, Polars, and DuckDB. Arrow powers vectorized query engines such as DuckDB, ClickHouse, and Velox, which process data in batches of rows (vectors) rather than one row at a time, using SIMD CPU instructions to dramatically reduce interpretation overhead. Row-oriented engines process operators one row at a time, suited for OLTP, while column-oriented engines process data column-by-column, ideal for analytics that touch few columns of many rows. Compression algorithms make further trade-offs: Snappy is fast with a modest ratio (Google-developed, common with Parquet), Gzip has higher ratio but is slower with universal compatibility, and Zstd achieves both high ratio and fast speed (Facebook-developed), making it increasingly preferred for analytics. A row group in Parquet (or stripe in ORC) is a horizontal batch of rows typically 64–256 MB in size, and min/max statistics stored per row group enable predicate pushdown to skip entire groups; larger row groups improve compression but worsen read amplification, while smaller ones give finer-grained skipping at the cost of more metadata.
Query performance also depends on intelligent data organization. Data partitioning divides a large dataset into smaller segments based on a key such as date, region, or ID range, enabling partition pruning—only scanning relevant partitions. Common strategies include range, hash, and list partitioning, and pruning works best when queries always include the partition key. Predicate pushdown pushes filter conditions down to the storage layer so irrelevant rows are excluded before being read into memory. Zone maps (min/max indexes) extend this idea to ORC stripes and other columnar formats. Bloom filters provide a probabilistic test for set membership and are used in HBase, RocksDB, and Parquet to skip whole row groups or data files that cannot contain a value. Data sharding horizontally splits data across independent servers, while partitioning within a single system is internal to one engine. The small files problem occurs when a distributed file system holds millions of tiny files, each consuming metadata slots (in HDFS, the NameNode RAM) and causing excessive seek overhead; solutions include compacting into larger files, using Parquet or sequence files, and using columnar table formats like Iceberg or Delta that pack many records per file. Hadoop HDFS stores large files across commodity cluster nodes with block-level replication managed by a NameNode, while cloud object stores like S3 provide virtually unlimited object storage over HTTP APIs with eleven nines of durability and no cluster to manage—now preferred for data lakes due to scalability, cost, and decoupling of storage from compute.
Modern warehouses also rely on physical design choices that affect performance. In shared-nothing architectures (Snowflake, Redshift, BigQuery), each node has its own CPU, memory, and disk; queries parallelize across nodes that exchange data over the network. In shared-disk architectures (traditional Oracle RAC), all nodes access the same storage and the bottleneck is usually the storage network. Separation of storage and compute decouples where data is persisted (object storage) from where queries are processed (ephemeral compute clusters), letting you scale each independently and run multiple engines against the same data. Snowflake uses micro-partitions—immutable 50–500 MB compressed columnar units with min/max metadata—to enable pruning without requiring user-defined indexes, and a virtual warehouse can be resized or scaled out on the fly. Redshift uses a sort key to determine physical row order (with zone-map pruning) and a distribution key to control how rows spread across nodes: KEY co-locates joins, ALL copies small tables to every node, and EVEN round-robins. A workload manager (WLM) divides a warehouse's compute into queues with different priorities and concurrency levels, protecting critical SLAs from noisy neighbors. A star join optimization recognizes a star schema and broadcasts small dimension tables to all nodes while distributing the large fact table, avoiding shuffling the fact table. Tables can be internal (managed), where the engine controls storage and dropping the table deletes the data, or external, where the data lives outside the engine's control and dropping only removes the registration—standard for data lakes where data should outlive any single engine.