The growing interest in artificial intelligence has created workloads that
require both sequential and random access. At the same time, NVMe-backed
storage solutions have emerged, providing caching capability for large columnar
datasets in cloud storage. Current columnar storage libraries fall short of
effectively utilizing an NVMe device’s capabilities, especially when it comes
to random access. Historically, this has been assumed an implicit weakness in
columnar storage formats, but this has not been sufficiently explored. In this
paper, we examine the effectiveness of popular columnar formats such as Apache
Arrow, Apache Parquet, and Lance in both random access and full scan tasks
against NVMe storage.
We argue that effective encoding of a column’s structure, such as the
repetition and validity information, is the key to unlocking the disk’s
performance. We show that Parquet, when configured correctly, can achieve over
60x better random access performance than default settings. We also show that
this high random access performance requires making minor trade-offs in scan
performance and RAM utilization. We then describe the Lance structural encoding
scheme, which alternates between two different structural encodings based on
data width, and achieves better random access performance without making
trade-offs in scan performance or RAM utilization.

Questo articolo esplora i giri e le loro implicazioni.

Scarica PDF: