Introduction

At Uber, several offline export workloads retrieve small record sets from large historical datasets. Today, these selective queries can still trigger full-table scans. On a Google Cloud Storage™-backed lakehouse like that at Uber, repeated reads keep large table regions hot, increasing storage, retrieval, metadata-operation, GCS egress, and latency costs. This blog shows how Apache Hudi™ column stats in conjunction with sorting/clustering can reduce scan volume and make these workloads more cost-efficient in that lakehouse environment.

What Are Export Workloads?

Export workloads are common in compliance, privacy, and operational export flows, where the result set is small but the search space spans the entire history of data.

Export workloads typically have the following characteristics:

• They issue point queries or narrow predicate queries over large transaction or fact tables stored in Hudi and partitioned by date
• They run repeatedly, often multiple times per week.
• They repeat the same query pattern, but with different parameters each time.
• The target record set is only a small fraction of the overall dataset
• Despite their selectivity, they perform full-table scans to find the relevant records

One concrete example of this workload pattern is a DSAR request. A DSAR (Data Subject Access Request) is a legal request by an individual to access the personal data an organization holds about them. In practice, fulfilling a DSAR often translates into a narrow query pattern like the one shown in Figure 1.

Figure 1: Narrow query pattern.

GCS Auto-Class Tiering

Uber uses GCS (Google Cloud Storage) as the storage layer for its lakehouse. GCS offers multiple storage classes that trade off storage cost, retrieval cost, and access latency. Understanding these tiers is important because the cost efficiency of export workloads depends not only on how much data is stored, but also on how often that data is touched. 

Figure 2: Storage tier distinctions in GCS.

Export Workloads Break Auto-Class Tiering

Export workloads disrupt auto-class tiering because they combine high selectivity with a large historical search space. A query may return only a few records while still touching all the partitions and files.

Analytics workloads often process more data but over recent partitions, letting older data go cold. Export workloads do the opposite: they repeatedly search across history for a small target set. This repeated needle in a haystack pattern where the haystack is the full table will keep data hot, prevent auto-class transitions, and increase storage and retrieval costs.

Consider the example of the trip_table again. 

Figure 3 shows what a typical analytics query might look like.

Figure 3: A typical analytics query.

In contrast, Figure 4 shows what an export query looks like.

Figure 4: A typical export query.

The result is a much larger storage access footprint than ‌output size alone would suggest. Repeated execution of this pattern increases storage cost, retrieval cost, and metadata-heavy Class B operations such as objects.get and objects.list. It can also degrade query performance by forcing repeated scans over files that’d otherwise remain cold.

A visualization of the scan surface area in a Hudi table makes this contrast clear.

Figure 5: Diagram contrasting typical analytics versus an export workload’s file scanning patterns.

Secondary Index Doesn’t Help

To achieve our optimization and efficiency goals, we focused on minimizing the number of data files touched during query execution. We evaluated the Hudi Secondary Index as a potential solution. Secondary indexes in Hudi are designed to accelerate predicates on non-primary-key columns by helping query engines prune files before scanning data. Hudi supports these indexes through its metadata-table indexing framework, and secondary indexes can be created with SQL when the table has the required record-index setup.

For our export workload, the key question wasn’t whether a secondary index could prune files, but whether it’d prune enough files to justify its additional metadata and operational cost. On our target tables, records for the same subject were spread across most files. A secondary index helps locate candidate rows, but it can still map to a large file set. The engine therefore still pays a high file-scan and metadata cost.

What Works ?

To achieve the goal of reducing the number of file scans, we combine two ideas: 

• Hudi column stats to identify which files are relevant for a predicate without touching the underlying Parquet data files
• Table sorting on the predicate column so that matching records are clustered into a much smaller subset of files, making file pruning significantly more effective since the query remains constant and filters on a consistent predicate.

Figure 6: Architecture diagram.

Hudi Column Stats

Hudi maintains column-level statistics in its metadata table, including values such as min, max, null count, and total count for tracked columns. Because this metadata is stored separately from the underlying Parquet data files, the query engine can use it to prune irrelevant files without directly reading them.

For export workloads that repeatedly run selective predicates over large historical datasets, this avoids touching much of the table and reduces scan surface area.

Parquet stores useful column statistics, but they live in the file footer. In our environment, relying on those footers isn’t sufficient, because reading them still touches the data files. Over time, those repeated reads can keep files hot and prevent auto-class from moving them into colder, cheaper tiers.

Hudi addresses the footer reads problem by maintaining file-pruning metadata at the table level, rather than requiring the engine to inspect each Parquet file individually. Instead of opening many files to discover their statistics, the engine can query Hudi’s metadata table and determine which files are relevant before it begins scanning data.

This makes file pruning both faster and more storage-efficient for export workloads. It reduces the need to touch cold files, preserves the benefits of auto-class tiering, and lowers the operational cost of repeated historical lookups.

Table sorting

Export workload queries are usually driven by subject identifiers such as rider ID, driver ID, or courier ID. If a subject’s records are scattered across many files, even column stats may still leave the engine inspecting much of the table.

Sorting on the predicate column clusters similar keys together, tightens file-level min/max ranges, and makes pruning more selective. Column stats provide the pruning mechanism; sorting improves its effectiveness. Together, they turn broad file scans into narrow file selection, reducing reads, shuffle, and execution time for row-level queries.

Benchmark

We sorted a real table partition and  took 100, 200, 500, 1,000, and 5,000 predicates (user IDs), sorted the partition, compared the number of Parquet files where the target rows are spread across.

Figure 7: Target files percentage across different number of predicates in non-sorted versus sorted partitions.

Figure 8:  Benchmark results for files pruned, egress, compute reduction and storage reduction.

Why Not Sorting Paired With Secondary Index?

Sorting makes pruning much more effective. And at this point Hudi column stats already provide effective file pruning with a smaller metadata footprint than a secondary index. Column stats scale primarily with file count and tracked columns, while a secondary index must maintain lookup mappings for indexed values. 

In our workload, sorting provides the physical clustering, and column stats index provides a lightweight pruning mechanism vs a much heavier secondary index.

Comparison Matrix

Figure 8: Target files in a sorted table.

Applicability in Other Industry Use Cases

The underlying export access pattern appears in many domains where systems need to retrieve a small set of records from very large historical datasets, often under strict compliance, operational, or investigative requirements. The common pattern is a narrow, subject-oriented lookup over a long retention window.

Examples include:

• Finance: Regulatory, audit, and dispute workflows often require multi-year histories for specific accounts or entities
• Healthcare and life sciences: Compliance reviews, patient-history access, and audit workflows often require selective lookups over long-retained records
• E-commerce and retail: Fraud investigations, customer support workflows, and order-history exports frequently retrieve records for a small set of users across long time ranges
• Ad tech and measurement: Timeline reconstruction for attribution, personalization, or compliance often requires subject-level lookups across historical event data
• Security and identity systems: Investigations, access reviews, and compliance workflows often require full login or activity histories for specific users or principals

Next Steps

We’re focused on expanding these optimizations across Uber’s broader infrastructure.

Our first objective is tackling multi-column lookup scenarios. For datasets where various clients use different predicate columns, single-column sorting often fails to provide uniform benefits. To overcome the limitations of standard lexicographic sorting as column counts increase, we’re exploring Z-order space-filling curves for more effective data layout optimization.

Additionally, we plan to develop an automated framework to detect these kinds of needle-in-a-haystack access patterns. This system will weigh the costs of indexing and sorting against potential query savings, triggering data layout optimization pipelines only when the ROI is justified.

Conclusion

Here are some of the lessons we learned from this effort: 

• Access pattern drives storage cost. Narrow, history-spanning lookups behave very differently from typical analytics queries, and naive table layouts can create disproportionate storage and egress costs.
• File layout matters as much as tiering. GCS auto-class only delivers savings when files are left untouched long enough to go cold. Sorting plus Hudi metadata helps minimize unnecessary file touches.
• Metadata-driven pruning is critical. Relying on Parquet footers works against tiering because footer reads still touch the files. Hudi column stats enable pruning without repeatedly accessing data files.
• A one-time rewrite can unlock long-term savings. Sorting and rewriting tables introduces upfront cost, but it reduces scan surface area and enables sustained storage savings over time.
• The best results come from an end-to-end design. Storage layout, metadata, query execution, and orchestration all need to work together; otherwise, cost is simply shifted from one layer to another.
• Hudi Table Service as the primary enabler. The Hudi Table Service plays a pivotal role in enabling these access-pattern-driven optimizations, serving as the central engine required to manage and execute all backfill sorting, index construction, and incremental sorting operations.

Overall, optimizing export workloads at Uber shows that cloud storage costs depend heavily on access patterns and file layout. By combining Hudi column stats with predicate-column sorting, we transformed broad historical scans into selective, low-touch lookups that sustain storage and egress savings.

Acknowledgments 

Meenal Binwade Vamshi Pasunuru Jiashen Zhang Uber's Hudi team has been instrumental in productionizing the column stats at Uber.

Cover Photo Attribution: Generated using Gemini

Apache®, Apache Hudi™, and the star logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

Google Cloud Storage™ is a trademark of Google LLC and this blog post is not endorsed by or affiliated with Google in any way.