Freight Search: Composing a Shared Search Index from Independently Owned Fragments

Introduction

Searching is the process of finding relevant information from a large collection of data based on what a user is looking for. At its core, a search system takes a user’s input—a keyword, phrase, or set of criteria—and returns the items that best match their intent, often ranked by how relevant each result is. 

This article presents a generic way to design such a platform. Its core is a declarative fragment composition framework that lets dozens of teams contribute independently to a shared search index. Over several years of business expansion and infrastructure change, the design has held up as:

• Flexible. New business use cases, and later a full search engine migration, were absorbed without redesigning the platform.
• Sharable. Reusable building blocks, including an annotation DSL and connector abstractions, are shared across business objects and across both the batch and streaming pipelines.
• Composable. A clear boundary between platform space and user space means platform components can be swapped independently, without touching business code.

Problem Statement

Freight is unusual within Uber in how heavily it relies on microservices: the organization runs over 40 of them, and search has to pull from almost a hundred data sources and tables rather than the handful most Uber use cases deal with. What makes it harder is that these services are heterogeneous, spanning transactional stores, tracking systems, and external APIs, so data arrives in different shapes and under different SLAs. And the questions users ask, such as "all loads for shipper X picking up in Chicago in the next 24 hours," routinely span several services at once.

With that in mind, we designed the Freight Search platform to be:

• A centralized search infrastructure for Uber Freight. All Uber Freight microservices use this search engine, rather than building their own.
• Able to search heterogeneous sources. Our organization has over 40 microservices, and each may have their own data source. For example, we want to be able to filter by fields in microservices A and B backed by MySQL® , then sort by fields in microservice C backed by Apache Hive™. 
• Cheap to extend. It should be simple for new fields or data sources to be ingested, without requiring complex code.
• Fresh and fast.  Loads and related data change constantly, and those changes should be searchable within seconds, with query latency low enough for interactive use.
• Transparent data lineage. With all sorts of data sources, anyone should be able to trace any index field back to its system of record and understand why it exists..
• Open to ad hoc queries. Callers compose their own queries against the index at request time instead of choosing from a fixed set.

Today, Freight Search ingests over 70 source topics that denormalize into 300 fragments spanning over 20 indices. We serve thousands of complex search queries per second across every sector of the Freight business—brokerage, transportation management systems, operations portals, and more—at hundreds of milliseconds p99 query latency and a data freshness of a few seconds. 

Architecture

Uber has a central Search Platform that provides the infrastructure for hosting OpenSearch® clusters. Freight Search is the business layer on top of it: it bridges our microservice landscape and that infrastructure, so that individual Freight services never talk to the search engine directly.

Data model: indices composed of fragments

To onboard data, a developer works with two core abstractions: the index and the fragment.

An index represents a searchable entity, and maps to an actual OpenSearch® index. For example, in the world of Freight, a load is an index, with possible fields like UUID, shipper, operating locations, external ID, cargo value, created at, and so on. A load represents a shipment from location A to B. We can run search queries like, “give me all loads under shipper X within 24h”, or “give me all loads starting at Chicago and ending at Dallas.”

One additional thing to note is that our index definition also defines how the field itself is ingested into OpenSearch. This includes things like analyzers to apply, sortability, data type, tokenizers, and so on. This allows us to construct the full OpenSearch mapping for that index, while keeping it in a central place.

A fragment is a slice of an index sourced from one data source. It declares where the data comes from, which fields to ingest, and how those fields map onto the index. In a microservice architecture no single service owns a whole load: the Load service owns load_uuid and created_at, the Facility service owns facility locations, and the Financial service owns cargo value. Each of these is its own fragment.

Freight Search is a framework that allows an index to be flexibly composed of multiple fragments, as shown in Figure 1. It merges them into a single denormalized document that the cluster serves with no joins at query time, which is what lets a single query span fields owned by several services.  Field ownership is explicit and conflict-free: each field in an index entity comes from exactly one fragment. This also allows us to perform partial index updates easily; the fields changed in a fragment (for example, from a CDC Apache Kafka® event) can be directly sent to OpenSearch, without needing to reconstruct the whole document.

Figure 1: Freight Search data model.

Freight Search platform Offering

Given how heterogeneous our services are, the platform supplies everything around the business definition:

• Common data connectors for different data sources. A data source just needs to specify the right connector; all the plumbing work is already done for it.
• Interfaces and default implementations that plug into Uber’s search ingestion pipelines. The default implementations are designed to satisfy most ‌use cases, while still allowing customization if needed.
• An annotation DSL for defining an index and data source. Combined with the above, a developer only needs to implement code that’s business-specific, while all internals are hidden.
• A Data Validator. This continuously validates that the data in our index matches the source data.

Data pipeline:  bootstrap and live ingestion

Ingestion has two stages, shown in Figure 2. Bootstrap establishes the initial state of an index, and live ingestion keeps it fresh from then on.

Figure 2: Freight Search data ingestion pipeline.

Once data is loaded from the source, it’s denormalized and merged, essentially combining all the different fragments for a given UUID into a single document.  Some fragments fan out: many loads share a facility, so a facility's location has to be copied into every load that references it. The resulting documents make up the index and are written to OpenSearch.

Live ingestion is an Apache Flink® streaming job that consumes CDC events, starting from an offset aligned with the bootstrap timestamp, and propagates each change to the affected documents within seconds. It follows the same steps as bootstrap: convert the source row into its fragment, denormalize where needed and write to OpenSearch. Because ingestion is per fragment, an update rewrites only the fields that fragment owns rather than rebuilding the whole document, which keeps updates fast and simple.

Both stages reduce to the same operation: turning source data into a fragment, and the framework performs that conversion by default. In most cases defining the fragment is the only work a data source requires.

Similarly, our Data Validator is built on the same principles. The fragment names the connector to use for data validation, and that connector fetches the source rows and compares them against what Freight Search holds, continuously.

Code Examples

Up to now, we’ve covered data models like index and fragment, and data pipelines like bootstrap and live ingestion. We also mentioned platform offerings of different tools like the denormalizer. In this section, we use some code examples to show how developers use these concepts via annotations, demonstrating the composability of the platform. Freight Search is implemented in Java, given that the ingestion pipelines we use (Spark and Flink) are JVM-based.

We will use Load throughout this section. A load is one of the core concepts in the freight world, standing for a shipment from location A to location B. It is heterogeneous from the fact that so many microservices will update some piece of information, meaning load will be good to demonstrate how index is built out of different fragments. 

Example 1: Index Schema as Code

In Freight Search, every searchable entity is just a Java class with an Index annotation. The code example in Figure 3 defines a load index with 3 fields: reference, weight, and a geolocation. The  AnalyzableField annotation describes how this field is indexed in OpenSearch, in other words,  what kind of search does this field support. For example, load.reference is treated as case-insensitive terms, meaning “abc” will match “AbC”. At the same time, termOptions doesn’t include PREFIX, meaning “ab” won’t match this reference. Also notice that the load.origin uses GeoField, meaning load.origin supports all geolocation search, such as all matched loads whose origin is within 10 miles of a city.

Figure 3: Load index example.

Example 2: Compose Index From Fragment 

The index is composed of fragments. A document like a “Load” isn’t owned by one team—its data is scattered across many source systems, each owned by a different domain and team. Rather than forcing a single team to own the whole document, Freight Search assembles each document from fragments. Every data source contributes one fragment. Teams add new data to an index by dropping in a new fragment class independently, without touching anyone else’s code or coordinating a schema migration. 

Figure 4: Load fragment example.

Fragments are basically just POJOs, and customized getters can also be defined. When annotated with MappingField also, these getters allow us to define custom data transformations.

Example 3: Creating a Live Ingestion Pipeline From a Single Config

Following the fragment, we come to the final missing piece: config for live ingestion, where the developer specifies the CDC stream as one of the source topics of Flink live ingestion. After adding this config, the worker automatically registers a Kafka consumer on this topic and wires it in as one of the data sources of our Flink job. As a platform, we own and have a centralized view of our Flink job, and can tune operator parallelism, cluster size, and checkpointing accordingly, while as a user, a full pipeline is set just by providing the minimal business-specific setup.

Figure 5: Flink config example.

Use Cases at Uber

In the following section, we use three examples to demonstrate the flexibility of this design at Uber and the benefits of a clear boundary between platform space and user space.

Declare Once, Derive Everything 

A carrier's DOT number (abc-123456) is an opaque identifier: upstream services search it by prefix (abc-1) or by a substring anywhere in the middle (bc-12). A carrier’s legal name (Swift Transport LLC) is plain text, and services expect prefix matching (swift t), case-insensitivity (SWIFT = swift), and phrase matching (wift trans). These are fundamentally different search contracts, and in Freight Search they are declared explicitly on the index field:

Figure 6: Search contracts defined on the index field.

The annotation is simultaneously three things: the schema declaration, the analyzer configuration, and the documentation. A developer reading code knows immediately what search behaviors it supports, without tracing through analyzer wiring elsewhere in the codebase. There’s no separate mapping file to keep in lockstep and no place for the declared intent and the actual behavior to drift apart—the Java class defining the index is the single source of truth.

Declare a Relationship, Get Denormalization For Free

A parent-child hierarchy is a very common pattern in search. For example, changes happening in a shipper impact all its ongoing loads. When a change happens in a parent entity, it must propagate to all of its child entities. Doing this correctly requires knowledge of the relationship between the entities, as well as how to fan-out the changes.

Freight Search hides all of that behind the @LinkedEntity annotation. The developer declares the relationship and implements an ID selector interface; then the platform takes care of the denormalization and propagation. The hard, error-prone part becomes invisible—defining a data relation can be easily declared via one annotation as shown in Figure 7.

Figure 7: Defining a linked shipper fragment. 

Figure 8: Denormalizing a shipper on a load fragment. 

Declare a Fold, Get 1:N Aggregation For Free

The previous example covered the many-to-one direction: one shipper change fans out to many load documents. The one-to-many direction shows up just as often—a load has many stops, a bid has many charge lines—and it flips the problem: instead of propagating one row into many documents, we need to fold many child rows into a single document.

Freight Search models this with an aggregated fragment, split into two classes. The child fragment (StopFragment) is a normal fragment describing one raw source row, with one addition: a @ParentId field declaring which parent it rolls up into.

Figure 9: Defining a stop fragment with the parent UUID.

Figure 10: Folding different stop fragments into an aggregation under the load index.

Swap the Engine Once, Not Once Per Entity

We're ‌ in the middle of migrating our search engine from SIA (a Lucene®-based platform) to OpenSearch, for native distributed shards, a more modernized architecture, vector search, and so on. Given that all domain business logic is self-contained in the fragments and index definitions, it allows us to make platform-wide changes and have it apply to all entities. 

For example, for the migration to OpenSearch, we updated our platform to emit all documents and changes to both SIA and OpenSearch. This change only needed to be applied in a few places (for example, bootstrap and live ingestion), but works for all entities automatically. Similarly, we could reuse the existing index definitions and annotations (which existed before OpenSearch), in order to generate the corresponding OpenSearch mapping.

Conclusion

While Freight Search is implemented in Java using annotations and runtime reflection, the underlying idea is language-agnostic: declare a searchable entity as a business object, and let the platform derive the index, the analysis, and the data pipelines from it. The same pattern maps naturally onto other ecosystems. In Python, decorators and typed field descriptors provide the metadata, and runtime introspection plays the role of reflection. In Rust, attributes (#[search(...)]) carry the metadata and derive macros generate the mapping code at compile time—which‌ strengthens the "code is the single source of truth" guarantee by enforcing it through the compiler. What stays constant across all of these is the principle that matters: a clear boundary between the user’s business definition and the search internals, so the platform can evolve independently of the entities built on top of it.

This boundary fell out of two questions we kept returning to: what’s business-specific and should stay with the service, and what’s common and can be abstracted away by the platform. Answering them consistently—across schema definition, analysis, data relations, and engine migration—is what produced a framework that’s extensible to new business use cases, sharable in its common implementations, and flexible enough to swap components independently. The benefit compounds over time: every capability the platform gains is one every entity inherits for free, and every business team’s work stays focused on what’s genuinely theirs.

Acknowledgments 

Cover Photo Attribution: Licensed by Uber. 

Apache®, Apache Spark™, Lucene, Flink®, Apache Kafka®, Apache Hive™, and the star logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

Java and MySQL are registered trademarks of Oracle® and/or its affiliates.

OpenSearch is a registered trademark of the LF Projects, LLC  (under Linux Foundation) in the United States and/or other countries.  No endorsement by LF Projects, LLC is implied by the use of these marks.