Translations at Scale: Memory Optimization Techniques That Kept Uber's P99 Under 1ms
Sr Staff Engineer
Introduction
At Uber, we serve millions of translations per second. To save costs and reduce latency, we serve all these translations from memory. Maintaining resource efficiency is a sustained priority for us. To that end, we’re focusing on GC (Garbage Collection), which continues to be a top consumer of compute for our Go services. In the past, we’ve used things like GoGCTuner to reduce the cost of garbage collection. However, while the tuner only helps by using all the memory available for the application, it does nothing when free memory is already low.
After a series of memory profile analyses for our fleet, we found that our translation library was one of the top contributors in terms of size and number of objects. This blog presents two major optimizations that we’ve done over the years to improve how the library interacts with garbage collection.
First Problem: High Number of Objects Created
The original problem we faced was that the translation service had a very high volume of garbage collection. Over 20% of the service’s CPU was consumed by GC. Most of the created objects came from translations.
Figure 1. Memory profile showing that 70% of the objects came from translations.
The following explanation focuses on Go versions before Go 1.24 (there was a major change to sync.Map). Before we made any optimizations, the translation store used sync.Map. This implementation provided the best performance for our use case because:
- It required very few updates: most services don’t use the live updates feature, and even for the ones using it, the rate of updates is extremely low
- It involved no deletions: the map gets re-created every couple of hours
- It offered a high number of reads
Figure 2 shows the sync.Map schema.
Figure 2. sync.Map schema.
Let’s walk through an example of inserting an item (a simple key and value pair of strings). Imagine that you want to insert a key (“hello-es”) and a value (“hola”). The dirty and readOnly are pretty much the same, so for now let’s focus on the readOnly struct.
To store a string as an interface, the Go runtime creates a pointer to the string because interfaces only store pointers (first object created).
For the value, there’s *entry, which is a pointer (second object created). Within the entry there’s an atomic.Pointer (third object created). To store the string value in an interface, we need another pointer (fourth object created).
Overall, to store a single translation we require 4 objects to be created. For the example above, the translation service has over 2 million translations (a translation is a <key>-<language> combination).
To address this problem, we moved to a simple map implementation using values (map[string]string) with a RwMutex to support live updates.
Figure 3. Simple schema to reduce pointers.
Figure 4. CPU usage comparing week after and before the deployment.
Second Problem: High Memory Consumption
Figure 4. CPU usage comparing week after and before the deployment.
With the optimization presented above, we reduced CPU usage significantly for many services, but it didn’t address other problems:
- High memory consumption: Translations no longer created many objects (references needed to be tracked by the garbage collector), but they still consumed a significant amount of memory since we reduced pointers, not the actual data being stored. This caused a higher frequency of garbage collection, and depending on whether the service has many non-translation objects, GC compute time could still be quite high.
- Out-of-memory issues: As mentioned above, refreshes happen every couple of hours, and the way they work is by creating a background cache and then doing an atomic swap with the current cache. This created a double-consumption problem for up to one minute, which in some services translated to running out of memory.
One important thing we realized was that the percentage of translations used was pretty low. We explored different options to improve the situation. An external service had higher costs due to overhead from network calls, higher latency, and higher chances of failures. We also explored generating smaller subsets for each service so that they only put the translations they need. However, this approach would have a higher maintenance cost, like having to update a key in many repositories. We landed on a hybrid solution, where a small portion was in memory, everything else on disk.
Hybrid Solution
Figure 5. High-level overview of the disk storage solution.
When a service container starts, it first downloads the translations from an external blob storage. Once downloaded, we decompress it and we read the JSON using streaming APIs to avoid loading a big JSON all into memory. Then for each new translation, we call the Put() method to ensure the key-value pair gets added. Once we process all the translations in the JSON, we do the atomic swap so we can start serving translation requests.
The only part we changed was the cache storage mechanism, and everything else remained the same. The most important thing we need to keep an eye on is that latency doesn’t go up significantly.
The requirements for our storage are:
- All keys must remain in memory for quick look ups
- Sharding to reduce lock contention
- LRU for in-memory eviction
All Keys in Memory
Most keys are small, but they’d still be consuming (25-30%) of the total memory. To reduce memory usage further, we decided to have integer keys, which are basically the hash codes of such keys.
Our biggest translations repositories have around 2-3 million keys, but even considering a 10 million repository, our collision rate is:
Bytes | Probability of collision | Expected collisions |
8 | 0.00027% | ~0 |
7 | 0.069% | ~0 |
6 | 16% | ~0.177 |
5 | 100% | ~45 |
We concluded that we can use 6 bytes, where the first byte would be used for sharding (256 shards) and the remaining 5 bytes for the shard map. Using 6 bytes keeps the number of collisions pretty low and memory becomes around 5% of the original size.
To deal with collisions, we keep a fallback mechanism that uses the key in its original format. So only a couple of keys hit that path, and if we ever need to expand the number of bytes, it’s just a few lines of change.
Sharding to Reduce Lock Contention
As mentioned above, 1 byte of the hashcode would be used to do sharding. This translates to 256 shards, where each shard has its own RWMutex, in-memory entries, and disk file.
LRU for In-Memory Eviction
We decided to implement a simple LRU eviction policy, but we had 2 scenarios:
- Global LRU: then what’s the point of having shards?
- Per-shard LRU: a bit inaccurate because you could be evicting from a hot shard
Our final solution was to use a global LRU, but time-based. Instead of updating the LRU on every access, we store the last time an entry was accessed and if it exceeds the threshold (in our case, we’re doing it at least once a second) update the Global LRU. So far, this solution has proven to work well for the amount of load we get on a single instance.
One challenge we experienced while doing this was high CPU usage coming from getting the current time. On our benchmarks, it showed getting the time was adding 3-5% extra CPU, which is prohibitively expensive for this library because the current usage by most services is less than 2%. Therefore, we needed to come up with a workaround to unblock us. We ended up adding a background goroutine that just keeps an atomic variable fresh with the last reported time.
This enabled an additional optimization: we used the LRU access time to know what entries were recently accessed. When we run a refresh cycle, we use that information to only pre-load the entries that have been used in the last 15 minutes. We did this because our hit ratio was going down (and disk latency was going up) every time we hit the refresh cycle.
Disk storage
Now that we’ve presented the main requirements of our solution, let’s discuss how we’re going to handle writing and reading from the disk. We’ve already presented that our keys are composed of 6 bytes, but what are we going to store for the values? We decided to store 5 bytes:
- 2 bytes: index for in-memory entries, which allows 65,000 items per shard
- 3 bytes: disk offset where the entry is located
Our final implementation translates to 11 bytes per entry. So a 2 million repository is consuming around 30MB (consider the map overhead).
Let’s discuss disk offset. We’re going to use 3 bytes, which means we can index up to 16MB per shard (4GiB total). Our biggest repositories are using around 1GB, so 3 bytes is more than enough, but we need to consider that repositories will keep growing and that shards can be unbalanced.
Therefore, to improve how much we can index, we’re going to use CompressedOffsets, which translates to enforcing that any offset is divisible by 8 (if not we add sufficient padding). Then the 3 least significant bits are going to be empty, so we can shift any offset by 3 bits, which allows references up to 16MB<<3 which is 128MB per shard (32 GB total).
To push for the highest performance and lowest disk usage, we’re using manual serialization since we only store a couple of fields on disk per entry.
Example
For the example presented above (memory profile) the compressed JSON is around 150MB. Once uncompressed and loaded into memory, we saw memory consumption going up to 800MB.
After moving to the hybrid solution, we saw the initial size sitting at around 40MB (metadata only, an over 90% improvement). We configured max memory items to be 50,000 which usually puts memory usage in the 80-120MB.
When reloads happen, we only need an additional 40MB for the metadata instead of having an additional copy of 800MB.
Metrics
Finally, let’s review how our solution is performing.
Figure 6. 18,000 disk items per shard.
Figure 7. Total items in memory per instance for the example above.
Figure 8. Global hit rate for the example above.
Figure 9. Total global hits per minute for the example above.
Figure 10. Total global misses per minute for the example above.
Figure 11. Latency for misses for the example above.
Figure 12. Memory went from ~2.7GB to ~750MB.
Next Steps
We’re looking into other cache solutions that consume a lot of memory and may be impacting garbage collection. We’d like to see if there’s enough interest to build a generic solution that could work for more scenarios.
Conclusion
In-memory cache solutions are a great way to reduce service latency. Due to our scale, having unoptimized in-memory cache solutions across thousands of instances gets expensive.
To optimize costs, it’s important to:
- Create the smallest possible amount of references (less work for GC) which was achieved by moving from sync. Map approach to a simple value based map.
- Keep an eye on the total memory consumed and how often those objects are truly used. If your actual usage of them is a small percentage of the total usage then there is a big opportunity to do a hybrid solution.
Acknowledgments
Special thanks to the Internationalization team, who helped us with the reviews and approvals to roll out changes to all of Uber.
Cover Photo Attribution: Generated with ChatGPT by OpenAI; no external images, logos, or third-party assets used.
Go is a trademark of Google LLC.
Cristian Velazquez
Sr Staff Engineer
Cristian Velazquez is a Sr Staff Software Engineer on the Production Engineering team at Uber. He mainly works on Go and Java and is currently working on multiple efficiency initiatives across organizations.