Building the #UberBot on CodeFights for Uber Engineering

Math, mapping, modeling, and movement: Uber Engineering works on a wide range of challenging problems. Now, we’re challenging you: if you think you could out-solve us, here’s the chance to prove it. We’ve partnered with CodeFights to bring you the #UberBot.

Players on Codefights participate in timed, competitive coding challenges against human players and automated bots. When you codefight against the UberBot, you’ll solve some of the core challenges we face in growing our business, such as finding the optimal route for a car, or how to best match riders for uberPOOL.

How did CodeFights develop the bot with Uber in mind? We had 3 main constraints:

Make the tasks relevant to Uber. Many coding challenges can be abstracted to the point where you don’t see the end-impact and business application in solving the task. We wanted to include enough context to show the bigger picture of the problem space that each task represents.

Make well-defined tasks solvable in 3-10 minutes. Some of our problems take a while to work on; breaking apart a large task into smaller, self-contained modules is a good approach when tackling large problems, and gives the idea of the bigger picture without all of the upfront work.

Have a mix of easy, medium and difficult tasks. A range of difficulties is what we realistically work on, and gives us more resolution to measure performance.

Each CodeFight against the UberBot consists of 3-5 tasks with a mix of code writing and code review:

For simplicity’s sake, scoring is based purely on correctness and speed. The faster you answer, the more points you accrue.

One of the big concerns CodeFights had when creating the content was to make the tasks difficult enough to be challenging, yet simple enough to keep it fun. “As an experiment, we added the bot to the CodeFights interview process in early October, where the first part of the in-person interview was to fight the UberBot. The interviewee crushed most of the questions very quickly, so we started thinking the we might have erred on the easy side…”, CodeFights CTO Aram Shatakhtsyan told us, “but then we realized that our interviewee was an IOI gold medalist. So that was a very biased sample!”

Once the bot’s content was created, it was time to calibrate the bot’s performance to Uber engineers. UberBot, like other bots on CodeFights, has two core components: Training and Playing.

In the lead up to launch, software engineers from across Uber solved tasks by fighting against the UberBot. Components of each attempted solution were systematically measured and monitored. The UberBot tracked how often Uber engineers answered correctly, and if successful, the time it took to complete the task.  (`totalCount, rightCount, wrongCount` and `avgAnsTime`). As right answers accumulate, we recalculate the benchmark:

`newAvgAnsTime  = (avgAnsTime * rightCount + ansTime) / (rightCount + 1)`

where `ansTime` is the time it took the trainer to give the correct answer. We repeated this dozens of times to accumulate a large sample and to produce the UberBot’s playing strategy, further described below.

In order to make this simulation realistic for how an Uber engineer would perform, when faced with an opponent the UberBot simulation replicates a range of answers it encountered during training. For every task, the UberBot probabilistically determines its score using the following:

`Pright = rightCount / totalCount`

`Pwrong = wrongCount / totalCount`

`Pskip = 1 – Pright – Pwrong`

`Pright, Pwrong, Pskip` represent the probability that the UberBot will answer the given task correctly, incorrectly or not at all. If the UberBot decides to answer correctly, it applies a random deviation from the average Uber engineer answer time to arrive at its final performance.

Now that you have some background on how the UberBot was built, are you ready for a CodeFight? Challenge the #UberBot here: https://codefights.com/bots/bot_uber. If you can beat ‘em, join ‘em: surpass the bot and we’ll invite you to learn more about what it’s like to work in Uber Engineering. Good luck!