eBay taps computer vision to transform online shopping

When eBay rolled out image-based search for shoppers nearly four years ago, the company was among the first to deploy computer vision technology in ecommerce. eBay has learned many lessons since then, and it’s now working on bringing that innovation to sellers on its platform.

In a conversation at VentureBeat’s Transform 2021 virtual conference, eBay’s new chief AI officer, Nitzan Mekel-Bobrov, shared some insights with Maribel Lopez, founder and principal analyst at Lopez Research. Mekel-Bobrov discussed the challenges of developing image-based search technology, the essential elements of any AI strategy, and what’s next in computer vision.

The challenges of image-based search

Offering buyers the ability to look for similar or exact items using images instead of text has become fairly standard in the industry, but launching the feature in 2017 was a big deal, Mekel-Bobrov said.

Data presented some of the hardest obstacles, especially with the range of items eBay is known for.

“The adage that data is king is true, probably nowhere more than in training CD models for item-matching,” Mekel-Bobrov said. “And for us, this was compounded at the time [by] the fact that we don’t control all the ways in which our sellers photograph their items.”

Hardware also posed issues in the first years, Mekel-Bobrov said, and the company had to find creative solutions.

“First, we did our training separately,” he explained. “We trained the models on the public cloud, where we could access [graphic processing units]. Then we did the inference and deployment locally. We also set up a series of what were essentially stateless systems, holding the entire index in memory.”

eBay was also one of the early users of product quantization. That reduced the dimensionality and size of the problem during search.

Organizational challenges remain eBay’s biggest undertaking. Launching a product that requires many different teams with “tremendous” upstream and downstream dependencies presents myriad issues, Mekel-Bobrov said. “A lot of the workflows needed reworking to allow for integration of computer vision capabilities.”

Additionally, ecommerce category taxonomy changes based on the market and inventory. “As the taxonomy innovation progresses, we have to keep up with it on the image search site and continuously adapt and evolve not just our models but our entire approach sometimes,” Mekel-Bobrov said.

The essential elements of AI strategy

Having only been with the company for two months, Mekel-Bobrov said he’s been spending most of his days thinking about eBay’s AI strategy. His plan focuses on three broad categories: taking a customer-first approach, democratizing AI among teams, and nailing down the data infrastructure.

The question should never be what new technology can be adopted, but what customer problems can be solved, Mekel-Bobrov said. “For us, as a two-sided marketplace … the mediation of that relationship — there’s a ton of opportunity and complexity there that AI can really transform.”

The rapidly changing nature of AI also calls for companywide upskilling and training. “We want to put these technologies into the hands of developers much more broadly than AI specialists, even product managers and analysts,” Mekel-Bobrov said. “The question is, how do we organize ourselves such that we can have a smaller cohort of AI specialists build differentiated core capabilities that can then enable teams across the company to leverage those and build unique products in their own domains?”

Finally, a key facet of any AI strategy is data. Mekel-Bobrov said a company must ask where the data is while offline and streaming, as well as how it’s collected, labeled, managed, and searched. “The more we’re doing at scale, the more we’re doing with real time, the more we’re doing with dynamic learning and reinforcement learning, the more we have to push the boundaries of these infrastructure considerations,” Mekel-Bobrov added.

What’s next?

The ecommerce giant is now “turbocharging” its platform to make computer vision an integral part of the experience for both buyers and sellers, Mekel-Bobrov said.

eBay recently launched an image-scanning tool on its app, starting with trading cards that sellers can scan with their phone to auto-populate and create listings.

“If you think about an individual seller or small business that needs to continuously create new listings at a high volume but wants to maintain the highest quality of information, selling tools based on computer vision can be a real game-changer,” Mekel-Bobrov said. The ultimate goal is to maintain trust between buyers and sellers.

During the COVID-19 pandemic, eBay has used computer vision detection and classification models on items like hand sanitizers and masks to identify listings with inflated prices and products that make false health claims.

Since November 2020, eBay has removed 50 million listings that violated its COVID-19 policies, Mekel-Bobrov said.

“We’re continuously making every effort to ensure that anyone who sells on our platform follows local laws, as well as eBay’s policies,” Mekel-Bobrov said. “Doing that at a scale requires a lot of creative thinking and advanced technologies like computer vision.”