Machine Learning Powering the Future of Retail

Forward-thinking brands and retailers are looking to image recognition and machine learning to analyze enormous data sets (online catalogs) with vast assortments of visual features (fit/stitch/etc.) down to a single product. The results are powering a new level of personalization for better customer experience.

The problem with current online shopping is that experiences are more performance-driven than service-oriented. While brands and retailers are most likely saving user interaction and clickstream data, that data doesn’t capture the complexity of details that influence customer purchasing decisions...like body perception for example.

Lily AI is one company powering a new retail revolution by using image recognition and machine learning to help brands and retailers understand the “why” behind what their customers buy. The company’s technology stack (which is proprietary for “hyper-personalization”) was built and implemented by Sowmiya Chocka Narayanan, Co-founder, and CTO at Lily AI.

Narayanan holds a Masters in Electrical and Computer Engineering from UT Austin and a Bachelor's in IT from PSG College of Technology (India). She worked in various areas of the tech stack for big players like Yahoo! and Box, then became passionate about the intersection of emotional intelligence and artificial intelligence.

Lily AI’s co-founder and CEO Purva Gupta came from the ad agency Saatchi & Saatchi. Early on in life, Gupta had learned how finding just the right item of clothing could help her overcome self-doubt and when the two female founders’ paths crossed, Gupta had just completed a mountain of in-person research looking into a business built on the premise in New York. Gupta’s research showed that women look for clothes by body type, to find what they feel most comfortable in.

One woman might decide on a specific blouse because the cut hides a rounder stomach for instance. The two set to work on creating a business that would guide brands and retailers about these preferences in the digital realm. Narayanan convinced Gupta the best way to build the kind of preference-driven personalized shopping service she wanted was to use machine learning

Lily AI is now using computer vision and artificial intelligence to identify the most granular attributes of every SKU in a retailer's portfolio. By starting with these dozens of attributes per item, brands and retailers can then hyper-personalize their customers’ experience online by tapping into their customers’ affinity for these specific, very granular product attributes.

Deep tags play a critical role in improving site navigation by improving filters and facets to narrow results, improve the accuracy of site search, and more. To achieve that, Narayanan stacked an ensemble of deep learning models created from convolutional neural networks with different architectures and trained with nearly 1 billion data points manually curated by human experts.

The first custom models were created using a 3rd party deep learning-as-a-service platform and over 100,000 labeled images. The co-founders quickly realized that if they wanted deeper granular and style-driven attributes, they needed to experiment and fine-tune the models themselves. Gupta and Narayanan agreed it was time to toss out the 3rd-party approach and build AI models in-house.

To give an idea of scale, Lily AI has now created tens of millions of tags for just one of the company’s retailer customers. They continue to delight their clients by identifying the shoppable items in an image, and then predicting meta-tags from a database of thousands of attributes that range from color and cut to the most minute embellishments like seams and material weight.

The human feedback loop which has been integrated into the training pipeline enables the team to add multiple millions of data points per day to increase the accuracy of these finer details, and models are hierarchically organized such that each level of prediction adds finer tags.

By feeding the product attributes and clickstream data of users to Lily AI’s recommendation algorithms, the team is extracting the affinity and sensitivity of the user to different product elements and features and apply them, and can then be recommended.

In this process, they are also evaluating and incorporating every other style and product feature that would be flattering for the user (or not) to prevent the wrong recommendation.

In this way, two women who have the same body type and measurements, who may both be looking at a pair of jeans, but have different preferences for curves and visible stitching, will surface different results.

When asked about off-the-shelf options from Amazon and Microsoft to try and recreate the results, Narayanan has always found it best to build in-house.

“The best use case for us was on the application infrastructure side - spinning off a streaming pipeline, ETL on gigabytes of data, serverless APIs, etc. We could focus on the core algorithm part without spending too much time and resources building the rest of the infrastructure required to support the retail applications,” states Narayanan. “Because of the nature of our industry being retail, we have decided to be cloud-agnostic.”

To differentiate, the Lily AI platform provides an end-to-end solution that can be applied to a broad set of applications over time. The team has customized the deep learning models (layers and filters) to enable them to go deep to learn and predict more than thousands of granular product attributes with a high expectation of minimum precision in identifying the attributes. Similarly, they feed the recommendation algorithms with custom user attributes extracted from their clickstream and feature affinities, then experiment by assembling different approaches. These are impossible to do with off-the-shelf platforms.

According to Narayanan, traditional machine learning laid the foundation for learning logical rules from input data without being explicitly programmed for it, and deep learning delivers the horsepower to extract features from massive unstructured data sets and learn without human intervention.

Inspired by the biological structure of the human brain, deep learning uses neural networks to analyze patterns, and find correlations in unstructured data such as images, audio, video, and text.

“The predictive power of deep learning in visual perception, natural language understanding and the ability to predict purchase intent makes it possible for Lily AI to deliver highly personalized recommendations, optimize pricing strategy and inventory planning, AI assistants among other applications,” said Narayanan.

In order to provide the most relevant experience, a retailer needs to capture and store every single interface with the user - what the user did in the online store, how they bought the item, if they returned the product in-store, did they connect with customer care, what were their main concerns, etc. They need to be able to identify the offline store interactions of an online user, online user logging in from another device, an online user shopping for someone else.

This step is most critical to ensure that any machine learning is performed on the correct dataset; several CDPs (customer data platforms) promise to do this. On the other side, they also need to have granular data about the products to not just serve in recommendations but also to analyze trends and help forecast demand. It is highly important for the retailer to make the right choice of tools and platforms that will help them capture and interpret the humongous data generated by the consumers.

Shoppers' expectations of relevant and engaging online experiences across channels will continue to drive retailers to embrace and implement AI-driven solutions for customer service, smarter search, digital navigation, recommendations, virtual assistants etc.

The adoption of AI-powered services like Lily AI will not only improve customer touchpoints, but can also positively impact other areas such as inventory management, sales forecasting, out-of-stock issues, and better-optimized marketing plans. This helps retailers not only increase their sales and operational efficiency but in a way that customers appreciate and reward with loyalty.