Lin Zhu | Sr. Staff Machine Learning Engineer; Manan Kalra | Machine Learning Engineer II; Logan Jeon | Sr. Machine Learning Engineer; Ye Liu | Staff Machine Learning Engineer; Xiangyi Chen | Sr. Machine Learning Engineer; Jaewon Yang | Principal Machine Learning Engineer; Jinwen Xu | Manager II, Machine Learning Engineering; Tingting Zhu | Sr. Manager, Engineering; Sudarshan Lamkhede | Director, Machine Learning EngineeringPinterest is built to get inspired and then turn the inspiration into realization — a dinner, a renovation, a wedding, a new skill. That only works if we understand more…
Machine learning models increasingly support decisions that carry real consequences. For example, they help fraud analysts identify suspicious transactions, assist doctors in assessing patient risk, and inform hiring decisions that can shape people’s careers.As these models become more complex, explainability has become a cornerstone of trustworthy AI. The idea is straightforward: if people understand why a model reached a particular prediction, they should be able to make better, more informed decisions.But there is a fundamental question that often goes unasked: How do we know whether an…
The short story Consider a Friday evening. A food order arrives from a mall in the city center. One driver is nearby; another is finishing a drop-off and will be available shortly; a second order from the same mall may or may not appear in the next two minutes. Dispatch the nearby driver now, or hold briefly for a batching opportunity? The decision window is short. A fulfillment marketplace makes these decisions continuously. Each one is small. Across a city, those decisions determine whether your dinner arrives hot and whether a driver’s hour is well spent. And that is one decision. There…
How Airbnb uses proximity signals to personalize without relying on individual user history.By: Wei Jiang, Bin Xu, Bharathi Thangamani, Weiwei Guo, Sundar Srinivasavaradhan, Tracy Yu, Huiji Gao, Swapnil Ghike, Michael KinotiGreat personalization starts with knowing your user. But what happens when the user is a stranger?A significant share of Airbnb users arrive without a login, without a recent search history, or without any prior booking — especially those landing from paid advertising or organic search. For these users, the ML models that power search ranking, destination recommendations,…
How two new Chronon capabilities, Push Mode and NRT Model Transform, allows us to provide more relevant search results instantly as a guest explores, rather than waiting for the next batch run.By: Pengyu Hou, Yuli Han, Daochen Zha, Haozhen Ding, Xin Liu, Sophie Wang, Pallavi Adusumilli, Sherry Li, Henry Saputra, Chun How Tan, Huiji Gao, Yan Zhang, Stephanie Moyerman, Yi Li, and Sanjeev KatariyaA guest’s interaction with Airbnb doesn’t pause to wait for a nightly batch job. Someone might browse a dozen listings on a Tuesday afternoon, run a new search that evening, and expect the next search…
Authors: Longyu Zhao (Staff Machine Learning Engineer), Gwendolyn Zhao (Staff Machine Learning Engineer), Peng Yan (Senior Machine Learning Engineer), Yuanlu Bai (Senior Machine Learning Engineer), Yuan Wang (Senior Machine Learning Engineer), Yao Cheng (Staff Machine Learning Engineer), Ang Xu (Principal Machine Learning Engineer), Zhaohong Han (Manager II, Ads Lightweight Ranking)IntroductionPreviously¹, we launched the next-generation serving stack for standard ads, which we call Nexus. Nexus decoupled candidate generation from scoring and moved us beyond the classic two-tower-only world,…
Reusing what we already had, treating embeddings as data, and lowering the next team’s costToday, an ML engineer at Thumbtack can stand up production vector search without negotiating database access, building a custom ETL, or writing a query service. The team brings their choice of embedding model, the data, and the query; the platform handles what connects them. It took several iterations to get to this point. In this post we’ll walk through how we got there and the three principles that shaped what we built.A vector database stores high-dimensional numeric arrays (embeddings) and serves…
Tilman Drerup, Moe Moazzami, Shih-Ting Lin, Greg Reda (and many more)IntroductionAt Instacart, artificial intelligence is fundamentally changing the way our machine learning engineers operate. In a prior blog post, we used one of our teams as a case study to illustrate how the emergence of agents has reshaped what machine learning engineers spend their time on. The post below goes a few levels deeper and zooms in on the machine learning modeling process itself, an area where recent developments in AI-assisted research have opened up exciting new frontiers that we are now actively exploring.…
instacartaiexcerpt only · body stays at the source
By Emma Yanyang Kong, Aditya Deshpande, Asad Abbasi, Bowei Yan, David Fagnan, Ashish Rastogi, Dhaval Patel, Ray ZhangIntroductionThe Netflix experience is a journey of discovery. Every visual cue, from the artwork on a title to the video previews that autoplay while you browse, is there to connect you with a story you will love. We call these visual cues assets, and choosing the right one for each member is a personalization problem of its own. But which image or video preview of Squid Game should we show you? And what do we do right after a title launches, when there’s far too little…
Why we think LLMs can be useful and why we will not replace all of our models with themContextAt Medium, we have many Machine Learning models that we use to label stories automatically. These affect what stories we recommend to readers.Here’s some examples:a few of our text classification models. All diagrams and charts made by the authorSome Clarifications on our Machine Learning policyBefore we go deep on this project, I just wanted to clarify a few things about how we stand regarding AI in general.Medium has been training internal models with user and post data for a long time now. We…
When we retrain, when we rebuild, and when we leave a model alone.By: Harrison KatzA forecast that carries weightThe Forecasting Data Science team at Airbnb produces many of the forecasts the rest of the company plans around: demand, bookings, cancellations, and a range of finer cuts by market and segment, refreshed continuously across thousands of markets. The targets differ, and the models differ, but they have one thing in common: Other teams build on top of them.This means a forecast that is casually wrong is not a clean miss, as it might be in an academic setting. That’s because a small…
Key Idea Human validation is not only for evaluating an LLM. It can also calibrate how the LLM is used as a scalable measurement instrument for population estimation. An LLM can classify thousands of records at low cost, but the proportion it classifies as positive is not necessarily the true proportion in the population. By […]
Introduction What worried us wasn’t the hallucination, it was the subtle plausibility. Answers an engineer could easily read past and accept: a right-looking Structured Query Language (SQL) query, a plausible tool call, an innocent profile update, or a patch that satisfied the surface tests. When we analyzed the row-level failures, a clear pattern emerged: SQL generation: kept the query shape but changed the underlying metric. Tool calling: selected the right tool family but drifted on parameters. Profile updates: cited every event instead of only the evidence that supported the claim. Coding…
Expedia Group Technology — DataWhat happened when we treated a framework migration as an architecture modernisation — and cut P99 inference latency by two-thirdsSt Paul’s and millennium bridge, LondonExpedia Group™ has always been a market leader in providing personalised search experiences for travellers. As our ranking models evolved, we saw an opportunity not just to migrate to Keras 3, but to modernise the broader stack around it so we can better serve travellers. This led us to rewrite key parts of our pipelines that made model training 30% faster and cut P99 inference latency by…
IntroductionAt OLX, professional sellers pay for higher-tier packages because they promise more exposure. More visibility, and, in theory, better results. But when we looked at the data, something unexpected happened. In some cases, ads published with premium packages appeared to perform worse than ads using cheaper packages.That raised an uncomfortable question: If higher-tier packages provide more exposure, shouldn’t they consistently perform better?At first glance, there were several possible explanations. Perhaps the extra visibility weren’t creating as much value as we expected. Perhaps…
Introduction The efficacy of semantic search relies on the accuracy of the underlying Knowledge Graph (KG). In high-velocity domains like on-demand food delivery or e-commerce, the catalog of entities like dishes, products, and merchants changes rapidly. Current methods for KG construction and maintenance face three critical challenges: Inaccuracy and hallucination from Large Language Models (LLMs): Automated models often infer relationships based on statistical text co-occurrence rather than semantic reality. For instance, an LLM might incorrectly classify “Pho” as a child of “Italian Noodle…
Part 1: From one support bot to a framework At Grab, AI agents have evolved from interesting team prototypes into production services used every day by millions of merchants, drivers, and consumers. Today, more than 500 services run on our internal agent framework, over 50 Model Context Protocol (MCP) servers are registered on our remote MCP framework, and a single Large Language Model (LLM) gateway fronts every model call across the company, handling billions of tokens each month. None of this was designed up front. It began as the plumbing behind one internal support bot, which then…
How we built a Transformer-based sequence model that encodes years of guest behavior to surface the right listings at the right time.By: Daochen Zha, Chun How Tan, Xin Liu, Bin Xu, Han Zhao, Xiaowei Liu, Jun Shi, Tracy Yu, Hui Gao, Huiji Gao, Liwei He, Michael Kinoti, Stephanie Moyerman, and Sanjeev KatariyaIntroductionPlanning a trip on Airbnb rarely happens in a single session. A guest searching for a place to stay in San Francisco might browse dozens of listings over several days, leaving behind a trail of views. Typically, over a period of years, that same guest will have accumulated many…
A look at compensating for position bias in recommender systems using negative sampling strategiesBy: Andrew Morss, Senior Applied ScientistIntroductionA recommender system is a machine learning model that, given a user and a catalog of items, predicts which items that user is most likely to want. Recommenders set your YouTube playlist, determine what items Amazon suggests for you, push you songs on Spotify and customize your Steam store. If you’re a homeowner, Thumbtack’s recommender systems can suggest home projects for you such as house cleaning or lawn mowing.Thumbtack connects users with…
Expedia Group Technology — InnovationA framework for how we build, deploy, and evolve AI systems for impact and scalePhoto by Florian Wehde on UnsplashThere’s an important distinction between Artificial Intelligence (AI) that just works today and AI that lasts at scale. Many companies optimize hard for the first one without ever asking whether they’re building the second.Velocity without discipline and strategic direction is a liability, not an asset. The hardest part of building AI at scale isn’t getting a model to work once. It’s building systems that continue to work, scale beyond…
Introduction As Large Language Models (LLMs) move from research prototypes to production systems, the developers of these systems need rigorous performance evaluation. In particular, we need confidence intervals around estimates of system accuracy. However, LLMs introduce a challenge that is unusual for ML systems: they are (operationally) non-deterministic. Even with the temperature set to zero, […]
Sergio Camelo, Caitlin Kearns, Matias Cersosimo, and Tilman DrerupAs artificial intelligence increases the velocity of engineering and science teams, experimental throughput is set to become a bottleneck for many product decisions. Many companies can now build faster than they can experiment, with queues of good ideas running the risk of not being tested because of lack of experimental capacity.This problem is particularly severe in marketplaces, where the presence of spillover and cannibalization effects between experimental units requires cluster-level randomization techniques. That…
Authors: Lequn Wang, Jiangwei Pan, and Linas BaltrunasFigure 1. Autoregressive homepage generation. GenPage builds a Netflix homepage one row or entity at a time, each one conditioned on what’s already on the page and the user’s context.IntroductionThe Netflix homepage is the first thing users see when they open the app and the primary way they discover content to enjoy. Almost every part of it is personalized, including which rows appear, which entities show up within those rows, and how everything is arranged on the page.Constructing that homepage is a genuinely hard problem. It is not…
Sheng Huang | Software Engineer, AI Platform; Pong Eksombatchai | Machine Learning Engineer, Applied Sciences; Saurabh Vishwas Joshi | Software Engineer, AI Platform; Gaurav Arora | Software Engineer, AI Platform; Karthik Anantha Padmanabhan | Engineering Director, AI PlatformAt Pinterest, foundation models power recommendations for over 600 million monthly active users. Our latest Foundation Model (ACM RecSys 2025) pre-trains on two years of user activity data and is deployed into Home feed and Related Pins ranking, the platform’s two most important recommendation systems. Multi-node…
Authors : Marsan Ma, Nikhil Lopes, Raj Amrit, Hong Lu, Dipankar Biswas, Trent KyonoLeadership: Iris Wang, Madhu Kurup Recommendation and ranking systems power many of the most important experiences on large internet platforms. Yet the models that run in production are rarely the largest models we can train. They are usually compact, latency-sensitive supervised models […]
Key Contributors: Karuna Ahuja, Marko Avdalovic, Soroush Sobhkhiz, Shrikar Archak, Xiyu Wang, Ji Chao Zhang, Hao YanIntroductionEvery time a user opens Instacart, they see product recommendations: on the retailer home page, in search results, and alongside their cart. Many of these recommendations are sponsored products surfaced by a retrieval model that decides which products to show from a vast ads product catalog. A relevant ad helps users discover products they didn’t know they needed; a less relevant one generates friction.Two years ago, we introduced Contextual Recommendations (CR), a…
How Airbnb used sequential geographic recovery signals and prior propagation to generate reliable corridor-level forecasts when local data was scarce.By: Harrison KatzThe problem with unprecedented shocksAlmost every forecasting system is built on the same implicit assumption: the future will resemble the past. You train on historical data, you validate on holdout periods, and you trust that past patterns will at least roughly indicate future performance. When this assumption breaks, the model does not gracefully degrade; it fails confidently. It produces precise, well-calibrated intervals…
aitechnologyexcerpt only · body stays at the source
Key Contributors: Shrikar Archak, Karuna Ahuja, Soroush Sobhkhiz, Marko Avdalovic, Xiyu Wang, JiChao Zhang, Hao Yan, Chris HartleyIntroductionOperating a grocery catalog at Instacart’s scale means managing millions of products across thousands of categories. Every product is assigned to a category in our hierarchical taxonomy like “Dairy > Cheese > Parmesan”. These categories provide broad classification, but they miss the connections that drive how customers actually shop.For example, a customer is building a cheese board. They’ve added Parmigiano Reggiano, and now they need accompaniments.…
Levi Boxell, Tilman Drerup, Alexandr LenkThe Economics Team at Instacart is an applied science team that operates at the intersection of machine learning engineering and economics. Similar to other applied science teams, our work involves a good chunk of engineering, steeped in statistics, math, theory, and strategy. And while that is still at the heart of what we do today, the surprisingly rapid emergence of artificial intelligence has also fundamentally altered our work in ways that we did not see coming.With this post, we want to provide a brief check-in and share an analysis of the…
Authors (listed alphabetically)Ads Feature Engineering Infra team: Ajay Venkatakrishnan, Le ZhangCore ML Infra team: Eric Shang, Pihui WeiML Data team: Connor Votroubek, Yi HeUser Understanding team: Camilo Munoz, Simin LiIf you work on ranking, retrieval, or recommendation systems, you’ve probably asked for some version of the same thing: “Give me the last N meaningful actions this user took, with the right enrichments, in a format that’s easy to train and serve ML models.”On paper, that sounds simple. In practice, “user sequences” often become one of the most expensive and fragile parts of…
Authors: Trey Zhong, Xiyu WangContributors: Joseph Haraldson, Sharad Gupta, Sarah LamacchiaIntroductionCarrot Ads is Instacart’s omnichannel retail media solution that allows retailer partners to build and scale their own advertising businesses on either their owned-and-operated (O&O) websites and apps or their whitelabel Storefront hosted by Instacart. Carrot Ads empowers retailers and CPG brands to accelerate revenue, while improving the customer experience, engagement and Ads return on investment. It features enterprise-grade infrastructure, AI-powered optimization, years of proprietary…
Guangtong Bai | Staff Software Engineer, Product ML Infrastructure*; Shantam Shorewala | Software Engineer II, Product ML Infrastructure*; Chi Zhang | Staff Software Engineer, AI Platform*; Neha Upadhyay | Software Engineer II, AI Platform*; Haoyang Li | Director, Product ML Infrastructure*These authors contributed equally to this article.BackgroundAt Pinterest, our online ML serving systems employ a root-leaf architecture. On a high level, the architecture looks as follows:Figure 1: Root-leaf Architecture of Online ML Serving Systems at PinterestIn the diagram, “Client Service” is…
At Lyft, understanding how riders go through our user experience is fundamental to operating a healthy marketplace. Specifically, it is important to have a robust model determining if a rider will actually request a ride after entering a destination and viewing a price and ETA. Accurately predicting this decision, that we call conversion, informs countless decisions across our platform. Whether it is to better balance supply and demand, improve user experiences, optimize recommendations and advertisement, understand long-term engagement, decide how to distribute coupons… rider conversion…
Key Contributors: Moein Hasani, Hamidreza Shahidi, Trace Levinson, Guanghua ShuIntroductionAt Instacart, we are laser-focused on improving the user experience by making shopping feel easy, engaging, and personalized. Our discovery surfaces play a central role in bringing this to life. Alongside explicit Search intents, discovery is our opportunity to meet customers’ implicit needs, presenting them with the most relevant and inspiring content we have to offer. The main discovery surface within the Instacart app, referred to here as the “Shopping Hub”, is one of the most critical in this…
Written by Stefan ZierFor years, Lyft’s localization infrastructure relied exclusively on human translation. While this model usually ensured excellent quality, it was bound by multi-day turnarounds and costs that scaled linearly with every new language. For the few languages Lyft initially supported (Spanish, Portuguese, and French), these limits were acceptable.However, Lyft’s expansion goals quickly outpaced what traditional workflows could support. Lyft’s recent Québec launch required compliance with Bill 96 (legislation mandating French-first user experiences) which demanded faster…
Written by Rohan Varshney, with support from Devon Mittow & Janice Lee.This article expands upon a presentation from the Feature Store Summit 2025, which can be viewed in full here. There is also another video available on the evolution of Lyft’s Feature Store from DE4AI 2024.Introduction and Core PurposeLyft’s Feature Store stands as a core infrastructural pillar within its Data Platform organization, designed to optimize the management and deployment of Machine Learning (ML) features at massive scale. Its primary objective is to centralize feature engineering efforts, guaranteeing…
IntroductionIn an age where artificial intelligence (AI) and machine learning (ML) are integral to almost every aspect of our lives, ensuring the effectiveness, fairness, and reliability of ML models is paramount. Observability plays a crucial role in maintaining the performance of these models, allowing us to detect and resolve issues promptly. At Helpshift, we recognized the need for robust ML observability to keep our models running smoothly and efficiently.This blog post explores our journey in building a custom ML observability solution tailored to our specific needs. We’ll delve into…
By Jean V. Alves and Ferran Pla FernándezMoving beyond binary classification provides novel insights.In the real world, scams rarely present themselves in black and white. Fraudsters exploit nuance, impersonate legitimate brands, and mask malicious intent with seemingly ordinary behavior. That’s why Feedzai has launched ScamAlert (patent pending), a Generative AI-based system innovating on the current paradigm of scam prevention, in response to this growing challenge.Traditional detection systems treat the problem as a binary choice: scam or not a scam, often outputting an estimated “scam…
How real-time algorithms and causal ML transformed our digital subscription funnel from static paywalls to dynamic, millisecond decision-makingIllustration by Mathieu LabrecqueThe New York Times became a subscription-first news and lifestyle service with the launch of its paywall in 2011. Since then, our subscription strategy has evolved substantially. Initially, users could access a limited number of free articles per month before they encountered the paywall. In 2019, we began personalizing this number using a Machine Learning (ML) model — The Dynamic Meter. In the past few years, we have…
How we made our email story recommendations betterIn this Part 1, you’ll understand how we improved one of the main ways our users are exposed to our product and how that led to a massive 7% increase on the average reading time for the digest users.Intro: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.By the end of these projects, the…
Cross-Digest diversificationIn this part 4, we’ll see how we went from investigating a few complaints from digest power users to improving our digest recommendations across the board.Intro: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.By the end of these projects, the digest was 10% more likely to convert users to paying members, less…
By Sofia Guerreiro, Ricardo Ribeiro Pereira, Iker Perez, Jacopo BonoDetecting financial fraud is like finding a moving needle in a shifting haystack. Fraud accounts for a tiny fraction of financial transactions, often less than 0.1%. At the same time, fraudsters are constantly adapting their tactics to evade detection. And this happens within a live and dynamic environment, where financial behaviors and technologies are changing over time. In short, this is an exceptionally difficult problem for financial institutions.With the rise of digital banking and new technologies like GenAI, this…
Optimizing FastAPI at Scale: Lessons from OLX’s Recommendation PlatformPhoto by Rosy KoIn distributed systems, there is a motto that says ‘you are as slow as your slowest tasks’. In Python, thanks to the notorious Global Interpreter Lock (GIL), this issue is amplified: ‘your slowest task will make every other task slower’. In this article, I’ll walk you through the optimizations we made to scale a FastAPI service that now handles tens of thousands of requests per second, achieving a p99 latency under 10ms.IntroductionOLX is a global online marketplace that enables users to buy and sell goods…
Schematics of the PCB module In my previous post, Developing a Fall Detection Device with Raspberry Pi, I delved into the initial steps of building a fall detection system using the Raspberry Pi Zero 2W. Today, I’m excited to share the next phase of this journey: designing and fabricating a custom PCB that integrates essential sensors to create a compact, robust, and efficient fall detection device. The Quest for Compactness The primary goal of this project was to shrink the device’s size, making it thinner and more wearable for our target users. Bulky prototypes are a hindrance, especially…
Your browser does not support the audio element. ** Dive into an AI-generated podcast where two virtual hosts discuss the key findings and implications of the featured article and its groundbreaking research." The prototype with the cover removed Falls are a significant concern for the elderly population, often leading to serious injuries and a decrease in the quality of life. Detecting falls promptly can enable quick assistance, potentially reducing the severity of injuries and providing peace of mind for both seniors and their families. In my recent project, I set out to create a highly…
Use Cases for Generative AI For the past year, almost every week the Software Engineering world is asked about Artificial Intelligence (AI). “What should Capgemini’s AI offerings be?”, “Aren’t you worried about the role of Capgemini software engineers now that AI can write code?” Specifically in these cases, the questions are around Generative AI. To put that in context, here is a recap of some terms used to describe different implementations of AI. Machine Learning A nice dictionary definition of machine learning from Oxford Languages is: “the use and development of computer systems that are…
By Sérgio Jesus, Inês Silva, Pedro Saleiro, Hugo Ferreira, Pedro BizarroIn this blog post we will visit Aequitas Flow, an Open-Source framework designed to run complete and standardized experiments of Fair ML algorithms. We encourage you to try Aequitas Flow with the Google Colab Notebooks, which are available in the project’s GitHub repository.This blog post is based on the paper by Sérgio Jesus, Pedro Saleiro, Inês Silva, Beatriz M. Jorge, Rita P. Ribeiro, João Gama, Pedro Bizarro, and Rayid Ghani.Table of Contents:1. What is Aequitas Flow?- 1.1. For Practitioners selecting a model- 1.2.…
In the world of financial services, the bank or financial institution’s relationship with the customer relies on digital trust, which is anchored in two fundamental principles. First, it must ensure the person engaging through digital banking channels is genuinely the individual they claim to be. Second, it must confirm that this person is authorized to complete the intended financial transaction.Addressing these crucial requirements is the core mission of Feedzai’s Digital Trust solution. The solution collects and analyzes comprehensive user behavioral data, scrutinizes device information…
Optional Google Analytics helps us understand visits. Microsoft Clarity records masked interactions to improve the site. Optional tools stay off unless you choose them. Privacy details.