In the entertainment sector, content significantly influences viewer decisions, necessitating a Large Language Model (LLM) with specialized knowledge of the entertainment catalog. Traditional LLMs, while proficient in capturing broad factual information, struggle with static knowledge and lack domain-specific expertise. This paper explores post-training knowledge injection via instruction tuning to address these challenges. By converting content into diverse Question-Answer pairs, the model gains a nuanced understanding of titles, enhancing its ability to generalize to new instructions. This enriched context is expected to improve the model's accuracy in recommending the next title, aligning more closely with individual viewer preferences and histories.

The poster LinkedIn’s large-scale, GPU-based retrieval system for out-of-network feed. The new retrieval system supports a billion-sized index on GPU models where both items and model weights are integrated into the model binary. Viewing index construction as a form of model training, we describe scaling our system for large indexes, incorporating full scans and efficient filtering. A key focus is on enabling attribute-based pre-filtering for exhaustive GPU searches, addressing the common challenge of post-filtering in KNN searches that often reduces system quality. We believe this represents one of the industry’s first Live-updated model-based retrieval indexes. Applied to out-of-network post recommendations on LinkedIn Feed, it has contributed to a +0.1% lift in daily unique professional users. We envisage this as a step towards integrating retrieval and ranking into a single GPU model, simplifying complex infrastructures and enabling end-to-end optimization of the entire differentiable infrastructure through gradient descent.

Recommender systems exemplify sequential decision-making under uncertainty, strategically deciding what content to serve to users, to optimise a range of potential objectives.
To balance the explore-exploit trade-off successfully, Thompson sampling provides a natural and widespread paradigm to probabilistically select which action to take.
Questions of causal and counterfactual inference, which underpin use-cases like offline evaluation, are not straightforward to answer in these contexts.
Specifically, whilst most existing estimators rely on action propensities, these are not readily available under Thompson sampling procedures.

We derive exact and efficiently computable expressions for action propensities under a variety of parameter and outcome distributions, enabling the use of off-policy estimators in Thompson sampling scenarios.
This opens up a range of practical use-cases where counterfactual inference is crucial, including unbiased offline evaluation of recommender systems, as well as general applications of causal inference in online advertising, personalisation, and beyond.

In today’s fast-evolving digital space, personalization is key to enhancing user experience and satisfaction. As LLM demonstrates remarkable capabilities in NLP tasks, we explore fine tune open source LLMs for member taste summarization to improve personalization.

Modern recommendation systems aim to increase click-through rates (CTR) for better user experience, through commonly treating ranking as a classification task focused on predicting CTR. However, there’s a gap between this method and the actual objectives of businesses across different sectors. In video recommendation services, the objective of video on demand (VOD) extends beyond merely encouraging clicks, but also guiding users to discover their true interests, leading to increased watch time. And longer users’ watch time will leads to more revenue through increased chances of presenting online display advertisements. This research addresses the issue by redefining the problem from classification to regression, with a focus on maximizing revenue through user viewing time. Due to the lack of positive labels on recommendation, the study introduces Tweedie Loss Function, which is better suited in this scenario than the traditional mean square error loss. The paper also provides insights on how Tweedie process capture users’ diverse interests. Our offline simulation and online A/B test revealed that we can substantially enhance our core business objectives: user engagement in terms of viewing time and, consequently, revenue. Additionally, we provide a theoretical comparison between the Tweedie Loss and the commonly employed viewing time weighted Logloss, highlighting why Tweedie Regression stands out as an efficient solution. We further outline a framework for designing a loss function that focuses on a singular objective.

Recommender systems operate in closed feedback loops, where user interactions reinforce popularity bias, leading to over-recommendation of already popular items while under-exposing niche or novel content. Existing bias mitigation methods, such as Inverse Propensity Scoring (IPS) and Off- Policy Correction (OPC), primarily operate at the ranking stage or during training, lacking explicit real-time control over exposure dynamics. In this work, we introduce an exposure- aware retrieval scoring approach, which explicitly models item exposure probability and adjusts retrieval-stage ranking at inference time. Unlike prior work, this method decouples exposure effects from engagement likelihood, enabling controlled trade-offs between fairness and engagement in large-scale recommendation platforms. We validate our approach through online A/B experiments in a real-world video recommendation system, demonstrating a 25% increase in uniquely retrieved items and a 40% reduction in the dominance of over-popular content, all while maintaining overall user engagement levels. Our results establish a scalable, deployable solution for mitigating popularity bias at the retrieval stage, offering a new paradigm for bias-aware personalization.

When logging into Netflix, members are greeted with a diverse array of titles on the homepage. To enrich member’s browsing and discovery experience, we present diverse evidence assets for each title placed in different locations. For example, the most prominent asset is the artwork, or box-art image, which showcases the main characters and underscores the title's theme. We also provide a comprehensive synopsis and evidence cards to outline the content, eye-catching badges to draw users' attention, and supplemental videos (trailers) to offer a sneak peek into the shows.

With the thrilling advancements on the multimodal models, computer visions, and large language models (LLMs), we are motivated to incorporate rich asset information, such as images, text, and videos, into our recommendation algorithms.

Our initial move is to leverage multimodal embeddings in the evidence personalization domain. We have established the paved path to experiment various types of embedding in our image/video/text evidence personalization, and specific applications like query-aware evidence on the search page.

Recent advancements in Generative AI (GenAI) empower us to generate free-text personalized explanations that connect a recommended title given a member's viewing history. These explanations are not only more expressive but can also incorporate a broader context, offering an unprecedented level of personalization with evidence creation that surpasses the conventional evidence assets. Our goal is to harness AI to empower our recommendation system with reasoning and explainability, make our recommendation algorithm more transparent for our Netflix members.

This work presents a novel approach to real-time personalization of recommendations by integrating user-specific goals through linear optimization. Traditional recommendation systems often rely on static models trained on delayed data, limiting their ability to adapt to dynamic user preferences and immediate needs. We address this limitation by introducing a two-stage architecture: (1) A genAI agent invokes recsys tool and generates an initial ranked list of recommendations based on a robust but potentially outdated feature set. (2) Upon user interaction, we capture their specific goal in real-time and employ a lightweight linear optimization layer to dynamically re-rank the initial recommendations by treating the user-specified goal as a constraint within the optimization problem. This approach allows us to maintain the efficacy of the pre-generated recommendations while ensuring precise alignment with the user's immediate objectives. Our method offers a computationally efficient and scalable solution for real-time personalization, enhancing user experience and engagement within the FinTech domain.

Our mission at Netflix is to entertain every member by recommending the right shows to them. While conventional recommendation systems typically focus on immediate metrics such as clicks or short-term engagement, these measures often fail to reflect a user's enduring satisfaction. Our approach aims to recommend content that delivers both immediate appeal and lasting enjoyment, ultimately providing greater value to our members and fostering long-term retention. Over time, achieving this leads to each surface ranker optimizing its own set of reward-maximizing policies, making it harder to innovate on the entire recommendation stack holistically. Moreover, this significantly increases the compute costs associated with training and maintaining bespoke reward-aligned surface policies.

To address these challenges, we are transitioning from independent policies for each surface recommendation model to implementing a centralized "core behavior value" that aligns all ranking models through a consistent algorithmic procedure—two-stage training.

We propose this two-stage training process that decouples reward optimization from training extensive recommendation models. The first stage involves pre-training a policy without specific reward optimization, while the second stage incorporates engineered proxy rewards that better reflect long-term satisfaction. This approach offers backward compatibility and multiple implementation options, including fine-tuning the base policy on smaller datasets optimized for longer-term objectives.

By adopting this two-stage training methodology, we've achieved significant improvements in compute resource efficiency—approximately 70% savings in pre-processing and model training pipelines across various surface models including homepage, category tabs, and candidate generation. Additionally, this approach enables independent innovation tracks for researchers working separately on engagement models and long-term satisfaction metrics.

The Messaging Selection Algorithm (MESA) is the core algorithm used by the Netflix Messaging Personalization System (MPS) to create a personalized ranking of messages for a user based on their interests. MESA relies on accurate predictions of the individual treatment effect of a message on a user’s engagement. MESA faces challenges with unpredictable fluctuations in the treatment effect estimation when there is insufficient historical data for specific messages which can negatively impact user experience when specific message intents are under- or over-selected. We present CATE-based Treatment Covariate Calibration, a technique for calibrating treatment effects of any blackbox model based on IPS-weighted causal effect estimates. Our approach is designed with extensibility in mind, allowing for straightforward incorporation of treatment covariates such as message intent or delivery channel into the calibration action space. This flexibility enables more granular calibration across multiple dimensions of the recommendation system. In practical applications to MESA, this algorithm has demonstrated improvements in the calibration of individual treatment effect estimates for models trained using both S-Learner and Double Machine Learning causal learning frameworks.

The Messaging Personalization System (MPS) is the Netflix algorithm responsible for providing a personalized message experience for each user by choosing the best messages and delivery channel (e.g. push or email) as well as the timing and frequency of messages. Personalizing the frequency of messages is critical for a high quality user experience as sending too many messages may lead to fatigue or opt-out, while too few messages may lead to reduced user engagement. We present the design of a two-layer algorithm that simultaneously optimizes for both short- and long-term user engagement. The first layer, a slow policy, runs on a weekly cadence to determine the optimal message pacing for each user across different delivery channels and targets the causal effect of messaging on long-term engagement signals (e.g. weekly aggregated). The personalized message pacing is then sent as an input to the second layer, a ranking policy that is triggered on a more frequent cadence (e.g. daily). This fast ranking policy targets short-term engagement signals and makes the final ranking and targeting decisions for a user while adhering to the constraints established by the slow policy.

Large Language Models (LLMs), pre-trained on extensive internet discussions, present a new paradigm for improving title (i.e., shows and movies) recommendations. Generative recommenders (GenRec) using LLMs have already shown promising results compared to previous baselines. However, the frequent release of new content introduces the challenge of title cold-starting: new titles, not present in interaction sequence datasets, are not learned by the model for recommendation. To alleviate this issue, this work explores methods to incorporate new titles into LLM-based recommendation systems without compromising their inherent strengths, aiming to preserve the advantage LLMs offer in predicting and recommending newly released titles.

This work introduces a novel, multi-stage framework for robust intent detection in production dialogue-based search assistants and query rerouting systems. It addresses challenges such as ambiguity in multi-turn interactions, malicious intent detection, continuous adaptation to new intents, and initial data scarcity. The framework uses a staged approach: starting with few-shot learning via Large Language Models (LLMs) and a high-level ontology, it transitions to a supervised classifier with a sliding window for improved contextual awareness as data grows. Unsupervised clustering and weak supervision enable continuous learning, granular intent detection, and identification of harmful requests. A final hybrid architecture integrates the classifier, clustering, and LLMs to handle edge cases and novel intents, ensuring rapid deployment and long-term scalability.

Ads recommendation is a prominent service of online advertising systems and has been actively studied. Recent studies indicate that scaling-up and advanced design of the recommendation model can bring significant performance improvement. However, with a larger model scale, such prior studies have a significantly increasing gap from industry as they often neglect two fundamental challenges in industrial-scale applications. First, training and inference budgets are restricted for the model to be served, exceeding which may incur latency and impair user experience. Second, large-volume data arrive in a streaming mode with data distributions dynamically shifting, as new users/ads join and existing users/ads leave the system. We propose the External Large Foundation Model (ExFM) framework to address the overlooked challenges. Specifically, we develop external distillation and a data augmentation system (DAS) to control the computational cost of training/inference while maintaining high performance. We design the teacher in a way like a foundation model (FM) that can serve multiple students as vertical models (VMs) to amortize its building cost. We propose Auxiliary Head and Student Adapter to mitigate the data distribution gap between FM and VMs caused by the streaming data issue. Comprehensive experiments on internal industrial-scale applications and public datasets demonstrate significant performance gain by ExFM.

We discuss how we built a more interactive discovery experience for Netflix using LLMs to understand complex user queries for providing useful recommendations. Audience will be able to try out the Beta as well.