Deep Learning Based Causal Inference for Large-Scale Combinatorial Experiments: Theory and Empirical Evidence
Dr. Philip Renyu Zhang
Associate Professor
Department of Decisions, Operations and Technology
The Chinese University of Hong Kong
ABSTRACT
Large-scale online platforms launch hundreds of randomized experiments (a.k.a. A/B tests) every day to iterate their operations and marketing strategies, while the combinations of these treatments are typically not exhaustively tested. It triggers an important question of both academic and practical interests: Without observing the outcomes of all treatment combinations, how to estimate the causal effect of any treatment combination and identify the optimal treatment combination? We develop a novel framework combining deep learning and doubly robust estimation to estimate the causal effect of any treatment combination for each user on the platform when observing only a small subset of treatment combinations. Our proposed framework (called debiased deep learning, DeDL) exploits Neyman orthogonality and combines interpretable and flexible structural layers in deep learning. We prove theoretically that this framework yields efficient, consistent, and asymptotically normal estimators under mild assumptions, thus allowing for identifying the best treatment combination when only observing a few combinations. To empirically validate our method, we then collaborate with a large-scale video-sharing platform and implement our framework for three experiments involving three treatments where each combination of treatments is tested. When only observing a subset of treatment combinations, our DeDL approach significantly outperforms other benchmarks to accurately estimate and infer the average treatment effect (ATE) of any treatment combination, and to identify the optimal treatment combination.
Key words: Deep Learning; Double Machine Learning; Causal Inference; Field Experiments; Experimentation on Online Platforms
Based on the joint work with Zikun Ye, Zhiqi Zhang, Dennis Zhang, and Heng Zhang.
BIOGRAPHY
Renyu (Philip) Zhang has been an Associate Professor (with tenure) at the Department of Decisions, Operations and Technology, The Chinese University of Hong Kong Business School since September 2022. He is also an economist and Tech Lead at Kwai, one of the world’s largest online video-sharing and live-streaming platforms. Philip’s recent research focuses on developing data science methodologies (e.g., data-driven optimization, causal inference, and machine learning) to evaluate and optimize the operations strategies in the contexts of online platforms and marketplaces, sharing economy, and social networks, especially their recommendation, advertising, pricing, and matching policies. His research works have appeared in top journals such as Management Science, Operations Research, and Manufacturing & Service Operations Management, and have been recognized by various research awards of the INFORMS and POMS communities. His research projects have been funded by various funding agencies including HK RGC, NSFC, SMEC, STCSM, and Tencent. Philip serves as a Senior Editor for Production and Operations Management, and an Associate Editor for Naval Research Logistics. He has also developed data science and economics frameworks to evaluate and optimize the user growth strategy and the platform ecosystem of Kwai. Prior to joining CUHK, Philip was an Assistant Professor of Operations Management at New York University Shanghai between 2016 and 2022. Please visit Philip’s personal website for more about him: https://rphilipzhang.github.io/rphilipzhang/
