# NeurIPS 2019 Workshop

*“Do the right thing”: machine learning and causal inference for improved decision making*

*“Do the right thing”: machine learning and causal inference for improved decision making*

December 14, 2019, Vancouver, Vancouver Conventional Center, West Level 1, West Ballroom C

**Workshop Summary**

In recent years, machine learning has seen important advances in its theoretical and practical domains, with some of the most significant applications in online marketing and commerce, personalized medicine, and data-driven policy-making. This dramatic success has led to increased expectations for autonomous systems to make the right decision at the right target at the right time. This gives rise to one of the major challenges of machine learning today that is the understanding of the cause-effect connection. Indeed, actions, intervention, and decisions have important consequences, and so, in seeking to make the best decision, one must understand the process of identifying causality. By embracing causal reasoning autonomous systems will be able to answer counterfactual questions, such as “What if I had treated a patient differently?”, and “What if had ranked a list differently?” thus helping to establish the evidence base for important decision-making processes.

*The purpose of this workshop is to bring together experts from different fields to discuss the relationships between machine learning and causal inference and to discuss and highlight the formalization and algorithmization of causality toward achieving human-level machine intelligence.*

**Invited Speakers**

**Program Schedule**

#### 08:45am – 09:00am

#### West Ballroom C

### Opening Remarks

#### 09:00am – 09:30am

#### West Ballroom C

### Confidence Intervals for Policy Evaluation in Adaptive Experiments

By Susan Athey

##### Invited speaker

#### 09:30am – 10:00am

#### West Ballroom C

### Optimal adjustment sets in non-parametric graphical models

##### Invited speaker

#### 10:00am – 10:15am

#### West Ballroom C

### Poster Spotlights

##### Two minute poster presentation

*Bounds on the Conditional and Average Treatment Effect with Unobserved Confounding Factors*

* Marie-Laure Charpignon, Elizabeth Manrao, Min Kang, Kirti Kamboj, Kamal Choudhary, John Hoegger – *Moving beyond correlation: Using a causal inference approach to measure the impact of a customer’s attribute on product retention A case study on Microsoft Teams collaborative app*

* Maja Rudolph – *Multi-Cause Inference with Sequential Treatments*

* Amanda Coston, Alexandra Chouldechova, Edward Kennedy – *Counterfactual Risk Assessments, Evaluation, and Fairness*

* Yuta Saito – *Unbiased Pairwise Learning from Implicit Feedback*

* Jeremy Yang , Dean Eckles , Paramveer Dhillon, Sinan Aral – *Optimizing Targeting Policies via Sequential Experimentation for User Retention*

* Alexander Markham, Moritz Grosse-Wentrup – *Measurement Dependence Inducing Latent Causal Models*

#### 10:15am – 11:00am

#### West Ballroom C

### Coffee break, posters, and 1-on-1 discussions

##### Poster Reminders

– Please make your posters 36W x 48H inches or 90 x 122 cm.

– Posters should be on lightweight paper, not laminated.

#### 11:00am – 11:30am

#### West Ballroom C

### Challenges in Developing Learning Algorithms to Personalize Treatment in Real Time

By Susan Murphy

##### Invited speaker

#### 11:30am – 12:00pm

#### West Ballroom C

### Estimation and inference on high dimensional individualized treatment rule using split-and-pooled de-correlated score

By Ying-Qi Zhao

##### Invited speaker

#### 12:00pm – 12:45pm

#### West Ballroom C

### How machine learning, and causal inference work together: cross-pollination and new challenges

By David Sontag, Susan Athey, Susan Murphy, Andrea Rotnitzky, Ying-Qi Zhao, Susan Shortreed

##### Panel discussion

#### 12:45pm – 02:30pm

#### West Ballroom C

### Lunch

#### 02:30pm – 03:00pm

#### West Ballroom C

### Variable selection for causal inference: outcome-adaptive lasso

##### Invited speaker

#### 03:00pm – 03:30pm

#### West Ballroom C

### Oral Spotlights

##### Five minute oral presentation

*Deep IV in Law: Analysis of Appellate Impacts on Sentencing Using High-Dimensional Instrumental Variables*

* Jörn Boehnke, Pietro Bonaldi – *Synthetic Regression Discontinuity: Estimating Treatment Effects using Machine Learning*

* Yixin Wang, Dhanya Sridhar, David Blei – *Equal Opportunity and Affirmative Action via Counterfactual Predictions*

* Divyat Mahajan, Amit Sharma – *Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers*

* Khashayar Khosravi, Vasilis Syrgkanis, Greg Lewis – *Non-Parametric Inference Adaptive to Intrinsic Dimension*

* Alexander D’Amour, Alexander Franks – *Covariate Reduction for Weighted Causal Effect Estimation with Deconfounding Scores*

#### 03:30pm – 03:45pm

#### West Ballroom C

### Poster Spotlights

##### Two minute poster presentation

*Efficient structure learning with automatic sparsity selection for causal graph processes*

* Rahul Singh, Liyang Sun – *De-biased Machine Learning for Compliers*

* Zichen Zhang, Qingfeng Lan, Lei Ding, Yue Wang, Negar Hassanpour, Russ Greiner – *Reducing Selection Bias in Counterfactual Reasoning for Individual Treatment Effects Estimation*

* Jon Richens, Ciarán M. Lee, Saurabh Johri – *Counterfactual diagnosis*

* Jesse Krijthe, Tom Heskes – *Estimating the long-term effect of early treatment initiation in Parkinson’s disease using observational data*

* Bradley Butcher, Vincent S Huang, Jeremy Reffin, Sema K Sgaier, Grace Charles, Novi Quadrianto – *Causal datasheet: An approximate guide to practically assess Bayesian networks in the real world*

* Victor Chernozhukov, Whitney Newey, Vira Semenova – *Inference on the weighted average value function with high-dimensional state space*

* Rahul Ladhania – *A Sequence of Two Studies to Propose & Test Sub-groups with Heterogeneous Treatment Effects*

* Miruna Oprescu, Vasilis Syrgkanis, Keith Battocchi – *EconML: A Machine Learning Library for Estimating Heterogeneous Treatment Effects*

#### 03:45pm – 04:30pm

#### West Ballroom C

### Coffee break, posters, and 1-on-1 discussions

##### Poster Reminders

– Please make your posters 36W x 48H inches or 90 x 122 cm.

– Posters should be on lightweight paper, not laminated.

#### 04:30pm – 04:45pm

#### West Ballroom C

### Closing Remarks

**Call for Papers**

We solicit submissions of novel research related to all aspects of causal inference, counterfactual prediction, and autonomous action. This includes, but is not limited to, the following topics:

- Predicting counterfactual outcomes
- Reinforcement Learning and Causal Inference
- Causal transfer learning
- Mediation analysis
- Estimation of (conditional) average treatment effects
- Contextual bandit algorithms and on-policy learning
- Batch/offline learning from bandit feedback
- Off-policy evaluation and learning
- Interactive experimental control vs. counterfactual estimation from logged experiments
- Online A/B-testing vs. offline A/B-testing
- De-biasing observational data and feedback cycles
- Fairness of actions and causal aspects of fairness
- Applications in online systems (e.g. search, recommendation, ad placement)
- Applications in physical systems (e.g. cars, smart homes)
- Applications in medicine (e.g. personalized treatment, clinical trials)

We suggest extended abstracts of 2 pages in the NeurIPS format, but no specific format is enforced. The review process is double-blind. A maximum of 8 pages will be considered. References will not count towards the page limit. PDF files only. At the discretion of the organizers, accepted contributions will be assigned slots as contributed talks and others will be presented as posters.

**Accepted Papers**

Daniel L Chen – *Deep IV in Law: Analysis of Appellate Impacts on Sentencing Using High-Dimensional Instrumental Variables*

Jörn Boehnke, Pietro Bonaldi – *Synthetic Regression Discontinuity: Estimating Treatment Effects using Machine Learning*

Yixin Wang, Dhanya Sridhar, David Blei – *Equal Opportunity and Affirmative Action via Counterfactual Predictions*

Divyat Mahajan, Amit Sharma – *Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers*

Khashayar Khosravi, Vasilis Syrgkanis, Greg Lewis – *Non-Parametric Inference Adaptive to Intrinsic Dimension*

Alexander D’Amour, Alexander Franks – *Covariate Reduction for Weighted Causal Effect Estimation with Deconfounding Scores*

Steve Yadlowsky, Hongseok Namkoong, Sanjay Basu, John Duchi, Lu Tian – *Bounds on the Conditional and Average Treatment Effect with Unobserved Confounding Factors*

Marie-Laure Charpignon, Elizabeth Manrao, Min Kang, Kirti Kamboj, Kamal Choudhary, John Hoegger – *Moving beyond correlation: Using a causal inference approach to measure the impact of a customer’s attribute on product retention A case study on Microsoft Teams collaborative app*

Maja Rudolph – *Multi-Cause Inference with Sequential Treatments*

Amanda Coston, Alexandra Chouldechova, Edward Kennedy – *Counterfactual Risk Assessments, Evaluation, and Fairness*

Yuta Saito – *Unbiased Pairwise Learning from Implicit Feedback*

Jeremy Yang , Dean Eckles , Paramveer Dhillon, Sinan Aral – *Optimizing Targeting Policies via Sequential Experimentation for User Retention*

Alexander Markham, Moritz Grosse-Wentrup – *Measurement Dependence Inducing Latent Causal Models*

Théophile Griveau-Billion, Ben Calderhead – *Efficient structure learning with automatic sparsity selection for causal graph processes*

Rahul Singh, Liyang Sun – *De-biased Machine Learning for Compliers*

Zichen Zhang, Qingfeng Lan, Lei Ding, Yue Wang, Negar Hassanpour, Russ Greiner – *Reducing Selection Bias in Counterfactual Reasoning for Individual Treatment Effects Estimation*

Jon Richens, Ciarán M. Lee, Saurabh Johri – *Counterfactual diagnosis*

Jesse Krijthe, Tom Heskes – *Estimating the long-term effect of early treatment initiation in Parkinson’s disease using observational data*

Bradley Butcher, Vincent S Huang, Jeremy Reffin, Sema K Sgaier, Grace Charles, Novi Quadrianto – *Causal datasheet: An approximate guide to practically assess Bayesian networks in the real world*

Victor Chernozhukov, Whitney Newey, Vira Semenova – *Inference on the weighted average value function with high-dimensional state space*

Rahul Ladhania – *A Sequence of Two Studies to Propose & Test Sub-groups with Heterogeneous Treatment Effects*

Miruna Oprescu, Vasilis Syrgkanis, Keith Battocchi – *EconML: A Machine Learning Library for Estimating Heterogeneous Treatment Effects*

Yangyi Lu, Amirhossein Meisami, Ambuj Tewari – *Learning Good Interventions Sequentially via Causal Bandits*

Julius von Kügelgen, Alexander Mey, Marco Loog, Bernhard Schölkopf – *Semi-Supervised Learning, Causality and the Conditional Cluster Assumption*

Niranjani Prasad – *Defining Admissible Rewards for High-Confidence Policy Evaluation*

David Simchi-Levi, Yunzong Xu, Jinglong Zhao – *Unifying Adaptivity and Switching Ability in Online Experimental Design*

Dmytro Mykhaylov, David J Rohde, Flavian Vasile, Martin Bompaire, Olivier Jeunen – *Three Methods for Training on Bandit Feedback*

Finnian Lattimore, David J Rohde – *Causal inference with Bayes rule*

Chirag Modi, Uros Seljak – *Generative Learning of Counterfactual for Synthetic Control Applications in Econometrics*

Negar Hassanpour, Russell Greiner – *Learning Disentangled Representations for Counter Factual Regression*

Ankit Sharma, Garima Gupta, Ranjitha Prasad, Arnab Chatterjee, Lovekesh Vig, Gautam Shroff – *MetaCI: Meta-Learning for Causal Inference in a Heterogeneous Population*

Yuta Saito, Gota Morishita – *Dual Learning Algorithm for Delayed Feedback in Display Advertising*

Shyngys Karimov, Gert Bijnens, Joep Konings – *The impact of automatic wage indexation on Belgian employment, a machine learning approach*

Jean Pouget-Abadie – *Variance Reduction in Bipartite Experiments through Correlation Clustering*

Jeroen Berrevoets, Sam Verboven, Wouter Verbeke – *Optimising Individual-Treatment-Effect Using Bandits*

Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Chris J Pal, Yoshua Bengio – *Fast Adaptation and Slow Meta-Learning of Neural Causal Models*

Peter Wirnsberger, Jovana Mitrovic, Melissa Tan, Charles Blundell, Lars Buesing – *Improved Decision Making in Structural Causal Bandits*

Julius von Kügelgen, Paul Rubenstein, Bernhard Schölkopf, Adrian Weller – *Optimal experimental design via Bayesian optimization: active causal structure learning for Gaussian process networks*

Logan Graham, Ciarán M. Lee, Yura Perov – *Copy, paste, infer: A robust analysis of twin networks for counterfactual inference*

Candice Schumann, Zhi T Lang, Nicholas Mattei, John P Dickerson – *Group Fairness in Bandit Arm Selection*

Anisha Zaveri, Victor Veitch – *Sense and Sensitivity Analysis: Simple Post-Hoc Analysis of Bias Due to Unobserved Confounding*

Anish Agarwal, Romain Cosson, Devavrat Shah, Dennis Shen – *Bridging Randomized Control Trials and Personalized Treatments*

Julie Josse, Imke Mayer, Jean-Philippe Vert – *MissDeepCausal: causal inference from incomplete data using deep latent variable models*

Victor Chernozhukov, Denis Nekipelov, Vira Semenova, Vasilis Syrgkanis – *Regularized estimation of high-dimensional semiparametric models*

Mert Demirer, Vasilis Syrgkanis, Greg Lewis, Victor Chernozhukov – *Semi-Parametric Efficient Policy Learning with Continuous Actions*

Vasilis Syrgkanis, Victor Lei, Miruna Oprescu, Maggie Hei, Keith Battocchi, Greg Lewis – *Heterogeneous Treatment Effects with Instruments*

Suraj Nair, Yuke Zhu, Silvio Savarese, Li Fei-Fei – *Visual Causal Induction for Goal Directed Tasks*

Aahlad Manas Puli, Rajesh Ranganath – *Generalized Control Functions for Flexible IV Estimation*

Dhanya Sridhar, Victor Veitch, David Blei – *Using Text Embeddings for Causal Inference*

Nathan Kallus, Masatoshi Uehara – *Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes*

Mohammadi Zaki, Avinash Mohan, Aditya Gopalan – *Towards Optimal and Efficient Best Arm Identification in Linear Bandits*

Issa Dahabreh – *Causally interpretable meta-analysis with double/debiased machine learning*

Yi Ding, Guillaume Basse, Panos Toulis – *Minimax Crossover Designs*

Ignavier Ng, Shengyu Zhu, Zhitang Chen, Zhuangyan Fang – *A Graph Autoencoder Approach to Causal Structure Learning*

Alan Mishler, Niccolo Dalmasso – *When the Oracle Misleads: Modeling the Consequences of Using Observable Rather than Potential Outcomes in Risk Assessment Instruments*

Eli Sherman, David Arbour, Ilya Shpitser – *Policy Interventions Under Interference*

Duncan Wadsworth – *Measuring Fairness in A/B Testing*

Shuxi Zeng, Pengchuan Zhang, Denis Charles, Eren Manavoglu, Emre Kiciman – *Robust Neural Network for Causal Invariant Features Extraction*

Aleksander Wieczorek, Volker Roth – *Information Theoretic Causal Effect Quantification*

Shuxi Zeng, Murat Bayir, Denis Charles, Joseph J Pfeiffer, Emre Kiciman – *Causal Transfer Random Forest: Leveraging Observational and Randomization Studies*

Hao Liu, Anqi Liu, Tongxin Li, Saeed Karimi, Yisong Yue, Animashree Anandkumar – *Disentangling Causal Effects from Latent Confounders using Interventions*

Khizar Qureshi, Tauhid Zaman – *Running Influence Campaigns on Twitter*

Kyra Gan, Andrew Li, Zachary Lipton, Sridhar Tayur – *Causal Inference with Selectively-Deconfounded Data*

Jason S Hartford, Kevin Leyton-Brown – *Identifying Valid Instruments via Effect Agreement*

**Organizers**

**Thorsten Joachims**

Cornell University

**Nathan ****Kallus**

Cornell University

**Adith Swaminathan**

Microsoft Research

**Michele Santacatterina**

Cornell University

**David ****Sontag**

MIT

**Angela ****Zhou**

Cornell University