Workshop on Deep Reinforcement Learning for Knowledge Discovery

August 5 (1:00-5:00PM), 2019
Summit 1 - Ground Level, Egan, Anchorage, Alaska - USA


While supervised and unsupervised learning have been extensively used for knowledge discovery for decades and have achieved immense success, much less attention has been paid to reinforcement learning in knowledge discovery until the recent emergence of deep reinforcement learning (DRL). By integrating deep learning into reinforcement learning, DRL is not only capable of continuing sensing and learning to act, but also capturing complex patterns with the power of deep learning. Recent years have witnessed the enormous success of DRL for numerous domains such as the game of Go, video games, and robotics, leading up to increasing advances of DRL for knowledge discovery. For instance, RL-based recommender systems have been developed to produce recommendations that maximize user utility (reward) in the long run for interactive systems; RL-based traffic signal systems have been designed to control traffic lights in real time to enhance traffic efficiency for urban computing. Similar excitement has been generated in other areas of knowledge discovery, such as graph optimization, interactive dialogue systems, and big data systems. While these successes show the promise of DRL, applying learning from game-based DRL to knowledge discovery is fraught with unique challenges, including, but not limited to, extreme data sparsity, power-law distributed samples, and large state and action spaces. Therefore, it is timely and necessary to provide a venue, which can bring together academia researchers and industry practitioners (1) to discuss the principles, limitations and applications of DRL for knowledge discovery; and (2) to foster research on innovative algorithms, novel techniques, and new applications of DRL to knowledge discovery.


We invite the submission of novel research paper (6 ~ 10 pages), demo paper (4 ~ 10 pages), visionary papers (4 ~ 10 pages) as well as extended abstracts (1 ~ 4 pages). Submissions must be in PDF format, written in English, and formatted according to the latest double-column ACM Conference Proceedings Template. All papers will be peer reviewed, single-blinded. Submitted papers will be assessed based on their novelty, technical quality, potential impact, insightfulness, depth, clarity, and reproducibility. All the papers are required to be submitted via EasyChair system. For more questions about the workshop and submissions, please send email to

We encourage submissions on a broad range of DRL for knowledge discovery in various domains. Topics of interest include but are not limited to theoretical aspects, algorithms, methods, applications, and systems, such as:

  • Foundation:
  • - Reinforcement Learning and Planning
  • - Decision and Control
  • - Exploration
  • - Hierarchical RL
  • - Markov Decision Processes
  • - Model-Based RL
  • - Multi-Agent RL
  • - Inverse RL
  • - Contextual Bandits
  • - Navigation
  • Business:
  • - Advertising and E-commerce
  • - Finance
  • - Marketing
  • - Markets and Crowds
  • - Recommender systems
  • Urban Computing:
  • - Smart Transportation
  • - Intelligent Environment
  • - Urban Planning
  • - Urban Economy
  • - Urban Energy
  • Computational Linguistics:
  • - Dialogue and Interactive Systems
  • - Semantic Parsing
  • - Summarization
  • - Machine Translation
  • - Question Answering
  • Graph Mining:
  • - Social and Network Sciences
  • - Graph Modeling and Embedding
  • - Graph Generation and Optimization
  • - Combinatorial Optimization and Planning
  • Big Data Systems:
  • - Systems for large-scale RL
  • - Environments for testing RL
  • - RL to improve Systems
  • Further target application areas:
  • - Health Care
  • - Computer Vision
  • - Education
  • - Security
  • - Time Series
  • - Multimedia


May 15, 2019: Workshop paper submission due (23:59, Pacific Standard Time)

June 1, 2019: Workshop paper notifications

June 16, 2019: Camera-ready deadline for workshop papers

August 5 (1:00-5:00PM), 2019: Workshop Date


  • 1:00 - 1:10PM -- Opening & Welcome
  • 1:10 - 1:50PM -- Keynote 1: Deep Reinforcement Learning in Ride-sharing Marketplace
  • Dr.Zhiwei (Tony) Qin, AI Research Lead, DiDi Labs - Silicon Valley
  • With the rising prevalence of smart mobile phones in our daily life, online ride-hailing platforms have emerged as a viable solution to provide more timely and personalized transportation service, led by such companies as DiDi, Uber, and Lyft. These platforms also allow idle vehicle vacancy to be more effectively utilized to meet the growing need of on-demand transportation, by connecting potential mobility requests to eligible drivers. In this talk, we will discuss our train of research on ride-hailing marketplace optimization at DiDi, in particular, order dispatching and driver repositioning. We will show single-agent and multi-agent RL formulations and how value function can be designed to leverage different amount of information and also facilitate knowledge transfer.
  • 1:50 - 2:30PM -- Keynote 2: Practical solutions to real-world reinforcement learning problems
  • Dr.Eytan Bakshy, Senior Scientist, Adaptive Experimentation group, Facebook
  • Rapid progress in deep reinforcement learning has produced stunning achievements in controlled environments, yet many challenges arise when attempting to apply such methods to real-world RL problems. Using examples from Facebook, I will discuss several problems faced by practitioners who aim to apply RL to their own situations. These include issues with problem specification, safety, off-policy evaluation, deployment, validation, and human factors. I will present recent work from my group which address these concerns: counterfactual inference, deep Bayesian modeling, Bayesian optimization, and opinionated tooling.
  • 2:30 - 3:00PM -- Coffee Break
  • 3:00 - 3:40PM -- Keynote 3: Improving Mild Cognitive Impairment Prediction via Reinforcement Learning
  • Dr.Jiayu Zhou, Assistant Professor, Department of Computer Science and Engineering, Michigan State University
  • Mild cognitive impairment (MCI) is a prodromal phase in the progression from normal aging to dementia, especially Alzheimer's disease. Even though there is a mild cognitive decline in MCI patients, they have normal overall cognition and thus is challenging to distinguish from normal aging. Using transcribed data obtained from recorded conversational interactions between participants and trained interviewers, and applying supervised learning models to these data, a recent clinical trial has shown a promising result in differentiating MCI from normal aging. However, the substantial amount of interactions with medical staff can still incur significant medical care expenses in practice. We propose a novel reinforcement learning framework to train an efficient dialogue agent on existing transcripts from clinical trials. Specifically, the agent is trained to sketch disease-specific lexical probability distribution, and thus to converse in a way that maximizes the diagnosis accuracy and minimizes the number of conversation turns. We evaluate the performance of the proposed reinforcement learning framework on the MCI diagnosis from a real clinical trial. The results show that while using only a few turns of conversation, our framework can significantly outperform state-of-the-art supervised learning approaches.
  • 3:40 - 4:20PM -- Keynote 4: From Importance Weighting to Reinforcement Learning: A Story of Pattern Exploration in Recommenders
  • Dr.Ed H. Chi, Principal Scientist and Research Lead, Brain Team, Google Research
  • What does Boltzmann have to do with Missing Data? I will take us on a journey connecting Boltzmann distribution and partition functions with importance weighting for learning better softmax functions in recommenders, and then further to reinforcement learning, where we can plan better explorations using off-policy correction with policy gradient approaches. These techniques enable us to reason about missing data features, labels, and patterns from our data.
  • 4:20 - 5:00PM -- Poster Session
  • Keynote Speakers


    Zhiwei (Tony) Qin, AI Research Lead, DiDi Labs


    Eytan Bakshy Senior Scientist, Facebook


    Jiayu Zhou Assistant Professor, Michigan State University


    Ed H. Chi Principal Scientist and Research Lead, Google Research

    ACCEPTED PAPERS Poster Instruction

  • Rel4KC: A Reinforcement Learning Agent for Knowledge Graph Completion and Validation [PDF]
  • Authors: Xiao Lin, Pero Subasic and Hongfeng Yin
  • From AlphaGo Zero to 2048 [PDF]
  • Authors: Yulin Zhou
  • Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform [PDF]
  • Authors: Edoardo Conti and Jason Gauci
  • Reinforcement Learning Driven Heuristic Optimization [PDF]
  • Authors: Qingpeng Cai, Will Hang, Azalia Mirhoseini, George Tucker, Jingtao Wang and Wei Wei
  • Deep Reinforcement Learning for List-wise Recommendations [PDF]
  • Authors: Xiangyu Zhao, Liang Zhang, Long Xia, Zhuoye Ding, Dawei Yin and Jiliang Tang
  • Deep Reinforcement Learning for Traffic Signal Control along Arterials [PDF]
  • Authors: Hua Wei, Chacha Chen, Kan Wu, Guanjie Zheng, Zhengyao Yu, Vikash Gayah and Zhenhui Jessie Li
  • Fair and Explainable Heavy-tailed Solutions of Option Prices through Reinforcement, Deep, and EM Learnings [PDF]
  • Authors: Chansoo Kim and Byoungseon Choi
  • CuSH: Cognitive ScHeduler for Heterogeneous High Performance Computing System [PDF]
  • Authors: Giacomo Domeniconi, Eun Kyung Lee and Alessandro Morari


    Jiliang Tang Michigan State University


    Dawei Yin


    Long Xia


    Alex Beutel Google Brain


    Minmin Chen Google Brain


    Shaili Jain Facebook


    Xiangyu Zhao Michigan State University