The DEEM workshop will be held on Sunday, 14th of June in Portland, OR in conjunction with SIGMOD/PODS 2020. DEEM brings together researchers and practitioners at the intersection of applied machine learning, data management and systems research, with the goal to discuss the arising data management issues in ML application scenarios.
The workshop solicits regular research papers describing preliminary and ongoing research results. In addition, the workshop encourages the submission of industrial experience reports of end-to-end ML deployments. Submissions can be short papers (4 pages) or long papers (up to 10 pages) following the ACM proceedings format. Please use the latest ACM paper format (2017) and change the font size to 10 pts (analogous to SIGMOD).
Follow us on twitter @deem_workshop or contact us via email at info[at]deem-workshop[dot]org. We also provide archived websites of previous versions of the workshop: DEEM 2017, DEEM 2018, DEEM 2019.
Abstract: As the deployment of automated decision tools in society continues to accelerate, their interactions with fundamental questions in law, in the social sciences, and in public policy have become impossible to ignore. Although this technology holds the promise of reducing costs, reducing errors, and improving objectivity, there is enormous potential for harm. As we train algorithms on biased data, we are amplifying, operationalizing, and, most insidiously, legitimizing the historical discrimination and opacity that the technology was in part intended to address. End-to-end data systems provide abstractions to hide complexity behind (hopefully) simpler contractual interfaces. But as these systems are deployed in complex and changing social contexts, their abstractions can reduce transparency and exacerbate equity issues: It gets easier to do the wrong thing. In this talk, I’ll argue that data systems research needs to broaden scope to explicitly model, manage, and communicate assumptions and requirements about the contexts in which they are deployed, making equity issues a first-class design consideration. I'll provide some examples of the tensions, and describe some projects in this space, including learning fair representations of heterogeneous data to combat bias and providing interactive warning labels to communicate fitness for use downstream. I’ll end with some thoughts about how academic research in this space can be conducted more responsibly.
Abstract: Machine learning (ML) applications become increasingly common in many domains. Existing ML systems to execute these workloads include numerical computing frameworks, ML algorithm libraries, and more specialized deep neural network frameworks. These systems focus primarily on efficient model training and scoring. However, the data science process is exploratory, and deals with underspecified objectives, a range of different algorithms, and a wide variety of heterogeneous data sources with various errors and inconsistencies. Therefore, additional tools are employed for data engineering, exploration, and debugging, which requires boundary crossing, unnecessary manual effort, and lacks optimization across the lifecycle. This talk introduces Apache SystemDS, an open source ML system for the end-to-end data science lifecycle from data integration, cleaning, and preparation, over local and distributed model training, to debugging and serving. In a first part, we describe the overall system architecture and major design decisions that are motivated by lessons learned from Apache SystemML. In a second part, we then share early ideas and preliminary results of selected features including (1) lineage-based reuse of intermediates, (2) federated ML over raw data, and (3) the automatic tuning of data cleaning pipelines.
Abstract: A lot of the most impactful MLSys work in recent years (e.g., Tensorflow, PyTorch, Michelangelo) has come from industry and particularly large companies. This is rather unusual compared to the field of databases where lots of high-impact work was done in academia too and this can leave academics wondering how to make a dent in this extremely interesting but also extremely competitive research area. In this talk, I will present some perspectives (supported by several examples) on why so much of the MLSys innovation has come from large companies so far and present a simple, anecdotal framework to assess whether a given MLSys problem is amenable to research in a large company, a startup, or academic setting. I will conclude by presenting several research problems, which based on this framework, are particularly well suited for academia and hope that the broader research community adopts them!
Abstract: XLA (accelerated linear algebra) is a compiler-based linear algebra execution engine. It is the backend that powers machine learning frameworks such as TensorFlow and JAX at Google, on a variety of devices including CPUs, GPUs, and TPUs. This talk will cover how ML growth has fueled accelerator architectures and the way XLA and related technologies help obtain high performance from the accelerators. Unlike traditional compilation methods, where high-level programs are first translated into a common intermediate representation (IR) and only during final code-generation architecture-specific optimizations are applied, XLA compiler maps the IR to accelerator architecture constructs much earlier. This talk will demonstrate how such early architecture mapping is essential for peak performance, how it performs on real-world workloads, and the related techniques under the hood.
Applying Machine Learning (ML) in real-world scenarios is a challenging task. In recent years, the main focus of the database community has been on creating systems and abstractions for the efficient training of ML models on large datasets. However, model training is only one of many steps in an end-to-end ML application, and a number of orthogonal data management problems arise from the large-scale use of ML, which require the attention of the data management community.
For example, data preprocessing and feature extraction workloads result in complex pipelines that often require the simultaneous execution of relational and linear algebraic operations. Next, the class of the ML model to use needs to be chosen, for that often a set of popular approaches such as linear models, decision trees and deep neural networks have to be tried out on the problem at hand. The prediction quality of such ML models heavily depends on the choice of features and hyperparameters, which are typically selected in a costly offline evaluation process, that poses huge opportunities for parallelization and optimization. Afterwards, the resulting models must be deployed and integrated into existing business workflows in a way that enables fast and efficient predictions, while still allowing for the lifecycle of models (that become stale over time) to be managed. Managing this lifecycle requires careful bookkeeping of metadata and lineage (which data was used to train this model?, which models are affected by changes in this feature) and involves methods for continuous analysis, validation, and monitoring of data and models in production. As a further complication, the resulting systems need to take the target audience of ML applications into account; this audience is very heterogeneous, ranging from analysts without programming skills that possibly prefer an easy-to-use cloud-based solution on the one hand, to teams of data processing experts and statisticians developing and deploying custom-tailored algorithms​ on​ the​ other​ hand.
Additionally, the importance of incorporating ethics and legal compliance into machine-assisted decision-making is being broadly recognized. Critical opportunities for improving data quality and representativeness, controlling for bias, and allowing humans to oversee and impact computational processes are missed if we do not consider the lifecycle stages upstream from model training and deployment. DEEM welcomes research on providing system-level support to data scientists who wish to develop and deploy responsible machine learning methods.
DEEM aims to bring together researchers and practitioners at the intersection of applied machine learning, data management and systems research, with the goal to discuss the arising data management issues in ML application scenarios. The workshop solicits regular research papers describing preliminary and ongoing research results. In addition, the workshop encourages the submission of industrial experience reports of end-to-end ML deployments.
The workshop will have two tracks for regular research papers and industrial papers. Submissions can be short papers (4 pages) or long papers (up to 10 pages). Authors are requested to prepare submissions following the ACM proceedings format. Please use the latest ACM paper format (2017) and change the font size to 10 pts (analogous to SIGMOD). DEEM is a single-blind workshop, authors must include their names and affiliations on the manuscript cover page.
Submission Website: https://cmt3.research.microsoft.com/DEEM2020
Bill Howe is an Associate Professor in the Information School and Adjunct Associate Professor in Computer Science & Engineering at University of Washington. He leads an interdisciplinary group in Responsible Data Science with emphasis on urban applications. His group's research aims to make the techniques and technologies of data science dramatically more accessible and reliable, particularly at scale. Their applied methods are rooted in database models and languages, though they sometimes work in machine learning, visualization, HCI, and high-performance computing. His group is an applied, systems-oriented group, frequently sourcing projects through collaborations in the physical, life, and social sciences.
Amit Sabne is a software engineer at Google Brain. He works on high performance compilers for Tensor Processing Units (TPUs), named XLA. Before that, he was a software engineer at Microsoft, working in the Visual C++ compiler team. He researched and designed novel optimization techniques to improve program performance while lowering binary size and compilation time. He earned a PhD in Computer Engineering from the School of Electrical and Computer Engineering, Purdue. His PhD research area was High Performance Computing, with a focus on heterogeneous computing systems. His dissertation proposed and developed efficient programming models for accelerators, and also provided compiler and runtime support for these models and formulated fast autotuning mechanisms for accelerator programs.
Manasi Vartak is the founder and CEO of Verta.AI, which is based on her PhD work at MIT CSAIL on systems for software to streamline the process of data science and machine learning.
Previously, she was a PhD student in the Database Group at MIT. She worked on systems for the analysis of large scale data, specifically on making machine learning and visual analysis faster, interactive, and more efficient. She worked and interned at Twitter, Google, Facebook and Microsoft, and is a recipient of the Facebook PhD Fellowship and Google Anita Borg Fellowship.
Matthias Boehm is a BMVIT-endowed professor for data management at Graz University of Technology, Austria, and a research area manager for data management at the colocated Know-Center GmbH, Austria. Prior to joining TU Graz in 2018, he was a research staff member at IBM Research - Almaden, CA, USA, with a major focus on compilation and runtime techniques for declarative, large-scale machine learning in Apache SystemML. Matthias received his Ph.D. from Dresden University of Technology, Germany in 2011 with a dissertation on cost-based optimization of integration flows. His previous research also includes systems support for time series forecasting as well as in-memory indexing and query processing. Matthias is a recipient of the 2016 VLDB Best Paper Award, a 2016 SIGMOD Research Highlight Award, and a 2016 IBM Pat Goldberg Memorial Best Paper Award.
Sebastian Schelter
Steven Whang
Julia Stoyanovich