Get real: How benchmarks fail to represent the real world
Vogelsgesang A, Haubenschild M, Finis J, Kemper A, Leis V, Muehlbauer T, Neumann T, Then M (2018)
Publication Type: Conference contribution
Publication year: 2018
Publisher: Association for Computing Machinery, Inc
Conference Proceedings Title: Proceedings of the Workshop on Testing Database Systems, DBTest 2018
Event location: Houston, TX
ISBN: 9781450358262
Abstract
Industrial as well as academic analytics systems are usually evaluated based on well-known standard benchmarks, such as TPC-H or TPC-DS. These benchmarks test various components of the DBMS including the join optimizer, the implementation of the join and aggregation operators, concurrency control and the scheduler. However, these benchmarks fall short of evaluating the "real" challenges imposed by modern BI systems, such as Tableau, that emit machine-generated query workloads. This paper reports a comprehensive study based on a set of more than 60k real-world BI data repositories together with their generated query workload. The machine-generated workload posed by BI tools differs from the "hand-crafted" benchmark queries in multiple ways: Structurally simple relational operator trees often come with extremely complex scalar expressions such that expression evaluation becomes the limiting factor. At the same time, we also encountered much more complex relational operator trees than covered by benchmarks. This long tail in both, operator tree and expression complexity, is not adequately represented in standard benchmarks. We contribute various statistics gathered from the large dataset, e.g., data type distributions, operator frequency, string length distribution and expression complexity. We hope our study gives an impetus to database researchers and benchmark designers alike to address the relevant problems in future projects and to enable better database support for data exploration systems which become more and more important in the Big Data era.
Authors with CRIS profile
Involved external institutions
How to cite
APA:
Vogelsgesang, A., Haubenschild, M., Finis, J., Kemper, A., Leis, V., Muehlbauer, T.,... Then, M. (2018). Get real: How benchmarks fail to represent the real world. In Proceedings of the Workshop on Testing Database Systems, DBTest 2018. Houston, TX, US: Association for Computing Machinery, Inc.
MLA:
Vogelsgesang, Adrian, et al. "Get real: How benchmarks fail to represent the real world." Proceedings of the 2018 Workshop on Testing Database Systems, DBTest 2018, Houston, TX Association for Computing Machinery, Inc, 2018.
BibTeX: Download