Sixteenth TPC Technology Conference on Performance Evaluation & Benchmarking

(TPCTC 2024)
August 30, 2024
in conjunction with VLDB 2024



Conference Program

August 30, 2024
All times are local times for
Guangzhou, China
Start Time End Time Paper Information
08:30 AM 09:00 AM Setting the Stage: Benchmarking in the AI Revolution
Raghunath Nambiar
09:00 AM 10:00 AM Keynote: Evolving the TPC benchmarks for Cloud Native Databases
FeiFei Li
10:00 AM 10:30 AM Coffee Break
10:30 AM 11:00 AM PDSP-Bench: A Benchmarking System for Parallel and Distributed Stream Processing
Pratyush Agnihotri, Boris Koldehofe, Roman Heinrich, Carsten Binnig, and Manisha Luthra
11:00 AM 11:30 AM A Survey of Stream Processing System Benchmarks
Wang Yue, Martin Boissier, and Tilmann Rabl
11:30 AM 12:00 PM CrypQ: A Database Benchmark Based on Dynamic, Ever-Evolving Ethereum Data
Vincent Capol, Yuxi Liu, Haibo Xiu, and Jun Yang
12:00 PM 01:30 PM Lunch
01:30 PM 02:30 PM Panel: Benchmarking Timeseries Databases: Current State and Future Perspectives
(for more information about this panel - see below)
02:30 PM 03:00 PM StarBench: A Fresh Approach On Star Schema Benchmarking
Hanumath Rao Maduri, Ahmad Ghazal, Alain Crolotte and Yuchao Li Hamish Nicholson, Andreea Nica, Aunn Raza, Viktor Sanca, and Anastasia Ailamaki
03:00 PM 03:30 PM Coffee Break
03:30 PM 04:00 PM Web3Bench: A Web3 Based HTAP Benchmark
Ahmad Ghazal, Zhongxin Ge, Hanumath Rao Maduri, Anita Shao, Guoxin Kang, Jingpei Hu, Huaiyu Xu, Ruoxi Sun, Li Shen, and Ed Huang
04:00 PM 04:30 PM Invited Talk: Performance Evaluation of TimechoDB using TPCx-IoT
Xinhao Gu, Xinyu Tan, Jialin Qiao, Junzhi Peng, Steve Yurong Su, Pengcheng Zheng, Xiangdong Huang, Shaoxu Song, and Jianmin Wang
04:30 PM 05:00 PM A Benchmarking Machine Learning Pipelines in PostgreSQL with TPCx-AI
Leonhard Liu and Patrick Erdelt
05:00 PM 05:30 PM Evaluation Considerations of Synthetic Natural Language Datasets for Question Answering Applications
Chris Van Buren, Ajay Dholakia, Xiaotong Jiang, David Ellison, Sachin Gopal Wani, and Jieyu Lin
05:30 PM 05:45 PM Closing Remarks
Meikel Poess
Panel Discussion

Topic:
Benchmarking Timeseries Databases: Current State and Future Perspectives
  • Why do we need Benchmarks?
  • Key Characteristics of Databases for IoT Scenarios
  • Integrating AI Technologies with Database Systems for IoT Scenarios
  • TPCx-IoT Benchmarking
  • Insights from Benchmarking TimechoDB (based on Apache IoTDB)
  • Future Optimization and Development of TPCx-IoT Benchmark

Schedule:
2024-08-30, 01:30 pm to 02:30 pm

Abstract:
The Internet of Things (IoT) is revolutionizing industries, generating an unprecedented volume of timeseries data. To effectively manage and analyze this data, robust and efficient timeseries databases are essential. But how do we evaluate and compare their performance? This panel will explore the crucial role of benchmarking in the timeseries database landscape, with a special focus on the unique demands of IoT scenarios.

Join us as experts delve into the key characteristics required for databases to handle the scale and complexity of IoT data. We will examine the integration of AI technologies, such as machine learning, for advanced analytics and anomaly detection. The TPCx-IoT benchmark, specifically designed for IoT workloads, will be discussed, including its strengths and limitations. Additionally, the panel will share practical insights gained from benchmarking TimechoDB built on Apache IoTDB, an open-source timeseries database.

Looking ahead, we'll explore the future of timeseries database benchmarking, addressing potential optimizations and developments in the TPCx-IoT benchmark to keep pace with the evolving IoT landscape. This panel is a must-attend for anyone seeking to navigate the current state and future perspectives of timeseries databases and make informed decisions in this rapidly advancing field.

Panelist:
Prof. Jianmin Wang - Tsinghua University
Prof. Jianmin Wang is the Dean of the School of Software, Tsinghua University, and the Executive Director of National Engineering Research Center for Big Data Software. His research interests include unstructured data management, workflow and BPM technology, and database system. Over the years, he has made significant contributions to the field, authoring numerous influential papers and leading critical research projects. His work has garnered several accolades, including the National Science and Technology Progress Award. He has published over 100 DBLP indexed papers in Journals, such as TKDE, TSC, DMKD, CII, DKE, FGCS, and IJIIS, and in conferences, such as VLDB, SIGMOD, SIGIR, ICDE, AAAI, IJCAI, ICWS, and SAC. Beyond his academic achievements, Professor Wang plays a vital role in guiding the next generation of scholars and engineers, shaping the future of technology and software development in China and globally.
Raghu Nambiar - AMD, General Chair of TPCTC
Raghu Nambiar is a Corporate Vice President at AMD where he leads a global engineering team responsible for the strategy, roadmap, and execution for AMD’s datacenter business. Prior to joining AMD, as the CTO of the Cisco UCS business he played an instrumental role in accelerating the growth of Cisco UCS to a top datacenter-compute platform. At Hewlett-Packard, where Raghu spent his early years as a technology leader, he was responsible for several industry-first and disruptive technology solutions and a decade of performance benchmark leadership. He has published more than 75 peer-reviewed papers and book chapters, 15 books in Lecture Series in Computer Science (LNCS) and holds ten patents with several pending. He holds dual Master’s degrees from University of Massachusetts and Goa University, and an advanced management program from Stanford University.
Prof. Lei Chen - HKUST, VLDB 2024 General Chair
Prof. Lei Chen is a Chair Professor in the Data Science and Analytic Thrust at HKUST (GZ), Fellow of the IEEE, a Distinguished Member of the ACM, and also a General Chair of the VLDB 2024. Currently, Prof. Chen serves as the Dean of Information Hub, the Director of Big Data Institute at HKUST, MOE/MSRA Information Technology Key Laboratory. His research interests include Data-driven AI, knowledge graphs, blockchains, data privacy, crowdsourcing, spatial and temporal databases and query optimization on large graphs and probabilistic databases. Prof. Chen received the SIGMOD Test-of-Time Award in 2015, Best research paper award in VLDB 2022. The system developed by his team won the excellent demonstration award in VLDB 2014. Prof. Chen had served as VLDB 2019 PC Co-chair. Currently, Prof. Chen serves as Editor-in-chief of IEEE Transaction on Data and Knowledge Engineering and an executive member of the VLDB endowment.
Prof. Hongzhi Wang - Harbin Institute of Technology
Prof. Hongzhi Wang is a distinguished professor and Ph.D. advisor at Harbin Institute of Technology (HIT). He serves as the Head of the Department of Computer Science and Engineering, Director of the Center for Mass Data Computing, and is responsible for the Data Science and Big Data Technology program. He also leads the Heilongjiang Province Key Laboratory of Big Data Science and Engineering and the HIT Youth Scientist Studio. Prof. Wang is a Distinguished Member of the China Computer Federation and a Senior Member of IEEE. His research focuses on databases, big data management and analysis, and big data governance. He has published over 350 papers, with more than 100 indexed by SCI and cited over 4,000 times, and has led more than 10 projects, including National Natural Science Foundation key projects and international collaborations.
Prof. Mingsheng Long - Tsinghua University
Prof. Mingsheng Long is the Director of Institute of Software System Engineering in the School of Software at Tsinghua University. His research spans machine learning theory, algorithms and models, with persistent dedication to creating strong learning machines from big data that adapt to complex real world. He has published a long article in Nature and received a feature report, was featured on the cover of Nature Machine Intelligence, and won the Test of Time Award of IJCAI. Prof. Long received the BE and PhD degrees from Tsinghua University, in 2008 and 2014 respectively. He was a visiting researcher with UC Berkeley from 2014 to 2015. He serves as an associate editor of IEEE Transactions on Pattern Analysis and Machine Intelligence and the Artificial Intelligence Journal, and as (Senior) Area Chairs of major machine learning conferences, including ICML, NeurIPS, and ICLR.
Qiang Li - CISDI Info
Qiang Li is the Deputy General Manager and CTO of CISDI Info, a distinguished talent under the “Golden Phoenix” program in the Western (Chongqing) Science City, Deputy Director of the Chongqing Key Laboratory of Industrial Software and Cloud Innovation, an expert in intelligent manufacturing of the China Iron and Steel Association (CISA), and Vice Chairman of the Chongqing Copyright Association. He has led the development of "CISDigital IIoT Platform", which won the national championship of the China Industrial Internet Contest, and recognized by the Ministry of Industry and Information Technology (MIIT) as a national-level "Cross-industry and Cross-regional" platform. The platform serves 18 industries and has established in-depth cooperation with more than 100 benchmark enterprises.
Pengcheng Zheng - (Moderator) Timecho
Pengcheng Zheng is the Managing Director of Timecho Europe, concentrating on time series database management system and related technologies to address challenges in data management in the era of IoT. Pengcheng previously worked at the Fraunhofer Institute for Industrial Engineering in Germany, where he participated in several EU research projects and contributed to multiple publications, with a focus on Industry 4.0 and automotive industry research. Additionally, he is dedicated to open-source software advocacy, particularly focusing on Apache IoTDB and PLC4X. At Timecho, his team maintains close collaboration with academic institutions like Tsinghua University and industry partners to establish a de facto standard for next-generation time series databases.


Call For Papers

The Transaction Processing Performance Council (TPC) is a non-profit organization established in August 1988. Over the past two decades, the TPC has had a significant impact on the computing industry’s use of industry-standard benchmarks. Vendors use TPC benchmarks to illustrate performance competitiveness for their existing products, and to improve and monitor the performance of their products under development. Many buyers use TPC benchmark results as points of comparison when purchasing new computing systems. The information technology landscape is evolving at a rapid pace, challenging industry experts and researchers to develop innovative techniques for evaluation, measurement and characterization of complex systems. The TPC remains committed to developing new benchmark standards to keep pace, and one vehicle for achieving this objective is the sponsorship of the Technology Conference on Performance Evaluation and Benchmarking (TPCTC). Over the last sixteen years we have held TPCTC successfully in conjunction with VLDB.


With the sixteenth TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC 2024) proposal, we strive to excel the success of previous workshops by encouraging researchers and industry experts to present and debate novel ideas and methodologies in performance evaluation and benchmarking for emerging technology areas. Authors are invited to submit original, unpublished papers that are not currently under review for any other conference or journal. We also encourage the submission of extended abstracts, position statement papers and lessons learned in practice. The accepted papers will be published in the workshop proceedings, and selected papers will be considered for future TPC benchmark developments.
 

Topics of interest include, but are not limited to:
  • Vector Processing
  • GenAI (e.g. LLM, Stable Diffusion)
  • Hyperscale Datacenter
  • Big Data Analytics
  • Cloud Computing
  • Social Media Infrastructure
  • Internet of Things
  • Blockchain
  • Lessons learned in practice using TPC workloads
  • Database Optimizations
  • Disaggregated Data Center
  • Sustainability
  • Virtualization
  • In-memory databases
  • Complex event processing
  • Hybrid workloads
  • General enhancements to TPC workloads

Submission Guidelines
Authors are invited to submit original, unpublished papers that are not currently under review for any other conference or journal. We also encourage the submission of extended abstracts, position statement papers, and lessons learned in practice. The length of a paper should not exceed 16 pages. Papers should follow Springer's Formatting Guidelines for LNCS
All papers should be submitted electronically in PDF format to: Easychair (TPCTC24)


Important Dates
Abstract due: May 17th, 2024
Papers due: June 7th, 2024
Notification of acceptance: June 27th, 2024
Conference day: August 30th, 2024                


Conference Venue and Registration
Please visit the VLDB2024 conference web site at: http://vldb.org/2024

Proceedings
Proceedings will be published by Springer-Verlag as Lecture Notes in Computer Science (LNCS). Selected papers may be considered for future TPC benchmark developments.
LNCS logo


TPCTC 2024 Organization (not finalized yet)
General Chairs and Contacts
Raghunath Nambiar, AMD, USA, raghu.nambiar@amd.com
Meikel Poess, Oracle, USA, meikel.poess@oracle.com

Program Committee
Ajay Dholakia, Lenovo, USA
Andrew Bond, Red Hat, USA
Anil Rajput AMD, USA
Hans-Arno Jacobsen, University of Toronto, Canada
Harry Le, University of Houston, USA
John Poelman, IBM, USA
Klaus-Dieter Lange, Hewlett Packard Enterprise, USA
Michael Brey, Oracle, USA
Miro Hodak, AMD, USA
Nicholas Wakou, Dell, USA
Paul Cao, Hewlett Packard Enterprise, USA
Rodrigo D. Escobar, Univ. Texas at San Antonio, USA
Shahram Ghandeharizadeh, University of Southern California, USA
Tariq Magdon-Ismail, VMware, USA
Tilmann Rabl, Hasso Plattner Institute, Germany

Publicity Committee
Meikel Poess, Oracle, USA
Paul Cao, HPE, USA
Rodrigo Escobar, Intel, USA
Gary Little, Nutanix, USA
Nirmala Sundararajan, Dell, USA
Michael Majdalany, SBIMS, USA
Forrest Carman, Owen Media, USA
Andreas Hotea, Hotea Solutions, USA


About the TPC
The Transaction Processing Performance Council (TPC) is a non-profit organization that defines transaction processing and database benchmarks and distributes vendor-neutral performance data to the industry. Additional information is available at: tpc.org.