tpctc2015/default5.asp

TPC Benchmark Status - November 2015
Overview

Published about 3 times per year, the TPC Benchmark Status Report is a digest of the activities of the TPC and its technical subcommittees.

The TPC held meetings in June in Waikoloa, HI, in August in Austin, TX, and in November in Palo Alto, CA. The TPC-DS specification were updated following enhancements and minor corrections and clarifications, and a Hadoop friendly V2 has been approved. A new Express benchmark, TPCx-V v1.0 was also released. The TPC’s seventh Technology Conference, TPCTC 2015, was held in conjunction with the annual conference on Very Large Databases (VLDB), in Waikoloa, HI. Lastly, the TPC would like your feedback on which benchmark opportunities to explore.

 

Updates to Current Benchmarks
TPC-DS

The maintenance subcommittee released Versions 1.4.0 and 1.4.1 of TPC-DS which included wording clarifications and editorial changes.

The subcommittee also finalized a version 2.0 of TPC-DS that can be run on Hive/Hadoop platforms, which was approved by mail ballot. The new version does not require the database product to enforce Atomicity, Consistency, Isolation and Durability. But it requires enforcement of Data Accessibility. All 99 SQL queries are part of this benchmark. Data maintenance portion of the workload is simplified to adjust for common practice on Hive/Hadoop platforms, i.e. it includes the inserting and deletion of fact table data. This was updated to version 2.1 to incorporate editorial changes to the documentation and bug fixes to queries and tools. The new version is available immediately

 
TPCx-HS

TPCx-HS is the TPC’s first Express Benchmark™ – developed to provide verifiable performance,  price / performance, availability, and optional energy consumption metrics of big data systems. Express-class benchmarks are being developed in response to overwhelming demand for a turnkey alternative to enterprise-class benchmarks, which have distinct advantages but are also substantially more time-intensive and costly to run.

The TPC approved an auditor for TPCx-HS.

 

TPCx-V

The TPC approved version 1.0.0 of the TPCx-V Benchmark Standard, the second TPC Express Benchmark™. The TPCx-V benchmark measures the performance of a server running virtualized databases. It is similar to previous virtualization benchmarks in that it has many VMs running different workloads. It is also similar to previous TPC benchmarks in that it uses the schema and transactions of the TPC-E benchmark. But TPCx-V is unique since unlike previous virtualization benchmarks, it has a database-centric workload, and models many properties of cloud servers, such as multiple VMs running at different load demand levels, and large fluctuations in the load level of each VM. Unlike previous TPC benchmarks, TPCx-V will have a publicly-available, end-to-end benchmarking kit.

 
Pricing

The TPC-Pricing Maintenance Subcommittee has continued discussing and developing wording to address the pricing of TPC workloads on cloud environments. The subcommittee is close to a release of the revised specification. The subcommittee is targeting January 2016 for the formal review.

 

Benchmarks Under Development
TPCx-BB (BigBench)

The TPC-BigBench subcommittee was created in November 2014 to investigate the development of a benchmark based on the Intel BigBench proposal. The Working Group is making progress on a final specification, with version 0.0.17 under formal review.

 

Other TPC Activities
TPC Technology Conference

The TPC remains committed to developing new benchmark standards to keep pace, and one vehicle for achieving this objective is the sponsorship of the Technology Conference on Performance Evaluation and Benchmarking (TPCTC). Over the last five years we have held TPCTC successfully in conjunction with VLDB. With the TPCTC 2015 conference, in conjuction with with VLDB 2015, the TPC invited researchers and industry experts to submit novel ideas and methodologies in performance evaluation, measurement, and characterization. The conference was held on August 31, 2015 in Waikoloa, Hawaii.

The TPC has approved spponsorship of TPCTC 2016 which will also be held in conjunction with VLDB 2016.

 
Survey

The TPC would like your assessment on which future benchmarks would have the most relevance to you or your company. Please take the survey. The survey will remain open until January 15, 2016.

The opportunities include:

Opportunity Definition Examples
Data Analytics (DA) is the science of examining raw data with the purpose of drawing conclusions about that information. Data analytics is used in many industries to allow companies and organization to make better business decisions and in the sciences to verify or disprove existing models or theories Machine learning
Non-SQL or No-SQL Non relational databases provide a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases. Motivations for this approach: simplicity of design, simpler "horizontal” scaling to clusters of machines, which is a problem for relational databases, and finer control over availability. Big data and real-time web applications
In-memory

An in-memory database (also main memory database system or memory resident database) is a database management system that primarily relies on main memory for computer data storage. Characteristics: High volume low latency with ACID properties, spiky behavior (must consume/eject the data), little or no locking, different data structures. Size of database is limited by the amount of memory. NOT a large cache for an on disk database

 Online gambling site, toll booth, SQL server in memory, Oracle times ten, SAP Hana
Cloud (private or public) On premise cloud (within the corporate firewall, run by the company IT dept), IaaS (infrastructure as a service) / PaaS (platform as a service) / SaaS (software as a service), Open Stack. Google perfkit is a harness workload
Data Streaming Streaming with realtime queries and analytics Traffic (with sensors in every car), medical monitoring

 

 
Other Updates

The TPC has updated its Policies along with its End User License Agreement (EULA) and Contributor License Agreement (CLA).

The TPC continues to release new benchmarks. To receive notifications when results are published, please add yourself to the mailing list at http://www.tpc.org/information/about/mailinglist_signup5.asp. There you will be able to specify which benchmarks you wish to be notified about.

 
All Benchmark Status Reports