TPC photo
The TPC defines transaction processing and database benchmarks and delivers trusted results to the industry
    Document Search         Member Login    
     Home
About the TPC
Benchmarks
      Newsletter
      Join the TPC
      Downloads
      Technical Articles
      TPCTC
      Performance-Pulse

TPC Benchmark Status
November 2008
Overview

Published 4 times per year, the TPC Benchmark Status Report is a digest of the activities of the TPC and its technical subcommittees.

The TPC held a General Council meeting on September 11 in Minneapolis, MN and June 6th in Frankfurt, Germany. On the OLTP front, the TPC-E Maintenance Subcommittee made editorial changes and clarifications to the TPC-E benchmark, resulting in Version 1.6.0, while the TPC-C Maintenance Subcommittee continued work on reconciling the list of Technical Advisory Board interpretations. With respect to decision support, important revisions were made to TPC-H resulting in Version 2.8.0. In addition, the TPC-DS specification draft is going through the TPC's evaluation process. Finally, the TPC has formed the Energy subcommittee to develop methodology to add energy metrics to all the existing TPC Benchmarks.

 

TPC 20th Anniversary

This year marks the 20th anniversary of the Transaction Processing Performance Council. The TPC was established in August 1988 by eight leading software and hardware companies; today there are 25 member companies that encompass major systems and software vendors throughout the industry.

Over its 20 year history, the TPC has developed benchmarks in the areas of Online Transaction Processing (OLTP), Decision Support, and Web/Application Servers including: TPC-A, TPC-B, TPC-C, TPC-D, TPC-H, TPC-R, TPC-W, TPC-App, and TPC-E. The TPC's volume of published results has risen significantly, along with the industry acceptance of TPC benchmarks. To date, more than 900 benchmark results have been published, and approximately 55 TPC benchmark results are published by member and non-member companies each year.

With its two-decade history, the TPC has developed several benchmark elements that define the organization and have earned it the reputation of providing the most credible results in the industry:

  • Consistent price/performance - the TPC is the only organization that provides consistent scores that enable end users to clearly understand the actual cost of performance of a system
  • Extensive auditing process - the TPC has extensive fair use policies and requires an independent audit of all results prior to publication
  • Broad industry support - the TPC has the support of major systems and software vendors, and benchmarks are not exclusive to any particular system or hardware
  • Whole system testing - the TPC tests entire system performance, not just one component, thus providing end users with cross platform performance comparisons

The TPC is conducting a series of workshops which will be held in various locations throughout the United States. At the workshop, the TPC will provide an overview of its current benchmarks and how they may be used in making server procurement decisions. The TPC will also preview several benchmarks that are in development for release next year, and will explore several promising areas including energy consumption, ETL (Extract/Transform/Load) and service-oriented architecture. If you would like to attend one of the workshops, you can express your interest here.

 

Current Benchmarks
TPC-C

The TPC-C subcommittee addressed three wording changes to the specification. The subcommittee modified the wording in Clause 3.4.2.9 to correct a reference in one of the isolation tests. The subcommittee also addressed modifying the graph in Clause 5.6.4 to better illustrate what is required to be reported in the FDR. In addition, wording is being developed for the Numerical Quantities (Clause 8.1.1.3) to ensure consistency across publications. These wording changes will be included in a future revision of the TPC-C Specification.  

 
TPC-H

The TPC-H maintenance subcommittee specified which component can be substituted in Clause 7.2 and updated dangling references in clauses 8.4.2.2, 9.2.6.1, 5.4.6 and 5.4. These changes resulted in Version 2.8.0 of TPC-H. Benchmark results can be published against Version 2.8.0 immediately. Version 2.8.0 will become mandatory 60 days beginning November 9, 2008.

 
TPC-App

The TPC-App subcommittee did not convene at these meetings. The subcommittee will continue to monitor the benchmarks, along with any interpretations and rulings, for discussion and possible inclusion in future versions of the specification.

 
TPC-E

The TPC-E subcommittee made changes to the procedure to calculate Business Recovery Time so that the procedure covered database clusters. The subcommittee also made changes to EGen to speed up data generation. The subcommittee also worked on some improvements to EGen. One of those improvements was to add status checking to EGenTxnHarness. Another improvement was to aid portability across platforms. The Subcommittee made some editorial changes to the TPC-E Specification and updated the specification to reflect the changes made to EGen. A new revision of the TPC-E benchmark (v1.6.0) with these changes was approved and is available immediately, and v1.5.1 becomes obsolete as of November 10, 2008. Prior results and Version 1.6.0 results are comparable. Prior results do not need to be upgraded to Version 1.6.0 results.

 

Benchmarks Under Development
TPC-DS

The TPC-DS Development Subcommittee concluded its work by resolving the remaining open comments in the comment database, performing a thorough review of the specification and addressing all cases in its tracking and reporting system.

 

Other TPC Activities
TPC Energy

The TPC Energy subcommittee has developed methodology to add energy metrics to all the existing TPC Benchmarks. This will utilize the existing TPC benchmarks as the workload and during the running of the benchmarks, measure the energy being consumed by the System Under Test. The new metrics will be of the form: Watts/Performance.

The TPC Energy Specification wording is approximately 60% complete. There remains significant work in developing procedures and measurement methodologies which require testing prior to finalization of the specification.

The subcommittee is developing code which will interface to the power analyzers and temperature probes. This code will collect the energy and temperature data of the components of the System Under Test and create a uniform log for reporting the results. The TPC will use the SPEC consortium Power and Temperature Daemon code as part of this benchmark. These programs will be available to Test Sponsors who will run the TPC-Energy option of the TPC Benchmarks.

 
Pricing Specification

The Pricing Maintenance Subcommittee has been examining specification language regarding what it means to have an orderable and/or available product. We are also reexamining specification language to ensure that our maintenance requirements match the type of offerings that are available to customers from most vendors. We anticipate a minor revision to the specification with some of these enhancements later this year. The current version of the Pricing Specification is Version 1.3.0.

 
Public Relations Committee

The TPC announced its 20th Anniversary Workshop Series. A series of analyst and press briefings were conducted to update the industry on the progress of the TPC benchmarks.

We welcome opportunities to present the benefits and status of TPC benchmarks: if you are interested in inviting someone from the TPC for a presentation, please contact Michael Majdalany (majdalany @ tpc.org)

 

 
All Benchmark Status Reports
 

Valid XHTML 1.0 Transitional Valid CSS!

©2001-2019 TPC. All rights reserved. Web-Design and Maintenance by Hotea Solutions