TPC Benchmark Status
November 1, 1998 |
By Kim Shanley, TPC Administrator
|
This is our inaugural issue of the TPC Benchmark Status, which will be published every two months. The first and primary purpose of the newsletter is to keep interested parties informed about the content, issues, and schedule of the TPC's benchmark development efforts. The second purpose is to invite new members to join these important development efforts. I’ve already outlined most of the reasons for joining the TPC in another article,
Why Join. I invite all readers of the newsletter to contact me directly at
info@tpc.org, and I will be more than happy to answer any questions you might have. Without further ado, then.
|
| Why New Versions of Benchmarks? |
This report will tell you about the TPC's new benchmark development efforts and its work in revising its current set of benchmarks. What's wrong with the old versions? Here's what's wrong.
|
- Benchmarks have a certain shelf-life. Hardware and software technology advances continually undermine the relevancy of a benchmark workload. In addition, once a benchmark has been published, companies optimize their systems to run the benchmark faster. There's nothing wrong with this--at least at first--as this process results in real-world performance increase for the average user. However, after a point, this process results in diminishing returns to the companies running the benchmarks and to the useful technology advances derived from the benchmarking exercise.
- Benchmarking is an experimental science. The goal of TPC benchmark workloads is to create a representative model of a complex, real-world workload. Benchmarks, then, are to performance engineers what experiments are to scientists. Just as the goal of running an experiment is not to ascertain any final truth, the goal of running a benchmark is not to define a completely representative model of the real-world once in for all. Benchmark workloads are developed with a complex array of experimental data, informed hypotheses and yes, educated guesses. As the benchmark results are submitted year after year, the performance engineers start to see where their original design was accurate and where it might be flawed--or even slightly flawed. When the evidence for a disparity between the model and reality reaches a certain critical mass, we see the call go out for a new version of the workload.
|
| The Next Version of TPC-D Submitted for TPC Approval |
Version 1 of TPC-D, the TPC's decision support benchmark, was approved in April 1995. The TPC-D Subcommittee, chaired by Susanne Englert of Compaq, has been working for some time on Version 2 of the benchmark. The goals of Version 2 are as follows:
|
- Requiring multiple simultaneous query streams, as the vast majority of real decision support systems must process query streams from more than a single user at a time.
- A simpler single reported performance metric, making it easier to compare results from different vendors.
- Increasing the number of queries from 17 to 22, to add additional query functionality and complexity to the workload.
- Reducing the number of query variants from which test sponsors are allowed to choose. This increases the comparability of the various results.
- More realistic requirements with respect to database durability.
- More efficient parallel generation of data (using the DBGEN program) for database load.
|
| TPC-D Version 2 Timetable |
At the September 1998 General Council Meeting in Portland, Oregon, the TPC’s General Council approved the submission of TPC-D Version 2 for approval by the general membership. The mail ballot concludes in late November 1998. If approved, TPC-D Version 2 will become effective in late January 1999. Thereafter, all TPC-D benchmark tests must be run in accordance with Version 2 of the benchmark specification. Version 1 results will remain official for six more months, and then will be removed from the official results list.
|
| TPC-D Version 3—Data Mart Workload and More |
TPC-D Version 2 makes virtually no changes to the original data warehouse schema. For the next major revision of the benchmark (Version 3), the TPC-D technical development subcommittee plans to substantially expand the warehouse schema to include multiple subject areas.
In addition, the subcommittee has committed to incorporating a new data mart workload into the benchmark. Data marts are characterized by a different type of database organization ("star" or "snowflake") that is well suited to a more limited class of queries representing particular types of business questions about a smaller subject area. Data marts are typically extracted or refreshed from processed warehouse or OLTP data and are not updated while being queried. For Version 3, the subcommittee envisions one or more data mart schemas derived from the data warehouse schema.
"The industry needs a level playing field to objectively evaluate the performance of both data warehouse and data mart solutions," said Susanne Englert, Chair of the TPC-D Subcommittee. "The development of Version 3.0 will be a formidable technical task, and we would like to solicit the participation of new members experienced in the data mart arena. Those interested should contact the TPC via the TPC web site at webmaster@tpc.org."
|
| TPC-C, Version 4 Moving to Completion |
TPC-C, the TPC’s on-line transaction processing (OLTP) benchmark, was approved in July 1992. Since then it has passed through two major revisions, with the revision number now standing at Version 3.4. The goals of Version 4 are as follows:
|
- Provide a system level benchmark that represents balanced performance.
- Increase CPU utilization per transaction.
- Increase the read/write ratio to better model customer implementations.
- Reduce required disk storage to a reasonable configuration.
- Reduce maximum number of supported users on a given platform.
- Increase ease of benchmarking; reduce the cost of each benchmark result; reduce the time to implement.
- Increase the value of reported data and improve readability.
|
At the 9/98 General Council meeting, TPC-C Subcommittee Chair, John Fowler of IBM, reported that three evaluations of the proposed Version 4 workload were now underway among subcommittee member companies. Here is the current milestones for TPC-C Version 4.0:
|
- 4/1999: TPC-C Version 4 submitted to general membership for approval.
- 8/1999: Version 4.0 becomes official benchmark (first results can be published).
|
| New Web Commerce (TPC-W) Benchmark Moving Forward Quickly |
The TPC-W Subcommittee (composed of 20 member companies), chaired by Jerry Buggert of Unisys, is working towards the completion of a new web Ecommerce benchmark (TPC-W) by Q2 1999. TPC-W is designed to represent any business (retail store, software distribution, airline reservation, electronic stock trades, etc.) that markets and sells over the Internet. It also represents Intranet environments that use Web based transactions for internal operations. The benchmark will measure the performance of systems supporting users browsing, ordering, and conducting transaction oriented business activities. For more information, see: TPC Launches New ECommerce Transactional Web Benchmark Effort.
|
| TPC-W features 3 workloads: |
- Web shopping. Primary metric is web interactions per second (WIPS)
- Web browsing. Secondary metric: web interactions per second-browsing (WIPSB)
- Web OLTP. Secondary metric: web interactions per second-OLTP (WIPSO)
|
A number of new companies have recently joined the TPC-W development effort, including InterWorld, Netscape, and Progress Software.
|
| TPC-W Milestones |
- 6/1999: TPC-W submitted for general membership approval.
- 10/1999: TPC-W becomes an official benchmark (first results can be published).
|
| Wrap-Up |
Before I sign off, I shouldn’t neglect to mention that the
TPC recently celebrated its 10th year anniversary, making the TPC the oldest
major benchmark standards body. Stay tuned for the January issue. Again, I invite all
readers of the newsletter to contact me directly
at shanley@tpc.org.
|
| All Benchmark Status Reports |
|
|
