TPC Benchmark Status
July 1999 |
By Kim Shanley, TPC Administrator
|
TPC Benchmark Status is published about every two months.
The first and primary purpose of the newsletter is to keep interested parties informed about the content, issues, and schedule of the TPC's benchmark development efforts. The second purpose is to invite new members to join these important development efforts.
We've already outlined most of the reasons for joining the TPC in another article, Why Join.
To receive the status report by email, please click here.
|
| Last Meeting |
The TPC held a General Council meeting June 24 in Grenoble, France. The primary focus at this meeting was on TPC-W, as the Council approved TPC-W Subcommittee's plan to accelerate the completion date of the benchmark specification. To that end, the Council approved the TPC-W's Subcommittee request to move the TPC-W specification to company and public review in July of this year (originally planned for early in 2000). There weren't any major changes in direction or status in the TPC-C, TPC-R, TPC-H Subcommittees, so I will briefly recap the status of these activities.
|
| TPC-C Subcommittee |
The TPC-C Subcommittee's primary focus is on completing Version 4 of the TPC-C specification. The overall goal of Version 4 is to increase the complexity of the workload to keep pace with today's more powerful systems and the evolving transaction processing landscape. The specific goals of Version 4 are as follows:
|
- Increase CPU utilization per transaction.
- Increase the read/write ratio.
- Reduce required disk storage.
- Reduce maximum number of supported users on a given platform.
- Increase ease of benchmarking; reduce the cost of each benchmark result; reduce the time to implement.
- Increase the value of reported data and improve readability.
|
The Subcommittee members have been running prototype tests on Version 4 to ensure that it meets the goals stated above. The plan of the Subcommittee is to submit Version 4 to company and public review at the next General Council meeting, which will August 25-26 in Portland, Oregon.
|
- 8/1999: TPC-C Version 4 (V.4) submitted TPC company review.
- 12/1999: TPC-C V. 4 submitted for TPC mail ballot approval.
- 2/2000: TPC-C V. 4 approved as an official TPC benchmark.
- 4/2000: First TPC-C V. 4 results can be published.
|
| TPC-H Subcommittee |
This was the first substantial face-to-face meeting for the TPC-H Subcommittee, which was formed when TPC-D was split into two benchmarks, TPC-H (ad-hoc) and TPC-R (business reporting). See the
May 1999 Newsletter for all the details of this decision. The TPC-H Subcommittee spent most of its time attempting to resolve certain ambiguities in the TPC-H specification, which seemed to permit the use of auxiliary data structures that could have circumvented the spirit of TPC-H. Simply put, the spirit of TPC-H is "ad-hocness," meaning users should not have any pre-knowledge of the queries and the software should not be optimized in advance to run a given set of queries. As a result of these discussions, the TPC-H Subcommittee did recommend certain changes to the TPC-H specification and testing process. The Council voted that these changes were minor (did not affect the general meaning or intent of the benchmark) and created Version 1.1 of TPC-H, effective June 24, 1999.
|
| TPC-R Subcommittee |
The TPC-R Subcommittee made some very minor changes to the TPC-R specification and the Council voted to created Version 1.0.1, effective June 24, 1999.
|
| TPC-W Subcommittee |
Responding to the TPC Steering Committee and General Council's requests to accelerate the TPC-W's completion schedule, the TPC-W Subcommittee shortened the schedule six months by asking the Council to allow TPC-W to go to company and public review at this meeting. TPC-W's Chair, Jerry Buggert of Unisys, made a very thorough presentation of the TPC-W specification to the Council, which included the following highlights:
|
- TPC Benchmark W (TPC-W) is a transactional web benchmark:
- The workload is performed in a controlled Internet Commerce environment that simulates the activities of a business oriented web server.
- The workload exercises a breadth of system components associated with such environments.
- The application portrayed by the benchmark is a Retail Store on the Internet with a customer browse-and-order scenario.
- TPC-W measures how fast an Ecommerce system completes various Ecommerce-type transactions but does not include the variable response time the user might experience when connected to the server via different Internet connections.
- TPC-W features:
- The simultaneous execution of multiple transaction types that span a breadth of complexity.
- On-line transaction execution modes.
- Databases consisting of many tables with a wide variety of sizes, attributes, and relationship.
- Multiple on-line browser sessions.
- Secure browser interaction for confidential data.
- On-line secure payment authorization to an external server.
- Consistent web object update.
- Transaction integrity (ACID properties).
- Contention on data access and update.
- 24x7 operations requirement.
- Three year total cost of ownership pricing model.
- There are three workloads in the benchmark, representing different customer environments. These workloads are run serially, but no significant hardware or software changes are permitted between running the three workloads. Each of the workloads will generate a performance metric, according the following scheme:
- Primarily shopping (WIPS). Representing typical browsing, searching and ordering activities of on-line shopping.
- Browsing (WIPSB). Representing browsing activities with dynamic web page generation and searching activities.
- Web-based Ordering (WIPSO). Representing intranet and business to business secure web activities.
- All references to results must include the primary metrics, which are: WIPS rate (WIPS), the associated price per WIPS ($/WIPS), and the availability date of the priced configuration.
|
After Jerry's presentation, the Council voted to move TPC-W to company and public review. What this means is that the TPC-W specification, along with a white paper explaining the benchmark, will be made available on the TPC web site. Anyone in the public may download the specification and submit comments to the TPC (instructions will be included in the downloaded file). At the same time, TPC member companies will review the specification. Comments are due back to the Subcommittee in approximately 60 days. Here is the TPC-W schedule:
TPC-W Schedule |
- 7/1999: TPC-W Version 1 (V.1) submitted TPC company review.
- 10/1999: TPC-W V. 1 submitted for TPC mail ballot approval.
- 12/1999: TPC-W V. 1 approved as an official TPC benchmark.
- 12/1999: TPC-W V. 1 results can be published.
|
| TPC and Ecommerce |
In a matter of only two years, Ecommerce has moved from an
experiment to what one leading CEO calls "the ultimate medium for business."
Is this hype? None of the independent market researchers think so. Forrester
Research reports that business-to-business ecommerce will grow from $43.1 billion
last year to $109.3 billion this year. Jupiter Communications expects consumer
ecommerce to grow from $7.1 billion last year to $12 billion in revenues this year.
So now that we've established that Ecommerce is big and is getting bigger, the next
question is, what type of systems can best handle these new levels of Ecommerce
transactional throughput and complexity? The answer to that question will require a
complete analysis of a number of factors. But in an area like Ecommerce where there
are no objective tools of measurement, TPC-W will be an invaluable resource.
Twenty-three companies were members of the subcommittee developing the TPC-W benchmark,
including internet, ecommerce, operating system, database and hardware platform
vendors.
|
| All Benchmark Status Reports |
|
|
