Origins of the TPC and the first 10 years |
by Kim Shanley, Transaction Processing Performance Council |
| February, 1998 |
| Preface |
In my view, the TPC's history can be best understood by focusing on two of its major organizational activities: 1) creating good benchmarks; 2) creating a good process for reviewing and monitoring those benchmarks. Good benchmarks are like good laws. They lay the foundation for civilized (fair) competition. But if we have good benchmarks, why do we need all the overhead process for reviewing and monitoring the benchmark results? Similarly, you might ask if we have good laws, why do we need police, lawyers and judges? The answer to both questions is the same. Laws and benchmarks are not, in of themselves, enough. And by this I don't mean to imply that it's simply human nature to break or bend the rules. The TPC has found that no matter how clear-cut the rules appear to be when the benchmark specifications are written, there are always gray areas, and yes, loopholes left in the benchmark law. There must be a way of addressing and resolving these gray areas and loopholes in a fair manner. And yes, even "good laws," said Aristotle, "if they are not obeyed, do not constitute good government." Therefore, there must be a means for stopping those who would break or bend the rules. While this book is primarily a technical overview of the industry's benchmarks, the TPC's history is about both benchmark law and benchmark order.
The State of Nature
In writing this early history of the TPC, I've drawn heavily upon the account by Omri Serlin published in the second edition of this handbook. It was through Omri's initiative and leadership that the TPC was founded.
In the early 1980's, the industry began a race that has accelerated over time: automation of daily end-user business transactions. The first application that received wide-spread focus was automated teller transactions (ATM), but we've seen this automation trend ripple through almost every area of business, from grocery stores to gas stations. As opposed to the batch-computing model that dominated the industry in the 1960's and 1970's, this new online model of computing had relatively unsophisticated clerks and consumers directly conducting simple update transactions against an on-line database system. Thus, the on-line transaction processing industry was born, an industry that now represents billions of dollars in annual sales.
Given the stakes--even at this point in the race--over who could claim the best OLTP system, the competition among computer vendors was intense. But, how to prove who was the best? The answer, of course, was a test--or a benchmark. Beginning in the mid-1980's, computer system and database vendors began to make performance claims based upon the TP1 benchmark, a benchmark originally developed within IBM that then found its way into the public domain. This benchmark purported to measure the performance of a system handling ATM transactions in a batch mode without the network or user interaction (think-time) components of the system workload (similar in design to what later turned out to be TPC-B). The TP1 benchmark had two major flaws. First, by ignoring the network and user interaction components of an OLTP workload, the system under test (SUT) could generate inflated performance numbers. Secondly, the benchmark was poorly defined and there was no supervision or control of the benchmark process. As a result, the TP1 marketing claims, not surprisingly, had little credibility with the press, market researchers (among them Omri Serlin), or users. The situation also deeply frustrated vendors who felt was their competitors' marketing claims, based upon flawed benchmark implementations, were ruining every vendor's credibility.
Early Attempts at Civilized Competition
In the April 1, 1985 issue of Datamation, Jim Gray in collaboration with 24 others from academy and industry, published (anonymously) an article titled, "A Measure of Transaction Processing Power." This article outlined a test for on-line transaction processing which was given the title of "DebitCredit." Unlike the TP1 benchmark, Gray's DebitCredit benchmark specified a true system-level benchmark where the network and user interaction components of the workload were included. In addition, it outlined several other key features of the benchmarking process that were later incorporated into the TPC process:
|
- Total system cost published with the performance rating. Total system cost included all hardware and software used to successfully run the benchmark, including 5 years maintenance costs. Until this concept became law in the TPC process, vendors often quote only part of the overall system cost that generated a given performance rating.
- Test specified in terms of high-level functional requirements rather than specifying any given hardware or software platform or code-level requirements. This allowed any company to run this benchmark if they could meet the functional requirements of the benchmark.
- The benchmark workload scaleup rules -- the number of users and size of the database tables -- increased proportionally with the increasing power of the system to produce higher transaction rates. The scaling prevented the workload from being overwhelmed by the rapidly increasing power of OLTP systems.
- The overall transaction rate would be constrained by a response time requirement. In DebitCredit, 95 percent of all transactions had to be completed in less than 1 second.
|
The TPC Lays Down the Law
While Gray's DebitCredit ideas were widely praised by industry opinion makers, the DebitCredit benchmark had the same success in curbing bad benchmarking as the prohibition did in stopping excessive drinking. In fact, according to industry analysts like Omri Serlin, the situation only got worse. Without a standards body to supervise the testing and publishing, vendors began to publish extraordinary marketing claims on both TP1 and DebitCredit. They often deleted key requirements in DebitCredit to improve their performance results.
From 1985 through 1988, vendors used TP1 and DebitCredit--or their own interpretation of these benchmarks--to muddy the already murky performance waters. Omri Serlin had had enough. He spearheaded a campaign to see if this mess could be straightened out. On August 10, 1988, Serlin had successfully convinced eight companies to form the Transaction Processing Performance Council (TPC).
|
| TPC-A |
Using the model and the consensus that had already developed around the DebitCredit benchmark, the TPC published its first benchmark, TPC Benchmark A (TPC-A) within one year (November 1989). TPC-A differed from DebitCredit in the following respects:
|
- The requirement that 95 percent of all transactions must complete in less than 1 second was altered to 90 percent of transactions must complete in less than 2 seconds.
- The number of emulated terminals interacting with the SUT was reduced to a requirement of 10 terminals per tps and the cost of the terminals was included in the system price.
- TPC-A could be run in a local or wide-area network configuration (DebitCredit has specified only WANs).
- The production-oriented requirements of the benchmark were strengthened to prevent the reporting of peak, unsustainable performance ratings. Specifically, the ACID requirements (atomicity, consistency, isolation, and durability) were bolstered and specific tests added to ensure ACID viability.
|
Finally, TPC-A specified that all benchmark testing data should be publicly disclosed in a Full Disclosure Report.
The first TPC-A results were announced in July 1990. Four years later, at the peak of its popularity, 33 companies were publishing on TPC benchmarks and 115 different systems had published TPC-A results. In total, about 300 TPC-A benchmark results were published.
The first TPC-A result was 33 tpsA at a cost of $25,500 per transaction or tpsA. The highest TPC-A result ever recorded was 3,692 tpsA with a cost of $4,873 per tpsA. In summary, the highest tpsA rating had increased by a whopping factor of 111 times and the price/performance had improved by a factor of five. Does this increase in the top tpsA ratings correspond to an identical increase in the real world performance of OLTP systems during this period? Even in keeping in mind that this is a comparison of peak, not average benchmark ratings, the answer is no. The increase in tpsA ratings is just too great. The increase can be attributed to four major reasons: 1) the first benchmark test is usually run for bragging rights and is grossly unoptimized compared to later results; 2) real performance increases of hardware and software products; 3) vendors improving their products to eliminate performance bugs exposed by the benchmark, 4) vendors playing the benchmarking game effectively--learning from each other on how best to run the benchmark. So yes, there is a gamesmanship aspect to the TPC benchmark competition, but it should not obscure the fact that TPC benchmarks have provided an objective measure of a truly vast increase in computing power of hardware and software during this period. Indeed, the benchmarks have accelerated some of these software improvements. And all the marketing gamesmanship should also not invalidate the legacy achievement of TPC-A that for the first time, it provided the industry with an objective and standard means of comparing the performance of a vast number of systems.
|
| TPC-B |
While TPC-A leveraged the industry consensus built up over DebitCredit, the TPC's was much more ambiguous about publishing a benchmark around the TP1 model. The ambiguity about what was to turn into the TPC's second benchmark, the TPC-B benchmark, lasted throughout the life of the TPC-B. Everyone within the TPC organization believed that one of TPC-A's principal strengths was that it was an end-to-end system benchmark that exercised all aspects of an OLTP system. Furthermore, this OLTP system representing users working on terminals, conducting simple transactions over a LAN connected to a database server, was a model of computing that everyone could intuitively understand.
As described earlier, TP1 (and later TPC-B) was the batch version of DebitCredit, without the network and user interaction (terminals) figured into the workload. A strong block of companies within the TPC, including hardware companies who sold "servers" (as opposed to end-to-end system solutions) and database software companies, felt that the TPC-B model was more representative of the customer environments they sold into. The anti-TPC-B crowd, on the other hand, argued that the partial-system model that TPC-B represented would reduced the stress on key system resources and would therefore produce artificially high transaction rates. In addition, while the tps rates would be artificially high, the total system cost since the network and terminal pricing would be eliminated would be artificially low, thereby artificially boosting TPC-B's price/performance ratings. Finally, the anti-TPC-B spokespeople argued, since TPC-A and TPC-B were so much alike, using identical a "tps" throughput rating system, the users and press would be confused. Whether they won the argument or not can be debated, but what cannot is that TPC-B proponents eventually won the day and in August 1990, TPC-B was published as the second TPC benchmark standard. TPC-B used the same TPC-A transaction type (banking transaction) but it cut out the network and user interaction component of the TPC-A workload. What was left was a batch transaction processing benchmark.
The first TPC-B results were published in mid-1991 and by June 1994, at the peak of its popularity, TPC-B results had been published on 73 systems. In total, about 130 TPC-B tests were published. In all, there were about 2.5 times more TPC-A results published. The first TPC-B result was 102.94 tpsB with a cost of $4,167 per tpsB. The highest TPC-B rating was a 2,025 tpsB result, and the best price/performance number was $254 per tpsB. In summary, the top TPC-B ratings increased by a factor of 19 and the price performance rating improved by a factor of 16.
It's more difficult to give the legacy of TPC-B the unconditional stamp of success. It never received the user or market analyst acceptance that TPC-A did, but many within the Council and the industry perceived a real value in this "database server" benchmark model. This belief fueled a later failed TPC efforts around TPC-S (more on this later), and continues to influence the development of TPC benchmarks today. In January 1998, the TPC announced the formation of a Web Commerce benchmark (TPC-W) which will measure OLTP and browsing performance only of the web server, excluding the network and human interaction components of the overall system. So, TPC-B proponents may not have won the public debate on the merits of TPC-A versus TPC-B, but they may take some measure of satisfaction that the back-end model of benchmarking lives on.
Political Reform Begins Immediately
The TPC was a major improvement over the state of nature that existed previously. However, as the Council was to learn, most of the work of building a successful benchmarking organization and process was ahead of them. In this sense, the early TPC was not unlike the early American colonies who idealistically believed that they could eliminate the endless political and legal conflict of the Old Word by passing laws abolishing lawyers. While such a law still appeals to a significant minority, in our more judicious moments, most would agree that it's not possible.
As soon as vendors began to publish TPC results, complaints from rival vendors began to surface. Every TPC result had to be accompanied by a Full Disclosure Report (FDR). But, what happened when people reviewed the FDR and didn't like what they read? How could protest be registered and how would it be adjudicated? Even if a member of the public or a vendor representative were, so to speak, make a citizen's arrest of a benchmark violator, there was no police or court system to turn the perpetrator over to for further investigation or if need be, prosecution. It became apparent to the Council that without an active process for reviewing and challenging benchmark compliance, there was no way that the TPC could guarantee the level playing field that the TPC had promised the industry.
Throughout 1990 and 1991, the TPC embarked on a political journey to fix this hole in its process. The Technical Advisory Board (TAB), which was originally constituted as just an advisory board, became the arm of the TPC where the public or companies could challenge published TPC benchmarks. The TAB process, which remains in place today, established a fair, deliberative mechanism for reviewing benchmark compliance challenges. Once the TAB has thoroughly researched and reviewed a challenge, the TAB makes a recommendation to the full Council. The full Council then hears the TAB's report, discusses and debates the challenge, and then votes on the challenge. If the Council finds the result non-compliant in a significant or major way, the result is immediately removed as an official TPC result.
Benchmarking Versus Benchmarketing
By the spring of 1991, the TPC was clearly a success. Dozens of companies were running multiple TPC-A and TPC-B results. Not surprisingly, these companies wanted to capitalize on the TPC's cachet and leverage the investment they had made in TPC benchmarking. Several companies launched aggressive advertising and public relations campaigns based around their TPC results. In many ways, this was exactly why the TPC was created: to provide objective measures of performance. What was wrong, therefore, with companies wanting to brag about their good results? What was wrong is that there was often a large gap between the objective benchmark results and their benchmark marketing claims--this gap, over the years, has been dubbed "benchmarketing."
So the TPC was faced with an ironic situation. It had poured an enormous amount of time and energy into creating good benchmark and even a good benchmark review process. However, the TPC had no means to control how those results were used once they were approved. The resulting problems generated intense debates within the TPC.
Out of these Council debates emerged the TPC's Fair Use policies adopted in June, 1991.
|
- When TPC results are used in publicity, the use is expected to adhere to basic standards of fidelity, candor, and due diligence, the qualities that together add up to, and define, Fair Use of TPC Results.
- Fidelity: Adherence to facts; accuracy
- Candor: Above-boardness; needful completeness
- Due Diligence: Care for integrity of TPC results
|
Have the TPC's Fair Use policies worked? By and large, they have been effective in stopping blatantly misuse or misappropriation of the TPC's trademark and good name. In other words, very few companies claim TPC results when, in fact, they don't have them. In general, TPC member companies have done a fair job in policing themselves to stop or correct fair use violations that have occurred. At times, the TPC has acted strongly, issuing cease and retraction orders, or levying fines for major violations.
It must be said, however, that there remains today among the press, market researchers, and users, a sense that the TPC hasn't gone far enough in stamping out benchmarketing. This issue has two sides. On the one hand, companies spend hundreds of thousands of dollars, even millions of dollars, running TPC benchmarks to demonstrate objective performance results. It's quite legitimate, therefore, for these companies to market the results of these tests and compare them with the results of their competitors. On the other hand, no company has a right to misrepresent or mislead the public, regardless of how legitimate the benchmark tests may be. So where does the "war" on benchmarketing stand today? Much like the war on crime, the war on benchmarketing persists, and the TPC continues to wage an active campaign to eliminate it.
Codifying the Spirit of the Law
With the creation of a good review and fair use process, and with dozens of companies publishing regularly on the TPC-A and TPC-B benchmarks, the TPC may be forgiven for lapsing into a self-satisfied belief that the road ahead was smooth. That sense of well-being was torpedoed in April, 1993 when the Standish Group, a Massachusetts-based consulting firm, charged that Oracle had added a special option (discrete transactions) to its database software, with the sole purpose of inflating Oracle's TPC-A results. The Standish Group claimed that Oracle had "violated the spirit of the TPC" because the discrete transaction option was something a typical customer wouldn't use and was, therefore, a benchmark special. Oracle vehemently rejected the accusation, stating, with some justification, that they had followed the letter of the law in the benchmark specifications. Oracle argued that since benchmark specials, much less the spirit of the TPC, were not addressed in the TPC benchmark specifications, it was unfair to accuse them of violating anything.
The benchmarking process, which sprang from the discredited TP1 and DebitCredit days, has always been treated with a fair degree of skepticism by the press. So the Standish Group's charges against Oracle and the TPC attracted broad press coverage. Headlines like the May 17, 1993 issue of Network World were not uncommon, "Report Finds Oracle TPC results to be misleading; says option discredits TPC-A as benchmark."
Whether Oracle's discrete transition option was truly a benchmark special was never formally discussed or decided by the TPC. The historical relevance of this incident was that it spurred the TPC into instituting several major changes to its benchmark review process.
|
| New Anti-Benchmark Special Prohibition |
TPC benchmark rules had always required companies to run the benchmark tests commercially available software. However, after the Standish Group charges, the Council realized that it had no real protection from companies that purposely designed a benchmark special component into their commercially available software. In other words, this special component could be buried in some obscure corner of overall product code and only be used when the vendor wanted to run a TPC test. If the TPC was formed to create fair, relevant measures of performance, then yes, the benchmark special was a violation of the TPC's spirit and thus had to be prohibited. In September, 1993, the Council passed Clause 0.2, which contains the sweeping prohibition against benchmark specials that become a bedrock of the TPC process to ensure, relevant benchmarks.
The Council drew a line in the sand with the passing of the Clause 0.2 anti-benchmark prohibition that has become part of the bedrock of the TPC process to ensure fair, relevant benchmarks:
|
- Specifically prohibited are benchmark systems, products, technologies or pricing...whose primary purpose is performance optimization of TPC benchmark results without any corresponding applicability to real-world applications and environments. In other words, all "benchmark special" implementations that improve benchmark results but not real-world performance or pricing, are prohibited.
|
Clause 0.2 in TPC-A and TPC-B went into effect in June, 1994. Oracle decided not to test its discrete transaction option against the new anti-benchmark special rules in the specifications and withdrew all of its results by October, 1994. Let it also be noted that Oracle remains a TPC member and strong supporter of the organization.
|
| New TPC Auditing Process |
As a result of the 1993 controversies, the TPC realized that the millions of dollars being invested in the running of TPC benchmarks would be completely wasted if the credibility of the results were challenged. The TPC's process of FDR review was fine, but it only was invoked after a result was published and publicized. Yes, the TPC could yank a result from the official results list after it was found to be non-compliant, and even fine a company for violating the specifications, but the damage to the company's competitors and TPC's credibility would already have been done. In summary, it wasn't enough to catch the bad horse after it had left the barn. The goal was to stop the bad horse from ever getting out of the barn.
The result of these discussions, passed in September and December, 1993, was the creation of a group of TPC certified auditors who would review and approve every TPC benchmark test and result before it was even submitted to the TPC as an official benchmark or publicized. While TPC benchmarks are still reviewed and challenged on a regular basis, the TPC auditing system has been very effective in preventing most of the bad horses from ever leaving the barn.
|
| New and Better Benchmarks |
From the outset, I have said that the TPC's history is both about benchmark law and benchmark order. From the last sections of this chapter, the reader might have received the false impression that the TPC is exclusively a political organization endlessly embroiled in public controversies and institutional reform. The "benchmark order" activity of the TPC is certainly important, but the TPC's day-to-day focus is to build better benchmarks.
TPC-A was a major accomplishment in bringing order out of chaos, but TPC-A was primarily a codification of the simplistic TP1 and DebitCredit workloads created in the mid-1980's. However, what was very clear even as the TPC members approved TPC-A in late 1989 was that better, more robust and realistic workloads would be required for the 1990's.
Two benchmark activities were launched in 1990, the development of TPC-C, the next generation OLTP benchmark, and TPC-D, a decision support benchmark.
Both TPC-C, which was approved as a new benchmark in July, 1992 and TPC-D, which was approved in April, 1994, are covered in other chapters of this book and therefore I'll only add a few comments on these benchmarks.
The first TPC-C result published in September, 1992 was 54 tpmC result with a cost per tpmC of $188,562. As of this date (January 1998) more than six years later, the top result is a 52,871 tpmC with a cost per tpmC of $135. We have witnessed the same tremendous improvement in the top TPC-C numbers as we did for TPC-A and for the same reasons: 1) real world performance and cost improvements and 2) an increased knowledge about how to run the benchmark. (Again, keep in mind that just by looking at peak numbers, and not averages, we're seeing an exaggerated inflationary effect). Currently, there 143 official TPC-C results, a higher total than TPC-A at the peak of its popularity.
The first TPC-D result was a 100 GB result in December, 1995 with a throughput performance rating of 84 QthD and a price/performance rating of $52,170 QphD. Today, the top 100 GB throughput result that has been produced is 1205 QthD and $1877 QphD. Currently, there are 28 official TPC-D results. Why so few? TPC-D is only 2.5 years old, compared to TPC-C's venerable 6 years, and TPC-D is more expensive and complex to run.
Both TPC-C and TPC-D have gained widespread acceptance as the industry's premier benchmarks in their respective fields (OLTP and Decision Support). But the increase in the power of computing systems is relentless, and benchmark workload must continually be enhanced to keep them relevant to real world performance. Currently, a new major revision of TPC-C is being planned for release in early 1999. A new major revision of TPC-D is being planned for mid-1998 and another one in 1999.
|
|
