Frequently Asked Questions
What is TPC-C?

TPC-C Benchmark Results

Q: What do TPC-C throughput numbers mean?

A: You must understand what the benchmark is intended to measure, before you can understand throughput. Throughput, in TPC terms, is a measure of maximum sustained system performance. In TPC-C, throughput is defined as how many New-Order transactions per minute a system generates while the system is executing four other transactions types (Payment, Order-Status, Delivery, Stock-Level). All five TPC-C transactions have a certain user response time requirement, with the New-Order transaction response time set at 5 seconds. Therefore, for a 710 tpmC number, a system is generating 710 New-Order transactions per minute while fulfilling the rest of the TPC-C transaction mix workload.

Q: What do the TPC-C's price/performance numbers mean?

A: TPC-C's price/performance numbers (e.g. $550 per tpmC) may not be what you think they are. When first analyzing the TPC price/performance numbers, most people mistakenly believe they are looking at the cost of the computer or host machine. That is just one component, and not always the major component of the TPC-C's pricing methodology. In general, TPC-C benchmarks are system-wide benchmarks, encompassing almost all cost dimensions of an entire system environment the user might purchase, including terminals, communications equipment, software (transaction monitors and database software), computer system or host, backup storage, and three years maintenance cost. Therefore, if the total system cost is $859,100 and the throughput is 1562 tpmC, the price/performance is derived by taking the price of the entire system ($859,100) divided by the performance (1562 tpmC), which equals $550 per tpmC.

Q: There are two ways to look at TPC-C results: performance and price/performance. Is one more important and how do I know which system has the best TPC result?

A: Performance or price/performance may be more important to you, depending on your needs. If your application environment demands very high, mission-critical performance, then obviously you may want to give more weight to the TPC-C's throughput metric. On the other hand, most users are shopping within a given price range and any throughput number must be balanced against the cost of the system. Generally, the best TPC-C results combine high throughput with low price/performance.

Q: With two metrics (performance and price/performance), how do I proceed with comparing different systems?

A: There is no easy answer to this. You have to decide on the general range of performance and price/performance you are looking for and the relative importance of each factor. Typically, users start their evaluation process by stating that they are interested in systems that offer, say, 4000-5000 tpmC, in a price/performance range of $300-400. Once they have extracted those systems from the TPC-C's results listing, they can proceed with a more thorough investigation of the merits of each system.

Q: How can I tell if one of the TPC-C benchmarks is relevant to my application and environment?

A: This is an extremely difficult question to answer as the range of customer application environments is almost infinite and benchmarks are necessarily abstract and simplified models of all those environments. For this reason, TPC-C benchmarks represent a good yardstick for comparing different systems rather than a precise tool for capacity planning for a given customer application environment. The best approach is to reach a deeper understanding of the benchmark and compare its model (user interaction, database design, database size, transaction complexity, processing requirements, storage/backup tests) with your own application environment. If there is a rough match, you probably have a useful and relevant tool for comparing different systems that may be installed in your environment.

Q: Can I compare TPC-C results with other TPC benchmark results, say TPC-W?

A: No. Each TPC benchmark is substantially different from any other and the results cannot be compared.

Q: I notice that some TPC-C results are labelled "withdrawn." Could you explain what that means?

A: The TPC felt that users should be made aware of what results drop from the TPC-C's official results list and why those results no longer appear. Some vendors withdraw results because they feel these results no longer have market relevance. Other vendors withdraw results after compliance to the benchmark specification has been challenged by someone within the TPC-C. Rather than defend their implementation (and perhaps expend further resources to demonstrate compliance), the vendor chooses to withdraw the result. Finally, if the Council votes that a result is non-compliant, the Council will drop the result from the official results list.

Q: How should I evaluate "estimated" TPC-C results if they are presented?

A: You should not accept "estimated" or "extrapolated" TPC-C results. The TPC-C does not recognize these results as valid or legitimate. In short, accept nothing less than the real thing.

Q: If the system I'm interested in does not appear in your benchmark results list, what do I do?

A: Call the vendor and ask for more information. They may be planning to release more TPC-C results in the near future.

Q: Why don't TPC members publish results on all their systems?

A: First, no vendor (or TPC member) is required to run and publish benchmark results. Second, vendors choose to run benchmarks that provide their customer base with results relevant to those customers. Third, vendors may choose to run one benchmark over another because they believe one gives them better performance or price/performance than another benchmark.

Q: How do I get more information beyond the TPC-C's results listing?

A: First, you can obtain a Full Disclosure Report (approximately 100 pages) from our web site under Benchmark Results. These reports provide all the documentation on the system configuration, benchmark implementation, and pricing the vendor used in producing a benchmark result. There is additional performance data contained in these reports that may help you select the right system. In addition, you may choose to contact the vendor who may be able to supply additional performance and pricing information.