| TPC-B |
| TPC-B measures throughput in terms of how many transactions per second a system can perform. (Obsolete as of 6/6/95) |
|
Summary
In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by:
|
|
Detailed Description
TPC Benchmark B - What It Means and How to Use Itby Robert J. Hanson, AT&T Global Information Solutions The Big Picture All Transaction Processing Performance Council (TPC) benchmarks are formulated to allow comparison of computer systems (hardware and software together) in the database and transaction processing market segment. There are currently three TPC benchmark specifications: TPC-A, TPC-B, and TPC-C. TPC-C, the Council's newest benchmark, is explained in another article in this Report. TPC-A, the Council's first benchmark, has been the subject of many articles. However, very little has been written about TPC-B and people often confuse it with TPC-A. This article will attempt to define what TPC-B is, how to use it, and what the results mean. What it is not First, let's look at what TPC-B is not. It is not TPC-A. The only thing TPC-B shares with TPC-A is the transaction profile and database schema, and this was done for the convenience of the users building the databases for these two tests. This does not mean that TPC-B is a subset of TPC-A, nor does it mean that TPC-B is "weaker" than TPC-A because the terminals have been left out. It is not TPC-A or TPC-C or any other benchmark and cannot be compared to them. It is not for front-end OLTP measurement: it is for back-end database server measurement. What It Is TPC Benchmark B is targeted at database management systems (DBMS) batch applications and the back-end database server market segment, either stand-alone or client-server. It can be used to measure how many total simultaneous transactions a system can handle. There are no "users," no communication lines, and no terminals used or priced other than the few needed to run the benchmark. This is analogous to electronic data processing (EDP) batch processing applications which run overnight when no customer users are logged on. Transactions are submitted by programs all executing concurrently. Each program submits one transaction, waits for its completion, and then submits another one. There is no human think time, so each program goes as fast as possible serially. Many benchmark programs can be started simultaneously, loading the system up with as many transactions as it can handle, subject to a constraining residence time (the maximum time transactions can take to run in the system). TPC-B may be used wherever simultaneous multiplexed transactions are present, and the maximum throughput performance value is to be measured. Get Stressed Out TPC Benchmark B is designed to be a stress test on the core portion of a database system. It focuses in on the portion of a system which performs the actual transaction processing, much like a magnifying glass. It stresses the CPU hardware because the transactions are not paced by human think times: each stream of transactions are generated by the benchmark programs as fast as possible. It stresses the memory and disk I/O subsystems of the hardware by causing large amounts of small read and write jobs. The system overhead associated with processing the small TPC-B transaction profile becomes apparent under this type of workload. Depending on how a system is configured, there can be much higher disk activity in TPC-B than in TPC-A. The operating system's handling of I/O buffers becomes key. TPC-B also stresses the operating system by causing unending context switches between front-end processes (the benchmark code) and back-end processes (the database manager), and by challenging the memory management and I/O drivers. It stresses the database manager because it is handling maximum-rate simultaneous streams of updates: both reads and writes are done, and logging or journaling must be performed (i.e., it is not just a simple read-only operation). Note that the purpose is to stress only the core portion of a database system, without "diluting" the activity with terminal-oriented I/O. Like all TPC benchmarks, TPC-B demands that a system demonstrate they can meet the reliability and security features of the ACID (atomicity, consistency, isolation, durability) tests. In particular, there are three durability tests. One requires the removal of power (or memory) while the system is operating fully loaded with ongoing transactions. This causes the database manager to exercise its recovery functions when power (or memory) is restored. A second test requires the failure of a disk volume containing a portion of the database. Here again, the database manager is forced to restore the database to a consistent state. The third test requires the failure of a logging or journaling device during transaction submittal, and of course the database manager must continue operating at the same stressful rate as in the performance runs. 
Database Server Tune UpMost customers already have an EDP shop complete with terminals and operators. So they may not need a benchmark which includes terminals, like TPC-A. The business question then becomes: "How do I get the most out of my shop? What upgrades do I put in place to maximize performance and productivity? How do I right-size my operation?" TPC-B may be the answer for quantifying how much back-end performance to add. It provides a focused view into the transaction-processing capacity of a system, which is extremely useful for capacity planning. It can quantify individual "boxes" in the back-end by running in the simplex configuration. It can also quantify the "box" in a networked setting by running in duplex mode (client-server). TPC-B is useful anywhere the "box" or "box plus network" piece of an EDP shop must be quantified. Performance monitoring and tuning is very easy using TPC-B. Because it is a simple suite of small programs, TPC-B provides a convenient way to put a repeatable, consistent, and stressful database load on a system. The system can then be monitored by the available performance monitoring tools. CPU utilization, I/O traffic, and storage layout can all be investigated and tuned for maximal performance. The operating system and database manager tunable parameters can be adjusted to clear resource bottlenecks. Shared memory segments, spin lock counts, logfile sizes, checkpoint intervals, etc. can all be investigated and tuned. Since TPC-B places a high rate of database activity on a system, it can be used to balance a system's configuration. Even though TPC-B does not represent all applications, other applications may perform better after some tuning work has been accomplished with TPC-B. If system balance is achieved with the maximal load of TPC-B, then less stressful database loads should be accommodated by the tuned configuration unless some exotic resource parameters are needed. For newly purchased systems, TPC-B can be used as a functional stress test (does this new box really work?) which also gives performance data. Two Heads are Better than One In laying out an OLTP customer solution with TPC-A, the question of overall balance of the configuration becomes important. Since TPC-A handles user contexts, the incoming stream of transactions will need to be sent to some number of back-end database processes. But how and how many? We have already seen (TPC Quarterly Report Issue of July 15, 1992) that a transaction monitor can multiplex transaction streams to match the processing profile of the database subsystem. For example, 1000 user terminals would present transactions with human think times and delays, and the transaction monitor will concentrate them down to a steady stream of, say, 50 concurrent processes. But how is the magic number 50 determined? Maybe it's 40? Or 60? Enter TPC-B. Since the goal of a transaction monitor is to maximize the database processing efficiency given an offered terminal context load, a test is needed to determine the maximum concurrency level of the database manager. This is exactly what TPC-B was designed for. Use TPC-B to focus in on the database subsystem performance and tune the transaction monitor with the resultant data. Then use TPC-A to test the overall OLTP configuration system performance. The Modern Server-Style Benchmark TPC-B is the TPC's official database stress test for simple-profile batch-mode applications in a back-end database server scenario. It provides answers to EDP shop capacity planning problems, client-server architectures, and SQL (or other internal database language) operations over a network. It should be used instead of TPC-A for certain test scenarios such as determining resource constraints or for database server tuning. It can also be used along with TPC-A to measure other scenarios such as setting transaction monitor tuning parameters. Use it to compare systems (hardware and software) wherever you need to focus in on the database system capabilities in your EDP situation. |
