With the petaflop barrier broken, is it time to change the benchmark?

At a presentation at the SC08 semi-annual supercomputing conference in Austin, Texas, an engineer with Oak Ridge National Laboratories in Tennessee who is an expert on the Linpack benchmark, suggested that the methodology used to determine supercomputer performance using Linmark may be behind the times. Specifically, Jack Dongarra -- the man credited with introducing the High-Performance Linpack (HPL) benchmark to the Top 500 program -- suggested that as supercomputers get bigger and can store more data, their lag times increase exponentially. This implies that making existing supercomputers bigger and faster eventually leads to a point of diminishing returns.

In the presentation (PDF available here), Dongarra states up front that it's only natural to test a supercomputer cluster with a problem size that's proportionate to its capacity. HPL expresses problem size in orders of 10; whereas desktop computers may be tested using Linpack set with a problem size of 1,000, the problem or matrix size for supercomputers is set into the millions. The #1 contender in this week's Top 500, Los Alamos National Laboratories' IBM Roadrunner, ran the HPL with a problem size (n) of 2,300,000. It was able to perform the HPL in about two hours. Oak Ridge had the #8 and #2 contenders, both of them Cray XTs and the #2 player -- Jaguar -- also beating the petaflop barrier this year.

Jaguar has more memory available to it than Roadrunner. So in one test, Oak Ridge increased the problem size proportionate to its capacity, for n of 4,700,000 (4.7 x 106). As Dongarra reported, that test took Jaguar 16 hours to complete.

At the rate at which supercomputers are presently scaling, he told attendees, it should be perfectly reasonable to adjust the problem size matrix to 33,500,000 by 2012. That would only seem fair, to scale the problem size to the extent that the clusters are scaling. But with current architectures, applying the current rate of performance falloff, the completion time could actually plummet, Dongarra predicted, to over two and a half days.

What's the solution? Dongarra's presentation showed how performance declines over the period of the entire benchmark's run, which it's supposed to do. Conceivably if a certain segment of the run were captured prior to the big dropoff late in the run, then a formula for the dropoff could be estimated based on the graph. That way, the test won't take days for each iteration...or conceivably even weeks.

But implementing the solution may mean changing the benchmark, and quite possibly impacting the results. This from a guy who works for the laboratory whose best contender placed second.

Dongarra's presentation closed with the following: "We are planning on making changes and will probably be ready after ISC in Hamburg." That's the very next supercomputing conference, slated for June 23 in Hamburg, Germany, and that's when the next Top 500 list will be published. At that time, we could see whether changing the way we measure tasks that scale up or down with their respective clusters, changes in turn the way we perceive supercomputing.

Source: BetaNews

Add comment

Your name:
Sign in with:
Your comment:

Enter code:

E-mail (not required)
E-mail will not be disclosed to the third party

Last news

Sales of new models way below those of 2017 generation
The new Windows 10 browser will run on the Chromium engine
Google will shut the service down in April of 2019 instead of August as initially planned
The regular S10 will sport a 6.1-inch panel with the same front-facing camera design
The smartphone has a 6.4-inch Full HD+ (2340 x 1080 pixel) Infinity-O display
Google Play Services will deprecate the aging OS in newer releases
Apple might be looking to trial the feature on the iPad before iPhone
Toshiba, which released the world’s first 14TB nearline 3.5-inch and 26.1mm-height HDDs with 9-disk
The Samsung Galaxy A5 (2017) Review
The evolution of the successful smartphone, now with a waterproof body and USB Type-C
February 7, 2017 /
Samsung Galaxy TabPro S - a tablet with the Windows-keyboard
The first Windows-tablet with the 12-inch display Super AMOLED
June 7, 2016 /
Keyboards for iOS
Ten iOS keyboards review
July 18, 2015 /
Samsung E1200 Mobile Phone Review
A cheap phone with a good screen
March 8, 2015 / 4
Creative Sound Blaster Z sound card review
Good sound for those who are not satisfied with the onboard solution
September 25, 2014 / 2
Samsung Galaxy Gear: Smartwatch at High Price
The first smartwatch from Samsung - almost a smartphone with a small body
December 19, 2013 /

News Archive



Do you use microSD card with your phone?
or leave your own version in comments (11)