Subject: Re: SMPs - clusters - db perf. - parallelism

From: [email-masked]
Date: Tue, 4 May 1999 15:30:27 -0500

Hi, Saleh.

Yup. It's a big problem.

In fact, it goes both ways: How do you design hardware without understanding how
the software can use it? This becomes fairly critical, since system performance
is used to determine expected revenue (how much you can charge), which
determines at least in part how much you can afford to spend on development now,
and determines a lot about how much manufacturing cost you can have later --
guess wrong, and your profits look sick.

So it's a thoroughy knotty chicken-and-egg question.

However, from what I've seen most commercial software developers ignore it, in
the sense that they wait for the hardware to be shipped before designing
software for it.

Example: Sun's UE10000, 64-way SMP. It's been sold for well over a year. Only in
the last month or so was there an OLTP (On Line Transaction Processing)
benchmark (TPC-C) published on it with all 64 processors. It took that long for
software (Oracle, in this case) to do the work needed to scale adequately with
that many SMP processors -- as you might imagine, that was a big job. Until
then, the biggest OLTP benchmark numbers Sun did were on its smaller, 24-way,
systems.

The exact same thing happened when HP first started shipping 12-way SMPs in
about 1992. Nothing commercial scaled to 12-way at that point; several years
later, the software caught up.

Overall, there are two things either hardware or software designers can do:

1. Ignore the problem. Assume that what's there now is what's going to be there
later, and design for what's there now. Much software appears to do that
(examples above). If hardware does this, it's slitting its throat in many cases,
unless the hardware targets the low end of the performance range, which is still
a big market.

2. Use your best judgement and extrapolate what's going to be available from
current trends, and design to that. For hardware, in many cases this amounts to
taking the position that "if we build it, they will come." Sometimes they comes,
and sometimes they doesn't.

There is, of course, a fairly obvious reason why software just takes what's
there: It's awfully hard to test or otherwise work on software that doesn't run
on any existing hardware.

If you want to figure out how some software organization will work against
future hardware, you'll just have to take approach 2, making some well-founded
extrapolations. Doing so qualitatively is relatively easy; here are a couple:
(1) Locality will become ever-increasingly important; even communicating to the
other side of a chip will have a performance cost in 6-7 years. Implication:
Every large SMP will inevitably be NUMA. (2) Communication overhead within
clusters will go down, particularly compared with current (TCP/IP) levels; I'd
say by 1-2 orders of magnitude. This is overhead, the cycles a processor must
spend to move data elsewhere.

Getting quantitative is harder. I'll leave that as an exercise. :-)

Once you have your assumptions, the next thing I'd do is construct a model that
captures the interesting aspects, feed in some numbers (like # of instructions
needed for this or that function, commo overhead, synch time, etc., beaten
against hardware characteristics) -- and do sensitivity testing: See what
happens if you're off by a little (or a lot) in your assumptions. Stare at the
results *a lot* and see if you can derive any implications from them. Then
repeat until you have something that's interesting.

Hope this running off at the mouth helps!

Greg Pfister
Senior Technical Staff Member, IBM Advanced Technology & Architecture, Server
Design
(512) 838-8338 | IBM tieline 678-8338 | FAX (512) 838-3418        Sic Crustulum
Frangitur

To:   Greg Pfister/Austin/IBM@IBMUS
cc:   Saleh Elmohamed <[email-masked]>
Subject:  Re: SMPs - clusters - db perf. - parallelism

Hi Greg,

I've been looking at some of those redbooks on performance.
There is an interesting one on performance tips and techniques for
the SanFrancisco project.

This is a tough issue and most likely will come down to that
hard-nut-to-crack issue of understanding the hardware implications of
designing software.

How one go about analyzing performance of any kind of software
(scientific or IT type) without first understanding of the impact
of the hardware that piece of software will run on and how the
hardware is put together, on the design, speedup, etc. of the software
???!!! ... In other words, have a clear idea of the limitation of
the hardware first before twisting and turning the software ....

Am I correct/half way correct on this view?

You probably have beaten this thing to death by now, and from reading
your book it sounds that way, but would really like to see some of
your insightful comments about it ... ?

best,
Saleh

On Wed, 28 Apr 1999 [email-masked] wrote:

>
>
> Try http://www.software.ibm.com/data/db2/ -- somewhere from there you may
> find something.
>
> or use "DB2 EEE" (that's one parallel version) in the search. Doing that is
> how I found the URL above. Somebody appears to have been working on the
> searches for the IBM site; they actually seem to be coming up with fairly
> intelligent results these days, which is a *big* change from their previous
> behavior.
>
> Greg Pfister
> Senior Technical Staff Member, IBM Advanced Technology & Architecture,
> Server Design
> (512) 838-8338 | IBM tieline 678-8338 | FAX (512) 838-3418        Sic
> Crustulum Frangitur
>
>
>
> To:   Greg Pfister/Austin/IBM
> cc:
> Subject:  Re: SMPs - clusters - db perf. - parallelism
>
>
>
> Greg,
>
> Thanks for the info. I plan to look into this a little further so any
> comments you might have on the subject would be appreciated.
> BTW, what is the DB2's url there at IBM?
>
> After doing some search/research I'll try to write up something.
> I'll keep you posted. In case you come across anything of interest, please
> drop me a line or two.
>
> best regards,
> Saleh
>
>
> On Sun, 25 Apr 1999 [email-masked] wrote:
>
> >
> >
> > Saleh,
> >
> > Sorry, I don't have any direct pointers for you on that topic. But I can
> > give you some indirections.
> >
> > I doubt there are any journal articles and the like. I would look on the
> > web sites of database vendors (Oracle, Sybase, DB2 (for 390 and for other
> > systemss -- two different programs, one name (marketing!)), Informix,
> > Tandem/Compaq). What I'd look for are white papers, "red books" (IBM),
> and
> > other things attempting to educate programmers on how to write SQL that
> > runs well in parallel.
> >
> > There *might* also be some information on the web sites of system vendors
> > who do parallel commercial systems (IBM, HP, Sun, Compaq/DEC,
> > Compaq/Tandem), but you're more likely to find it on the database
> vendors'
> > sites, I think.
> >
> > And hey, if you find anything interesting and/or useful, let me know.
> >
> > I agree with you that it is a key attribute. It's not the only one,
> > obviously -- making parallelism work is a large logical "AND": One thing
> > doesn't work and it doesn't work.
> >
> > Greg Pfister
> > Senior Technical Staff Member, IBM Advanced Technology & Architecture,
> > Server Design
> > (512) 838-8338 | IBM tieline 678-8338 | FAX (512) 838-3418        Sic
> > Crustulum Frangitur
> >
> >
> >
>
>
>
>
>
>
>

---

From: Saleh Elmohamed <[email-masked]>
Date: Sat, 8 May 1999 17:54:16 -0400

Hi Greg,

I quite appreciate the comments. Very insightful, as always :)

A bit surprised at example one though; I thought Sun conducted
an OLTP benchmark (TPC-C) few months ago on the 64-way SMP.
I recall coming across and announcement from them sometimes 
toward the end of last year.

> other side of a chip will have a performance cost in 6-7 years. 

Giving the current high emphasis on various issues of performance, I'd go
as far as halfing that number, to around 4 years or so.

> Implication:
> Every large SMP will inevitably be NUMA.

But do you believe that 'N' of cc-NUMA will come close to 1:1 ratio or
something of the neighbourhood within those number of years? That
will be one ideal thing to have but with that, is the model will still
be coined 'NUMA'? 

> Getting quantitative is harder. I'll leave that as an exercise. :-)

I'll assign it as something to do ;-)

Many thanks.

best regards,
Saleh