ongoing -· Do Databases Suck?
The Moral of the Story Databases suck. Except when they don’t
Databases don’t have to suck, but in many cases they do. Depends on how often you end up using the slowest components. Since Tim’s piece mentions bulk-loading as taking a long time, that suggests a lot of disk activity — writing records, updating indices — and that’s a performance killer.
Contrast that with this view:
“I just got a new Mac with two 2-gigahertz processors, 8 gigabytes of memory, and a half a terabyte of internal disk,” [Bill Joy] said, describing his Power Mac G5. “… So you have the ability to hold a huge simulation all in memory — a database becomes a data structure. Add 64-bit computing and I can do breathtaking visualization.”
Not that I want to plug OS X — there are other ways of doing this, of course — but this is reminiscent of the information retrieval startup I was working at 3 years ago (this is memory lane week, apparently. The salient lesson from that experience seems to be that any technology can be misused . . . . ).
RAM is always faster than disk, but more expensive, so in the context of your overall design, you end up with the age-old question: cheap, fast, or good, choose any two.
Going to an in-memory database or some way of scaling up from the limitations of a single box was the way to go. There was a recent article where Google may be hitting the 32 bit limit on article identifiers. I would guess they’re doing some really cool stuff in memory, like sharing cached results and the like.