When you say "passed memtest", how many runs did it pass? Leave it running overnight for preference, get yourself a dozen passes to be certain. That aside, I don't believe memtest fully stresses the memory controller, as opposed to a program like Prime.
Are you certain that your FSB is stable? Are you certain that your CPU speed is stable? Either of those can cause Prime to fail, but you'll have to use a process of elimination in order to determine which one.
To the best of my knowledge, Asus' transaction booster is a fancy name for adjusting the memory controller's tRD timings. If you're determined to run beyond the memory's rated speed, you should probably just disable the transaction booster, and pull it down a level or two. It'll cost you memory bandwidth, but it might just make the difference. Later on, once you've got a stable OC, you can try notching it up slightly to regain as much bandwidth as you can.
Personally, I suspect that any memory speed gains you see by using the relaxed transaction booster settings will be negated by the looser MCH timings. And since you're running a quad-core, you'll run out of FSB headroom before you run out of memory speed.
And DDR2-1200? That's fast. If you don't plan on aiming a fan at those modules, you should probably slow things down a little.
__________________ QX9650 @ 4.038GHz (425x9.5) | Asus Formula Maximus | GTX280 (777/1620/1404) | Mushkin XP2-8000 Redline 2x2GB
Q6600 @ 3.6GHz (400x9) | MSI P35 Platinum | 8800GTS 640MB (700/900) | Mushkin EM2-6400 2x2GB
|