what part of stopping performance are you trying to increase?
more resistance to fade?
shorter stopping distance?
Your correct that the brakes on the RX8 are already pretty amazing, the rx8 stops from 60mph in the same distance as a Mecedez/Mclaren SLR.
If you want to improve stopping distance, and increase fade resistance a little bit you should get some good tires, they will make a huge decrease in stopping distance, good pads will also help shorten the stopping distance while helping with fade resistance a little bit, lastly a good fluid (like motul) and some stainless steel lines will help increase fade resistance.
I would suggest trying some of all of the above first and then move onto a BBK kit if you still are not happy.
As far as a direct bolt on BBK, the only one I have seen is the Greddy Grex kit : its a 6pot caliper with 330mm rotors, i havent personal felt how well the RX8 Grex kit feels, but im sure it similar to the other 6pot-330mm kits they have for other cars. They have excelent fade resistance.
http://www.greddy.com/products/display/?Category=suspension&SubCategory=11
The thing to remeber with brakes is that even with a 12 pot caliper, you cant really apply much more stoping force to the brakes, as that is provided by the brake master cylinder. You can increase the leverage on the wheel with a larger rotor (and you need a larger/multi pot caliper to fit the larger rotor, and it also helps spread the braking force more evenly across the rotor.
Stainless lines and good fluid will reduce the 'spongy' feel of the brakes when they get really hot, as it will resist boiling untill a higher temp.
The brakes are already capable of stoping the wheel completly, so they have plently of power, but the tire is the weak link in the system, as it cant hold the road tight enough to use 100% of the power (just try driving at 50mph and hitting the brakes hard, you will feel the ABS fighting lockup).
Most of the big brake kits wont shorten you 60-0 or even 80-0 distance, since the weak link is the tires, what they will do is give you increased fade resistance over repeated braking (mostly due to the larger rotors and the drilled/slotted "vents". The BBK will shorten your stopping distance from higher speeds such as 100-0 or 120-0, ect.
If the car is mostly street driven you will want to choose a pad compound that is easy on rotors and also doesnt need a lot of heat to give good clamping.
Again I would suggest that you upgrade your pads and/or tires, as well as put on stainless lines and a quality fluid (motul/RBF600/Hawk SuperBlue, ect) and see how you like that.
New Pads will be 60-80 dollars per axle for a quality set (up to 120 per axle for a race style track pad).
New SS lines will be 100-150 depending on brand (as long as they are brass fitings, teflon lines wraped in SS sleves they are all the same, also look for lines that are DOT approved, or similar for your country).
Fluid will be around 30 for enough fluid to bleed the system and refill it
tires will be the biggest expense, 200-300 for a near oem size (i would run 255/35/18 or 245/40/18) per tire.
The big brake kit will run around 2500-3500 for most kits (and thats the front only) so you will still need to buy and change pads and SS lines in the back (some BBK do only the front but come with lines for all 4 wheels) and you will still need to get fluid and most important of all....you will still need good tires as they again are the weakest link in the system.
You can also look at upgrading to a 'oem' size rotor. Get one slotted OR cross-drilled BUT NOT BOTH! I can explain why if needed.
Also, stay away from the one peice all steel/iron rotors, they are often the same weight or heavier than stock. If your going to get new rotors anyway, spend the "little extra" and get a set that are 2 peice and have an aluminum top hat. This will help reduce unsprung weight, and often helps cool the brakes even more.