It's my understanding that that's only true when you're actively using the bit of extra power (air flow) that the exhaust/intake can provide. Cruising on the expressway at 70 mph, the fuel efficiency should, if anything, be very slightly better with higher performing intakes and exhausts.
Your understanding is in the right direction, but not quite there
Backpressure in the exhaust system prevents a significant amount of fuel/air from the intake stroke from being expelled out of the exhaust valve while both the intake and exhaust valves are open at the same time. This occurs for a short overlap duration, determined by the cam lobes.
When you install a low back-pressure exhaust system, the velocity and intertia of the exhaust gas leaving the cylinder is higher, which means the pressure at the exhaust valve is less than it normally is. The increased pressure gradient causes more intake gas to move toward and out of the open exhaust valve.
This is why lowering the back-pressure on the exhaust causes less torque and less power to be developed by the engine unless the intake mix is adjusted to compensate. You can also count on a marked decrease in mileage and efficiency as more of your fuel is leaving the engine without having been combusted in the power stroke.
The problem with bolt-on aftermarket low-pressure exhausts is that the control loop in the computer detects all of this unburned fuel in the exhaust and interprets it as a "rich burn" condition, and decreases the amount of fuel to the motor to compensate, further reducing torque and power (power is simply the time-average of torque). Most bolt-on exhausts probably don't change the pressure for this reason, and they are for sound only.
This phenomenon occurs regardless of engine speed.
If you had higher-flow intakes and exhausts, there is no reason to realistically expect higher economy and efficiency. The more free-flowing your engine is, the more fuel will leave through the exhaust port during the overlap.
The efficiency of the intake stroke is determined by the mass of air that enters the cylinder on the intake stroke. Ideally, zero air-flow resistance will result in a mass equivalent to having atmospheric pressure in the cylinder when the intake valve closes. This is the most air that is going to be available on a naturally aspirated motor. You are only going to be able to improve your intake flow so much, and if you only see an inch of pressure drop between ambient and cylinder, there isn't much you can do to improve. The mass of air in the cylinder is linearly proportional to the pressure for static volume and temperature, so a cylinder with 30 inches (ambient) is only going to have about 3.5% more mass of air than one with 29 inches. In reality, you might be able to go from 29 inches to 29.5 inches by spending $1000 on intake stuff, for a 1.75% improvement in available air mass.
Mass is inversely proportional to temperature, so decreasing temperature by 1% Kelvin (about 3 degrees C at standard atmosphere) will result in a 1% increase in available mass. This is why there is a cold-air intake market. You still won't be able to surpass atmospheric pressure by a significant amount, but it's still another 1%. If the air at the inlet is, for example, at 40C (313K) because it's hear all the "hot stuff" under the hood, and you move it such that it drops to 30C (303K), this is about a 3% drop, resulting in 3% more mass assuming the pressure does not change.
If I were modifying my car, I would not change the exhaust, but I would measure my intake-closed cylinder pressure to see what the drop was from ambient. If it turned out to be a couple of inches or more, I might consider trying to loosen up the intake path. This would allow more mass of air into the cylinder during the intake stroke (and therefore the computer would put more fuel in it, resulting in more power at wide-open-throttle), but allow me to keep the 35+MPG I've been getting with my car by making sure that extra fuel wasn't blown out the exhaust port (which is right below the main port - haha).
I would feel justified in changing the exhaust character if I were building a race car and didn't care about economy. One of the things that I didn't mention before is that there is always some exhaust gas that remains in the cylinder after combustion, even though the cylinder tries to push most of it out. By opening up the exhaust, more of that gas can leave, and more intake gas can come in. This is a huge decrease in efficiency, but you do get more fuel and air IF AND ONLY IF your computer is set up to do it. This is why you must buy an AFC if you really want to see power gains out of your motor. The stock computer will compensate for anything you do looking for the 14 to 1 air fuel mixture.
I guess the lesson of this really long lecture is that, if you're going to mod your car for higher power, you must necessarily change intake, exhaust, AND the control loop (AFC) if it is going to work. The stock ECU is a whole lot smarter than you think, and it will defeat you at every turn to make Federal Emissions.