Ok, so mine started the flickering last week, then speed #2 went out completely.
Took mine apart. Tip #1 for anybody else who does this - there are 3 snap in tabs holding the thing together, but they are not all the same. One of these has an indent on the back side, and the other two don't. (It is obvious with the switch in your hand, impossible to photograph though.) Start with the tab with the indent. There you can get a small screwdriver in and pop the tab out easily, then the others come off too. I tried for a while on one of the other tabs, and couldn't get anywhere. Tip #2, measure the resistance between the tabs on the back of the switch, and if all positions work as they should (open when they are supposed to be, shorted to the common in the appropriate position) don't take it apart. Tip #3, if you must take this switch apart open it over a tray of some sort - when the switch finally pops open the little springs and the ball bearing may fall out and bounce away.
Bottom line though, is that the AC problems are not due to this switch. At least not on my car, probably not on most of them. There are only 5 connectors on the back, these are common and speeds 1,2,3,4. If the fan is turning at the right speed then the switch is working. If the AC light is not also on, then the problem is elsewhere. Before opening the switch I had thought that these 5 connectors were used in a more complicated manner, with the AC part separate, but that is not the case.
Anyway, it appears that your switch was in fact defective. When I opened mine there were already two springs in the holes you found empty. Additionally, there is a spring that sits on a ball bearing that lives in a hole on the other side. The ball bearing provides the clicks for this switch as it rolls into the depressions in the casing. In the first picture the parts are laid out in order. Left to right, these are the front part of the case (on the screwdriver, with the 5 detentes for 0->5 visible), the ball bearing, the spring, the plastic piece that fits on the shaft and holds the inner switch plate (with its two springs seated in it), the anterior switch plate, the posterior switch plate and case.
In the second picture the back switch plate, the one attached to the external contacts, is shown. The contacts are all coper, everything else you see is plastic or grease. The inner wiper on the other switch plate moves across the 4 separate pads to select speeds 0 (no connection) or 1->4. It looks to me like in the 0 position both the inner and outer wipers are sitting on plastic.The outer wiper moves on the long continuous pad. There is gunk all over the innards which is probably dielectric grease.
While I was at it I pulled the blower resistor, and it appeared to be fine. (As expected, the blower never failed, only the AC). Once it was off I could see some bits of leaves in the chamber behind it - probably a good place to look for those mysterious "sounds like there is a leaf in my AC" sounds.
My best guess now is that either the controller board is defective, or the AC needs servicing so that it draws less current under high temperature conditions.