This is normal behavior.
Of course the water temperature is low if the car hasn't been running for a time.
It goes out before the engine is fully warm.
Just drive gently until the light goes out.
For maximum long-term reliability, I also recommend that you avoid full throttle operation for a few minutes beyond the light going out.
This allows the oil temperature to stabilize.