We present a scheme to compensate environmental polarization changes in deployed telecom single mode optical fibers such that they can be used for encoding optical qubits into polarization. Contrary to earlier approaches that actively measure the polarization transformation through reference and test fields, we accomplish this with an extremely lean hardware approach, just using four liquid crystal retarders and a stochastic algorithm that minimizes the quantum bit error in a quantum key distribution setting. We demonstrate this with a quantum key distribution process based on polarization-entangled photon pairs.