I'm having trouble with Color.Lerp()

I'm making a 2D game where the camera background changes color over time but there is a little glitch once in a few seconds. The color randomly changes a bit and then it smoothly continues lerping. Here is the code i tried first:

 float timeLeft;
 Color targetColor;
 void Update()
 {
     if (timeLeft <= Time.deltaTime)
     {
         Camera.main.backgroundColor = targetColor;
         targetColor = new Color(Random.Range(0.1f, 0.8f), Random.Range(0.1f, 0.8f), Random.Range(0.1f, 0.8f));
         timeLeft = 5.0f;
     }
     else
     {
         Camera.main.backgroundColor = Color.Lerp(Camera.main.backgroundColor, targetColor, Time.deltaTime / timeLeft);
         timeLeft -= Time.deltaTime;
     }
 }´

Here is the modified version of the code that still has the glitch:

 float timeLeft;
 float transitionDuration = 5f;
 Color initialColor;
 Color targetColor;
 void Update()
 {
     if (timeLeft <= Time.deltaTime)
     {
         Camera.main.backgroundColor = initialColor = targetColor;
         targetColor = new Color(Random.Range(0.1f, 0.8f), Random.Range(0.1f, 0.8f), Random.Range(0.1f, 0.8f));
         timeLeft = transitionDuration;
     }
     else
     {
         Camera.main.backgroundColor = Color.Lerp(targetColor, initialColor, timeLeft / transitionDuration);
         timeLeft -= Time.deltaTime;
     }
 }

Here is a video showing the glitch but its quality isn't the best. If you can't see it on the video, here is the built game.