Should I be worried about the yellowing leaves.

As of recently my Cayenne and ghost pepper plants have been turning a yellowish hue.