Trouble reading different coloured numbers

I’m trying to run hearthstone decklists through the OCR api. They look something like this:

It works fine for most of the content, but not the yellow numbers in the right side. It doesn’t recognize most of them, and I’m not sure why? Does the colouring throw the OCR off? Is there anything I could do about that? I tried making the image black and white, but that doesn’t help.

There can only be a blank, ‘2’ or a star in that position, and I’m only interested in when there’s a ‘2’. Could I use that limitation as an advantage somehow?