I understand that the color selection box itself move. But inside the color selection, does the order of the colors remain the same? So e. g. “red” is always the 7th element from the top?
Also, on your screenshot, note that XClick does distinguish the colors. The best matching color should have the highest confidence and get selected. But I guess it does not work reliable as you have a few very much alike colors?