Color Map all got jumbled up after listing? HELP!

I listed a parent product with multiple color variants.

For each variant, I correctly chose the Color Map and Color.

However, after listing and grouping them into the parent, ALL THE COLOR MAPS are 'resetted', and it defaulted to one of the colors. For example:

Variant 1: Color Map Black, Color Black

Variant 2: Color Map White, Color White

Variant 3: Color Map Green, Color Green

After a few days, I checked my listing and it changed into:

Variant 1: Color Map White, Color Black

Variant 2: Color Map White, Color White

Variant 3: Color Map White, Color Green.

What is happening? I can't change the Color map again. Why did this happen and how can I fix it? If I don't what issues will I encounter?