Royal had concerns about son's skin colour