White boys from disadvantaged backgrounds who do well at primary school are the most likely to see maths potential unfulfilled, a study has found.