New poll fans rumours of Veterans Affairs Minister leaving federal politics