A former Tory minister said families don't have a 'right' to low food prices and that they should live within their means