A former Norfolk County Council leader has lost his County Hall cabinet role because he did not support the call for May's elections to be cancelled.