You cannot remove the user or group from the delegation list because the user or group has inherited management permissions ...