You can remove a Delegate Manager from either the Delegate Manager's profile or the User's profile they are delegating for.
Navigate to Organisation and Users and search for the User. Click on their Username to go into their Profile (see view Users if you're unsure how to do this).
Click on the Reporting Lines tab on the left.
From the Manager's profile the Delegate Manager will show in the People reporting to...list.
From the Delegate Managers profile, the User they are delegating for will show in the People managing...list.
Click on the cross to the right of the Delegation relationship.
Check you have selected the correct person in the Remove Reporting Line Relationship window and, if so, click Confirm.
Cancel will return you to the Reporting Lines page and no one will have been removed.