I am going to replace an old server that has been upgraded to NW 6.5 Sp6
with a new Dell server. The old server always had little errors in the
installation to where certain administration tasks were always not the way
they were supposed to be, LDAP does not work, etc. If I do a migration
from the old server to the new server will it migrate thos settings as
well or does migrating install a new server and only copy over non system
related software.

Just wondering what exactly a migration does move over?

So I am just wondering if I should treat this as a dead server and rip it
out of the tree and reinstall with new server.

Its only a file/print server.