Do you need access to my server?

Not at all. ServerAuth provides an opensource client for you to install, which periodically checks for changes to your SSH Keys.

When you add your servers to ServerAuth we give you a one-line command to install and configure the agent. The agent then periodically checks for updates to your keys via the ServerAuth api. We don't store your servers ip address, location or any kind of identifiable information. 

This means that in the highly unlikely event that ServerAuth is compromised, your server will not be, nor will your team members access.