We maintain all of our server's /etc directories in subversion repositories and just disabled password logins on our subversion server. So that means you actually need ssh-agent to work when using sudo. This can be done by editting the /etc/sudoers file and adding the following line:

Defaults env_keep="SSH_AUTH_SOCK"