Where Are SSH Keys Stored in Linux — And How to Manage Them Securely

Common files in this directory:

SSH (Secure Shell) is a foundational tool in the Linux ecosystem, used for remote access, secure file transfers, automation, and server management. While most users interact with SSH via the ssh command, under the hood SSH relies on public and private key pairs for authentication — especially in environments where passwordless logins, automation, and DevOps practices are essential.

Default SSH Key Storage Location

The most common place SSH keys are stored is:

~/.ssh/

This refers to the .ssh directory in the user’s home folder, e.g.:

/home/username/.ssh/

Common files in this directory:

If you generate keys with ssh-keygen, they are stored here by default unless a path is specified.

System-Wide SSH Key Locations

SSH Server (sshd) Host Keys

System-wide keys used by the SSH daemon (server-side):

/etc/ssh/

Typical files:

The SSH daemon (sshd) presents the host public key during connection; clients compare it to ~/.ssh/known_hosts.

Custom Key Locations

You can generate or use SSH keys from any location, but you must specify the path:

ssh -i /path/to/custom_key user@host

You can also configure multiple keys via ~/.ssh/config:

Host myserver
HostName 192.168.1.100
User devops
IdentityFile ~/.ssh/devops_key

Where Are Keys Used?

Outbound (Client Side)

SSH clients look for private keys in ~/.ssh/ by default. They are used to initiate authentication when connecting to a remote server.

• ssh, scp, rsync over SSH, git (when using SSH remote)

Inbound (Server Side)

The server looks for public keys in:

~/.ssh/authorized_keys

This file lists which public keys are allowed to log in to that specific user account.

If user_a tries to SSH into a server as user_b, their public key must be present in ~user_b/.ssh/authorized_keys.

Permissions — Critical for Security

Correct permissions:

~/.ssh              → 700 (drwx------)
~/.ssh/authorized_keys → 600 (-rw-------)
~/.ssh/id_rsa       → 600 (-rw-------)
~/.ssh/id_rsa.pub   → 644 (-rw-r--r--)

Incorrect permissions may cause SSH to ignore your keys or reject logins entirely.

Managing SSH Keys Securely

• Use a passphrase when generating private keys:

  ssh-keygen -t ed25519 -C "your_email@example.com"

• Use ssh-agent to cache unlocked keys in memory:

  eval "$(ssh-agent -s)"
  ssh-add ~/.ssh/id_rsa

• Rotate keys regularly
• Remove unused or orphaned keys from “authorized_keys”
• Use separate keys per host/project
• Avoid using root keys across environments

Auditing and Debugging

To see what key is being used during SSH connection:

ssh -v user@host

This prints verbose logs, including which identity file was attempted.

To list loaded keys in your current agent:

ssh-add -l

To remove a key:

ssh-add -d ~/.ssh/mykey

Conclusion

Understanding where SSH keys are stored in Linux — and how to manage them securely — is crucial for system administrators, developers, DevOps engineers, and anyone working in multi-host or multi-user environments.

By knowing the difference between user keys, host keys, and authorized keys, you can:

• Troubleshoot authentication issues
• Set up secure automated workflows
• Manage access across teams and systems

On production systems or cloud platforms (e.g., VPS or dedicated servers), mismanaging SSH keys can lead to serious vulnerabilities. Ensure you follow best practices and audit access regularly.