In early stages telnet protocol was used to connect the remote server via client machine.

The disadvantage of using telnet was it did not support any encryption while transmitting data over the network, hence made it highly unsuitable for transmitting sensitive information over the public / untrusted  networks.

In order to overcome  shortcomings of telnet, ssh protocol was introduced. It is also short for Secure Socket Shell protocol. It makes use of 3 encryption  protocols for communicating with remote server. We will discuss the importance of each encryption protocol one by one.

Symmetric Encryption : In this, a single identical key is used to encrypt and decrypt the data between remote and client server. Along with authenticating the remote server with the client, this encryption is also used in the entire ssh connection to transfer the data in encrypted format.

Asymmetric Encryption : This encryption makes use of two keys. One referred to as a public key used for encryption and the other as the private key which is used for decryption.

Hash Encryption : The algorithm used in this protocol follows only one way of encryption i.e encrypted value cannot be decrypted back to its original value.

Asymmetric and Hash encryption are used in conjunction with each other in order to authenticate a client with a remote server.

SERVER AUTHENTICATION

  • First client give tcp connection request to a remote server on port 22, if the client is connecting for the very first time then it is prompted on its screen to manually authenticate the remote server, if approved entry gets saved in the known_hosts file of the client machine which avoids remote server prompt approvals in future.
  • In response, the remote server sends back the list of its supported symmetric encryption protocols like aes , arcfour, blowfish-cbc, cast..  etc. The client server will agree on most recent version that it can support out of the given list (typically aes). By making use of  Diffie-Hellman algorithm the session of symmetric encryption takes place in following steps
    • Both parties agree on a large prime number, which is shared between both the parties.
    • Independently, each party comes up with another prime number which is kept secret from each other. This number is used as the private key which is different from the private SSH key used for authentication.
    • The generated private key, the encryption protocol, and the shared prime number are then used to generate a public key which can be shared with the other party.
    • Both participants then exchange their generated public keys.
    • The receiving entity uses their own private key, the other party’s public key, and the original shared prime number to compute a shared secret key. Although this is independently computed by each party, using opposite private and public keys, it will result in the same shared secret key.
    • The shared secret is then used to encrypt all the communication takes place between both the parties in future.

CLIENT AUTHENTICATION

After symmetric encryption has been established, the authentication of the client happens as follows:

  • The client begins by sending an ID for the key pair it would like to authenticate with to the server.
  • The server checks the authorized_keys file of the account that the client is attempting to log into for the key ID.
  • If a public key with matching ID is found in the file, the server generates a random number and uses the public key to encrypt the number and sends this encrypted message.
  • If the client has the correct private key, it will decrypt the message to obtain the random number that was generated by the server.
  • The client combines the obtained random number with the shared session key and calculates the MD5 hash of this value.
  • The client then sends this MD5 hash back to the server as an answer to the encrypted number message.
  • The server uses the same shared session key and the original number that it sent to the client to calculate the MD5 value on its own. It compares its own calculation to the one that the client sent back. If these two values match, it proves that the client was in possession of the private key and the client is authenticated.

Asymmetry of the keys allows authentication of the client because client can only decrypt the messages if it has the correct associated private key