At this point, your new public key should be on your EC2 instance in the authorized_keys file, and all you have to do is remove the old one.

Nick Hardiman shows you how to set up a secure connection to an Amazon EC2 machine using PuTTY and Pageant to handle your private key.