There is currently no way to add the address for contacts. It shows as a field on the contact card, but it is not an optional field for CSV upload. When you try and edit the contact card, it tells you that you can't change it because it is part of the upload.