Remove delivered-to addreses from list of recipients

This is necessary so we don't add delivered-to addresses to the list of
addressee/collaborators.