It is not permissible for a man to form a relationship with a non-mahram woman, even if it is only with the aim of listening to some sweet words as you say. That is one of the traps of the shaytaan by means of which he gradually tempts people. Women are forbidden to soften their voices when speaking to men who are not permissible for them, as Allaah says (interpretation of the meaning):
“then be not soft in speech, lest he in whose heart is a disease (of hypocrisy, or evil desire for adultery) should be moved with desire, but speak in an honourable manner”
And it is haraam for a man to enjoy listening to the voice of a non-mahram woman. It is obvious that a believer would not approve of such a thing for his wife or sister or daughter, i.e., for her to get to know someone to whom she speaks on the phone, so why would he approve of that for the daughters of other people?
Hence what you must do is repent to Allaah and end this relationship completely. You do not have to tell your wife about the details; it is sufficient to tell her that it was a mistake and you have given it up.
We ask Allaah to forgive us and you.
And Allaah knows best.