Dear Soon Choon,
I think it is important to note that the interviewer is not the "creator" of the oral history. I like to use the term narrator for the person who is the subject of the interview. Without that person, we have nothing.
I have had people ask to withdraw from a project after the interview. I have always spoken to them directly, discussed their concerns, and have been able to keep the interview. Had I not ultimately won their consent, I would have destroyed the interview.