Using AI to reconstruct the voice of cancer patients

CoeFont’s service uses AI to learn the pitch, intonation of the user’s voice, and the speed of speech before surgery. The system then reconstructs a synthetic voice, which can read out the user-entered text aloud, with the same voice as the patient’s original voice.

CoeFont President Shogo Hayakawa, now a third-year student at the University of Tokyo Institute of Technology, said the company did not initially intend to develop the service, but only to provide content creators with videos posted on sites like YouTube, which use AI to poison the statements of broadcasters and voice artists. However, after Ms. Michiko Sakai, a patient diagnosed with tracheal cancer, visited Hayakawa before surgery, Hayakawa decided to use the service for medical purposes.

Within eight days of surgery, patient Sakai was asked to read 700 sentences in order for the AI ​​to record her voice. After the surgery, Ms. Sakai said: “The first person I talked to after the surgery was a nurse at the hospital. She was surprised to hear my voice sound the same as before, which made me very happy. “. Ms. Sakai’s husband also said that they can converse naturally and do not feel alienated.

With CoeFont’s service, after a user enters a text sentence on a smartphone, the system will automatically read it out loud in a rhythmic manner with the same intonation as the user and it is difficult for listeners to recognize this as a voice. say reproduced.

According to statistics of an organization representing people with voice loss in Tokyo, Japan has about 20,000 to 30,000 people who have lost their voice due to cancer. Masanori Matsuyama – the head of this organization – highly appreciated CoeFont’s service and expressed his hope that AI will further improve and move towards reproducing speech by reading mouth movements.