Hacker News Reader

Comment by stubish | Hacker News Reader

I had always thought a necessary step along the way to natural speech synthesis would be adding markup to the text. But I guess the use case being chased is reading factual information, where you don't use sarcasm or other emotional color. Trying to convert text to emotive speech requires markup IMO. Even the best actors will read their lines the wrong way and need correction by the director.

View on HN

> Even the best actors will read their lines the wrong way and need correction by the director.

And sometimes even the director doesn't notice. Even when they also wrote the screenplay! My favourite example is from The Matrix, @1:01:40, where Cypher says:

> The image translators work FOR the construct program, but there's way too much information to decode the Matrix

Stress on the "for", which makes no sense; he sounds like he's revealing an employee/employer relationship. The stress should be on the start of "CONstruct", since he's saying the tech works for that but not for the other. Subtle, but it changes the whole sense of the line.

Wow, I've seen that movie dozens of times and I've always thought that was the stupidest line of meaningless technobabble because of how it was delivered. It made me question whether I knew what was meant by the phrase "construct program". It's clearly the name for the training/utility simulation used by the crew, but this line always had me questioning whether it also referred to the "front end" of the Matrix itself or something. Now it makes sense!

Goddamn you Cypher, indeed.

Dumbledore said calmly

That's a good point. There are various xml markup formats for synthesizing text to speech that let you tag words for emphasis and pitch, but there's not granular or expressive enough to mark up individual syllables of words, and it's not useful for singing, for example.

It would get really messy of you had to put tags around individual letters, and letters don't even map directly to syllables, so you'd need to mark up a phonetic transcription. At that point you might as well use a binary file format, not xml.

But for that kind of stuff (singing), there are great tools like Vocaloid (which is a HUGE thing in Japan):

https://www.vocaloid.com/en/

https://en.wikipedia.org/wiki/Vocaloid

VOCALOID5 - Walkthrough

https://www.youtube.com/watch?v=UAtVGHl1AFM

(Check out the "Cool / Cute" slider at 8:22!)

Here's a much simpler and cruder tool I made years ago (when xml was all the rage) for editing and "rotoscoping" speech pitch envelopes, called the "Phoneloper" -- not quite as polished and refined as Vocaloid, but it was sure fun to make and play with:

Phoneloper Demo

The Phoneloper is a toy/tool for creating and editing expressive speech "Phonelopes" that Don Hopkins developed for Will Wright's Stupid Fun Club in 2003, using Python + Tkiter + CMU's "Flite" open source speech synthesizer. It modified Flite so it could export and import the diphone/pitch/timing as xml "Phonelopes", so you could synthesize a sentence to get an initial stream of diphones and a pitch envelope. Then you could edit them by selecting and dragging them around, add and delete control points from the pitch and amplitude tracks to inflect the speech, stretch the diphones to change their duration, etc. It was not "fully automatic", but you could load an audio file and draw its spectrogram in the background of the pitch track, so you stretch the diphones and "rotoscope" the pitch track to match it.

https://www.youtube.com/watch?v=qy5cqV8ypIs

> and it's not useful for singing, for example.

There's this: http://zeehio.github.io/festival/doc/Singing-Synthesis.html It won't sound as good as Vocaloid, but it uses XML with singing synthesis.

Do you know greats vocaloids for songs? the best I know being the soul crushing https://youtu.be/W2TE0DjdNqI

DECTalk (famously, the TTS used by Stephen Hawking) lets you mark up individual syllables, and it even had a singing feature circa 1985.[1] It is indeed messy though.

This is the example code to make it sing Happy Birthday:

>DECtalk Software Sample Singing Program

[:phoneme on]

[hxae<300,10>piy<300,10> brr<600,12>th<100>dey<600,10> tuw<600,15> yu<1200,14>_<120>] [hxae<300,10>piy<300,10> brr<600,12>th<100>dey<600,10> tuw<600,17> yu<1200,15>_<120>] [hxae<300,10>piy<300,10> brr<600,22>th<100>dey<600,19>dih<600,15>rdeh<600,14&g;ktao<600,12>k_<120>_<120>] [hxae<300,20>piy<300,20> brr<600,19>th<100>dey<600,15> tuw<600,17> yu<1200,15>]

---

[1] https://vt100.net/manx/details/1,1230 (92mb pdf so I won't link directly to it, but info on singing feature appears on p. 116 for the curious)

Here's a historic DECTalk Duet song from Peter Langston (which is actually quite lovely):

Eedie & Eddie (And The Reggaebots) - Some Velvet Morning (Peter Langston)

https://www.youtube.com/watch?v=1l0Ko1GUiSo

Peter S. Langston - "Some Velvet Morning" (By Lee Hazelwood) - Performed By Eedie & Eddie And The Reggaebots

http://www.wfmu.org/365/2003/169.shtml

Eedie & Eddie On The Wire

http://www.langston.com/SVM.html

Peter Langston's Home Page:

http://www.langston.com/

His 1986 Usenix "2332" paper:

http://www.langston.com/Papers/2332.pdf

How to use Eddie and Eedie to make free third party long distance phone calls (it's OK, Bellcore had as much free long distance phone service as they wanted to give away for free):

https://news.ycombinator.com/item?id=22308781

>My mom refused to get touch-tone service, in the hopes of preventing me from becoming a phone phreak. But I had my touch-tone-enabled friends touch-tone me MCI codes and phone numbers I wanted to call over the phone, and recorded them on a cassette tape recorder, which I could then play back, with the cassette player's mic and speaker cable wired directly into the phone speaker and mic.

>Finally there was one long distance service that used speech recognition to dial numbers! It would repeat groups of 3 or 4 digits you spoke, and ask you to verify they were correct with yes or no. If you said no, it would speak each digit back and ask you to verify it: Was the first number 7? ...

>The most satisfying way I ever made a free phone call was at the expense of Bell Communications Research (who were up to their ears swimming in as much free phone service as they possibly could give away, so it didn't hurt anyone -- and it was actually with their explicitly spoken consent), and was due to in-band signaling of billing authorization:

When you called (201) 644-2332, it would answer, say "Hello," pause long enough to let the operator ask "Will you accept a collect call from Richard Nixon?", then it would say "Yes operator, I will accept the charges." And that worked just fine for third party calls too!

>Peter Langston (working at Bellcore) created and wrote a classic 1985 Usenix paper about "Eedie & Eddie", whose phone number still rings a bell (in my head at least, since I called it so often): [...]

>(201) 644-2332 or Eedie & Eddie on the Wire: An Experiment in Music Generation. Peter S Langston. Bell communications Research, Morristown, New Jersey.

>ABSTRACT: At Bell Communications Research a set of programs running on loosely coupled Unix systems equipped with unusual peripherals forms a setting in which ideas about music may be "aired". This paper describes the hardware and software components of a short automated music concert that is available through the public switched telephone network. Three methods of algorithmic music generation are described.

FWIW the "Speech Synthesis Markup Language" (SSML) standard does have sections for "3.2 Prosody and Style" https://www.w3.org/TR/speech-synthesis11/#S3.2 & "3.1.10 phoneme Element" https://www.w3.org/TR/speech-synthesis11/#edef_phoneme in it which can enable quite granular control.

Of course, whether a particular speech synthesis system supports such features is another thing.

It also has a `pitch_contour` attribute: https://www.w3.org/TR/speech-synthesis11/#pitch_contour

Coincidentally enough I pretty much only know any of this because a few years ago I created a GUI for a client which enabled an assistive technology researcher to "draw" in the pitch contour required for a word/phrase (not unlike the project demonstrated in your video :) ) from which the SSML was then generated.

Given that a human is going to be listening to each sample, I wonder if a style-transfer style algorithm could be used to map the intent of a sentence to a simulated voice.

I tend to view most of these things through the perspective of what would help mod-maker's for video games: VA is the one thing you basically can't do yourself, but you also tend to have a decent data set to pull from (and I suspect various open source voice sample sets would become pretty popular).

Yes, this stuff will be pretty neat for game mods, particularly ones like the Elder Scrolls IMO. Can't wait!

> I wonder if a style-transfer style algorithm could be used to map the intent of a sentence to a simulated voice.

There's definitely research/proprietary software that can enable a person speaking in desired manner to have their voice control the expression of the generated speech.

Here's a related issue on a Open Source text to speech project which I only learned of today: https://github.com/neonbjb/tortoise-tts/issues/34#issue-1229...

> I tend to view most of these things through the perspective of what would help mod-maker's for video games

Yeah, I think there's some really cool potential for indie creatives to have access to (even lower quality) voice simulation--for use in everything from the initial writing process (I find it quite interesting how engaging it is to hear one's words if that's going to be the final form--and even synthesis artifacts can prompt an emotion or thought to develop); to placeholder audio; and, even final audio in some cases.

> (and I suspect various open source voice sample sets would become pretty popular).

That's definitely a powerful enabler for Free/Open Source speech systems. There's a list of current data sets for speech at the "Open Speech and Language Resources" site: https://openslr.org/resources.php

Encouraging people to provide their voice for Public Domain/Open Source use does come with some ethical aspects that I think people need to be made aware of so they can make informed decisions about it.

Given your interest in this topic you might be interested in this (rough) tool I finally released last week: https://rancidbacon.itch.io/dialogue-tool-for-larynx-text-to...

The W3C does have a standard for speech synthesis markup called "Speech Synthesis Markup Language" (SSML): https://www.w3.org/TR/speech-synthesis11/

Whether it's supported by a particular speech synthesis system and which set of features are supported varies.

The standard does include a section on "3.2 Prosody and Style" which covers this aspect to some degree: https://www.w3.org/TR/speech-synthesis11/#S3.2

https://15.ai/ uses emojis for emotional context and it works pretty good.

Reply on news.ycombinator.com