Hacker News Reader

Comment by derefr | Hacker News Reader

Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else.

But a stream of romaji furigana with no spaces is quite ambiguous—since there's nothing to indicate word boundaries, any substring of the input might turn out to have actually intended to be e.g. a katakana spelling of a name.

If CJK IMEs expected and required people to hit the spacebar between the "words" (lexer tokens) of the provided input for matching, they'd have a much simpler job. But as it is, they fall over quite badly when you type multiple words into the IME input box, and are mostly only usable if you resolve single words at a time (which, sadly, throws away a lot of the inter-word context that would otherwise be available for matching.)

View on HN

>because Japanese people pause between spoken words just like anyone else.

This is (surprisingly) not true. People do not pause between words, however when listening to a language that they understand, they do perceive pauses between words; even though such pauses do not exist.

I disagree. There's a subtle difference between breathing cadence and inflection, and a completely monotempo monotonal string of sounds.

It seems like it'd be useful for one or both of you to cite any research than has been done on this.

Seems more productive (and enlightening to all) than the agree/disagree dialogue here.

Unfourtuantly, this is so well established that it is hard to find research. I did find this paper [1] which looks into how babies acuire word boundries. As you identify, there are probably phonetic cues; but not pauses.

The best way to see this is to try listening to a language you do not understand, and try to identify word boundries.

Indeed, the paper I link argues that some phonetic cue must exist because babies can recognise word boundries.

[1] https://www.ncbi.nlm.nih.gov/pubmed/8176060

It seems I linked only to the abstract. Here [1] is the pdf.

[1] https://www.sissa.it/cns/Articles/94_doInfPerceiveWordB.pdf

Well for starters, a Japanese speaker can tell the differences between God "kami" and paper "kami"...

A monotonal, monotempo sound would not be able to make that difference audible

https://en.m.wikipedia.org/wiki/Japanese_pitch_accent

I think the claim being made is that the pause between consecutive syllables is (on average) the same whether they're part of the same word or not, so inter-syllable time is not a reliable indicator of word boundaries. Anecdotally, I think this is probably true of my speech most of the time (i.e. any time I'm not consciously enunciating). It might be less true for people with accents that have a slower cadence.

Wow, there are a lot of responses putting a lot of weight in my exact choice of words, here.

I'm not a linguist; I don't know what the proper name is for the thing people do between each pair of spoken words—that they don't do inside words—but I do know that there is something people do there. I would call it "a pause" because that is the function it serves. It's an overlap of lesser "terminal" sounds that forms something that is detectably a semantic gap—like the pause between crossfaded tracks on a gapless record, or between movements in a concerto.

Whatever it is, it is there, because speech-recognition systems use it to detect spoken word boundaries regardless of language. (This heuristic does screw up sometimes; spoken language does often "slur" particular word-pairs together. But it's rare enough that these can be trained as specific exceptions to the rule, rather than needing to throw out the rule.)

There isn't certainly pauses between words in Japanese. (Actually, I'm not aware of any language that has.)

The Japanese disambiguate word boundaries in spoken language using the pitch accent as the primary clue. Tokyo Japanese has a phenomenon called initial rise, which differentiates the pitch between the two first moras of an accent phrase – either the pitch rises or steeply falls.

Here's an example - upper case: high pitch, lower case: low pitch.

  KYOu,  kaINI  iKIMAshita
  today, to buy I went
  
  KYOu   KAini      iKIMAshita
  today, to meeting I went
  
  kyoUKAINI iKIMAshita
  to chuckh I went

> Spoken Japanese isn't any more ambiguous than English (for a human, or a speech-to-text AI) because Japanese people pause between spoken words just like anyone else.

Also intonation, which is not captured by the written system at all. Japanese isn't strongly tonal in the way Chinese is, but it has a regional prosody, like Swedish, which helps in disambiguating meaning.

It’s usually analysed as having both normal prosody and a pitch accent (similar to a stress accent) that varies somewhat by region. I’ve read that broadcasters are expected to use a standard (Tokyo?) pitch accent when speaking.

That is correct. The Tokyo dialect is considered "neutral" in the same way a Midwestern accent is for American broadcasters.

pause between each word? certainly not. Japanese are known to speak very fast and to remove any form of blank between words. like when asking お元気ですか? it sounds as if everything is attached. Please dont spread out inaccurate information.

Reply on news.ycombinator.com