Re: Isn't java.lang.Character.html#{ isLetterFromLang(int codePoint,
String ISOLangDef) missing from the spec?
On 12/04/2010 07:16 PM, lbrtchx@gmail.com wrote:
One possibly (and easily ;-)) could based on the Unicode code
points check the ranges for each language, but I think it would be
very useful for people parsing text from different languages.
Language is not so simple. First of all, code points don't necessarily
map to a `character' in a language--you can represent `??' as both the
"Latin small e with accent grave" and as "Latin small e" followed by a
"modifying accent grave". Second of all, what would you say makes a
character in a language? For the most part, ?? does not exist in English,
but, e.g., r??sum?? is the proper spelling. Then you get complicated cases
like Japanese, which can write in hiragana, katakana, kanji, or r??maji.
Technically, r??maji is merely Latin transliteration of Japanese, so it's
debatable how much it is or isn't Japanese.
Finally, you run into the ambiguities of Unicode codepoints. Are
fullwidth roman letters valid for en-US, even though English typography
doesn't distinguish between fullwidth and halfwidth? English also
borrows the characters of other languages for various purposes: remember
that the abbreviation for micrometer is `??m', so is `??' in en-US or not?
In my opinion, this is not generally useful enough to be worth having in
the standard library. Actually, I don't think Java even has Unicode
normalization functions, which are much more useful than divining
languages from code points.
Do you know of any java packages to address these NLP issues? or, if
you don't, is there something like that for text processing in ANSI C
or C++? ~ Thanks lbrtchx
What are you really trying to do? If you are trying to detect languages
based on codepoints, that is not going to work that well. You would be
far better trying to guess language based on letter frequency, or even
just parsing it different languages and seeing which language has the
least "misspelled" words.
--
Beware of bugs in the above code; I have only proved it correct, not
tried it. -- Donald E. Knuth