textToWords abstract method
- String text
Given unsegmented text, perform text segmentation particular to the
language and return a list of parsed words.
For example, in the case of Japanese, '日本語は難しいです。', this should ideally return a list containing '日本語', 'は', '難しい', 'です', '。'.
In the case of English, 'This is a pen.' should ideally return a list containing 'This', ' ', 'is', ' ', 'a', ' ', 'pen', '.'. Delimiters should stay intact for languages that feature such, such as spaces.
Implementation
List<String> textToWords(String text);