https://sms.linguistik.uzh.ch/ 2026-08-06T15:18:52+00:00 https://sms.linguistik.uzh.ch/ https://sms.linguistik.uzh.ch/_media/wiki/logo.png text/html 2022-06-27T07:21:32+00:00 Anonymous (anonymous@undisclosed.example.com) https://sms.linguistik.uzh.ch/03_processing/01_cleaning?rev=1656314492&do=diff Cleaning the data up No kind of censorship whatsoever was applied to the data. But still some SMS had to be removed from the original data, especially duplicate SMS that were created due to technical problems as well as SMS that were obviously not written by humans, such as notifications about new MMS etc. text/html 2022-06-27T07:21:32+00:00 Anonymous (anonymous@undisclosed.example.com) https://sms.linguistik.uzh.ch/03_processing/02_anonymization?rev=1656314492&do=diff Anonymization General privacy While the project did not have the intention of collecting private information about the informants (other than what they provided in the questionnaire), it could still not be assumed that the informants would not sent personal information in their SMS, so it was the team's task to remove specific pieces of information again. These steps were performed by means of computational linguistics. Phone numbers and names of the senders were never available to the team bu… text/html 2022-06-27T07:21:32+00:00 Anonymous (anonymous@undisclosed.example.com) https://sms.linguistik.uzh.ch/03_processing/03_languages?rev=1656314492&do=diff Language tagging Types of taggings Each SMS was tagged for the languages contained within. There are three possible tags: * Main language: For each SMS, a main language was defined as the dominant language, i.e. the language which provides most words to the SMS. text/html 2022-06-27T07:21:32+00:00 Anonymous (anonymous@undisclosed.example.com) https://sms.linguistik.uzh.ch/03_processing/04_tokenizing?rev=1656314492&do=diff Tokenizing In very general term, a token can be seen as a word. Every sentence consists of different words, from a technical point of view we call them tokens. However, that is not all there is to a sentence, there is punctuation, too, and maybe a number, an emoticon, a line break etc. All these are tokens, too. text/html 2022-06-27T07:21:32+00:00 Anonymous (anonymous@undisclosed.example.com) https://sms.linguistik.uzh.ch/03_processing/05_normalization?rev=1656314492&do=diff Normalization The spelling of many tokens in the SMS corpus deviate from the standard spelling rules of the according language and consequently, automated data processing such as the annotation of parts of speech (PoS) is difficult if not impossible. In order to overcome these problems, a second, normalised parallel corpus was created per language. In this normalised corpus, every token is spelled in a way that can be interpreted by tools used in computational linguistics. text/html 2022-06-27T07:21:32+00:00 Anonymous (anonymous@undisclosed.example.com) https://sms.linguistik.uzh.ch/03_processing/06_pos?rev=1656314492&do=diff Part of speech tagging Wikipedia defines PoS tagging as follows: "In corpus linguistics, part-of-speech tagging (POS tagging or POST), also called grammatical tagging or word-category disambiguation, is the process of marking up a word in a text (corpus) as corresponding to a particular part of speech, based on both its definition, as well as its context, i.e. relationship with adjacent and related words in a phrase, sentence, or paragraph. " In this corpus, we applied PoS tagging to the German…