Technical Report Series on Corpus Building

Transkript

1 Technical Report Series on Corpus Building Vol. 9 (June 2013) Swedish Corpora Uwe Quasthoff Dirk Goldhahn Abteilung Automatische Sprachverarbeitung, Institut für Informatik, Universität Leipzig

2 Affiliation of the authors: Uwe Quasthoff, Dirk Goldhahn: Institut für Informatik,Universität Leipzig {quasthoff, Copyright: Abteilung Automatische Sprachverarbeitung, Institut für Informatik, Universität Leipzig, Technical Report Series on Corpus Building Vol. 1: Deutscher Wortschatz 2013 Vol. 2: Danish Corpora Vol. 3: Dutch Corpora Vol. 4: Icelandic Corpora Vol. 5: Hungarian Corpora Vol. 6: Ukrainian Corpora Vol. 7: Indonesian Corpora Vol. 8: Czech Corpora Vol. 9: Swedish Corpora This PDF document was created using the open source tool mwlib. For more infotmation, see PDF generated at: 26. June 2013

3 Swedish corpora 1 Introduction to corpus creation 1 SWE - a processing related language description 2 SWE corpora 3 SWE corpus comparison 8 Processing details 10 Appendix to swe news 2007: Database summary 10 Appendix to swe news 2008: Database summary 10 Appendix to swe news 2009: Database summary 11 Appendix to swe news 2010: Database summary 11 Appendix to swe news 2011: Database summary 12 Appendix to swe news 2012: Database summary 12 Appendix to swe newscrawl 2011: Database summary 13 Appendix to swe newscrawl 2012: Database summary 13 Appendix to swe web 2002: Database summary 14 Appendix to swe web 2011: Database summary 14 Appendix to swe web 2012: Database summary 15 Appendix to swe wikipedia 2007: Database summary 15 Appendix to swe wikipedia 2012: Database summary 16 Appendix to swe mixed 2012: Database summary 16 Content details 17 Appendix to swe news 2007: Size of different TLDs 17 Appendix to swe news 2008: Size of different TLDs 17 Appendix to swe news 2009: Size of different TLDs 18 Appendix to swe news 2010: Size of different TLDs 18 Appendix to swe news 2011: Size of different TLDs 18 Appendix to swe news 2012: Size of different TLDs 19 Appendix to swe newscrawl 2011: Size of different TLDs 19 Appendix to swe newscrawl 2012: Size of different TLDs 20 Appendix to swe web 2002: Size of different TLDs 20 Appendix to swe web 2011: Size of different TLDs 20

4 Appendix to swe web 2012: Size of different TLDs 21 Appendix to swe mixed 2012: Size of different TLDs 21 Appendix to swe news 2007: Size of largest domains 22 Appendix to swe news 2008: Size of largest domains 22 Appendix to swe news 2009: Size of largest domains 23 Appendix to swe news 2010: Size of largest domains 24 Appendix to swe news 2011: Size of largest domains 24 Appendix to swe news 2012: Size of largest domains 25 Appendix to swe newscrawl 2011: Size of largest domains 26 Appendix to swe newscrawl 2012: Size of largest domains 26 Appendix to swe web 2002: Size of largest domains 27 Appendix to swe web 2011: Size of largest domains 28 Appendix to swe web 2012: Size of largest domains 28 Appendix to swe mixed 2012: Size of largest domains 29 Appendix to swe news 2007: Number of sources by time period 30 Appendix to swe news 2008: Number of sources by time period 31 Appendix to swe news 2009: Number of sources by time period 33 Appendix to swe news 2010: Number of sources by time period 34 Appendix to swe news 2011: Number of sources by time period 35 Appendix to swe news 2012: Number of sources by time period 37 Word details 39 Appendix to swe news 2007: Words by length without multiplicity 39 Appendix to swe news 2008: Words by length without multiplicity 41 Appendix to swe news 2009: Words by length without multiplicity 43 Appendix to swe news 2010: Words by length without multiplicity 45 Appendix to swe news 2011: Words by length without multiplicity 47 Appendix to swe news 2012: Words by length without multiplicity 49 Appendix to swe newscrawl 2011: Words by length without multiplicity 51 Appendix to swe newscrawl 2012: Words by length without multiplicity 53 Appendix to swe web 2002: Words by length without multiplicity 55 Appendix to swe web 2011: Words by length without multiplicity 57 Appendix to swe web 2012: Words by length without multiplicity 59 Appendix to swe wikipedia 2012: Words by length without multiplicity 61 Appendix to swe mixed 2012: Words by length without multiplicity 63 Appendix to swe news 2007: Words by length with multiplicity 65 Appendix to swe news 2008: Words by length with multiplicity 67 Appendix to swe news 2009: Words by length with multiplicity 69

5 Appendix to swe news 2010: Words by length with multiplicity 71 Appendix to swe news 2011: Words by length with multiplicity 73 Appendix to swe news 2012: Words by length with multiplicity 75 Appendix to swe newscrawl 2011: Words by length with multiplicity 77 Appendix to swe newscrawl 2012: Words by length with multiplicity 79 Appendix to swe web 2002: Words by length with multiplicity 81 Appendix to swe web 2011: Words by length with multiplicity 83 Appendix to swe web 2012: Words by length with multiplicity 85 Appendix to swe wikipedia 2007: Words by length with multiplicity 87 Appendix to swe wikipedia 2012: Words by length with multiplicity 89 Appendix to swe mixed 2012: Words by length with multiplicity 91 Appendix to swe news 2007: The most frequent 50 words 92 Appendix to swe news 2008: The most frequent 50 words 93 Appendix to swe news 2009: The most frequent 50 words 94 Appendix to swe news 2010: The most frequent 50 words 95 Appendix to swe news 2011: The most frequent 50 words 96 Appendix to swe news 2012: The most frequent 50 words 97 Appendix to swe newscrawl 2011: The most frequent 50 words 98 Appendix to swe newscrawl 2012: The most frequent 50 words 99 Appendix to swe web 2002: The most frequent 50 words 100 Appendix to swe web 2011: The most frequent 50 words 101 Appendix to swe web 2012: The most frequent 50 words 102 Appendix to swe wikipedia 2007: The most frequent 50 words 103 Appendix to swe wikipedia 2012: The most frequent 50 words 104 Appendix to swe mixed 2012: The most frequent 50 words 105 Appendix to swe news 2007: Longest words in top by rank 106 Appendix to swe news 2008: Longest words in top by rank 107 Appendix to swe news 2009: Longest words in top by rank 108 Appendix to swe news 2010: Longest words in top by rank 109 Appendix to swe news 2011: Longest words in top by rank 110 Appendix to swe news 2012: Longest words in top by rank 111 Appendix to swe newscrawl 2011: Longest words in top by rank 112 Appendix to swe newscrawl 2012: Longest words in top by rank 113 Appendix to swe web 2002: Longest words in top by rank 114 Appendix to swe web 2011: Longest words in top by rank 115 Appendix to swe web 2012: Longest words in top by rank 116 Appendix to swe wikipedia 2007: Longest words in top by rank 117 Appendix to swe wikipedia 2012: Longest words in top by rank 118

6 Appendix to swe mixed 2012: Longest words in top by rank 119 Character N-gram details 120 Appendix to swe news 2007: Alphabet as used in the top words 120 Appendix to swe news 2008: Alphabet as used in the top words 121 Appendix to swe news 2009: Alphabet as used in the top words 122 Appendix to swe news 2010: Alphabet as used in the top words 123 Appendix to swe news 2011: Alphabet as used in the top words 125 Appendix to swe news 2012: Alphabet as used in the top words 126 Appendix to swe newscrawl 2011: Alphabet as used in the top words 127 Appendix to swe newscrawl 2012: Alphabet as used in the top words 128 Appendix to swe web 2002: Alphabet as used in the top words 129 Appendix to swe web 2011: Alphabet as used in the top words 131 Appendix to swe web 2012: Alphabet as used in the top words 132 Appendix to swe wikipedia 2007: Alphabet as used in the top words 133 Appendix to swe wikipedia 2012: Alphabet as used in the top words 134 Appendix to swe mixed 2012: Alphabet as used in the top words 136 Abbreviation details 138 Appendix to swe news 2007: Most frequent abbreviations 138 Appendix to swe news 2008: Most frequent abbreviations 139 Appendix to swe news 2009: Most frequent abbreviations 140 Appendix to swe news 2010: Most frequent abbreviations 141 Appendix to swe news 2011: Most frequent abbreviations 142 Appendix to swe news 2012: Most frequent abbreviations 143 Appendix to swe newscrawl 2011: Most frequent abbreviations 143 Appendix to swe newscrawl 2012: Most frequent abbreviations 144 Appendix to swe web 2002: Most frequent abbreviations 144 Appendix to swe web 2011: Most frequent abbreviations 145 Appendix to swe web 2012: Most frequent abbreviations 145 Appendix to swe wikipedia 2007: Most frequent abbreviations 146 Appendix to swe wikipedia 2012: Most frequent abbreviations 147 Appendix to swe mixed 2012: Most frequent abbreviations 148 Appendix to swe news 2007: Left neighbors of the full stop 148 Appendix to swe news 2008: Left neighbors of the full stop 149 Appendix to swe news 2009: Left neighbors of the full stop 150 Appendix to swe news 2010: Left neighbors of the full stop 151 Appendix to swe news 2011: Left neighbors of the full stop 152

7 Appendix to swe news 2012: Left neighbors of the full stop 153 Appendix to swe newscrawl 2011: Left neighbors of the full stop 154 Appendix to swe newscrawl 2012: Left neighbors of the full stop 155 Appendix to swe web 2002: Left neighbors of the full stop 156 Appendix to swe web 2011: Left neighbors of the full stop 157 Appendix to swe web 2012: Left neighbors of the full stop 158 Appendix to swe wikipedia 2007: Left neighbors of the full stop 159 Appendix to swe wikipedia 2012: Left neighbors of the full stop 160 Appendix to swe mixed 2012: Left neighbors of the full stop 161 Appendix to swe news 2007: Left neighbors of the full stop with additional internal full stops 162 Appendix to swe news 2008: Left neighbors of the full stop with additional internal full stops 163 Appendix to swe news 2009: Left neighbors of the full stop with additional internal full stops 164 Appendix to swe news 2010: Left neighbors of the full stop with additional internal full stops 165 Appendix to swe news 2011: Left neighbors of the full stop with additional internal full stops 166 Appendix to swe news 2012: Left neighbors of the full stop with additional internal full stops 167 Appendix to swe newscrawl 2011: Left neighbors of the full stop with additional internal full stops 168 Appendix to swe newscrawl 2012: Left neighbors of the full stop with additional internal full stops 169 Appendix to swe web 2002: Left neighbors of the full stop with additional internal full stops 170 Appendix to swe web 2011: Left neighbors of the full stop with additional internal full stops 171 Appendix to swe web 2012: Left neighbors of the full stop with additional internal full stops 172 Appendix to swe wikipedia 2007: Left neighbors of the full stop with additional internal full stops 173 Appendix to swe wikipedia 2012: Left neighbors of the full stop with additional internal full stops 174 Appendix to swe mixed 2012: Left neighbors of the full stop with additional internal full stops 175 Sentences details 176 Appendix to swe news 2007: Shortest sentences 176 Appendix to swe news 2008: Shortest sentences 177 Appendix to swe news 2009: Shortest sentences 179 Appendix to swe news 2010: Shortest sentences 180 Appendix to swe news 2011: Shortest sentences 182 Appendix to swe news 2012: Shortest sentences 183 Appendix to swe newscrawl 2011: Shortest sentences 185 Appendix to swe newscrawl 2012: Shortest sentences 186 Appendix to swe web 2002: Shortest sentences 188 Appendix to swe web 2011: Shortest sentences 189 Appendix to swe web 2012: Shortest sentences 191 Appendix to swe wikipedia 2007: Shortest sentences 192

8 Appendix to swe wikipedia 2012: Shortest sentences 194 Appendix to swe mixed 2012: Shortest sentences 195 Appendix to swe news 2007: Longest sentences 197 Appendix to swe news 2008: Longest sentences 199 Appendix to swe news 2009: Longest sentences 201 Appendix to swe news 2010: Longest sentences 203 Appendix to swe news 2011: Longest sentences 205 Appendix to swe news 2012: Longest sentences 207 Appendix to swe newscrawl 2011: Longest sentences 209 Appendix to swe newscrawl 2012: Longest sentences 211 Appendix to swe web 2002: Longest sentences 213 Appendix to swe web 2011: Longest sentences 215 Appendix to swe web 2012: Longest sentences 217 Appendix to swe wikipedia 2007: Longest sentences 219 Appendix to swe wikipedia 2012: Longest sentences 221 Appendix to swe mixed 2012: Longest sentences 223 Appendix to swe news 2007: Length of sentences in characters 225 Appendix to swe news 2008: Length of sentences in characters 226 Appendix to swe news 2009: Length of sentences in characters 227 Appendix to swe news 2010: Length of sentences in characters 228 Appendix to swe news 2011: Length of sentences in characters 229 Appendix to swe news 2012: Length of sentences in characters 230 Appendix to swe newscrawl 2011: Length of sentences in characters 231 Appendix to swe newscrawl 2012: Length of sentences in characters 232 Appendix to swe web 2002: Length of sentences in characters 233 Appendix to swe web 2011: Length of sentences in characters 234 Appendix to swe web 2012: Length of sentences in characters 235 Appendix to swe wikipedia 2007: Length of sentences in characters 236 Appendix to swe wikipedia 2012: Length of sentences in characters 237 Appendix to swe mixed 2012: Length of sentences in characters 238 Appendix to swe news 2007: Length of sentences in words 239 Appendix to swe news 2008: Length of sentences in words 240 Appendix to swe news 2009: Length of sentences in words 241 Appendix to swe news 2010: Length of sentences in words 242 Appendix to swe news 2011: Length of sentences in words 243 Appendix to swe news 2012: Length of sentences in words 244 Appendix to swe newscrawl 2011: Length of sentences in words 245 Appendix to swe newscrawl 2012: Length of sentences in words 246

9 Appendix to swe web 2002: Length of sentences in words 247 Appendix to swe web 2011: Length of sentences in words 248 Appendix to swe web 2012: Length of sentences in words 249 Appendix to swe wikipedia 2007: Length of sentences in words 250 Appendix to swe wikipedia 2012: Length of sentences in words 251 Appendix to swe mixed 2012: Length of sentences in words 252 Oddities details 253 Appendix to swe news 2007: Longest words 253 Appendix to swe news 2008: Longest words 253 Appendix to swe news 2009: Longest words 254 Appendix to swe news 2010: Longest words 254 Appendix to swe news 2011: Longest words 255 Appendix to swe news 2012: Longest words 255 Appendix to swe newscrawl 2011: Longest words 256 Appendix to swe newscrawl 2012: Longest words 256 Appendix to swe web 2002: Longest words 257 Appendix to swe web 2011: Longest words 257 Appendix to swe web 2012: Longest words 258 Appendix to swe wikipedia 2007: Longest words 258 Appendix to swe wikipedia 2012: Longest words 259 Appendix to swe mixed 2012: Longest words 259 Appendix to swe news 2007: Sentences with high average word length 260 Appendix to swe news 2008: Sentences with high average word length 261 Appendix to swe news 2009: Sentences with high average word length 262 Appendix to swe news 2010: Sentences with high average word length 263 Appendix to swe news 2011: Sentences with high average word length 264 Appendix to swe news 2012: Sentences with high average word length 265 Appendix to swe newscrawl 2011: Sentences with high average word length 266 Appendix to swe newscrawl 2012: Sentences with high average word length 267 Appendix to swe web 2002: Sentences with high average word length 268 Appendix to swe web 2011: Sentences with high average word length 269 Appendix to swe web 2012: Sentences with high average word length 270 Appendix to swe wikipedia 2007: Sentences with high average word length 271 Appendix to swe wikipedia 2012: Sentences with high average word length 272 Appendix to swe mixed 2012: Sentences with high average word length 273 Appendix to swe news 2007: Problems with sentence segmentation - words ending in a stopword 274 Appendix to swe news 2008: Problems with sentence segmentation - words ending in a stopword 275

10 Appendix to swe news 2009: Problems with sentence segmentation - words ending in a stopword 275 Appendix to swe news 2010: Problems with sentence segmentation - words ending in a stopword 276 Appendix to swe news 2011: Problems with sentence segmentation - words ending in a stopword 277 Appendix to swe news 2012: Problems with sentence segmentation - words ending in a stopword 278 Appendix to swe newscrawl 2011: Problems with sentence segmentation - words ending in a stopword 278 Appendix to swe newscrawl 2012: Problems with sentence segmentation - words ending in a stopword 279 Appendix to swe web 2002: Problems with sentence segmentation - words ending in a stopword 280 Appendix to swe web 2011: Problems with sentence segmentation - words ending in a stopword 281 Appendix to swe web 2012: Problems with sentence segmentation - words ending in a stopword 282 Appendix to swe wikipedia 2007: Problems with sentence segmentation - words ending in a stopword 283 Appendix to swe wikipedia 2012: Problems with sentence segmentation - words ending in a stopword 283 Appendix to swe mixed 2012: Problems with sentence segmentation - words ending in a stopword 284

11 1 Swedish corpora Introduction to corpus creation The Leipzig Corpora Collection (LCC) collects Web based corpora for many different languages. The main text genres are newspaper texts, Wikipedias and randomly collected web pages. All corpora are processed in the same way: Crawling Web pages HTML stripping Language identifikation Sentence segmentation Cleaning: Removal of ill-formed sentences Duplicate removal Calculation of word frequences and word co-occurrences As result we have a corpus containing only well-formed sentences in the language under consideration. The sentences are in random order; hence, sharing the corpus does not violate copyright law because it is impossible to reconstruct the original texts. The pre-processing steps contain both language independent steps (like HTML stripping and duplicate removal) and language dependent steps (like language identification and sentence segmentation). Especially the language specific parts are vulnerable to specific processing problems. The aim of the paper is to identify possible problems and evaluate the results. The following problems are adressed: A processing-focused language description Language size: How much text is available for this language? What are the biggest sources? Corpus description: Genre, size, crawling and processing date. Possible problems in language identification: Which languages are similar? Character set and alphabet Inspecting the word list: Most frequent words, longer high frequent words and longest words at all. Word length distribution. Can abbreviations confuse sentence segmentation? Information about the abbreviation list. Inspecting sentences: Inspect shortest and longest sentences to identify possible segmentation problems. Sentence length distribution. The paper describes the result of these inspections; the appendices show the exact results for the different corpora. This helps to compare the corpora with respect to quality. In the section quality overview, an overall quality description for each corpus is given. All corpora contain only minor problems which are irrelevant for most applications. Otherwise the corpus creation has been iterated.

12 SWE - a processing related language description 2 SWE - a processing related language description General properties of the Swedish language Native Name: Svenska Classifiation: Indo-European, Germanic, North, East Scandinavian, Danish-Swedish, Swedish Total Number of Speakers: 8.4M Largest countries with number of speakers: Sweden(8.0M). Also spoken in parts of Finland, where it has equal legal standing with Finnish. Largely mutually intelligible with Norwegian and Danish. Source: / www. ethnologue. com/ language/ swe Processing summary Latin alphabet with some additional characters full stop is used as sentence boundary and for abbreviations apostrostophes used rarely Properties important for processing Alphabet and punctuation The alphabet is Latin based, with the following specialities (source: / en. wikipedia. org/ wiki/ Swedish_alphabet): Swedish includes all 26 base letters and Å, Ä, Ö. In the alphabetic ordering, the letters Å, Ä, Ö follow Z at the end of the alphabet. Usual Latin punctuation Usage of uppercase letters: At sentence beginnings and for proper names (of persons, organisations, countries etc.). Sentence segmentation and word tokenization Sentence beginnings Sentences begin with a capitalized first word. Abbreviations Abbreviations confusing with sentence boundaries: Special abbreviation list has to be inspected. Sources for abbreviations:??? Abbreviations with full stop may appear in the word list without full stop. Apostrophes: The use of apostrophes is infrequent.

13 SWE - a processing related language description 3 Sources and ranking (2012) Estimated number of webpages containing text Google.com top-5 words: 337,000,000 results for "och" "i" "att" "som" "på" Google.com top-10 words: 232,000,000 results for "och" "i" "att" "som" "på" "är" "en" "av" "för" "med" Rank according to number of speakers (Ethnologue): 86 Rank according to Wikipedia size (see / de. wikipedia. org/ wiki/ Wikipedia:Sprachen): Rank 5 with articles ( ). Rank according to number of newspapers as found by AbyZ (5/2012): 256 newspapers, rank 10. Rank according to number of newspapers with RSS feeds (5/2012): 122 newspapers, rank 13. Rank according to our corpus size (9/2012): 13 SWE corpora Quality Overview Quality Ratings A: Very good quality. Ready to use (or already used) for frequency dictionary. Size as large as possible Only minimal errors Multiple genres (if possible) A-: Small problems identified. They should not affect usage. B: Native speaker quality. Information about abbreviations and sentence boundaries by native speaker Resulting statistics checked by native speaker, possible errors corrected C: Non-native speaker quality Obvious problems shown in corpus statistics are corrected D: First version Pre-processing with default abbreviation list and default sentence boundaries E: Poor Quality: Old, outdated or faulty. Corpus Quality The quality of the corpora differes slightly because the corpus processing toolchain changed slightly during several years. Moreover, original data are often no more available. Hence, improvement of quality often means removing incomplete or doubtful sentences. Forthcoming editions of all corpora thus might have a slightly smaller number of sentences. This especially applies to near duplicate sentences which are removed only sparingly. The following table shows the quality of the corpora. Minimal errors are still possible and described in the sections below. All possible major improvements are mentioned here.

14 SWE corpora 4 Corpus Quality rating Known problems to-dos swe_news_2007 A - - swe_news_2008 A - - swe_news_2009 A- Some uplicate sentences - swe_news_2010 A - - swe_news_2011 A - - swe_news_2012 A - - swe_newscrawl_2011 A- several near duplicate peaks - swe_newscrawl_2012 A - - swe_web_2002 A- max. 255 bytes instead characters - swe_web_2011 A - - swe_web_2012 A - - swe_wikipedia_2007 A- max. 255 bytes instead characters - swe_wikipedia_2012 A - - swe_mixed_2012 A - - Processing Overview For more details, see Appendix: Database Summary and Appendix: Number of sources by time period. Corpus Size (M sentences) Size (M running words) Multiwords Crawling date Production date swe_news_ mainly 2005 and swe_news_ daily 2008, 17% without date 2011 swe_news_ daily swe_news_ daily swe_news_ daily swe_news_ daily swe_newscrawl_ / swe_newscrawl_ / swe_web_ batch crawl swe_web_ / / swe_web_ / / swe_wikipedia_ / swe_wikipedia_ / swe_mixed_ see above 2013

15 SWE corpora 5 Content Overview For more details, see Appendix: Size of different TLDs and Appendix: Size of different domains. Corpus Type of sources Countries Number of sources Publishing date Biggest source swe_news_2007 News.se (93%),.fi(3%),.com(2%) 113 mainly 5/ / swe_news_2008 News.se swe_news_2009 News.se swe_news_2010 News.se swe_news_2011 News.se swe_news_2012 News.se(95%),.ax(5%) swe_newscrawl_2011 News.se(80%),.com(18%) and before swe_newscrawl_2012 News.se(82%),.fi(8%),.com(7%),.nu(2%) and before swe_web_2002 Web.se and before swe_web_2011 Web.se(88%),.com(5%),.fi(4%) and before swe_web_2012 Web.se(86%),.com(7%),.fi(3%) and before swe_wikipedia_2007 Wikipedia and before wikipedia.org swe_wikipedia_2012 Wikipedia and before wikipedia.org swe_mixed_2012 Mixed Sources.se(80%),.com(6%),.fi(3%) and before Words Appendix: Words by Length without multiplicity shows a plot of the corresponding length distribution. A smooth asymetric bell-shaped curve is expected. Appendix: Words by Length with multiplicity shows a plot of the corresponding length distribution. A smooth asymetric bell-shaped curve is expected. Appendix: The Most Frequent 50 Words shows the most frequent stopwords as well as one or more words related to the region. Appendix: Longest Words in Top-1000 by rank shows the 25 longest words within the top The usually give an impression of the main topics treated in the corpus. Appendix: Longest Words with minimum frequency 2 should give an idea of very long words. In the case of processing problems, different types of non-words may appear. This might help to improve the word definition.

16 SWE corpora 6 Corpus Word length graph without multiplicity Word length graph with multiplicity Most Frequent 50 Words Longest Words in Top-1000 Longest Words with minimum frequency 2 swe_news_2007 okay okay okay okay URLs, missing blanks swe_news_2008 okay okay okay okay missing blanks swe_news_2009 okay okay okay okay missing blanks, routes swe_news_2010 okay okay okay okay missing blanks, junk swe_news_2011 okay okay okay Rank 636: 71000@aftonbladet.se URLs, missing blanks swe_news_2012 okay okay okay okay okay swe_newscrawl_2011 okay okay okay okay Missing blanks, routes, junk, URLs swe_newscrawl_2012 okay okay okay okay URLs, missing blanks, junk, etc. swe_web_2002 okay okay okay okay URLs, missing blanks, chemicals swe_web_2011 okay okay okay okay Routes, URLs, missing blanks, junk swe_web_2012 okay okay okay okay Routes, missing blanks, URLs, junk swe_wikipedia_2007 okay okay okay Rank 971: RobotQuistnix Routes, URLs swe_wikipedia_2012 okay okay okay okay URLs swe_mixed_2012 okay okay okay okay all of the above Abbreviations Abbreviations are usually not used as sentence boundaries. Conversely, missing abbreviations can overgenerate sentence boundaries. Due to limitations in the processing chain, the list of abbreviations used for sentence boundary detection can differ from the abbreviations in the word list. Appendix: Most Frequent Abbreviations shows possible under-generation of sentence boundaries by wrong abbreviations (i.e. words ending in a full stop) in the word list. Sentences Appendix: Shortest sentences shows the shortest declarative, exclamatory and interrogative sentences. In preprocessing, a minimal length for sentences might be specified. And missing abbreviations are often visible as faulty sentence engings. Appendix: Longest sentences shows the longest declarative, exclamatory and interrogative sentences. Usually, the maximun sentence length is defined as 256 characters (not 256 bytes). Very long exclamatory or interrogative sentences often contain an overseen sentence boundary. Appendix: Length of sentences in characters shows the distribution of the sentence length. A large and balanced corpus will result in a smooth and bell-shaped curve. Isolated local maxima usually result from large sets of near duplicate sentences.

17 SWE corpora 7 Corpus Shortest sentences Longest sentences Length distribution (in characters) Length distribution (in words) swe_news_2007 okay max. 255 bytes instead characters okay okay swe_news_2008 okay okay okay okay swe_news_2009 Some uplicate sentences okay okay okay swe_news_2010 okay okay okay okay swe_news_2011 okay okay okay okay swe_news_2012 okay okay okay okay swe_newscrawl_2011 okay okay several near duplicate peaks okay swe_newscrawl_2012 okay okay okay okay swe_web_2002 okay okay max. 255 bytes instead characters okay swe_web_2011 okay okay okay okay swe_web_2012 okay okay okay okay swe_wikipedia_2007 okay okay max. 255 bytes instead characters okay swe_wikipedia_2012 okay okay okay okay swe_mixed_2012 okay okay okay okay Oddities Appendix: Sentences with high average word length: Average sentences contain many stopwords, and these stopwords are usually short. Hence, they restrict the average word length in a sentence. Conversely, sentences with high average word length are often ill formed. They may be used to improve pre-processing. Appendix: Problems with sentence segmentation - Words ending in a stopword: If there are many ill-formed word or sentence boundaries witout a blank between two words, they will generate new ill-formed words. The appendix shows the most frequent words ending in an uppercase stopword. If they are infrequent then the date were of high quality. Corpus Sentences with high average word length Words ending in a stopword swe_news_2007 missing blanks maxfreq=48 swe_news_2008 routes, proper names okay, maxfreq=8 swe_news_2009 okay okay, maxfreq=11 swe_news_2010 URLs, missing blanks, routes maxfreq=19 swe_news_2011 okay maxfreq=17 swe_news_2012 okay okay, maxfreq=4 swe_newscrawl_2011 URLs, missing blanks, junk maxfreq=203 swe_newscrawl_2012 missing blanks, junk maxfreq=94 swe_web_2002 URLs, junk, special characters maxfreq=33 swe_web_2011 URLs, missing blanks, routes, junk maxfreq=32 swe_web_2012 URLs, missing blanks, routes, junk okay, maxfreq=12 swe_wikipedia_2007 URLs, chemicals, routes okay

18 SWE corpora 8 swe_wikipedia_2012 URLs, Japanese, routes okay swe_mixed_2012 as above maxfreq=203 SWE corpus comparison Automated Corpus comparison For the following comparisons, the following tests on the top-1000 words are performed: Vectors based on the frequencies of the top-1000 words are created for the analysed languages. The cosine of the angle between these vectors is computed. Identical languages receive a value of 0, distinct languages get a value of 1. The same analysis is conducted using the frequencies of the top-1000 typical letter trigrams of the languages. Monolingual word list comparison (top-1000 words) As one can expect the comparisons show: The different news corpora have different word lists with maximum distance 0.23 (swe_newscrawl_2011 and swe_news_2011) The wikipedia corpora are similar with maximum distance 0.09 The web corpora have maximum distance 0.18 (swe_web_2002 and swe_web_2012) The mixed corpus hun_mixed_2012 holds a central position with maximum distances of 0.32 to the other corpora. Multilingual word list comparison (top-1000 words) Both the comparison of the top-1000 words and the comparison of the letter trigrams used in these words show that there are similar languages in our data, mainly members of the north germanic family. The distance of the mixed corpus to the next language, Slovak, is 0.47 for the words and 0.54 for the letter trigrams. Both distances are below average. The average value for the most similar language is 0.58 for trigrams. The most similar languages based on words: Danish, Norwegian (Bokmål), Norwegian (Nynorsk) source language_short_name language_name cos_logfreq swe dan Danish swe nob Norwegian, Bokmål swe nno Norwegian, Nynorsk swe loy Loke swe cat Catalan-Valencian-Balear The most similar languages based on letter trigrams: Danish, Norwegian (Bokmål), Norwegian (Nynorsk) source language_short_name language_name cos_logfreq swe dan Danish swe nob Norwegian, Bokmål swe nno Norwegian, Nynorsk

19 SWE corpus comparison 9 swe eng English swe nld Dutch

20 10 Processing details Appendix to swe news 2007: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords 0 Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences Appendix to swe news 2008: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences