Swedish Named-Entity Recognition in the Nomen Nescio Project the work conducted during 2002

Transkript

1 Outline Swedish Named-Entity Recognition in the Nomen Nescio Project the work conducted during 2002 Dimitrios Kokkinakis 1 Current Status of the Swedish System What has been accomplished in 2002 Approach, architecture, individual modules, examples... The (Swedish) Resources in more detail Anatomy of the grammars and the DCA approach NE Tagset and some General/Specific Guidelines mainly Some Preliminary Evaluations based on SUC2 Evaluation on some difficult/frequent cases Work in 2003 Continuous development, better integration of modules and better DCA implementation Extensive testing & evaluation (second half of 2003) Extensions: Web-Interface, New Groups (e.g. Money) 2... Status of the Swedish System: The Work in Status of the Swedish System: General Approach Gathered Lists of Single and Multiword Names (www, corpora, available name lists) Studied large corpora and extracted a number of (incomplete) patterns using reg. expressions, which were then generalised and formalised into fs-rules; tested, extended......those rules were implemented as sets of context- sensitive finite-state grammars semantic grammars, one for each NE-group Implemented a first version of the DCA-algorithm Defined an XML-schema for the validation of NE- annotated data Application on unannotated texts (portability, ease of reusability ) Modular and scalable architecture Since no individual criterion can achieve both high precision and recall; a combination of criteria (external & internal evidence) are applied Separation of lexical (internal) and grammatical (external) resources Five major components (see later slide...) 3 4 Evidence: McDonald (1996) Internal: is taken from within the sequence of words that comprise the name, such as the content of lists of proper names (gazetteers), abbreviations and acronyms External: provided by the context in which a name appears the characteristic properties or events in a syntactic relation (verbs, adjectives) with a proper noun can be used to provide confirming or criterial evidence for a name s category important complementary information since internal evidence can never be complete... Operational definition of NE: NE are words or sequences of words which usually cannot be found in common (defining) dictionaries, and yet encapsulate important info that can be useful for the semantic interpretation of Source Texts Tokenization... Status of the Swedish System: Architecture FS- Grammars Single-Word Lists Multiword Lists Filter/ DCA Revision texts Output.xml 5 6

2 Approximate number of resources: CATEGORY PERSON LOCATION ORGANIZATION OBJECT EVENT WORK&ART Current Status of the Swedish System MULTI LISTS , SINGLE LISTS 80,000 4,500 1, # GRAMM. RULES internal external 7 8 internal external Lexical Resources (1) (Internal Evidence) Name Lists (Gazeteers) Single names Organiz. (FIN): 42 Organiz. (TVR): 15 Organiz. (POL): 35 Organiz. (ATH): 9 Multiword names Organizations (PLT): 35 Organizations (ATH): 80 Organizations (FIN): 70 Organizations (TVR): 25 Organizations (CRP): 1100 Organiz. (CRP):1,800 Locat. (non-s): 2,400 Locat. (countr): 320 Locat. (S): 1,700 Persons: 80,000 Events: 20 Objects: 30 Work&Art: 50 Locations (GPL): 15 Locations (FNC): 75 Locations (PPL): 320 Events: 20 Objects: 150 Lexical Resources (2) (Internal Evidence) Lexical Resources (3) (Internal Evidence) Designators, affixes, short phrases and trigger words Titles, premodifiers, appositions... Trigger words, affixes, MOVE-verbs and prepositions Trigger words, graphical criteria Organizations Trigger words: bolaget X, föreningen X, institutet X, X Agency, X Biotech, X Chemical, X Consultancy, Affixes:+kollegium,+verket,... Designators: AB, A/S, IFK,... Phrases: *match mot, VD för Persons PostMods Mods: Jr, Junior, PreTitles: VD, Dr, sir, Nationality: belgaren, brasilianaren, dansken, Profess.: amiral, inspektor,... Honor.: von, van der, af,... Verbs: säger, frågar... Locations Trigger words: floden, ö,... Affixes: *väg,*gatan,*byn, *staden, *... Prepos.: längs X, söder om X, väster om, utanför,... Short Phrases: ligger nära MOVE-verbs verbs: resa till,... Events Trigger words: turneringen, tävlingen, spel, utställning, mässa, festival, kongress, workshop, Games, Final, Cup,... Graph. criteria: tävlingen Lexical Resources (4) (Internal Evidence) Trigger words, verbs, graphical criteria and affixes Trigger words, small phrases, verbs Work&Art Trigger words: boken, filmen, dikten, plattan, Affixes:+posten,+journalen, +kuriren,... Verbs: regisserat X, föreläsa om, komponerat X, sjöng X... Objects Trigger words & phrases: ombord på, sorten, märket, medlet, sjukdomen,... Verbs: flyger en, äter en, dricker en, lanserar, tillverkar, Grammars: Anatomy of PERSONs /*OCCUPATION-TRIGGERS TRIGGERS PRECEEDING*/ EDING*/ [^\n ]*(bil jur bas kem...)(ist isten)(" \'" " \:" ",")?(" {U}[^ \n]*)+ [^\n ]*(bil jur bas kem...)(ister isterna)(" \'" " \:" ",")? (" {U}[^ \n]*)+([ \,]+{U}[^ \n]*)*(" "(och eller)(" "{U}[^ \n]*)+)? /*OCCUPATION-TRIGGERS TRIGGERS FOLLOWING*/ {U}[^ \n]+" "{U}[^ \n]+" \, "(professor expert...) /*NATIONALITY-TRIGGERS*/ TRIGGERS*/ (bulgari fransy japan...)(skor skorna)(" \'" " \:" ",")? (" {U}[^ \n]*)+([ \,]+{U}[^ \n]*)*(" "(och eller)(" "{U}[^ \n]*)+)? /*TYPICAL VERBS FOLLOWING NAME*/ {U}[^ \n]+" "{U}[^ \n][^ \n]+" "(jobbade arbetar ) /* TYPICAL VERBS PRECEEDING NAME*/ (säger förklarar...)" "{U}[^ \n]+(" "{U}[^ \n]*)* /*TRIGGER PHRASES*/ (gift kamrat...)" med "{U}[^ \n]*(" "{U}[^ \n]*)* (manus regi...)" av "{U}[^ \n]*(" "{U}[^ \n]*)* U=[A-ZÅÄÖÈÉ] 12

3 Rules capture local patterns that characterize entities, extracted from part-of of-speech annotated data and semi- automatic analysis of corpora: XXX köpte YYY: XXX and YYY are with high probability organizations EMI köpte Virgin Music Group Grundin köpte Hornline Moyne köpte Trustor Optiroc köpte Stråbruken Pandox köpte Park Avenue Hotel SF köpte Europafilm Stagecoach köpte Swebus Trelleborg köpte Intertrade Grammars: General Pattern Examples.*(trafiken banan tåget vägen bygggen bygg e sträckan tunnel...) mellan XXX och YYY : XXX and YYY are with high probability locations tåget mellan Warszawa och Berlin sträckan mellan Bornholm och Gotland ^XXX YYY, [0-9] 9]+ +,:, XXX and YYY are with high probability humans Robert Broberg, 51, Maria Bäckström, 25, Hillary Clinton, 44, 13 DCA (1)...for names not covered by contextual patterns No evidence to disambiguate Idea: Important words are typically used in a document more than once and in different contexts. Ambiguous words and phrases are usually unambiguously introduced at least once in the text unless they re part of common knowledge presupposed to be known by the readers ; Mikheev 00 Accordingly: look at unambiguous usages and then assign these to words in ambiguous positions A kind of inference & generalization process used to accurately tag isolated unknown names; prior knowledge (e.g. rules) is used in order to make the performance increase Important step in case of only UPPER/lowercase texts see later the Matti & Julius examples es 14 Annotated Text [... <LPG Telecom AB>...] DCA (2) - Process Array of Annotations <LPG Telecom AB> Partial Orderings of Annotations (if>1tagged) (excl. Single occ. of den, det, a, AB...) <LPG Telecom> <Telecom AB> <LPG> <Telecom> also <Lpg><TELECOM> Apply the Newly Learned Tags on Unannotated Tokens Produce (Hopefully) New Annotations Filter/Revision Capabilities Technique inspired by Mooney 93 which he calls theory refinement Tags put on NEs at previous steps are considered as default (preferential) tags & can be modified These can be updated given an appropriate context e.g. isolated occurrences of Sverige Sverige are tagged as LOC/GPL but in a sequence such as Sverige klar favorit i EM, Sverige Sverige is an ORG/ATH and thus the LOC/GPL annotation will be replaced by a new tag NE Tagset and General Guidelines e.g.: produce a single, unambiguous output for any relevant string; no nested expressions will be marked no extra whitespaces or carriage returns to be inserted in the markup; only tags in angled brackets; the markup has the form of the entity type and attribute information <ELEMENT-NAME NAME ATTR-NAME= ATTR NAME= ATTR-VALUE > VALUE >text-string</enamex> definite articles commonly associated with entity names are tagged: <Den < Norske Opera> romance numerals & arabic ordinals associated with entity names are tagged: <Gustav IV Adolf> defined an XML-schema for the validation of tagged texts ptoduced by the system <ELEMENT-NAME NAME ATTR-NAME= ATTR NAME= ATTR-VALUE > VALUE ><METHOD= METH- VALUE />text text-string</enamex> 17 Annotation Guidelines First draft specifications for the creation of simple guidelines for the NER work as applied on Swedish data have been written Ideas from MUC, ACE,, Eckhard s s reports, own experience... The guidelines have been evolved during the course of the project, refined and (might be) extended The purpose of the guidelines is to try and impose some consistency measures for annotation and evaluation, and for giving the potential future users of the system a clearer picture of what the recognition components can offer 18

4 Some Name Taxonomies in the Litterature Bauer G Namenkunde des Deuschen e.g. Anthroponyms, Ergonyms,, etc Jonasson K Le Nom Propre. Cosntructions et Interprétations Paik et al Categorizing and Standardizing Proper Nouns for Efficient IR Sheremetyeva et al A Multilingual Onomosticon as a Multipurpose NLP Resource Sekine et al Extended Named Entity Hierarchies... Guidelines - PERSONS HUM: : humans (alive or dead), fictional human characters appearing in TV, movies etc. appositives (Jr) or structural words (von) are part of the name MTH: : names of saints, apostles, gods, mythical names, humanoids, religious figures e.g. Gud, Allah, Jesus, Djävulen,... ANM: : names of animals, pets and mythical beasts e.g. Pegasus, pudeln Zeb,... CLC: : collective names of tribes, dynasties, ethnical and race names e.g. Yorubafolket, Yarlung-dynastin, Guidelines - LOCATION AST: : astronomical defined location with physical extent; planets, comets, galaxies GPL: : geographically/geologically defined location; bodies of water, archipelagos, rivers, mountains, continents, oceans PPL: : geo-social social-political entities, politically defined geographical regions; nations, states, towns, villages, territories, provinces FNC: : man-made made artefacts falling under the domains of architecture, transportation infrastructure and civil engineering; museums, parks, tunnels, stadiums, galleries, factories, bridges, airports, mil. bases STR: : names of streets, avenues, roads and postal addresses 21 Guidelines - ORGANIZ. CRP: : corporations, company groups, governmental, non- profit org., businesses, unions FIN: : financial institutions, banks, capital management and funding organizations ATH: : sports teams, athletic organizations, even mentions of regions in sport-related related contexts CLT: : artistic and music groups, circus, orchestras, bands PLT: : political parties, groups and movements, terrorist and criminal organizations, liberation armies TVR: : organization with a media profile, tv-channels and radio stations 22 Guidelines - EVENTS HPL: : historical or political events and manifestations, wars, scandals, battles e.g. Gulfkriget, Förintelsen,... WTH: : events that include some kind of natural motion; weather phenomena, hurricanes, cyclones, storms e.g. El Niño, orkanen George,... CLU: : festivals, fairs and conferences, workshops e.g. Vattenfestivalen, Expo 98,... ATL: : sports races, games and competitions, tournaments e.g. US Open, VM 1997, Formel 3000, Guidelines - OBJECTS MDC: : medical and pharmaceutical products, names of medicines, diseases, proteins e.g. Xenical, Alzheimers, Omega-3,... FWP: : food and wine products, names of drinks, fruits e.g. Granny Smith, Biff Oscar, Fanta,... CMP: : S/W and H/W products, as well as telephony VH(AGW AGW): vehicles and transportation means, water, land and air/space PRZ: : scholarships, honoursh and prizes PRD: : general products and artefacts that do not fit in the previous subcategories e.g. Marlboro Lights, Libress Invisible,... 24

5 Guidelines - WORK&ART WRT: : names that deal primarily with written material, magazines, newspapers, journals, books e.g. Bibeln, Vanity Fair,... RTV: : names that denote radio and tv-programs; tv-series, soap-operas operas e.g. TV-serien Glappet, Lilla Sportspegeln,... WAA: : names of paintings, songs, films, operas, movies e.g. Filmen Titanic, operan Staden,... PERSON LOCATION ORGANIZATION OBJECT EVENT ENTITY attr.: TYPE PRS LOC ORG OBJ EVN NE Tagset attr: SBT HUM,ANM ANM,MTH,CLC AST,STR,GPL,PPL,FNC FIN,TVR TVR,ATH,CLT,PLT, CRP VHW,VHA,VHG,CMP, MDC,PRZ,FWP,PRD HPL,CLT,ATH,WTH WORK&ART WRK WRT,RTV RTV,WAA Notes on the Evaluation Evaluation consists of (at least) three parts: Entity Detection (of the string that names an entity): <ENAMEX>Fj Fjärran Östern</ENAMEX> Attribute Recognition/Classification (of the entity); <ENAMEX TYPE= LOCATION SBT= XXX >Fj Fjärran Östern</ENAMEX> Extent Recognition (measure the ability of a system to correctly determine an entity s extent partial correctness): Fjärran <ENAMEX TYPE= LOCATION LOCATION >Östern</ENAMEX> 27 Notes on the Evaluation cont d Existing Existing systems identify names on the scale ~90-95% 95% in newswire texts (several languages) Metrics: Metrics: Vary from test case to test case; the commonest definitions are: Precision = #CorrectReturned# CorrectReturned/# /#TotalReturned Recall = #CorrectReturned# CorrectReturned/# /#CorrectPossible Quite Quite high figures in P&R can be found in the litterature based exclusively on these metrics... Papers Papers lack a thorough discussion on metonymy or other difficult cases; while the evaluations have been solely based on monolithic types of texts Notes on the Evaluation cont d SUC-2 Guidelines for more rigid evaluation criteria have been imposed by the MUC; e.g. Precision = Correct + ( 0.5 * Partially Correct ) Actual Correct: two single annotations are considered identical Partially Correct: two single annotations are not identical, but partial credit should still be given Actual = Correct + Incorrect + Partially Correct + Spurious Spurious: a response object has no key object aligned with it Recall = Correct + ( 0.5 * Partially Correct ) Possible Effetiveness: : (Meadow et al. 00) ((1-P) 2 + (1-R) 2 ) Eff = The SUC-2 2 (appx 1,1 mil. tokens) has been semi- automatically? annotated with NAME-tags PERSON 8771 PLACE 6309 INST 1887 WORK Här har <NAME TYPE=ANIMAL>Nalle 638 PRODUCT </NAME> frukosterat OTHER 364 ANIMAL 280 MYTH 245 EVENT 242 FORMULA...årsmöte i <NAME TYPE=OTHER> Kristiansborgskyrkan</NAME>...ber <NAME TYPE=MYTH>Herren </NAME> välsigna vår......till nitrat ( <DISTINCT TYPE=FORMULA> NO3-</DISTINCT> ) och därefter... 30

6 SUC-2; Problems... Though manually checked?, SUC-2 2 is not completely error-free; for instance:...over 80 cases of PERSON-tags in which the role names are part of the annotation <NAME TYPE=PERSON>professor Ulf af Trolle</NAME> <NAME TYPE=PERSON>fröken Lundmark</NAME> <NAME TYPE=PERSON>komminister Harry Åström</NAME>...cases of inconcistencies in the annotations Plåtslagarns hus under de [...] <TYPE=PERSON>Plåtslagarmästarens</NAME> hus, hade han [...]...sonsöner i bön till den Allsmäktige, glad kunde väl inte ens <NAME TYPE=MYTH>Den Allsmäktige</NAME>...han påpekade att <NAME TYPE=PERSON>Jesus Kristus</NAME> i själva...sin uppenbarelse från <NAME TYPE=MYTH>Jesus Kristus</NAME>...erroneous tags? Efter slaget på <NAME TYPE=PLACE>Vita berget</name> 1620 [...]...ballong med inskriptionen <NAME TYPE=EVENT>Teaterdagar</NAME> 31 The EVENT s in SUC-2 Tested the current status of the Swedish system that recognizes & annotates EVENT entities A rather open category with difficulties to reach consensus on its content The SUC-2 2 annotations of EVENT (245) seems at a first glance to closely resemble the NN-group of EVENT Hypothesis: does our system recognize more events than those annotated in SUC2? Answer: YES Mixed-All: F=82,65% Eff=0,77 P=97,6%(166/170) R=67,7%(166/245) 32 EVENT : Errors Analysis 1 partial correct: <EVENT>U.S.-Japan Workshop on Smart</> / Intelligent Materials and Systems; 3 annotations were not in the SUC2 sentences with EVENT-tags tags: VM-kvalet, märkes-vm, världscupen There where a few questionable cases in the SUC2:...ballong med inskriptionen <SUC EVENT>Teaterdagar</> Dubbeltriumfen i <SUC EVENT>Wimbledon</> var inte sämre... Some unrecognized cases by the swedish system:... torsdag, är det <SUC EVENT>Qingming</>, dagen då gravarna... " <SUC EVENT>Globengrejen</> " förra året var ju... <SUC EVENT>Barcelona</>, då?...såg dock till att de dominerade <SUC EVENT>Chiquitatalangen</>,......Kulturhusets annonserade " <SUC EVENT>Titelmatch i 15 satser</> "......en minneslapp där det står " <SUC EVENT>Canossa</>...sammankomster där " <SUC EVENT>Catechismi Förhör</> " hölls... EVENTs not Marked in SUC2 FOUND 168 EVN"" not marked as such in SUC2; 32 of those found were wrong; of which ATG (12),, Paris All Stars (3), Warszawapakten (2) e.g. Musikerna i Paris All Stars strålar samman... ;...konstruerade men aldrig avslutate Multi-User-Dungeonspel ( MUD )... ; Researrangören heter Holiday on Ice och då SBT="HPL">andra världskriget</enamex> 9 SBT="ATL">Allsvenskan</ENAMEX> 8 SBT="HPL">första världskriget</enamex> 4 SBT="CLU">D-föreställning</ENAMEX> 2 SBT="WTH">orkanen Andrew</ENAMEX> 2 SBT="HPL">Vietnamkriget</ENAMEX> 2 SBT="HPL">Tjernobylolyckan</ENAMEX> 2 SBT="HPL">finska inbördeskriget</enamex> 2 SBT="CLU">Karl XII-utställning</ENAMEX> 2 SBT="CLU">Dionysosfesten</ENAMEX> 2 SBT="CLU">1992 års FN-konferens</ENAMEX>...skedde på <SUC EVENT>Yom Kippur</> nittonhundratrettisex SBT="ATL">VM</ENAMEX>... EVENT-discrepancies in SUC2 egentligen inte har i <NAME TYPE=EVENT>OS</NAME> att göra. bara några dagar före <NAME TYPE=EVENT>OS</NAME>... första matchen här i OS-turneringen... Den här säsongen satsar jag på <NAME TYPE=INST>AIK</NAME> och OS, Men de har inte hört om <NAME TYPE=EVENT>Tjernobyl</NAME>,...svårt av effekterna av <NAME TYPE=EVENT>Tjernobylolyckan</NAME>...minska effekterna av Tjernobylolyckan, agerandet under <NAME TYPE=EVENT>andra världskriget</name> och han kom själv hit som flykting under andra världskriget. och hörde till slutet av första världskriget till... The MYTH s in SUC-2 Tested the current status of the Swedish system that recognizes & annotates MYTH entities The SUC-2 2 annotations of MYTH (280) seems at a first glance to closely resemble the NN-group of PRS:MTH Hypothesis: does our system recognize more myths than those annotated in SUC2? Answer: YES Mixed-All: F=83,2% Eff=0,82 P=88,6%(218/246) R=77,8%(218/280) 35 36

7 MYTH : Errors Analysis Strange cases <NAME TYPE=MYTH>Paris</NAME>, vrålar den bedragne <NAMETYPE=PERSON>Menelaos</NAME>, är en horbock......placerade där<name TYPE=MYTH>Atlantis</NAME>, den sjukna kontinenten...att <NAME TYPE=PERSON>Läraren</NAME> själv skulle återuppstå......tolkas som <NAME TYPE=PERSON>Jesus</NAME>, eftersom han liksom...fungerade den <NAME TYPE=MYTH>Gudomliga Uppenbarelsen</NAME> som en mall......beroende av den <NAME TYPE=MYTH>Gudomliga Uppenbarelsen</NAME>...aldrig ätit av <NAME TYPE=MYTH>Kunskapens träd</name> ska skriva <NAME TYPE=MYTH>Den stora amerikanska romanen</name>...pocketversionen av <NAME TYPE=MYTH>Baskervilles hund</name> MYTHs not Marked in SUC2 FOUND 1461 "MTH" not marked as such in SUC2 (tagged as PERSON); 5 of those found were wrong; ; e.g....översättningar är extremt ordagranna, t_ex Klagovisorna, Hesekiel 85 SBT="MTH">Jesus</ENAMEX> 33 SBT="MTH">Jesu</ENAMEX> 8 SBT="MTH">Gud</ENAMEX> 4 SBT="MTH">Guden</ENAMEX> 2 SBT="MTH">Jungfru Maria</ENAMEX> 2 SBT="MTH">guden</ENAMEX> 1 SBT="MTH">profeten Mohammeds</ENAMEX> 1 SBT="MTH">profeten Jesus</ENAMEX> 1 SBT="MTH">Johannes Döparen</ENAMEX> 1 SBT="MTH">Jesus Kristus</ENAMEX> 1 SBT="MTH">heliga Anden</ENAMEX> 1 SBT="MTH">den Allsmäktige</ENAMEX> 1 SBT="MTH">Buddha</ENAMEX> 38...sonsöner i bön till den Allsmäktige,...glad kunde väl inte ens <NAME TYPE=MYTH>Den Allsmäktige</NAME>...är att profeten <NAME TYPE=PERSON>Jesus</NAME> verkligen...den försoning som <NAME TYPE=PERSON>Jesus</NAME> död innebär...himmelske <NAME TYPE=MYTH>Jesus</NAME> hade uppenbarat sig...frälsande tron på <NAME TYPE=MYTH>Jesus</NAME>... påpekade att <NAME TYPE=PERSON>Jesus Kristus</NAME> i själva...sin uppenbarelse från <NAME TYPE=MYTH>Jesus Kristus</NAME> Gode Gud, om det syntes på henne. <NAME TYPE=PERSON>David</NAME> skrek till å Gud...och den<name TYPE=MYTH>Gud</NAME> välsigne våra tappra...bekant med <NAME TYPE=MYTH>Guds</NAME> röst MYTH-discrepancies in SUC2 39 The ANIMAL s in SUC-2 Tested the current status of the Swedish System that recognizes & annotates ANIMAL entities The SUC-2 2 annotations of ANIMAL (364) seems at a first glance to closely resemble the NN-group of PRS:ANM Hypothesis: does our system recognize more animals than those annotated in SUC2? Answer: NO! Mixed-All: F=51,9% Eff=0,36 P=93%(40/43) R=10,9%(40/364) WHY?? 40 ANIMAL : Errors Analysis Two evaluations were conducted. One using only the "ANIMAL" part of the PERSON-grammar and one using the grammar plus the DCA. In the first case: Precision=93% (40/43) & Recall=10,9% (40/364). Using the grammars and DCA module the Precision slightly increased (97,4%) while the Recall was substantially raised to 81,6%. Why is that? The vast majority of the "ANIMAL" occurrences in SUC are found in i texts from the area "K"="Imaginative prose" and particularly "KK"="General fiction" (~60%; 218 of 364) Simple DCA on these texts raises Recall! See the two examples of "Matti" and "Julius" Mixed-All: F=89,5% Eff=0,73 41 The Volvo/Saab Evaluation (1) the Volvo/Saab case (can be generalized with other meton. cases) a typical, frequent and fairly difficult example that illustrates s how metonymic distinctions are captured in the grammars searched in corpora (872 inst.); idea: : capture and model difficult & unusual cases, tag the rest as default (in this case <ORG:CRP>); e.g....saab mellanklassbilar som Volvo,......att köra Volvo i en Volvostad som... Object: <VHG>... i en stor svart Volvo och blinkade......tjuven försvinner i en stulen Saab...tappat kontrollen över sin Volvo Volvo steg med 12 kronor Object: <PRD> Saab backade med 1 peocent...gick Volvo ned med 10 kronor... rest of cases are tagged as default... Organization: <CRP> 42

8 The Volvo/Saab Evaluation (2) The Volvo/Saab Evaluation (3) Sense1: object; product (vehicle); Patterns after corpusc analysis 1. (Poss-Pron Color-Adj Partcpl)* (Saab Volvo) NUM+ [A-Z] Z]? 2. (Poss-Pron Color-Adj Partcpl)* (Saab Volvo) NUM? [A-Z] Z]? (coupé turbo dieselcabriolet corvette transporter cc... turbo dieselcabriolet corvette transporter cc...) 3. bil(ar)? (av märket som) m (Saab Volvo) NUM? [A-Z] Z]? 4. (Typical-Verbs:köra Typical-Nouns) (Saab Volvo) Sense2: object; share; Patterns after corpusc analysis 1. (Saab Volvo) AUX? VERB(steg/stig*/backa /backa*) 2. (Saab Volvo) AUX? VERB(öka ka*/minska*) /minska*)? med NUM procent 3. (Saab Volvo) gick (tillbaka kraftigt mot strömmen upp ned) 4. (Saab Volvo) NUM procent no rules without exception: dagarna kör r Volvo två fälttester i Sverige but the above simplistic patterns return, in the 143 (of 872) occurrences that were OBJ/VHG, a 95,1% (118/124) precision and 82,5% (118/143) recall the above patterns return, in the 11 (of 872) occurrences that were OBJ/PRD, a 100% (10/10) precision and 90,9% (10/11) recall Sense3: location; share; No patterns could be found after corpusc analysis Sense4: organization; was the Default sense since the vast majority in corpora are of this sense The Volvo/Saab Evaluation (4) the default pattern (organization) returned, in the 710 (of 872) occurrences that were ORG/CRP, a 95,9% (708/738) precision and 99,7% (708/710) recall In the 872 instances there were also 4 cases not captured by the patterns, and not belonging to the 4 sense distinctions discussed: d: e.g. Kungsträdgården <VHG>Volvo</> showroom 19/ The Volvo/Saab Evaluation (5) In the extracted 872 fragments from corpora containing either Volvo/Saab: 143 were OBJ/VHG, 4 were OBJ/VHA, 710 were ORG/CRP, 11 were OBJ/PRD, 3 were LOC/FNC and 1 were ORG/ATH LOC/FNC Error: Kungsträdgården <VHG>Volvo</> showroom 19/9... OBJ/ORG Error:...i dagarna kör <VHG>Volvo</> två fälttester i... OBJ/VHG=>VHA Errors:...och <VHG>Saab 2000</> trappas ner......av typen <VHG>Saab 2000</> med utrustning för kontroll...jag flugit <VHG>Saab B 17</>! Herregud... OBJ/VHG Error:...svågerns <ORG>Volvo</> under S:t Eriksbron... OBJ/VHA Error:...bilsätena till <ORG>Volvo</> skärs och sys... Mats Salomons <ORG>Saab</> på tvären och voltar... <ORG>Volvo</> har sin instrumentmiljö, trist... UNDECIDABLE:... I say <ORG>Saab</>, I say Bjorn and Benny. 46 Problems: Metonymy, Adjacent Names a speaker uses a reference to one entity to refer to another entity or entities related to it; in ALL groups we found words metonymic with other groups! OBJ#LOC:... A-influensan som är ganska lik Asiaten och Hongkong OBJ#PRS:...gäver av pipspännarkonstruktion som bär namn som Carl Gustaf... WRK#W&A:...släpptes första singeln Château d amour... ANM#PRS: Underbara Barbara galloperade... EVN#PRS: Maria Marathon vs Stockholm Marathon Adjacent names might be a problem in certain contexts:...på Karolinska sjukhuset i Solna Gunilla Bolinder...enligt Zhang Yimov Kinas mest berömda komikern...grundaren av Ruter Dam Gunilla Arhén <GP030121>Efter bilen kommer turen till hunden ALTEA : På en bensinstation i kuststaden Altea i sydöstra Spanien ( cirka tio mil söder om Valencia ) kan hundägare slå två flugor i en smäll. På stationen erbjuds inte bara tvätt av bilen, här kan även hunden få sig en grundlig rengöring. Peter Ardon och hans till synes skeptiska hund Pia var i går på platsen och undersökte automatens valmöjligheter. Man kunde välja mellan fyra olika tvättningsprogram : tvätt, skölj, avlusning och torkning. Vad Peter Ardon och Pia valde förtäljer inte historien. Wrapping UP: example 1 MULTIS GRAMMARS DCA FILTER SINGLE 47 48

9 Wrapping UP: example 2 Wrapping UP: example 3 <GP030121>Erdogan friad från mutmisstankar ANKARA : Den turkiske ledaren Recep Tayyip Erdogan, ledare för det regerande Rättvise- och utvecklingspartiet ( AKP ), friades i går från anklagelser om korruption. En domstol i Ankara har undersökt påståenden om att Erdogan skaffat sig en förmögenhet genom mutor under sin tid som borgmästare i Istanbul på 1990-talet. Erdogan och hans parti AKP tog hem en jordskredsseger i valet i november. Men Erdogan har inte tillåtits att ta plats i parlamentet. Orsaken är att han 1998 dömdes för religiös uppvigling. En lagändring gör att Erdogan nu kan ställa upp i ett fyllnadsval den 9 februari. ( TT-DPA ) MULTIS GRAMMARS DCA FILTER SINGLE <GP030121>LGP Telecom gör nytt försök att köpa Allgon LGP Telecom gör ett nytt försök att köpa antenntillverkaren Allgon. För tre år sedan blev det nobben till fusionen. I dag när verkligheten ser annorlunda ut och priset är en femtedel verkar Allgons ägare beskedligare. LGP Telecom lade i går ett bud på samtliga aktier i Allgon. Köpet ska ske genom betalning med egna nyutgivna aktier. Budet är värt 760 miljoner kronor baserat på börskurserna i de bägge bolagen som båda är underleverantören till telekomindustrin. Budpremien - det erbjudna priset jämfört med Allgons börsvärde - angav LGP till 62 procent. Men den premien åts snabbt upp när LGP:s aktie rasade med 20 procent sedan affären blivit känd i går. MULTIS GRAMMARS DCA FILTER SINGLE Wrapping UP: example 4 SUC-file file: : KK60 - "Matti" I hela sitt liv väntade Matti på att jag skulle ta initiativ. Jag överförde mänskliga egenskaper på Matti. Matti 55 occ. Matti fick tre promenader och mental träning varje dag. 2 matches! I början förutsatte jag att Matti var en normal hund. I början av januari fick Matti stygnen borttagna där hans testiklar hade opererats bort. En månad efter kastrationen var skillnaden i Mattis humör liten eller obefintlig. Matti hade ätit I/D i fyra och ett halvt år. Matti stank hel och hållen. Matti låg på sin madrass. Sigge kom in i rummet och befann sig omkring tre meter från Matti. <ENAMEX TYPE="PRS" SBT="ANM">Matti Matti</ENAMEX> morrade kraftigt. <ENAMEX TYPE="PRS" SBT="ANM">Matti Matti</ENAMEX> morrade ännu mer och reste sig. Matti ville ha sin liggplats ifred, det var naturligt. Sedan ville Matti gå in. Matti visste att Janne och jag ville honom väl. Jag hade ett sorgset och förtvivlat tonfall när jag pratade med och om Matti. Matti hörde mitt sorgsna tonfall och blev påmind om att han var sjuk Wrapping UP: example Julius" SUC-file file: : KK58 Julius. Julius hade ingen respekt för Matti. Julius ställde sig på bakbenen och lekte med Mattis svans. När Matti ville nosa satte Julius upp en tass : Stopp! På nätterna ville Julius vara hos mig. Julius fick lov att vara uppe hos mig igen. Julius låg och spanade efter kroppsdelar som rörde sig. Julius När Matti låg på sidan kröp Julius ihop intill Mattis mage. På morgnarna satte jag ut mat åt Julius i ladugården. Julius höll sig runt gården. Julius var ensam hemma. Det var som om Julius lockade till sig mössen. Jag ville inte att Julius skulle bo i lägenhet. Om inte Julius fanns... <ENAMEX TYPE="PRS" SBT="ANM">Julius Julius</ENAMEX> var en underbar katt. En morgon tog jag Julius med migi bilen. Han tog med sig Julius en bit in i skogen. Julius åt.... Julius 30 occ. 1 match! 52 Some Final Remarks One of the challenges with NER is creating a stable definition of what an entity is and creating a taxonomy of entities to map to... Having done that it becomes simpler to solve moste of the metonymy and other ambiguity problems... Problems still remain; where shall we draw the entity boundaries? Have we chosen the best approach? How can we best measure performance? (different text genres, formats etc)... Plan for 2003 February-May May: : continuous testing of modules in isolation and together; integration of modules (different permutations) May-August August: : work on the interface June-November November: : large-scale evaluation on an improved SUC2? and/or other material December: : wrapping-up Upcoming events: NODALIDA 03 & ACL 03 Workshop Multilingual NER 53 54