M Alzghool D Inkpen 2007 Evaluation of multilingual and multi-modal information retrieval 4730 778--785 C. Peters, (Ed.), Springer. Alzghool, Inkpen, 2007 Alzghool, M. & Inkpen, D. (2007). Experiments for the cross language speech retrieval task at CLEF 2006. In C. Peters, (Ed.), Evaluation of multilingual and multi-modal information retrieval (Vol. 4730/2007, pp. 778-785). Springer. G Amati C J Van Rijsbergen 2002 Probabilistic Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, Seattle, Washington, United States. Amati, Van Rijsbergen, 2002 Amati, G. & Van Rijsbergen, C. J. (2002). Probabilistic Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, Seattle, Washington, United States. D W Oard D Soergel D Doermann X Huang G C Murray J Wang B Ramabhadran M Franz S Gustman 2004 Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, Sheffield, United Kingdom. by an expert in the field. A set of 63 training topics and 33 test topics were generated for this task. The topics provided with the collection were created in English from actual user requests. Topics were structured using the standard TREC format of Title, Description and Narrative fields. To enable CL-SR experiments the topics were translated into Czech, German, French, and Spanish by native speakers; Figure 2 and 3 show two examples for English and its translation in French respectively. Relevance judgments were generated using a search-guided procedure and standard pooling methods. See (Oard et al., 2004) for full details of the collection design. We present results on the automatic transcripts for English queries and translated queries (cross-language) for two combination methods; we also present results when manual summaries and manual keywords are indexed. <DOC> <DOCNO>VHF[IntCode]-[SegId].[SequenceNum]</DOCNO\> <INTERVIEWDATA>Interviewee name(s) and birthdate</INTERVIEWDATA> <NAME>Full name of every person mentioned</NAME> <MANUALKEYWORD>Thesaurus keywords assigned to the segment</MANUALKEYWORD> <SUMMARY>3-sentence segment summary</SUMMARY> <ASRTEXT2004A>ASR transcript produced in 2004</AS Oard, Soergel, Doermann, Huang, Murray, Wang, Ramabhadran, Franz, Gustman, 2004 Oard, D. W., Soergel, D., Doermann, D., Huang, X., Murray, G. C., Wang, J., Ramabhadran, B., Franz, M., & Gustman, S. (2004). Building an information retrieval test collection for spontaneous conversational speech, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, Sheffield, United Kingdom. D W Oard J Wang G J F Jones R W White P Pecina D Soergel X Huang I Shafran 2007 Evaluation of multilingual and multi-modal information retrieval 4730 744--758 C. Peters, (Ed.), Springer, Heidelberg. se the text collection is automatically transcribed spontaneous speech, with many recognition errors. Also, the topics are real information needs, difficult to satisfy. Information Retrieval systems are not able to obtain good results on this data set, except for the case when manual summaries are included. 1. Introduction Conversational speech such as recordings of interviews or teleconferences is difficult to search through. The transcripts produced with Automatic Speech Recognition (ASR) systems tend to contain many recognition errors, leading to low Information Retrieval (IR) performance (Oard et al., 2007). Previous research has explored the idea of combining the results of different retrieval strategies; the motivation is that each technique will retrieve different sets of relevant documents; therefore combining the results could produce a better result than any of the individual techniques. We propose new data fusion techniques for combining the results of different IR models. We applied our data fusion techniques to the Mallach collection (Oard et al., 2007) used in the Cross-Language Speech Retrieval (CLSR) task at Cross-Language Evaluation Forum (CLEF) 2007. The Mallach collection comprise Oard, Wang, Jones, White, Pecina, Soergel, Huang, Shafran, 2007 Oard, D. W., Wang, J., Jones, G. J. F., White, R. W., Pecina, P., Soergel, D., Huang, X., & Shafran, I. (2007). Overview of the CLEF-2006 cross-language speech retrieval track. In C. Peters, (Ed.), Evaluation of multilingual and multi-modal information retrieval (Vol. 4730/2007, pp. 744-758). Springer, Heidelberg. I Ounis G Amati V Plachouras B He C Macdonald D Johnson 2005 In Advances in information retrieval 3408 517--519 Springer, Heidelberg. topic in CL-SR test collection. <top> <num>1159 <title>Les enfants survivants en Suède <desc>Descriptions des mécanismes de survie des enfants nés entre 1930 et 1933 qui ont passé la guerre en camps de concentration ou cachés et qui vivent actuellement en Suède. <narr>... </top> Figure 3. Example for French topic in CL-SR test collection. 2. System Description Our Cross-Language Information Retrieval systems were built with off-the-shelf components. For the retrieval part, the SMART (Buckley, Salton, &Allan, 1992; Salton &Buckley, 1988) IR system and the Terrier (Amati &Van Rijsbergen, 2002; Ounis et al., 2005) IR system were tested with many different weighting schemes for indexing the collection and the queries. SMART was originally developed at Cornell University in the 1960s. SMART is based on the vector space model of information retrieval. We use the standard notation: weighting scheme for the documents, followed by dot, followed by the weighting scheme for the queries, each term-weighting scheme is described as a combination of term frequency, collection frequency, and length normalization components where the schemes are abbreviated according to its components variations (n no normalization, zghool &Inkpen, 2007; Inkpen, Alzghool, &Islam, 2006); lnn.ntn means that lnn was used for documents and ntn for queries according to the following formulas: weightlnn= ln(tf)+1.0 (1) weight ntn= tf × log (2) N nt where tf denotes the term frequency of a term t in the document or query, N denotes the number of documents in the collection, and nt denotes the number of documents in which the term t occurs. Terrier was originally developed at the University of Glasgow. It is based on Divergence from Randomness models (DFR) where IR is seen as a probabilistic process (Amati &Van Rijsbergen, 2002; Ounis et al., 2005). We experimented with the In_expC2 (Inverse Expected Document Frequency model with Bernoulli after-effect and normalization) weighting model, one of Terrier’s DFR-based document weighting models. Using the In_expC2 model, the relevance score of a document d for a query q is given by the formula: sim(d, q) qtf .w ( t , d) = ∑ t q ∈ where qtf is the frequency of term t in the query q, and w(t,d) is the relevance score of a document d for the query term t, given by: 1 w t d ( , ) ( = ) ( log × tfn × ) (4) where -F is the term frequency of t in the whole collection. -N is the number of document i Ounis, Amati, Plachouras, He, Macdonald, Johnson, 2005 Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., & Johnson, D. (2005). Terrier information retrieval platform In Advances in information retrieval (Vol. 3408/2005, pp. 517-519). Springer, Heidelberg. P Pecina P Hoffmannov´a G J F Jones Y Zhang D W Oard 2007 Evaluation, . CLEF2007, Budapest-Hungary. Pecina, Hoffmannov´a, Jones, Zhang, Oard, 2007 Pecina, P., Hoffmannov´a, P., Jones, G. J. F., Zhang, Y., & Oard, D. W. (2007). Overview of the CLEF-2007 cross language speech retrieval track, Working Notes of the CLEF- 2007 Evaluation, . CLEF2007, Budapest-Hungary. G Salton C Buckley 1988 Information Processing and Management, 24 5 513--523 Salton, Buckley, 1988 Salton, G. & Buckley, C. (1988). Term weighting approaches in automatic text retrieval. Information Processing and Management, 24(5): 513-523. J A Shaw E A Fox 1994 In Third text retrieval conference (trec-3) 105--108 Shaw, Fox, 1994 Shaw, J. A. & Fox, E. A. (1994). Combination of multiple searches. In Third text retrieval conference (trec-3) (pp. 105-108). National Institute of Standards and Technology Special Publication.