A Comparative Analysis of Text Summarization in Hindi and English Languages Using the TF-IDF Approach

In India, which is a multilingual society with a huge amount of digital information, tools which can summarize the data are required. This research explores the use of TF-IDF (Term Frequency-Inverse Document Frequency) method for creating extractive summaries in two different languages, namely Indo-Aryan (Hindi) and Germanic (English). The aim of the study is to compare the effectiveness of model in both monolingual contexts. A set of news articles in Hindi and English were prepared. The strategy adopted was to extensively preprocess the texts (tokenization and removal of stop words and stemming/lemmatization) and then use TF-IDF (term frequency-inverse term frequency) to numerically represent the sentences based on their importance. The most representative sentences were chosen from the top-n sentences sorted by aggregated TF-IDF scores to make up the summary. The summaries generated automatically were compared with the reference summaries prepared by human using ROUGE-N measure, and informative and coherent extractive summaries in both languages were generated by TF-IDF model. The quantitative evaluation performed with ROUGE-1 and ROUGE-2 gave similar results at English and Hindi language, however, there was a slight decrease in Hindi because of the morphological complexity, which can be explained primarily. The model was shown to be robust and efficient to compute. The results obtained from this work demonstrate that TF-IDF algorithm is a decent baseline approach for bilingual summarization, particularly in resource poor environments. It is used as the basis for more complex models and shows the need for language-specific pre-processing for multi-language applications in NLP. The results are relevant to the implementation of user-friendly information retrieval systems in linguistically varied parts of the world.