Towards Proactively Forecasting Sentence-Specific Information Popularity within Online News Documents

ACM source attribution: Sayar Ghosh Roy, Anshul Padhi, Risubh Jain, Manish Gupta, Vasudeva Varma. “Towards Proactively Forecasting Sentence-Specific Information Popularity within Online News Documents.” Proceedings of the 33rd ACM Conference on Hypertext and Social Media (HT ’22), 2022. DOI: 10.1145/3511095.3531268. Full text, figures, and tables are reconciled from the authorized ACM publisher HTML and proceedings PDF.
Sayar Ghosh Roy, Information Retrieval and Extraction Lab, International Institute of Information Technology, Hyderabad, India, sayar.ghosh@research.iiit.ac.in
Anshul Padhi, Information Retrieval and Extraction Lab, International Institute of Information Technology, Hyderabad, India, anshul.padhi@research.iiit.ac.in
Risubh Jain, Information Retrieval and Extraction Lab, International Institute of Information Technology, Hyderabad, India, risubh.jain@students.iiit.ac.in
Manish Gupta, Information Retrieval and Extraction Lab, International Institute of Information Technology, Hyderabad, India and Microsoft, India, manish.gupta@iiit.ac.in
Vasudeva Varma, Information Retrieval and Extraction Lab, International Institute of Information Technology, Hyderabad, India, vv@iiit.ac.in
DOI: https://doi.org/10.1145/3511095.3531268 HT '22: Proceedings of the 33rd ACM Conference on Hypertext and Social Media, Barcelona, Spain, June 2022
Multiple studies have focused on predicting the prospective popularity of an online document as a whole, without paying attention to the contributions of its individual parts. We introduce the task of proactively forecasting popularities of sentences within online news documents solely utilizing their natural language content. We model sentence-specific popularity forecasting as a sequence regression task. For training our models, we curate InfoPop, the first dataset containing popularity labels for over 1.7 million sentences from over 50,000 online news documents. To the best of our knowledge, this is the first dataset automatically created using streams of incoming search engine queries to generate sentence-level popularity annotations. We propose a novel transfer learning approach involving sentence salience prediction as an auxiliary task. Our proposed technique coupled with a BERT-based neural model exceeds nDCG values of 0.8 for proactive sentence-specific popularity forecasting. Notably, our study presents a non-trivial takeaway: though popularity and salience are different concepts, transfer learning from salience prediction enhances popularity forecasting. We release InfoPop and make our code publicly available1.
CCS Concepts: • Information systems; • Computing methodologies → Neural networks; • Information systems → Content ranking;
Keywords: Sentence Popularity Forecasting,
Sentence Salience Prediction, Supervised Transfer Learning
ACM Reference Format: Sayar Ghosh Roy, Anshul Padhi, Risubh Jain, Manish Gupta, and Vasudeva Varma. 2022. Towards Proactively Forecasting Sentence-Specific Information Popularity within Online News Documents. In Proceedings of the 33rd ACM Conference on Hypertext and Social Media (HT '22), June 28-July 1, 2022, Barcelona, Spain. ACM, New York, NY, USA 10 Pages. https://doi.org/10.1145/3511095.3531268

1 INTRODUCTION

In a typical document popularity prediction task, the objective is to estimate the would-be popularity of some content, say, a news article that is published on an online platform. A popularity prediction system might evaluate the number of pageviews that a particular online document would receive, or a social-media popularity prediction model might estimate the number of signatures, comments, shares, or likes that a specific social media post would accrue over a certain period of time.

Though popularity prediction is a well-studied machine learning task, with approaches ranging from traditional feature-based methods to recent Deep Neural Network architectures, all of the existing works on popularity prediction have focused exclusively on the document-level prediction of prospective popularity labels [18, 31]. When a system predicts that a specific article will be popular over time, a natural follow-up is to ask why. Which information pieces in the document would receive the most notice? Could we identify the informative sentences that would dominate the popularity level of the article? Being able to generate such insights proactively would have wide-ranging business applications in article promotion, popularity-guided summarization and pull quote selection, etc.

But sentence-specific forecasting of popularity has not been attempted, not even in a simplified binary classification of sentences into popular or not setting. The difficulty in obtaining reliable sentence-level labeled data for information popularity has been a crucial driving factor for the same.

An online article's popularity can be defined in terms of its page views [34, 40], which is a by-product of the regular Internet browsing activities of the global population. Similarly, within a particular online document, information popularity of a unit sentence is proportional to the number of received requests actively seeking its contained information. To capture this, we leverage incoming search queries encountered by a popular search engine over a sufficient period to ascertain sentence-specific popularity labels based on how queried-after each sentence is within a document. We present InfoPop, a dataset of over 50,000 online news articles with over 1.7M sentences, mapped to popularity scores.

Figure 1

Figure 1: Task outline. Looking solely at document text as input, forecast prospective sentence-specific popularity scores.

Taking inspiration from existing research on document-level popularity prediction, we frame the problem of sentence-specific popularity forecasting as a regression task similar to previous studies [17, 35] (who model document-level popularity) as opposed to a simple binary classification of a document's sentences into popular or not. Unlike some document-popularity forecasting approaches that rely on post-publication signals like pageview hits in the first half-hour after publication [2, 17], we forecast sentence popularity scores proactively (before a document's publication). Note that we utilize incoming search queries only to define sentence-specific popularity and construct the InfoPop dataset. However, our problem formulation is one of query-insensitive relative normalized scoring of sentences. Formally, given just a document as input, our task (Fig. 1) is of assigning normalized scores in the range [0, 1], to every sentence indicating their intra-document relative information popularity, without utilizing any external signals.

The key novelty of our approach lies in the design of auxiliary transfer learning subtasks adapting the STILTs (Supplementary Training on Intermediate Labeled-data Tasks) [28] technique. In brief, we create auxiliary subtasks of sentence salience prediction in the domain of online news, framed under an identical problem formulation as that of our primary task of popularity forecasting (Sec. 5). We experiment with a baseline neural model using RNNs (Recurrent Neural Networks) and a robust sequence regression architecture based on BERT (Bidirectional Encoder Representations from Transformers) [6], showcasing our proposed approach's efficacy. Also, we handle arbitrarily long documents with our neural models employing a sliding window mechanism over a document's sentences. In our analysis, we illustrate the differences in the concepts of popularity and salience, both qualitatively and quantitatively (Sec. 8). The empirically demonstrated efficacy of our approach brings out an interesting yet non-trivial takeaway – though salience and popularity are varying concepts, transfer learning from salience prediction improves popularity forecasting.

Overall, in this paper, we make the following contributions.

We introduce the task of proactively forecasting relative information popularities of sentences within online news documents solely based on their natural language content, without the use of any external features.

We present InfoPop, the first labeled dataset containing over 50,000 online news documents from 26 news websites with over 1.7M sentences, each mapped to supervised popularity scores.

Through a novel STILTs-based transfer learning approach, we build high-performance neural models reaching nDCG scores over 0.8 for sentence-specific popularity forecasting.

The remainder of the paper is organized as follows. In Section 2, we discuss related work on document-level popularity prediction, automatic text summarization and snippet generation. We formally present the concept of sentence-specific information popularity in Section 3. We describe details of curation of our dataset, InfoPop in Section 4. Next, we outline our proposed methods in Section 5. Further, we discuss evaluation metrics in Section 6, experimental details in Section 7 and present detailed results with analysis in Section 8. Finally, we conclude with a brief summary in Section 9.

2 RELATED WORK

In this section, we discuss related work on document-level popularity prediction, automatic text summarization and snippet generation.

2.1 Document-level Popularity Prediction

Existing works in popularity prediction have considered a piece of content as an atomic unit whose prospective popularity is inferred. Based on the choice of popularity surrogate, we distinguish between two kinds of document popularity prediction problems.

Popularity based on Internet browsing: Studies on forecasting popularities of online news documents typically consider the number of received page load requests over sufficient time (accumulated pageviews or hit count) as a surrogate for article-level popularity [17, 18, 40].

Social Media Popularity: These approaches aim to estimate the prospective engagement of a piece of content put up on a particular social media platform. Previous works have dealt with multi-modal posts [41], images [42], movies [1], petitions [35], etc. and have utilized user-behavior based markers such as the number of comments [32, 36], shares [25, 38], etc. as surrogates of social-media popularity. There has also been limited work on time-aware [14] and time-series prediction of social-media popularity [20, 29].

Unlike all previous approaches that determine popularity at the document-level, we focus on sentence-specific popularity forecasting (as shown in Fig. 1) with our domain of information being online news, not social-media posts.

For informative documents (including online news), a preferred surrogate of popularity has been the number of pageview hits [3, 31, 34]. Intuitively, pageviews captures the generic browsing trends of the population not limited to social media actions. Our task, however, requires knowledge of more fine-grained Internet browsing activity for annotating specific sentences with popularity labels.

2.2 Automatic Text Summarization

Text summarization aims to recognize information that is central to the core idea of one or more document(s). Approaches are roughly divided into two categories, namely, abstractive: generate a concise, coherent, and cogent summary that captures the central idea expressed by the text piece [19, 33], and extractive: select and arrange salient plus diverse ranges of text to form a summary [43, 44]. Extractive summarization is often framed as a sequence classification problem – binary labels are assigned to sentences indicating their presence in the summary [11, 21]. In contrast, our task necessitates sequential regression (numerical sentence-specific scores are forecasted). Also, text summarization has the additional objective of generating text without repeated information. In that, an otherwise summary inclusion-worthy sentence may have a binary label of 0 if a similar sentence is already present in the oracle summary [15] (in an extractive setting). However, we expect almost identical sentences in an article to have very similar information popularity values.

2.3 Snippet Generation

A page snippet can be a document excerpt that lets a user understand whether a document is pertinent to their query without accessing it in its entirety. Snippet-generation [4] has a ranking-based problem formulation somewhat related to sentence popularity forecasting. However, our task is one of query-insensitive scoring of sentences. In contrast, tasks such as snippet generation (and document retrieval [12]) are query-sensitive [10] – typical methods require a particular query tailored to which an appropriate snippet is generated (except for naïve approaches that output initial tokens).

3 SENTENCE-SPECIFIC INFORMATION POPULARITY

Document popularity magnitudes and popular-or-not labels based on pageviews capture the amount of notice that the document receives on the Internet over a period of time [18, 40] and is a consequence of the everyday Internet browsing actions of the worldwide population.

Table 1: Selected sentences from a document in InfoPop with their true and forecasted popularities and predicted salience. Popularity forecasts are from our best performing model on nDCG (BERT${{textrm{Reg}}}$ with TL = S${{textrm{L}}}$). Salience predictions are based on BERT${{textrm{Reg}}}$ trained on S1. [TPL: True Popularity Label, FPL: Forecasted Popularity Label, PSL: Predicted Salience Label, TPR: True Popularity Rank, FPR: Forecasted Popularity Rank, PSR: Predicted Salience Rank].

Sentence	TPL	FPL	PSL	TPR	FPR	PSR
Local carriers and drivers will be able to book and move their first loads with Uber Freight in the next few weeks, CEO Lior Ron wrote in a blog posted Wednesday.	0.1099	0.1522	0.1237	5	4	1
Uber Freight plans to expand to more European countries this year.	0.2046	0.1914	0.0924	2	2	4
The EU and U.S. freight markets have problematic similarities.	0.0092	0.0274	0.0736	10	6	7
They're both huge – the EU truckload market is a $400 billion marketplace and third after China and the U.S. – and inefficient.	0.0067	0.0048	0.0818	12	7	6
“The European trucking market is experiencing a severe shortage of drivers, and of the time drivers are on the road, 21 percent of total kilometers travelled are empty”, Ron wrote.	0.0110	0.0041	0.0892	9	8	5
Inefficiency of this scale results in shippers struggling to find available drivers to move their goods.	0.0132	0.0015	0.0530	7	12	10
Uber Freight has been scaling up its business since launching in May 2017, growing from limited regional operations in Texas to the rest of the continental U.S.	0.1356	0.1301	0.1018	4	5	2
The company has offices in San Francisco and Chicago.	0.0158	0.0024	0.0590	6	10	9
In August, Uber announced that it would make Uber Freight a separate unit and more than double its investment into the business.	0.2703	0.1968	0.0988	1	1	3
It's also made some key hires, one of which intimated the company's global ambitions.	0.0076	0.0015	0.0382	11	11	12
The company has made headway breaking into the U.S. market.	0.0116	0.0025	0.0675	8	9	8
Uber Freight had about 30,000 active users last quarter.	0.1923	0.1906	0.0427	3	3	11

With increased Internet penetration, the average number of queries encountered daily by commercial search engines has exceeded the billion mark2. Google Trends3 showcases popular topics of interest segregated by region and timespan based on the pool of encountered search queries, and the collection of worldwide queries serves as a guide to mark information of universal interest. Correspondingly, within the local context of a single news document (say D), if an information piece, say I1, is more queried-after by the global population than, say, piece I2 in D, then: popularity(I1) > popularity(I2). Popularity levels of sentences within D can be derived based on this base principle, wherein the popularity of a specific sentence (say s) in D is incremented based on the lexical similarity between s and an encountered query.

For a news document D with sentences [s1, s2,..., sN], comparing each sentence $lbrace sirbrace {i=1}^N$ to every search query encountered by a commercial search engine would be computationally infeasible. Moreover, only a negligibly small percentage of all encountered queries would positively contribute to sentences’ popularities within D. Such queries would precisely be the ones for which D could be deemed relevant. Thus, to derive scores of si’s within D, we filter incoming queries and consider the sublist, say Q = [q1, q2,..., q|Q|], for which the document D was significantly relevant.

We regard D as a significantly relevant document for query q if D was shown within the top 10 search results (which roughly translates to the first page of news results) when the search engine encountered q. We assign a base score of $sum {j=1}^{|Q|}similarity(qj, si)$ to each sentence $lbrace sirbrace {i=1}^N$. Sentence popularity scores are then normalized to a [0, 1] range such that the popularity labels of sentences within D sum up to 1.

4 InfoPop Dataset

In this section, we outline the creation of the InfoPop dataset and present its basic statistics.

4.1 Data Acquisition

We scraped 82,540 news documents from 26 reputed online news websites (websites listed in Fig. 2). We accessed incoming queries of Microsoft Bing Search and mapped each document to the global assemblage of queries that deemed the document as significantly relevant (Sec. 3) at the time when it was encountered. We extracted each article's text content and split that into an ordered list of sentences. With news articles as unit data points, individual sentences served as our atomic information pieces. We discounted articles with over 100 sentences as we found many of these to be long non-news documents.

Figure 2

Figure 2:

InfoPop: news sources (on x-axis) with their corresponding #documents (on y-axis).

4.2 Preprocessing

We encountered a considerable amount of noise in the extracted webpage texts. These primarily manifested as strings of seemingly random word tokens. We removed such heavily non-grammatical sentences using two dependency parsing-based heuristics.

Remove a sentence if its dependency graph is not a tree. The dependency graph of a grammatical sentence is supposed to be a tree and not a forest. We evaluated whether the produced dependency graph of a sentence was both connected and acyclic or not.

Remove a sentence if its dependency parse contains some xcomp branch leading to a single participle. The X complement or xcomp is a dependency label used to mark dependent clauses that do not contain a subject. Consider a noisy unit such as ‘I would like to, I would like to, I would like to.’ The parse tree of such a text string contains an xcomp branch leading to a participle at the leaf node. We observed a positive correlation between such phenomena and text repetition.

Though the above cleaning heuristics are computationally expensive, we employ these owing to the one-time nature of the operation. After these cleanup steps, we further removed articles containing less than three grammatical sentences.

4.3 Popularity Labels

For a particular online document D (say), we considered the collection of all queries, say Q = [q1, q2,..., q|Q|], for which D was significantly relevant. We generated supervised popularity labels for every sentence in D using the method outlined in Sec. 3. More formally, Pi ← $frac{sum {j=1}^{|Q|}similarity(qj, si)}{sum {i=1}^{N} sum {j=1}^{|Q|}similarity(qj, si)}$, ∀i ∈ {1, 2,..., N} where Pi denotes the normalized popularity label assigned to the ith sentence in D, assuming D contains N sentences: s1, s2,..., sN.

We used cosine-similarity between corresponding TF-IDF vectors as the measure of similarity. Though our similarity function might feel primitive, we uphold that utilizing lexical similarity is optimum in this case since search engine ranking algorithms still utilize word match as the basic principle and most relevant pages typically contain the query terms (that usually do not have well defined semantic embeddings) themselves. That said, we plan to explore semantic embedding based similarity as part of future work.

Figure 3

Figure 3:

InfoPop: Distribution of #sentences per document.

4.4 InfoPop Dataset Statistics

InfoPop contains 51,770 news documents containing 1,711,890 sentences annotated with normalized popularity scores. Each document contains a minimum of 3 and a maximum of 100 sentences, with the average number of sentences per article being 33.07. Fig. 3 shows the distribution of the number of sentences per document. On average, each sentence contains 18.23 word tokens. The minimum and maximum document lengths stand at 15 and 2,516 tokens, respectively, with 602.92 tokens being the average document length. The news articles are sourced from 26 reputed news websites, with the average number of articles per website being 1991.15. Fig. 2 illustrates the specific number of documents sourced from each news website. We observe that popularity scores and sentence lengths have a weak positive correlation of 0.168. We divide our dataset into train, validation, and test splits in an 8:1:1 ratio.

To ensure reproducibility, we make InfoPop (and our defined train, validation, and test splits) publicly available here4. Table 1 shows an excerpt from a document within InfoPop with sentence-specific popularity labels (TPL).

5 METHODS

We describe various methods for proactive sentence-specific popularity forecasting, including unsupervised sentence ranking baselines, neural sentence sequence regression architectures, and our proposed supervised Transfer Learning approach from an auxiliary task of salience prediction.

5.1 Unsupervised sentence ranking baselines

News articles typically utilize the pyramid structure of reporting with primary information contained within the initial sentences. We experiment with a position baseline where we score sentences in descending order of their position from the beginning of the article. Specifically, we assign the score of $1 - frac{i}{n}$ to the ith sentence of an article with n sentences. We also evaluate the performance of TextRank [24] and LexRank [8] algorithms that exploit graph-based similarities between all sentence pairs through PageRank.

5.2 Our Proposed Approach

The primary novelty of our proposed approach lies in the design of a STILTs-based [28] Transfer Learning (TL) setup using a constructed task of sentence salience prediction. Recent advances in NLP have utilized transfer learning in two broad ways, (a) unsupervised pre-training of text encoders using specific Language Modeling objectives [9, 23] and (b) supervised pre-training of architectures on tasks with similar problem formulations [28, 30]. In STILTs (Supplementary Training on Intermediate Labeled-data Tasks), a model pre-trained on unsupervised corpora (like BERT) is further pre-trained on an intermediate, supervised task for which ample labeled data is available before fine-tuning on its primary task.

We observe how sentence salience is well studied in context of text summarization, an area that enjoys availability of sizeable amounts of news domain data and hypothesize that STILTs-based transfer learning from an auxiliary task of salience prediction would boost our models’ popularity forecasting ability.

5.2.1 Auxiliary Transfer Learning (TL) Subtasks. For document summarization, a sentence is considered salient if it contains information related to the primary semantics of the document and can be a worthy inclusion for the document's summary [26]. Given an article and its gold standard summary, the salience of an individual sentence is computed as the ROUGE overlap between sentence and summary. For oracle creation in a typical extractive summarization setting, binary summary-inclusion labels are computed for sentences greedily to maximize the ROUGE overlap between the complete oracle and the true summary [26]. Such a greedy labeling scheme implicitly takes minimization of information redundancy into account. In that, a salient sentence might receive a summary-inclusion label of 0 if a lexically similar sentence was previously included in the oracle summary [15]. However, we only capture the salience of sentences and desire very similar sentences to have similar labels. Similar to sentence-specific popularity forecasting, we package our auxiliary task as a sentence sequence regression problem. This lets us (a) train our architectures adapting the STILTs approach and (b) perform an empirical cross-task evaluation (Sec. 8.2.2).

Since our domain of interest is online news, we utilize the publicly available CNN-DailyMail news summarization dataset (with the same splits as in [11]). We compute three weakly supervised salience scores for each sentence based on its ROUGE 1, ROUGE 2, and ROUGE L overlap with the corresponding article's summary. Like InfoPop labels, we normalize salience labels across a document by dividing by the sum of individual sentence scores. Thus, we create three auxiliary subtasks due to the three labeling schemes, which we tag as S1, S2, and S${{textrm{L}}}$, respectively.

Figure 4

Figure 4: Base${{textrm{Reg}}}$: RNN for Sentence Sequence Regression.

5.2.2 Supervised Neural Architectures. We frame information popularity forecasting as a sentence sequence regression task where a sentence's score is relative to the entire article. Thus, effective neural models require the global context. We pre-train a neural architecture on a supervised TL subtask (S1, S2, or S${{textrm{L}}}$), and then fine-tune the model for popularity forecasting. High-level overviews of Base${{textrm{Reg}}}$, our rudimentary neural baseline, and BERT${{textrm{Reg}}}$, our BERT-based sentence-sequence regression model are illustrated in Fig. 4 and Fig. 5, respectively. We use MSE (Mean Squared Error) loss between true and inferred sentence scores for training our neural models. Note that we handle arbitrarily large documents using both BERT${{textrm{Reg}}}$ and Base${{textrm{Reg}}}$ employing a sliding window mechanism over a document's sentences with a preset stride.

Base${{textrm{Reg}}}$: As a simple baseline neural model for sentence scoring, we build upon the rudimentary SummaRunner [16, 26] architecture with Convolutional Neural Network (CNN) based sentence encoders. We input the concatenated padded sequence of GloVe [27] embeddings for tokens in a sentence to a two-layered CNN sentence vectorizer. Internally, a CNN layer applies a sequence of convolution and batch normalization followed by a Leaky ReLU activation. The output from the two stacked CNNs is then max pooled. Inside, three CNN stacks with kernel sizes of 3, 4, and 5 are used whose outputs are concatenated to form the sentence embedding.

Individual CNN-based sentence vectors are passed through a layer of bi-GRUs (bidirectional Gated Recurrent Units) [5] that produce a sequence of contextual sentence embeddings. All such contextual embeddings are max pooled and then passed through a fully connected (FC) layer to derive a document embedding vector that captures the global context of the complete document. Each sentence score is then computed based on aspects such as their content, novelty, etc., [26] which are in turn expressed as functions of their contextual embedding and the document embedding vector.

BERT${{textrm{Reg}}}$: The use of Transformer-based [39] models such as BERT [6] have yielded state-of-the-art results on various NLP tasks. We adapt BertSumExt’s [22] architecture of sentence classification to our sequential regression setting. To generate contextual sentence embeddings, we add [SEP] tokens to mark sentence ends and distinctive [REG] tokens at sentence beginnings. Downstream, the contextual embeddings of the [REG] tokens are used for sequence regression.

Figure 5

Figure 5: BERT${{textrm{Reg}}}$: BERT for Sentence Sequence Regression.

For each input token, its token, position, and segment embeddings are generated [6]. Here, the segment embeddings allow differentiation between odd and even positioned sentences within BERT (since traditional BERT has been trained with only two segments). The token, position, and segment embeddings of the defined input tokens are passed to BERT, which produces a sequence of refined token embeddings. Now, the sequence of BERT-based sentence embeddings (embeddings corresponding to the [REG] tokens) is summed with their sinusoidal sentence positional embeddings and passed through a two-layer Transformer model to yield contextual sentence embeddings aware of multi-sentence discourse. [21] showed the usefulness of adding a two-layer Transformer that captures inter-sentence relationships as opposed to directly using BERT-generated embeddings for sentence-sequence labeling. Finally, the generated contextual embeddings of the individual sentences are passed through a linear layer with a sigmoid activation to generate regression scores ∈ [0, 1]. Fig. 5 shows an overview of BERT${{textrm{Reg}}}$.

6 EVALUATION METRICS

We utilize the following automatic evaluation metrics to measure the performance of our models.

Top k overlap Let Ak and Pk be the sets of actual and predicted top-k highest scored sentences, respectively. We define top k overlap (topk) as |Ak ∩ Pk|/k and report its percentage. Thus, top1 marks whether a method correctly identifies the highest scored sentence.

Regression errors We report Mean Squared Error (MSE) and Mean Absolute Error (MAE) between arrays of actual and predicted sentence labels.

Rank Correlation Metrics Top k overlap focuses only on the identification of the highest scored sentences. To get a picture of how well various algorithms rank and order the complete set of sentences in a document, we use Spearman's rank correlation (ρ) and Kendall's Tau (τ). ρ, τ ∈ [ − 1, 1].

nDCG Top k overlap, ρ, and τ only consider sentence ranking, while regression errors only account for sentence scores. Normalized Discounted Cumulative Gain (nDCG) [13] captures the normalized gain or usefulness of a sentence based on both its position in the inferred rank list and its actual score. nDCG ∈ [0, 1].

Intuitively, the penalty for downscoring and inverting ranks of two truly high scored sentences should be lower than that of inverting ranks of two negligibly scored ones. Rank Correlation metrics do not take scores into account and fail to capture such insights. Whereas regression errors only focus on true and forecasted scores ignoring the score-induced rank ordering. Thus, broadly, nDCG is the most holistic metric for our task.

Table 2: Sentence Popularity Forecasting Results.

Method	TL	Top1	Top2	Top3	MSE	MAE	τ	ρ	nDCG
Position	−	6.92	10.81	16.46	0.0079	0.0530	0.0334	0.0424	0.5804
TextRank	−	6.08	12.58	19.08	0.0316	0.1486	0.0345	0.0474	0.6313
LexRank	−	18.76	29.95	37.84	0.0072	0.0503	0.0545	0.0705	0.7324
Base${{textrm{Reg}}}$	×	9.12	15.49	20.93	0.0083	0.0452	0.0534	0.0746	0.6228
	S1	9.35	14.91	20.49	0.0075	0.0478	0.0428	0.0592	0.6323
	S2	10.08	16.63	22.83	0.0073	0.0468	0.0545	0.0751	0.6465
	S${{textrm{L}}}$	9.16	14.85	20.76	0.0075	0.0475	0.0465	0.0638	0.6307
BERT${{textrm{Reg}}}$	×	27.53	38.89	45.89	0.0055	0.0335	0.0704	0.0955	0.7921
	S1	27.54	38.73	45.86	0.0052	0.0342	0.0734	0.0988	0.8009
	S2	28.34	39.08	46.17	0.0053	0.0332	0.0646	0.0876	0.8007
	S${{textrm{L}}}$	28.11	39.53	46.96	0.0053	0.0331	0.0510	0.0674	0.8025

7 EXPERIMENTAL DETAILS

We used publicly available implementations of TextRank5 and LexRank6. We normalized inferred LexRank scores to a [0, 1] range (by dividing by the sum of all sentence scores) for computing regression errors. To create the auxiliary transfer learning subtasks, we utilized the non-anonymized version of CNN-DailyMail dataset [11]. We used spaCy's7 dependency parser and NLTK's8 senttokenize function during preprocessing.

We used a maximum sequence limit of 100 sentences in the layer of bi-GRUs within our Base${{textrm{Reg}}}$ model. The CNN-sentence encoder had an upper input limit of 100 tokens per sentence. For training our Base${{textrm{Reg}}}$ model, we used Adam optimizer with a batch size of 256 documents, an initial learning rate of 10− 5, with default hyperparameters. We used the 6-layer bert-base-uncased model in our BERT${{textrm{Reg}}}$ experiments and considered a maximum sequence length of 1536 tokens. Adam optimizer was used with a learning rate of 2.10− 3, with other hyperparameters set to their default values. We split documents exceeding the maximum sequence length into overlapping sliding windows with a predefined maximum stride of 10 sentences. During inference, we adopted the same splitting technique. For a sentence s within the stride of two consecutive windows, we scored s as the mean of the computed scores from the two windows.

We trained all of our models on NVIDIA RTX 2080Ti(s). Base${{textrm{Reg}}}$ models were trained on a single GPU for a maximum of 4 epochs with early stopping. A single epoch on our transfer learning task took over 6 hours, while one epoch for popularity forecasting training took close to 45 minutes. We trained BERT${{textrm{Reg}}}$ based models on 4 GPUs for 50,000 optimizer steps with early stopping turned off for both the popularity forecasting and the transfer learning tasks. The training time ranged between 10 to 12 hours.

8 RESULTS

In this section, we analyze the performance of baselines and our proposed approach for proactive sentence-specific popularity forecasting. We then share some insights on the interrelations between our primary task of sentence popularity forecasting and our auxiliary Transfer Learning (TL) task of salience prediction.

Table 3: Performance of various approaches on auxiliary transfer learning subtasks (S1, S2, S${{textrm{L}}}$).

Task	Approach	Top1	Top2	Top3	MSE	MAE	τ	ρ	nDCG
S1	Position	12.86	26.30	34.04	0.0010	0.0215	0.2030	0.2855	0.8682
	TextRank	15.55	24.28	30.89	0.0243	0.1487	0.0771	0.1082	0.8999
	LexRank	13.14	21.91	28.45	0.0007	0.0178	0.0561	0.0798	0.8740
	Base${{textrm{Reg}}}$	17.01	26.60	34.00	0.0006	0.0167	0.1387	0.1967	0.8933
	BERT${{textrm{Reg}}}$	26.41	36.64	43.20	0.0004	0.0130	0.1372	0.1891	0.9274
S2	Position	11.24	25.03	32.77	0.0044	0.0418	0.1496	0.2101	0.7113
	TextRank	9.29	17.59	23.43	0.0280	0.1541	0.0407	0.0578	0.6665
	LexRank	11.74	20.68	26.88	0.0045	0.0422	0.0473	0.0669	0.6847
	Base${{textrm{Reg}}}$	17.19	27.56	35.83	0.0041	0.0373	0.1391	0.1989	0.7382
	BERT${{textrm{Reg}}}$	23.32	36.26	43.68	0.0034	0.0332	0.1108	0.1559	0.7946
S${{textrm{L}}}$	Position	13.60	27.57	35.18	0.0009	0.0211	0.2050	0.2881	0.8760
	TextRank	11.72	20.05	26.77	0.0245	0.1488	0.0719	0.1016	0.8776
	LexRank	12.40	21.71	27.95	0.0007	0.0182	0.0546	0.0778	0.8657
	Base${{textrm{Reg}}}$	15.13	24.75	32.36	0.0007	0.0175	0.1385	0.1966	0.8780
	BERT${{textrm{Reg}}}$	24.24	34.96	41.76	0.0005	0.0141	0.1329	0.1847	0.9152

8.1 Sentence-Specific Popularity Forecasting

8.1.1 Unsupervised sentence ranking. Table 2 shows performance of unsupervised sentence ranking baselines on sentence-specific popularity forecasting. LexRank turns out the best unsupervised method beating Position and TextRank on all metrics.

8.1.2 Proposed Methods. We present results of our supervised models in Table 2. As expected, BERT${{textrm{Reg}}}$ is the best architecture for the task. We see performance upgrades across the board upon employment of the Transfer Learning setup. Though there is no consistent winner among the three Transfer Learning subtasks9, the best result for each metric is achieved using some form of transfer learning. BERT${{textrm{Reg}}}$ with TL = S${{textrm{L}}}$ boosted average nDCG over 1% from plain BERT${{textrm{Reg}}}$ and over 7% from the best unsupervised baseline. We performed t-tests which showed that BERT${{textrm{Reg}}}$ with TL = S${{textrm{L}}}$ significantly outperformed vanilla BERT${{textrm{Reg}}}$ on nDCG at significance level p < 0.01. Though Base${{textrm{Reg}}}$ is ineffective for popularity forecasting, we observe the same trend of transfer learning as a positive addition. These results demonstrate that transfer learning from salience prediction significantly improves sentence popularity forecasting.

We attribute the performance enhancement to two factors. Firstly, datasets used for both tasks are sourced from online news documents, and transfer learning allows the model to witness more domain-specific data [9]. Secondly, though popularity forecasting differs from salience prediction (Sec. 8.2), they have certain similarities, such as penalizing not lexically dense sentences or understanding that specific sentences do not carry any notable information. Our proposed approach grounds the network to capture such relationships.

Table 4: Cross-task evaluation − performance of BERT${{textrm{Reg}}}$ trained for popularity forecasting (PF) evaluated on salience prediction and vice-versa.

Train	Eval	Top 1	Top 2	Top 3	MSE	MAE	τ	ρ	nDCG
S1	PF	10.97	18.31	25.69	0.0068	0.0475	0.0430	0.0598	0.6864
S2		10.74	19.83	27.89	0.0077	0.0455	0.0380	0.0531	0.6942
S${{textrm{L}}}$		11.44	19.09	27.02	0.0068	0.0476	0.0428	0.0595	0.6937
PF	S1	8.36	16.63	22.64	0.0020	0.0301	0.0600	0.0847	0.8603
	S2	8.51	16.34	22.53	0.0053	0.0430	0.0373	0.0525	0.6579
	S${{textrm{L}}}$	9.07	16.62	22.86	0.0020	0.0304	0.0563	0.0798	0.8524

8.2 Popularity and Salience

Salient sentences effectively capture summary-inclusion worthy ideas that are central to the core semantics of an article [44]. Unlike salient information bits, an information piece might deviate significantly from an article's primary topic yet be popular. Consider the following sentence from a particular article10 (with ID 34499) within InfoPop: ‘Weinsheimer has spent 27 years at DOJ, where he tried homicide and public corruption cases.’ The sentence is not salient enough to be included in a summary as it is barely related to the article's core topic (Scott Schools’ resignation). But, it contains one of the most popular information pieces in the document.

8.2.1 Quantitative analysis. We evaluate our unsupervised baselines and supervised neural models on the transfer learning subtasks (Table 3). The position baseline achieves the best rank correlation scores, explainable based on the pyramid structure of news reports. BERT${{textrm{Reg}}}$ beats other approaches on nDCG and in identifying the most salient sentences (Top k).

Comparing with Table 2, we see how sentence ranking baselines are more capable of capturing information salience than forecasting information popularity. Position baseline's scores for salience prediction are significantly greater than for popularity forecasting, indicating that the salience of the initial sentences in news articles is typically higher than their popularity.

Comparison with Table 2 shows that supervised salience prediction models achieve much better results on ρ, τ, and nDCG compared to popularity forecasting models utilizing the same underlying architecture. Also, note how Base${{textrm{Reg}}}$, which performed poorly on popularity forecasting, achieves respectable scores for salience prediction. These experiments identify salience prediction as the less complicated problem, i.e., given a document, it is easier to compute how central and summary-worthy a contained information piece is than forecasting its popularity. Table 1 shows an excerpt from an InfoPop article with actual and forecasted popularity plus predicted salience labels for sentences showcasing the variation between information popularity and salience within the same document.

8.2.2 Empirical cross-task evaluation. We performed an empirical cross-task analysis [7, 37] to capture the degree of relatedness between primary and auxiliary tasks. We used the BERT${{textrm{Reg}}}$ popularity forecasting model to infer sentences’ labels for documents in the salience-prediction dataset. We evaluated these scores against actual sentence salience labels of the three types – S1, S2, and S${{textrm{L}}}$. Similarly, we evaluated models trained on S1, S2, and S${{textrm{L}}}$ upon our primary popularity forecasting task. We show results in Table 4. Predictably, we observe a massive drop in performance when we switch to the cross-task setting. Values across all metrics fall below those achieved by some unsupervised ranking baseline. This further experimentally showcases the distinction between information popularity and information salience.

9 CONCLUSION

In this work, we introduced the task of proactively forecasting sentence-specific information popularity. We contribute InfoPop, a dataset containing 51,770 news articles from 26 news websites with over 1.7 million sentences labeled with normalized popularity scores. We experimented with both unsupervised and supervised baselines and demonstrated the efficacy of our novel STILTs-based Transfer Learning approach involving an auxiliary supervised task of salience prediction. Our best models achieved nDCG values over 0.8 for sentence-specific popularity forecasting. Our analysis presents an interesting takeaway: though popularity forecasting and salience prediction are quite different problems, transferring the learning from a salience prediction task enhances a model's popularity forecasting proficiency. In future, we aim to explore potential business applications of sentence popularity forecasting in problems such as pull quote extraction, popularity-guided text summarization, etc. We also plan on exploring a multi-task learning approach considering the popularity forecasting and salience prediction problems.

REFERENCES

Raza Abidi, Yonglin Xu, Jianyue Ni, Wang Xiangmeng, and wu Zhang. 2020. Popularity prediction of movies: from statistical modeling to machine learning techniques. Multimedia Tools and Applications 79 (12 2020), 1–35. https://doi.org/10.1007/s11042-019-08546-5

Mohamed Ahmed, Stella Spagna, Felipe Huici, and Saverio Niccolini. 2013. A Peek into the Future: Predicting the Evolution of Popularity in User Generated Content. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (Rome, Italy) (WSDM ’13). Association for Computing Machinery, New York, NY, USA, 607–616. https://doi.org/10.1145/2433396.2433473

Anthony Chen, Pallavi Gudipati, Shayne Longpre, Xiao Ling, and Sameer Singh. 2021. Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 4472–4485. https://doi.org/10.18653/v1/2021.acl-long.345

Wei-Fan Chen, Shahbaz Syed, Benno Stein, Matthias Hagen, and Martin Potthast. 2020. Abstractive Snippet Generation. Proceedings of The Web Conference 2020(2020).

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arxiv:1412.3555 [cs.NE]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018).

Elozino Egonmwan, Vittorio Castelli, and Md Arafat Sultan. 2019. Cross-Task Knowledge Transfer for Query-Based Text Summarization. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering. Association for Computational Linguistics, Hong Kong, China, 72–77. https://doi.org/10.18653/v1/D19-5810

Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research 22 (2004), 457–479.

Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. arxiv:2004.10964 [cs.CL]

Jing He, Pablo Duboue, and Jian-Yun Nie. 2012. Bridging the gap between intrinsic and perceived relevance in snippet generation. In Proceedings of COLING 2012. 1129–1146.

Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In NIPS. 1693–1701. http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend

Sebastian Hofstätter, Bhaskar Mitra, Hamed Zamani, Nick Craswell, and Allan Hanbury. 2021. Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking. arXiv preprint arXiv:2105.09816(2021).

Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated Gain-Based Evaluation of IR Techniques. ACM Trans. Inf. Syst. 20, 4 (Oct. 2002), 422–446. https://doi.org/10.1145/582415.582418

Yitong Ji, Aixin Sun, Jie Zhang, and Chenliang Li. 2020. A Re-Visit of the Popularity Baseline in Recommender Systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20). Association for Computing Machinery, New York, NY, USA, 1749–1752. https://doi.org/10.1145/3397271.3401233

Ruipeng Jia, Yanan Cao, Haichao Shi, Fang Fang, Yanbing Liu, and Jianlong Tan. 2020. DistilSum: Distilling the Knowledge for Extractive Summarization. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (Virtual Event, Ireland) (CIKM ’20). Association for Computing Machinery, New York, NY, USA, 2069–2072. https://doi.org/10.1145/3340531.3412078

Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018. Content Selection in Deep Learning Models of Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 1818–1828. https://doi.org/10.18653/v1/D18-1208

Yaser Keneshloo, Shuguang Wang, E. Han, and Naren Ramakrishnan. 2016. Predicting the Popularity of News Articles. In SDM.

Sotiris Lamprinidis, Daniel Hardt, and Dirk Hovy. 2018. Predicting News Headline Popularity with Syntactic and Semantic Knowledge Using Multi-Task Learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 659–664. https://doi.org/10.18653/v1/D18-1068

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 7871–7880. https://doi.org/10.18653/v1/2020.acl-main.703

Dongliang Liao, Jin Xu, Gongfu Li, Weijie Huang, Weiqing Liu, and Jing Li. 2019. Popularity Prediction on Online Articles with Deep Fusion of Temporal Process and Content Features. In AAAI.

Yang Liu. 2019. Fine-tune BERT for Extractive Summarization. arxiv:1903.10318 [cs.CL]

Yang Liu and Mirella Lapata. 2019. Text Summarization with Pretrained Encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 3730–3740. https://doi.org/10.18653/v1/D19-1387

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arxiv:1907.11692 [cs.CL]

Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing. 404–411.

Nuno Moniz and Luís Torgo. 2018. Multi-Source Social Feedback of Online News Feeds. CoRR https://archive.ics.uci.edu/ml/datasets/abs/1801.07055 (2018).

Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2016. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. arXiv preprint arXiv:1611.04230(2016).

Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532–1543.

Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks. ArXiv abs/1811.01088(2018).

Julia Proskurnia, P. A. Grabowicz, Ryota Kobayashi, C. Castillo, P. Cudré-Mauroux, and K. Aberer. 2017. Predicting the Success of Online Petitions Leveraging Multidimensional Time-Series. Proceedings of the 26th International Conference on World Wide Web (2017).

Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?. In ACL.

Alexander Pugachev, Anton Voronov, and Ilya Makarov. 2020. Prediction of News Popularity via Keywords Extraction and Trends Tracking. Recent Trends in Analysis of Images, Social Networks and Texts 1357 (2020), 37 – 51.

Georgios Rizos, Symeon Papadopoulos, and Yiannis Kompatsiaris. 2016. Predicting News Popularity by Mining Online Discussions. In Proceedings of the 25th International Conference Companion on World Wide Web (Montréal, Québec, Canada) (WWW ’16 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 737–742. https://doi.org/10.1145/2872518.2890096

Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. arxiv:1509.00685 [cs.CL]

Yun-Zhu Song, Hong-Han Shuai, Sung-Lin Yeh, Yi-Lun Wu, Lun-Wei Ku, and Wen-Chih Peng. 2020. Attractive or Faithful? Popularity-Reinforced Learning for Inspired Headline Generation. Proceedings of the AAAI Conference on Artificial Intelligence 34, 05 (Apr. 2020), 8910–8917. https://doi.org/10.1609/aaai.v34i05.6421

Shivashankar Subramanian, Timothy Baldwin, and Trevor Cohn. 2018. Content-based Popularity Prediction of Online Petitions Using a Deep Regression Model. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Melbourne, Australia, 182–188. https://doi.org/10.18653/v1/P18-2030

Alexandru Tatar, Panayotis Antoniadis, Marcelo Dias de Amorim, and Serge Fdida. 2012. Ranking News Articles Based on Popularity Prediction. In 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. 106–110. https://doi.org/10.1109/ASONAM.2012.28

Shubham Toshniwal, Haoyue Shi, Bowen Shi, Lingyu Gao, Karen Livescu, and Kevin Gimpel. 2020. A Cross-Task Analysis of Text Span Representations. In Proceedings of the 5th Workshop on Representation Learning for NLP. Association for Computational Linguistics, Online, 166–176. https://doi.org/10.18653/v1/2020.repl4nlp-1.20

Md. Taufeeq Uddin, Muhammed Jamshed Alam Patwary, Tanveer Ahsan, and Mohammed Shamsul Alam. 2016. Predicting the popularity of online news from content metadata. In 2016 International Conference on Innovations in Science, Engineering and Technology (ICISET). 1–5. https://doi.org/10.1109/ICISET.2016.7856498

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998–6008.

Anton Voronov, Yao Shen, and Pritom Kumar Mondal. 2019. Forecasting Popularity of News Article by Title Analyzing with BN-LSTM Network. In Proceedings of the 2019 International Conference on Data Mining and Machine Learning (Hong Kong, Hong Kong) (ICDMML 2019). Association for Computing Machinery, New York, NY, USA, 19–27. https://doi.org/10.1145/3335656.3335679

Kai Wang, Penghui Wang, Xin Chen, Qiushi Huang, Zhendong Mao, and Yongdong Zhang. 2020. A Feature Generalization Framework for Social Media Popularity Prediction. In Proceedings of the 28th ACM International Conference on Multimedia (Seattle, WA, USA) (MM ’20). Association for Computing Machinery, New York, NY, USA, 4570–4574. https://doi.org/10.1145/3394171.3416294

Bo Wu, Tao Mei, Wen-Huang Cheng, and Yongdong Zhang. 2016. Unfolding Temporal Dynamics: Predicting Social Media Popularity Using Multi-Scale Temporal Decomposition. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (Phoenix, Arizona) (AAAI’16). AAAI Press, 272–278.

Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization. arxiv:1905.06566 [cs.CL]

Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive Summarization as Text Matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 6197–6208. https://doi.org/10.18653/v1/2020.acl-main.552

FOOTNOTE

1 https://github.com/sayarghoshroy/InfoPopularity

2 https://searchengineland.com/google-now-handles-2-999-trillion-searches-per-year-250247

3 https://trends.google.com

4 https://github.com/sayarghoshroy/InfoPopularity

5 https://github.com/summanlp/textrank

6 https://github.com/crabcamp/lexrank

7 https://spacy.io/

8 https://www.nltk.org/

9We prefer TL = S${{textrm{L}}}$ as it outperforms others on Topk (k = 2, 3), MAE, and most importantly, nDCG. A downstream application focused purely on accurate ranking could use TL = S${{textrm{1}}}$, while another requiring identification of the most popular sentence could use TL = S${{textrm{2}}}$.

10 https://www.npr.org/2018/07/03/625581627/another-top-justice-department-lawyer-steps-down-following-earlier-departures

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

HT '22, June 28–July 01, 2022, Barcelona, Spain

ACM ISBN 978-1-4503-9233-4/22/06…$15.00.

DOI: https://doi.org/10.1145/3511095.3531268