Evaluation of clustering and topic modeling methods over health-related tweets and emails (2024)

The publisher's final edited version of this article is available at Artif Intell Med

2.1. Clustering of short texts

Clustering is an unsupervised machine learning method that seeks topartition objects into a certain number of clusters (i.e., groups or subsets)based on a similarity metric. Generally, clustering methods applied on text dataare based on vector representations; such as bag-of-words (BoW) or termfrequency-inverse document frequency (TF-IDF); and then grouping texts based ontheir similarity [85,89,90,21,84]. These techniques are frequently applied to several informationretrieval tasks such as event detection [65,93,115,82] andtext summarization [105,128,86,148]. There are several works usingclassic clustering methods on short texts, such as tweets, for instance, a study[120] compared three well-knownclustering algorithms: k-means, Singular Value Decomposition,and Affinity Propagation on over 600 tweets, and found that Affinity Propagation[46] had better performance. However,its complexity associated with the number of documents was quadratic, thusAffinity Propagation is not suitable for larger datasets. Other approachesfocused on variations of classic clustering techniques considering several tweetcomponents such as texts, hashtags, users, and temporal aspect (e.g., streamclustering) [129,86,75].

Short text clustering represents a big challenge due to the datasparsity, since most words co-occur once or twice in the dataset [10]. Several sparseness-resistant methodswere proposed to face this challenge such as text augmentation [19,156],topic modeling [147,30], neural networks [145,52], andDirichlet Mixture Model [149,150,75]. Data augmentation methods seek to enrich data representationwith external resources, such as Wikipedia [19,146]; or similar words byexploiting related text documents [69,133,35,156]; orthe incorporation of semantic features from ontologies, terminologies, anddictionaries, such as WordNet, DBpedia, Freebase [59,45,26,141].

Moreover, recent approaches based on low-dimension representations withneural network [145] proved to beeffective to tackle the sparsity problem in short text clustering [135,38,47,52], for instance using word embeddings [96,110], sentences embeddings [77,72] and documentembeddings [34]. Also, several studiesexplored sophisticated models for short text clustering. For instance, a workproposed a Dirichlet Multinomial Mixture model-based approach for short textclustering (GSDMM) [149] which alsoinfers the number of clusters and obtained the best performance when comparedwith clustering algorithms such as k-means [92], Hierarchical Agglomerative Clustering (HAC)[94], and DMAFP [60].

2.2. Topic modeling

Topic Modeling is also an unsupervised machine learning method mainlybased on statistical properties of the data to discover “topics”that describe the collection of documents. Topic modeling methods seek toextract topics from a set of documents based on statistical techniques. Eachtopic is defined as a distribution over a set of words. Diverse topic modelingmethods have been proposed to analyze texts from different fields includingpolitics, medicine, and psychology. Two of the most popular methods are thelatent dirichlet allocation (LDA) [22]and latent semantic indexing (LSI, a.k.a. LSA) [56]. A recent work has exhaustively listed LDA-based techniquesproposed from 2003 to 2016 applied to text from various domains, includingbiomedicine [67].

Some topic modeling algorithms were proposed to work on general healthand medical text [70,71]. Moreover, other proposed methods specificallyaimed to predict therapy outcomes from emails sent by patients under treatmentfor a social anxiety disorder [58];predict protein-protein [17], gene-drug[143] relations from biomedicalliterature; discover concepts in patients’ health records [16]; detect depression [124,137];recognize genuine suicide notes from notes written by healthy subjects [111]; classify patient issues from theirexperience and the result of using a particular drug [68]; improve the automating classification of patientportal messages through the use of semantic features and word context [132]; identify patterns of events frommedical reports of brain cancer patients [15]; treatment behaviors [62,63]; treatment activities[28]; determine patient mortality[49]; discover models of disease andphenotypes [112]; extract biologicalterminology [97]; discover biologicalprocesses [81]; among others. Topicmodeling methods have also been applied over health-related tweets to identifylatent health topics [107]. Hybridapproaches have also allowed to extract health trends in tweets by theintegration of visualization approaches with classical topic models [114,113]. Also, diverse studies addressed specific tasks such asgrouping opinions about HPV vaccines leveraging also community structure methods[134]; identify commonobesity-related themes through a combination of geographic information systemsand topic modeling methods [50]; identifythe associations of Zika-related topics, such as attitudes, knowledge, andbehaviors [44].

As clustering methods, topic modelling can also be used for clusteringby giving a probability distribution over a number of topics for each document.Indeed, clustering and topic modeling methods have been used for clusteringtasks and have been compared in different studies. For instance, the authors in[117] proposed three categories fortopic modeling based on: (1) DMM, (2) global word co-occurrence, and (3)self-aggregation. Then they compared nine different topic modeling techniquesfrom these categories: (1) GSDMM [149],LF-DMM [102], GPU-DMM [80], GPU-PDMM [79], (2) BTM [30], WNTM[158], and (3) SATM [118], PTM [157]. They found that: (i) strategies that use wordembeddings (LF-DMM, GPU-DMM, and GPU- PDMM) are very promising in short texttopic modeling, (ii) the highest computation costs were obtained with LF-DMM andGPU-PDMM (i.e., they are not suitable for large datasets), and (iii) simplermethods (GSDMM and BTM) are the most suitable with respect to effectiveness andefficiency. For the clustering task, GSDMM achieved the best results. Anotherrelated work [116] described an approachwith Gibbs sampling called PYPM. This model was tested on four short textdatasets and compared with five well-known techniques (Non-negative MatrixFactorization [78], LDA, DMAFP, GSDMM,and FGSDMM [151]). Results showed thatPYPM had the best results followed by GSDMM.

Moreover, another recent study [33] evaluated the performance of four clustering algorithms(k-means, k-medoids, HierarchicalAgglomerative Clustering, and Non-negative Matrix Factorization) and a topicmodeling method (LDA) on short texts from social networks such as Twitter andReddit. The paper also evaluated four different feature representationsincluding TF-IDF, Word2Vec, Word2Vec weighted with the top 1,000 TF-IDF scores,and Doc2Vec. The experiments showed that the best performance was achieved byk-means with Doc2Vec on both datasets.

Of note, topic modeling methods can be evaluated from several aspectssuch as from cluster evaluation, topic coherence, and classification evaluation.To compare clustering and topic modeling methods, we need to apply clustervalidity indices. For this purpose, after using topic modeling to compute topicprobabilities, the maximum topic probability of each document is selected to getthe cluster label of each document [24,88,144,147,116,117]. Then, cluster validity indices are applied to evaluate theirperformances.

2.3. Validity indices

Cluster validity indices are metrics to validate clustering results andto find natural structures for a given dataset [152,14]. In other words,validity indices seek to find optimal partitions that are well compacted andwell-separated from other partitions [54]. There are two kinds of cluster evaluation metrics which are calledexternal and internal validation indices [53]. External indices measure the quality based on ground-truthlabels, for instance Rand [119],Adjusted Rand Index [64],Fowlkes–Mallows, Variation of Information [11]. Internal indices evaluate the result oninformation intrinsic to the data alone. The latter is useful when there is noannotated dataset available and the usual approach focus on evaluate thecompactness and separation of clusters, such as Dunn [37], Calinski–Harabasz [25]. Several studies on cluster validity indicesconcluded that there is not a single best metric [42,95,14]. Also, they found that the performanceof cluster validity indices decrease considerably when there is noise orclusters overlap.

Recent works have compared topic modeling and clustering methods onshort text clustering. Most of them used annotated datasets for the experiments,therefore, they mainly used external indices to measure the quality of clusters,such as hom*ogeneity (H) [122],Completeness (C) [122], V-Measure (V)[122], Adjusted Rand Index (ARI)[64], Normalized Mutual Information(NMI) [29], Adjusted Mutual Information(AMI), Accuracy (ACC), F-measure, Entropy, Purity. For instance, the authorsthat introduced GSDMM [149] used fiveexternal indices such as H, C, ARI, NMI, and AMI to evaluate their model.Another study proposed an online semantic-enhanced Dirichlet model for shorttext clustering (OSDM) [75] andconsidered several indices such as NMI, V, H, and ACC to validate their results.In [52] the authors reported theevaluation of various text representations and self-training methods with ACCand NMI. A recent study [117] comparednine topic modeling techniques in clustering tasks using two external metrics:NMI and Purity. Moreover, a recent work evaluated one topic modeling and foursclassic clustering methods [33] usingthree external indices such as NMI, AMI, ARI.

On the other hand, there are also several commonly used internal indiceswhen assessing the goodness of short text clustering such as Calinski-Harabasz(CH) [25], Silhouette Coefficient (SC)[123], Dunn [37], Duda [43], Elbow [136], among others.The internal indices can also be used to determine the optimal number ofclusters in the data [98]. Internalindices in comparison to external ones usually detect improvements in theclustering distribution which have positive implications in the systemevaluation [66]. In our previous work[87], we evaluated topic modeling andclustering methods using only tweets and two internal indices (CH and SC).However, most studies previously cited that compared topic modeling andclustering methods did not use internal validity indices to evaluate theirresults.

Therefore, in this paper, we use seven validity indices: five external(NMI, ARI, H, C, and V) and two internal (CH and SC). We selected the five mostcommon external indices used in the literature that are also independent of theabsolute values of the labels in comparison to F-measure, ACC, among others.Moreover, we included two internal indices: CH index which has demonstrated inseveral works to be effective [12], andSC which is one of the most well-known measures and provides graphicalrepresentations of how well each element has been classified. We used theimplementation of these metrics in sklearn [109] in the experimental study.

3.1. Datasets

3.2. Applications

We used several available online implementation of topicmodeling² and clusteringmethods. To cover every aspect, we briefly describe all used systems along withour experiments.

3.3. Analysis of applications

Note that to evaluate the performance of clustering and topic modelingmethods we need to apply cluster validity indices. Thus, after executing topicmodeling to compute topic probabilities, the highest probability of eachdocument is selected to get the cluster label of each document [117]. Then internal and external validity indices areapplied to assess their performances. Therefore, we performed an evaluation ofall topic modeling and clustering applications in terms of: 1) experiments andresults over the tweets dataset, a complete calculation of two internal and fiveexternal indices, and 2) experiments and results over the emails dataset, also acomparison of the internal and external indices. In the paragraph below, weexplain the configuration for our evaluation, as well as the seven indicesused.

4.1. Results on the tweets dataset

4.1.1. Internal indices

We perform experiments using CH and SC to measure the performance ofthe topic modeling and clustering algorithms. Tables 6, ​,7,7, ​,88 and ​and99 show the CH and SC results for the seven topicmodeling methods and k-means algorithm with Doc2Vec andTF-IDF for 2, 5, 10, and 50 number of clusters/topics(“k”) respectively. Of note, CH and SCseek to evaluate the clusters/topics based on two aspects: the similarity oftweets within the same cluster (cohesion), and the difference between thetweets of different clusters.

Table 6

Internal index results for k=2.

	100 iterations		500 iterations		1,000 iterations
	CH	SC	CH	SC	CH	SC
LSI	86,260	0.41	86,260	0.40	86,260	0.40
BTM	634,412	0.74	661,242	0.75	604,629	0.72
LDA	515,737	0.68	486,522	0.68	521,198	0.69
GibbsLDA	10,767,060	0.97	10,514,730	0.98	9,932,722	0.97
Online LDA	849,068	0.77	834,351	0.76	938,428	0.78
Online Twitter LDA	50,110,500	0.99	53,291,260	0.99	50,730,260	0.99
GSDMM	16,232,540	0.97	18,030,810	0.97	17,961,280	0.97
k-means+Doc2Vec	31,196	0.20	31,196	0.20	31,196	0.20
k-means+TF-IDF	5,764	0.04	5,764	0.04	5,764	0.04

Open in a separate window

Table 7

Internal index results for k=5.

	100 iterations		500 iterations		1,000 iterations
	CH	SC	CH	SC	CH	SC
LSI	50,641	0.40	51,961	0.40	51,961	0.40
BTM	165,515	0.60	171,041	0.60	175,937	0.61
LDA	69,526	0.37	71,240	0.38	71,741	0.38
GibbsLDA	967,683	0.91	1,016,640	0.93	1,010,773	0.92
Online LDA	173,255	0.61	170,659	0.60	185,005	0.62
Online Twitter LDA	6,554,339	0.98	10,117,270	0.98	10,989,530	0.99
GSDMM	2,208,731	0.91	2,660,268	0.92	2,302,045	0.91
k-means+Doc2Vec	13,998	0.04	13,998	0.04	13,998	0.04
k-means+TF-IDF	4,722	0.07	4,722	0.07	4,722	0.07

Open in a separate window

Table 8

Internal index results for k=10.

	100 iterations		500 iterations		1,000 iterations
	CH	SC	CH	SC	CH	SC
LSI	25,346	0.34	25,539	0.34	25,532	0.35
BTM	55,110	0.41	61,790	0.43	67,856	0.53
LDA	24,498	0.27	24,102	0.27	24,860	0.27
GibbsLDA	239,457	0.83	266,065	0.85	272,035	0.86
Online LDA	70,824	0.54	69,418	0.55	69,892	0.55
Online Twitter LDA	1,400,045	0.96	1,925,903	0.97	2,035,547	0.97
GSDMM	878,294	0.88	957,856	0.89	955,849	0.89
k-means+Doc2Vec	7,617	0.02	7,617	0.02	7,617	0.02
k-means+TF-IDF	3,758	0.07	3,758	0.07	3,758	0.07

Open in a separate window

Table 9

Internal index results for k=50.

	100 iterations		500 iterations		1,000 iterations
	CH	SC	CH	SC	CH	SC
LSI	3,894	0.21	3,925	0.22	3,907	0.19
BTM	9,501	0.35	10,089	0.37	10,507	0.37
LDA	3,006	0.14	2,801	0.12	2,960	0.14
GibbsLDA	18,188	0.65	21,256	0.68	22,014	0.69
Online LDA	9,835	0.44	10,322	0.45	10,299	0.46
Online Twitter LDA	39,014	0.79	62,749	0.85	66,051	0.87
GSDMM	162,280	0.88	173,979	0.88	169,109	0.89
k-means+Doc2Vec	2,028	−0.02	2,028	−0.02	2,028	−0.02
k-means+TF-IDF	2,200	0.17	2,200	0.17	2,200	0.17

Open in a separate window

In addition, Figure 2 shows ageneral overview of the performance of the applications based on SC and CHfor 100, 500, and 1,000 iterations; and for“k” ranging from 2 to 50. In all cases, thebest values are obtained by Online Twitter LDA followed by GSDMM.

Open in a separate window

Fig. 2

Silhouette Coefficient and Calinski-Harabasz metrics with 100, 500, and1,000 iterations, for “k” ranging from 2 to50.

4.1.2. External indices

We perform experiments using the five external indices (NMI, ARI, V,H, and C) to measure the performance of the topic modeling and clusteringalgorithms. Tables 10, ​,11,11, ​,1212 and ​and1313 depict theNMI, ARI, V, H, and C values obtained with all topic modeling and clusteringmethods for 2, 5, 10, and 50 number of clusters/topics(“k”) respectively. Of note, externalindices measure the extent to which cluster labels match externally suppliedclass labels.

Table 10

External index results for k=2.

	100 iterations					500 iterations					1,000 iterations
	NMI	ARI	V	H	C	NMI	ARI	V	H	C	NMI	ARI	V	H	C
LSI	0.05	−0.02	0.09	0.16	0.07	0.05	−0.02	0.09	0.16	0.07	0.05	−0.02	0.09	0.159	0.07
LDA	0.00	0.02	0.01	0.01	0.01	0.05	0.07	0.09	0.16	0.07	0.00	0.01	0.00	0.00	0.00
GibbsLDA	0.02	0.01	0.03	0.05	0.02	0.02	0.01	0.04	0.08	0.03	0.02	−0.00	0.04	0.07	0.03
Online LDA	0.03	0.06	0.07	0.12	0.05	0.00	0.00	0.00	0.00	0.00	0.06	0.09	0.12	0.21	0.09
BTM	0.11	0.23	0.24	0.38	0.17	0.11	0.23	0.23	0.37	0.17	0.09	0.14	0.18	0.30	0.13
Online Twitter LDA	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
GSDMM	0.17	0.13	0.17	0.12	0.28	0.16	0.12	0.16	0.11	0.26	0.05	−0.06	0.05	0.04	0.08
k-means + Doc2Vec	0.00	−0.01	0.00	0.00	0.00	0.00	−0.01	0.00	0.00	0.00	0.00	−0.01	0.00	0.00	0.00
k-means + TF-IDF	0.04	−0.04	0.09	0.14	0.06	0.04	−0.04	0.09	0.14	0.06	0.04	−0.04	0.09	0.14	0.06

Open in a separate window

Table 11

External index results for k=5.

	100 iterations					500 iterations					1,000 iterations
	NMI	ARI	V	H	C	NMI	ARI	V	H	C	NMI	ARI	V	H	C
LSI	0.27	0.09	0.19	0.21	0.17	0.28	0.09	0.19	0.22	0.18	0.28	0.09	0.19	0.22	0.18
LDA	0.10	0.03	0.07	0.08	0.07	0.13	0.04	0.09	0.09	0.08	0.13	0.03	0.09	0.09	0.08
GibbsLDA	0.09	0.03	0.07	0.08	0.06	0.13	0.05	0.09	0.10	0.09	0.14	0.05	0.09	0.11	0.09
Online LDA	0.09	0.03	0.07	0.07	0.06	0.08	0.02	0.05	0.06	0.05	0.07	0.01	0.05	0.06	0.05
BTM	0.35	0.15	0.25	0.27	0.25	0.37	0.15	0.27	0.28	0.26	0.35	0.14	0.26	0.27	0.25
Online Twitter LDA	0.00	0.00	0.00	0.00	0.00	0.01	0.00	0.00	0.00	0.00	0.01	0.00	0.01	0.01	0.01
GSDMM	0.23	0.11	0.23	0.21	0.25	0.24	0.13	0.24	0.22	0.26	0.24	0.14	0.24	0.23	0.25
k-means + Doc2Vec	0.01	−0.01	0.01	0.01	0.01	0.01	−0.01	0.01	0.01	0.01	0.01	−0.01	0.01	0.01	0.01
k-means + TF-IDF	0.47	0.29	0.36	0.36	0.35	0.47	0.29	0.36	0.36	0.35	0.47	0.29	0.36	0.36	0.35

Open in a separate window

Table 12

External index results for k=10.

	100 iterations					500 iterations					1,000 iterations
	NMI	ARI	V	H	C	NMI	ARI	V	H	C	NMI	ARI	V	H	C
LSI	0.59	0.17	0.31	0.37	0.27	0.59	0.17	0.32	0.37	0.28	0.58	0.17	0.31	0.36	0.27
LDA	0.16	0.02	0.08	0.10	0.07	0.16	0.02	0.08	0.10	0.07	0.19	0.03	0.09	0.12	0.08
GibbsLDA	0.14	0.02	0.07	0.09	0.06	0.19	0.04	0.10	0.12	0.08	0.21	0.04	0.10	0.13	0.09
Online LDA	0.16	0.03	0.08	0.10	0.07	0.16	0.02	0.08	0.10	0.07	0.12	0.02	0.06	0.08	0.05
BTM	0.40	0.08	0.21	0.25	0.18	0.42	0.09	0.22	0.26	0.19	0.44	0.11	0.24	0.28	0.21
Online Twitter LDA	0.01	0.00	0.00	0.01	0.00	0.03	0.00	0.01	0.02	0.01	0.04	0.00	0.02	0.02	0.01
GSDMM	0.24	0.12	0.24	0.21	0.27	0.22	0.10	0.22	0.19	0.26	0.22	0.12	0.22	0.20	0.25
k-means + Doc2Vec	0.04	0.00	0.02	0.02	0.01	0.04	0.00	0.02	0.02	0.01	0.04	0.00	0.02	0.02	0.01
k-means + TF-IDF	0.53	0.18	0.29	0.33	0.26	0.53	0.18	0.29	0.33	0.26	0.53	0.18	0.29	0.33	0.26

Open in a separate window

Table 13

External index results for k=50.

	100 iterations					500 iterations					1,000 iterations
	NMI	ARI	V	H	C	NMI	ARI	V	H	C	NMI	ARI	V	H	C
LSI	0.87	0.10	0.30	0.38	0.25	0.87	0.11	0.31	0.38	0.26	0.84	0.09	0.29	0.37	0.24
LDA	0.40	0.01	0.13	0.17	0.10	0.47	0.01	0.15	0.20	0.12	0.44	0.01	0.15	0.19	0.12
GibbsLDA	0.40	0.01	0.13	0.17	0.10	0.44	0.01	0.14	0.19	0.11	0.45	0.01	0.14	0.20	0.11
Online LDA	0.36	0.01	0.11	0.16	0.09	0.36	0.01	0.11	0.16	0.09	0.34	0.00	0.11	0.15	0.09
BTM	0.62	0.03	0.21	0.27	0.17	0.64	0.02	0.22	0.28	0.18	0.63	0.02	0.21	0.28	0.17
Online Twitter LDA	0.19	0.00	0.06	0.08	0.04	0.28	0.00	0.09	0.12	0.07	0.34	0.00	0.11	0.15	0.08
GSDMM	0.22	0.04	0.22	0.18	0.28	0.22	0.04	0.22	0.19	0.28	0.23	0.04	0.23	0.19	0.30
k-means + Doc2Vec	0.15	0.00	0.05	0.07	0.04	0.15	0.00	0.05	0.07	0.04	0.15	0.00	0.05	0.07	0.04
k-means + TF-IDF	0.69	0.04	0.23	0.30	0.18	0.69	0.04	0.23	0.30	0.18	0.69	0.04	0.23	0.30	0.18

Open in a separate window

We also plotted the values obtained with NMI, ARI, V, H, and C with100 iterations only as shown in Figure3. In general, the best values are obtained by LSI followed byk-means with TF-IDF. Also, note that Online Twitter LDAsignificantly decreased its performance in comparison to the values obtainedin the internal indices evaluation. It obtained the lowest performance,while other algorithms such as LSI, BTM, and techniques such as TF-IDFimproved and in general, they are the three methods with the bestperformance.

Open in a separate window

Fig. 3

Normalized Mutual Information, Adjusted Rand Index, V-measure,hom*ogeneity, and Completeness metrics with 100 iterations for“k” ranging from 2 to 50 over the tweetsdataset.

4.2. Results on the emails dataset

4.2.1. Internal indices

We trained each model for integer values of kranging from 2 to 50, for 100 iterations and measured each model’sperformance with SC and CH scores. Online Twitter LDA was initially the bestperforming model both in terms of CH and SC scores of 8.2 million and 0.94,respectively, but in contrast to the results of the previous experiments, wesaw a greater decrease in the model’s performance, relative to LDAand Online LDA, as k increased. The GSDMM model had themost stable, high level performance, with a SC that never dropped below0.86, well above the next best performing model, LDA, which had a SC scoreof 0.65 (for k = 50). LDA and Online LDA achieved thesecond and third best SC scores once k exceeded 30. LSImodel had the worst performance, achieving negative SC values for almost allvalues of k.

The CH scores rapidly and substantially decreased with increasingvalues of k for all models. The rates of decreasesperformance were not uniform as the Online Twitter LDA stated out well abovethe rest at 8.2 million for k = 2 and dropped to 5,719 fork = 50 behind the LDA model, which had a CH score of8,062. The GSDMM model became the best performing model oncek exceeded 12 and was the only model for which all CHscores were greater than 10,000.

Table 14 lists the values ofthe SC and CH scores for the models with 2, 5, 10, and 50 clusters/topicswhile Figure 4 illustrates the SC andCH values for all values of k.

Open in a separate window

Fig. 4

Silhouette Coefficient and Calinski-Harabasz metrics over the emailsdataset with 100 iterations, for “k” ranging from2 to 50.

Table 14

Internal index results after 100 iterations.

See Also

What Is Topic Modeling? A Beginner's Guide

	k = 2		k = 5		k = 10		k = 50
	CH	SC	CH	SC	CH	SC	CH	SC
LSI	1,316	0.06	529	−0.19	696	−0.22	517	−0.24
BTM	121,814	0.62	19,884	0.33	9,200	0.24	1,173	0.11
LDA	237,684	0.74	118,955	0.75	54,848	0.75	8,062	0.65
GibbsLDA	59	0.04	1,128	−0.02	706	−0.02	103	−0.07
Online LDA	222,296	0.76	128,584	0.78	47,653	0.70	7,974	0.64
Online Twitter LDA	8,197,541	0.99	699,231	0.95	136,077	0.88	5,719	0.56
GSDMM	1,576,441	0.94	342,109	0.90	122,952	0.87	27,889	0.86
k-means+Doc2Vec	13	0.48	13	0.52	21	0.32	77	0.09
k-means+TF-IDF	556	0.01	401	0.01	260	−0.01	88	0.01

Open in a separate window

4.2.2. External indices

We also analyzed the same email clustering/topic models for externalvalidity indices. Table 15 displaysthe NMI, ARI, V, H, and C for models with 2, 5, 10, and 50 clusters/topics.It is worth noting that all metrics of external validity range between 0 and1. Figure 5 illustrates our findingsfor all values of k between 2 and 50. Notably, we see thatthe k-means with TF-IDF embedding and LSI the are bestperforming algorithms across all metrics while Online Twitter LDA andk-means with Doc2Vec embedding showed the worstperformance consistently across all 5 metrics of external validity. Overall,we see that all models improve along measures of external validity withincreasing values of k. This trend is rather gradualstarting at values 0.01–0.05 and increasing by one or two hundredths,with each value of k, reaching values between 0.04 and0.08. Two exceptions to this trend are the best models: LSI andk-means with TF-IDF, which start at 0.01–0.02for k = 2, quickly increase to vales 0.12–0.20 fork = 10, and then gradually increase to reach values0.18–0.30. Another two exceptions are the worst models: OnlineTwitter LDA and k-means with Doc2Vec, which start at valesless than or equal to 0.01 and never grow larger than 0.02 for any value ofk.

Open in a separate window

Fig. 5

Normalized Mutual Information, Adjusted Rand Index, V-measure,hom*ogeneity, and Completeness indices with 100 iterations for“k” ranging from 2 to 50 over the emaildataset.

Table 15

External index results after 100 iterations.

	k =2					k =5
	NMI	ARI	V	H	C	NMI	ARI	V	H	C
LSI	0.02	0.01	0.01	0.02	0.01	0.09	0.10	0.16	0.09	0.12
LDA	0.02	0.03	0.02	0.02	0.02	0.03	0.02	0.03	0.03	0.03
GibbsLDA	0.05	0.08	0.05	0.05	0.05	0.02	0.02	0.02	0.03	0.02
Online LDA	0.03	0.05	0.03	0.03	0.03	0.02	0.01	0.02	0.02	0.02
BTM	0.04	0.08	0.05	0.04	0.05	0.04	0.03	0.05	0.04	0.04
Online Twitter LDA	0.01	0.02	0.01	0.01	0.01	<0.01	<0.01	<0.01	<0.01	<0.01
GSDMM	0.03	0.06	0.03	0.03	0.03	0.02	0.02	0.02	0.02	0.02
k-means + Doc2Vec	0.01	0.01	0.01	0.01	0.01	<0.01	<0.01	<0.01	<0.01	<0.01
k-means + TF-IDF	<0.01	<0.01	<0.01	<0.01	<0.01	0.08	0.07	0.14	0.08	0.1
	k =10					k =50
	NMI	ARI	V	H	C	NMI	ARI	V	H	C
LSI	0.18	0.19	0.24	0.18	0.20	0.21	0.21	0.30	0.21	0.25
LDA	0.06	0.04	0.06	0.06	0.06	0.06	0.02	0.07	0.07	0.07
GibbsLDA	0.05	0.02	0.05	0.05	0.05	0.08	0.02	0.08	0.09	0.08
Online LDA	0.03	0.01	0.03	0.03	0.03	0.03	0.01	0.04	0.04	0.04
BTM	0.09	0.03	0.10	0.09	0.09	0.13	0.04	0.14	0.14	0.14
Online Twitter LDA	<0.01	<0.01	<0.01	<0.01	<0.01	0.02	<0.01	0.02	0.02	0.02
GSDMM	0.05	0.02	0.05	0.05	0.05	0.02	0.06	0.09	0.06	0.08
k-means + Doc2Vec	0.01	<0.01	0.01	0.01	0.01	0.01	<0.01	0.02	0.02	0.02
k-means + TF-IDF	0.13	0.12	0.18	0.13	0.15	0.21	0.08	0.22	0.21	0.22

Open in a separate window

5.1. Tweets dataset

A popular area of study is supervised algorithms using unbalanceddatasets. However, skewed distributions also affect the learning process inunsupervised methods, especially in clustering [100] that are based on centroids [140,73]. Despite enormoussolutions, there is a reduced effectiveness when the groups have highlydifferent sizes [74], however, most ofthe models we used proved capable of creating a group of tweets bigger than theother that reflected the unbalanced nature of the tweets data set (94.6% and5.4% of tweets related to HPV and lynch syndrome, respectively).

Online Twitter LDA followed by GSDMM obtained the highest values of SCand CH, which indicates that those clusterings were more compact, more dense(within the cluster), and better separated than all other models. In general,topic models out performed clustering methods in terms of CH and SC, whichprovides insights into the interconnected nature of medical communication. Thisproblem is well suited for LDA proposed as improvements of LDA such as OnlineLDA.

For the evaluation of the external indices, a subset of tweets withhashtags was used. The external indices showed that LSI followed byk-means with TF-IDF obtained the best results. Note thatOnline Twitter LDA significantly decreased its performance in comparison to thevalues obtained in the internal indices evaluation. We did not find an obviousrelationship between the number of iterations and the performance each of thedifferent experimental configurations. In several cases, the performance isproportional to the number of iterations, although this is not a common patternfor all algorithms.

Several findings from the applications of topic modeling and clusteringmethods confirmed that many in society are using Twitter to share past andcurrent experiences of a disease (HPV and lynch syndrome in our case), symptoms,treatment information, side effects, emotions, research, among others, asdepicted in Table 2 and Figure 6. Figure6 shows the most important topics extracted from two clusters createdwith k-means and Table 2presents tweets extracted from these two clusters.

Open in a separate window

Fig. 6

Most important topics extracted from two clusters created withk-means with TF-IDF.

5.2. Emails dataset

In contrast to the tweets, the emails in our dataset characterize asmaller and more hom*ogeneous domain of language. Each email was sent by apatient with prostate cancer (or their caregiver) to a health care provider.Careful consideration of the broader context of our modeling task can explainthe findings of our experiments using emails as well as shed light on theresults of our experiments with the tweets data.

The questions and concerns that arise as patients undergo treatment forprostate cancer, from scheduling procedures to managing a sudden crisis, arerarely discrete issues. This poses a substantial problem for clusteringalgorithms that search for perfectly separated clusters.k-means (with either embedding) is such an algorithm, whichhelps explain why it did not find internally meaningful clusters for any valueof k. The nature of the emails may also help explain why theLSI model, which searches for a fixed low dimensional representation, generatedinternally inconsistent labels. In contrast, the LDA approach models document asa mixture of topics and seems to naturally represent an email that is primarilyabout family member introducing himself as the patient’s new primary caregiver while also mentioning several previously unreported health issues, forexample. We see in our results that the three best performing models, withrespect to internal indices, were a variation of LDA.

Despite the dramatic difference model performance over tweets and emailswith respect to internal indices, we observed very similar patterns inperformance with respect to external indices. Notably, LSI andk-means with TF-IDF, performed very well despite havemostly negative or near zero SC scores, respectively. BTM was always among thetop 3 or 4 models while Online Twitter LDA had the best and worst measures ofinternal and external consistency, respectively in both experimentaldesigns.

5.3. Error analysis

We also conducted an analysis of the types of error patterns found onshort text clustering tasks. For this, we used the Hamming loss metric, which isa loss function, so the optimal value is zero (i.e., closer Hamming distance tothe external classes and better performance) and its upper bound is one. Hammingloss measures the fraction of wrong labels to the total number of labels.Hamming Loss is relevant for an unbalanced classification tasks and relevant formulti-label classification. Thus, we computed the Hamming loss on the tweets andemails datasets. Note that this function depends on the labels of each document(tweet/email), thus, the interpretation of the output is very similar to theexternal validity indices.

For instance, Figure 7 illustratesthe Hamming loss for emails for all models trained with 100 iterations. We foundremarkably stable values of loss for all models for values of k> 15. The sable values of loss are consistent with the otherexternal indices of validity: LSI and k-means with a TF-IDFrepresentation being the first and second best performing models, and thevariants of LDA along with GSDMM among the poorest performing models.

Open in a separate window

Fig. 7

Hamming loss for topic modeling and clustering methods trained for 100iterations and “k” ranging from 2 to 50 overemails dataset.

We then manually evaluated a group of tweets and emails to assess theassignment of clusters. There are several factors that caused errors whenassigning emails or tweets to their respective clusters: (1) most of the tweetsand emails contained misspellings, that received a part-of-speech category of“noun” or “unknown”, thus, considered for theclustering tasks; (2) tweets and emails contain terms created by patients suchas “onco” instead of “oncology” which also affectedthe groups of texts; (3) most of tweets contain hashtags composed of two or morewords, for instance “#hpvvaccine”; (4) lack of more context (e.g.,semantic information), indeed, n-gram terms (n≥ 2) as features provide more context for clustering than a single wordterms, for instance “hpv vaccination” and ”fluvaccination” rather than “vaccination”; (5) thesubjectivity to tell for a tweet/email to what cluster it belongs, the morenumber of cluster the more subjective become this task; among others.

Our study has known limitations. First, the annotation of both datasetswas semi-automatically performed which directly affected the values of theexternal indices compared to the internal indices. We selected the most frequenthashtags (tweets) and words (emails), we then manually selected the mostrelevant hashtags/words and grouped them by their semantic similarity, finally,we executed a script that automatically annotated the tweets/emails for eachnumber of clusters, (i.e., dataset labeled with two groups only whenk = 2, then, dataset labeled with three groups only whenk = 3, until k = 50). Second, for theexternal validation of the tweets dataset, we have only considered thosecontaining hashtags which potentially affected the different results betweeninternal and external indices. A possible solution could be to also considervisualization methods that can intuitively reflect the validity of clusteringresults.

Conflict of interest

The authors declare that they have no conict of interest.

¹Hashtag is a word or phrase preceded by a hash sign (#) to identifymessages on a specific topic.

²Most of the implementations were extracted using gensim version 3.8.3[121], an opensource library forunsupervised topic modeling.

Publisher's Disclaimer: This is a PDF file of an uneditedmanuscript that has been accepted for publication. As a service to our customerswe are providing this early version of the manuscript. The manuscript willundergo copyediting, typesetting, and review of the resulting proof before it ispublished in its final form. Please note that during the production processerrors may be discovered which could affect the content, and all legaldisclaimers that apply to the journal pertain.

10. Aggarwal CC and Zhai C. A survey of text clusteringalgorithms. In Mining text data, pages77–128.Springer,2012. [Google Scholar]

11. Amigó E, Gonzalo J, Artiles J, and Verdejo F. A comparison of extrinsic clustering evaluationmetrics based on formal constraints. Inf. Retr.,12(4):461–486,Aug. 2009. [Google Scholar]

12. Anderson MJ. A new method for non-parametric multivariateanalysis of variance. Austral ecology,26(1):32–46,2001. [Google Scholar]

13. Antoun J. Electronic mail communication between physiciansand patients: a review of challenges and opportunities.Family practice,33(2):121–126,2016. [PubMed] [Google Scholar]

14. Arbelaitz O, Gurrutxaga I, Muguerza J, PéRez JM, and Perona I. An extensive comparative study of clustervalidity indices. Pattern Recognition,46(1):243–256,2013. [Google Scholar]

15. Arnold C and Speier W. A topic model of clinical reports. InProceedings of the 35th International ACM SIGIR Conference onResearch and Development in Information Retrieval, SIGIR‘12, page 1031–1032,New York, NY, USA, 2012.Association for Computing Machinery. [Google Scholar]

16. Arnold CW, El-Saden SM, Bui AA, and Taira R. Clinical case-based retrieval using latent topicanalysis. In AMIA annual symposium proceedings, volume 2010, page 26.American Medical Informatics Association,2010. [PMC free article] [PubMed] [Google Scholar]

17. Aso T and Eguchi K. Predicting protein-protein relationships fromliterature using latent topics. In Genome Informatics 2009: Genome Informatics Series Vol.23, pages 3–12.World Scientific,2009. [PubMed] [Google Scholar]

18. Atherton H, Sawmynaden P, Sheikh A, Majeed A, and Car J. Email for clinical communication betweenpatients/caregivers and healthcare professionals.Cochrane Database of Systematic Reviews,(11), 2012. [PubMed] [Google Scholar]

19. Banerjee S, Ramanathan K, and Gupta A. Clustering short texts using wikipedia. InProceedings of the 30th Annual International ACM SIGIR Conferenceon Research and Development in Information Retrieval, SIGIR‘07, page 787–788,New York, NY, USA, 2007.Association for Computing Machinery. [Google Scholar]

20. Bergmo TS, Kummervold PE, Gammon D, and Dahl LB. Electronic patient–provider communication:Will it offset office visits and telephone consultations in primarycare?International journal of medical informatics,74(9):705–710,2005. [PubMed] [Google Scholar]

21. Bicalho P, Pita M, Pedrosa G, Lacerda A, and Pappa GL. A general framework to expand short text fortopic modeling. Information Sciences,393:66–81,2017. [Google Scholar]

22. Blei DM, Ng AY, and Jordan MI. Latent dirichlet allocation.Journal of machine Learning research,3(Jan):993–1022,2003. [Google Scholar]

23. Braithwaite SR, Giraud-Carrier C, West J, Barnes MD, and Hanson CL. Validating machine learning algorithms fortwitter data against established measures of suicidality.JMIR mental health,3(2):e21,2016. [PMC free article] [PubMed] [Google Scholar]

24. Cai D, Mei Q, Han J, and Zhai C. Modeling hidden topics on document manifold. InProceedings of the 17th ACM Conference on Information andKnowledge Management, CIKM ‘08, page911–920, New York, NY,USA, 2008. Association for ComputingMachinery. [Google Scholar]

25. Caliński T and Harabasz J. A dendrite method for clusteranalysis. Communications in Statistics-theory and Methods,3(1):1–27,1974. [Google Scholar]

26. Cano AE, Varga A, Rowe M, Ciravegna F, and He Y. Harnessing linked knowledge sources for topic classification in social media. In Proceedings of the24th ACM Conference on Hypertext and Social Media, HT‘13, page 41–50,New York, NY, USA, 2013.Association for Computing Machinery. [Google Scholar]

27. Chang F, Paramsothy T, Roche M, and Gupta NS. Patient, staff, and clinician perspectives onimplementing electronic communications in an interdisciplinary rural familyhealth practice. Primary health care research & development,18(2):149–160,2017. [PubMed] [Google Scholar]

28. Chen JH, Goldstein MK, Asch SM, Mackey L, and Altman RB. Predicting inpatient clinical order patterns withprobabilistic topic models vs conventional order sets.Journal of the American Medical Informatics Association,24(3):472–480,2017. [PMC free article] [PubMed] [Google Scholar]

29. Chen W-Y, Song Y, Bai H, Lin C-J, and Chang EY. Parallel spectral clustering in distributedsystems. IEEE transactions on pattern analysis and machine intelligence,33(3):568–586,2010. [PubMed] [Google Scholar]

30. Cheng X, Yan X, Lan Y, and Guo J. Btm: Topic modeling over shorttexts. IEEE Transactions on Knowledge and Data Engineering,26(12):2928–2941,2014. [Google Scholar]

31. Chu K-H, Unger JB, Allem J-P, Pattarroyo M, Soto D, Cruz TB, Yang H, Jiang L, and Yang CC. Diffusion of messages from an electroniccigarette brand to potential users through twitter.PloS one,10(12):e0145387,2015. [PMC free article] [PubMed] [Google Scholar]

32. Coppersmith G, Harman C, and Dredze M. Measuring post traumatic stress disorder in twitter. In Proceedings of the AAAI Eighth InternationalConference on Weblogs and Social Media, ICWSM 2014,Ann Arbor, Michigan, USA, June 1–4,2014., 2014. [Google Scholar]

33. Curiskis SA, Drake B, Osborn TR, and Kennedy PJ. An evaluation of document clustering and topicmodelling in two online social networks: Twitter and reddit.Information Processing & Management,57(2):102034,2020. [Google Scholar]

34. Dai AM, Olah C, and Le QV. Document embedding with paragraphvectors. arXiv preprint arXiv:1507.07998,2015. [Google Scholar]

35. Dai Z, Sun A, and Liu X-Y. Crest: Cluster-based representation enrichment for short text classification. In Pacific-Asia Conference onKnowledge Discovery and Data Mining, pages256–267. Springer,2013. [Google Scholar]

36. Dash J, Haller DM, Sommer J, and Perron NJ. Use of email, cell phone and text message betweenpatients and primary-care physicians: cross-sectional study in afrench-speaking part of switzerland. BMC health services research,16(1):549,2016. [PMC free article] [PubMed] [Google Scholar]

37. Davies DL and Bouldin DW. A cluster separation measure.IEEE Trans. Pattern Anal. Mach. Intell.,1(2):224–227,Feb. 1979. [PubMed] [Google Scholar]

38. De Boom C, Van Canneyt S, Demeester T, and Dhoedt B. Representation learning for very short textsusing weighted word embedding aggregation. Pattern Recogn. Lett.,80(C):150–156,Sept. 2016. [Google Scholar]

39. de Jong CC, Ros WJ, and Schrijvers G. The effects on health behavior and healthoutcomes of internet-based asynchronous communication between healthproviders and patients with a chronic condition: a systematicreview. Journal of medical Internet research, 16(1):e19,2014. [PMC free article] [PubMed] [Google Scholar]

40. De Martino I, D’Apolito R, McLawhorn AS, Fehring KA, Sculco PK, and Gasparini G. Social media for patients: benefits anddrawbacks. Current reviews in musculoskeletal medicine,10(1):141–145,2017. [PMC free article] [PubMed] [Google Scholar]

41. Deerwester S, Dumais ST, Furnas GW, Landauer TK, and Harshman R. Indexing by latent semanticanalysis. J. of the American society for information science,41(6):391–407,1990. [Google Scholar]

42. Dimitriadou E, Dolničar S, and Weingessel A. An examination of indexes for determining thenumber of clusters in binary data sets.Psychometrika,67(1):137–159,2002. [Google Scholar]

43. Duda RO, Hart PE, et al. Pattern classification and scene analysis, volume3. Wiley; New York, 1973. [Google Scholar]

44. Farhadloo M, Winneg K, Chan M-PS, Jamieson KH, and Albarracin D. Associations of topics of discussion on twitterwith survey measures of attitudes, knowledge, and behaviors related to zika:probabilistic study in the united states. JMIR public health and surveillance,4(1):e16,2018. [PMC free article] [PubMed] [Google Scholar]

45. Fodeh S, Punch B, and Tan P-N. On ontology-driven document clustering using coresemantic features. Knowledge Information System,28(2):395–421,Aug. 2011. [Google Scholar]

46. Frey BJ and Dueck D. Clustering by passing messages between datapoints. science,315(5814):972–976,2007. [PubMed] [Google Scholar]

47. Ganguly D and Ghosh K. Contextual word embedding: A case study in clustering tweets about emergency situations. In CompanionProceedings of the The Web Conference 2018, WWW ‘18, page73–74, Republic and Canton ofGeneva, CHE, 2018. International World WideWeb Conferences Steering Committee. [Google Scholar]

48. Garrido T, Meng D, Wang JJ, Palen TE, and Kanter MH. Secure e-mailing between physicians and patients:transformational change in ambulatory care. The Journal of ambulatory care management,37(3):211,2014. [PMC free article] [PubMed] [Google Scholar]

49. Ghassemi M, Naumann T, Doshi-Velez F, Brimmer N, Joshi R, Rumshisky A, and Szolovits P. Unfolding physiological state: Mortality modelling in intensive care units. In Proceedings of the 20th ACMSIGKDD International Conference on Knowledge Discovery and Data Mining, KDD‘14, page 75–84,New York, NY, USA, 2014.Association for Computing Machinery. [PMC free article] [PubMed] [Google Scholar]

50. Ghosh D and Guha R. What are we ‘tweeting’aboutobesity? mapping tweets with topic modeling and geographic informationsystem. Cartography and geographic information science,40(2):90–102,2013. [PMC free article] [PubMed] [Google Scholar]

51. Haase R, Schultheiss T, Kempcke R, Thomas K, and Ziemssen T. Use and acceptance of electronic communication bypatients with multiple sclerosis: a multicenter questionnairestudy. Journal of medical Internet research, 14(5):e135,2012. [PMC free article] [PubMed] [Google Scholar]

52. Hadifar A, Sterckx L, Demeester T, and Develder C. A self-training approach for short text clustering. In Proceedings of the 4th Workshop onRepresentation Learning for NLP (RepL4NLP-2019), pages194–199,2019. [Google Scholar]

53. Halkidi M, Batistakis Y, and Vazirgiannis M. Cluster validity methods: Part i.SIGMOD Rec.,31(2):40–45,June2002. [Google Scholar]

54. Halkidi M and Vazirgiannis M. A density-based cluster validity approach usingmulti-representatives. Pattern Recognition Letters,29(6):773–786,2008. [Google Scholar]

55. Halko N, Martinsson P-G, and Tropp JA. Finding structure with randomness: Probabilisticalgorithms for constructing approximate matrixdecompositions. SIAM review,53(2):217–288,2011. [Google Scholar]

56. Hinton GE and Salakhutdinov RR. Reducing the dimensionality of data with neuralnetworks. science,313(5786):504–507,2006. [PubMed] [Google Scholar]

57. Hoffman M, Bach F, and Blei D. Online learning for latent dirichletallocation. advances in neural information processing systems,23:856–864,2010. [Google Scholar]

58. Hoogendoorn M, Berger T, Schulz A, Stolz T, and Szolovits P. Predicting social anxiety treatment outcome basedon therapeutic email conversations. IEEE journal of biomedical and health informatics,21(5):1449–1459,2016. [PMC free article] [PubMed] [Google Scholar]

59. Hu X, Sun N, Zhang C, and Chua T-S. Exploiting internal and external semantics for the clustering of short texts using world knowledge. InProceedings of the 18th ACM Conference on Information andKnowledge Management, CIKM ‘09, page919–928, New York, NY,USA, 2009. Association for ComputingMachinery. [Google Scholar]

60. Huang R, Yu G, Wang Z, Zhang J, and Shi L. Dirichlet process mixture model for documentclustering with feature partition. IEEE Transactions on knowledge and data engineering,25(8):1748–1759,2012. [Google Scholar]

61. Huang X, Smith MC, Jamison AM, Broniatowski DA, Dredze M, Quinn SC, Cai J, and Paul MJ. Can online self-reports assist in real-timeidentification of influenza vaccination uptake? a cross-sectional study ofinfluenza vaccine-related tweets in the usa,2013–2017. BMJ open,9(1):e024018,2019. [PMC free article] [PubMed] [Google Scholar]

62. Huang Z, Dong W, Duan H, and Li H. Similarity measure between patient traces forclinical pathway analysis: problem, method, andapplications. IEEE journal of biomedical and health informatics,18(1):4–14,2013. [PubMed] [Google Scholar]

63. Huang Z, Lu X, and Duan H. Latent treatment pattern discovery for clinicalprocesses. Journal of medical systems,37(2):9915,2013. [PubMed] [Google Scholar]

64. Hubert L and Arabie P. Comparing partitions.Journal of Classification,2:193–218,1985. [Google Scholar]

65. Ifrim G, Shi B, and Brigadir I. Event detection in twitter using aggressive filtering and hierarchical tweet clustering. In Second Workshop onSocial News on the Web (SNOW), Seoul,Korea, 8 April 2014. ACM,2014. [Google Scholar]

66. Ingaramo D, Pinto D, Rosso P, and Errecalde M. Evaluation of internal validity measures in short-text corpora. In International Conference on Intelligent TextProcessing and Computational Linguistics, pages555–567. Springer,2008. [Google Scholar]

67. Jelodar H, Wang Y, Yuan C, Feng X, Jiang X, Li Y, and Zhao L. Latent dirichlet allocation (lda) and topicmodeling: models, applications, a survey. Multimedia Tools and Applications,78(11):15169–15211,2019. [Google Scholar]

68. Jiang Y, Liao QV, Cheng Q, Berlin RB, and Schatz BR. Designing and evaluating a clustering system fororganizing and integrating patient drug outcomes in personal healthmessages. In AMIA Annual Symposium Proceedings, volume 2012, page 417.American Medical Informatics Association,2012. [PMC free article] [PubMed] [Google Scholar]

69. Jin O, Liu NN, Zhao K, Yu Y, and Yang Q. Transferring topical knowledge from auxiliary long texts for short text clustering. In Proceedings of the 20thACM International Conference on Information and Knowledge Management, CIKM‘11, page 775–784,New York, NY, USA, 2011.Association for Computing Machinery. [Google Scholar]

70. Karami A, Gangopadhyay A, Zhou B, and Karrazi H. Flatm: A fuzzy logic approach topic model for medical documents. In 2015 Annual Conference of the NorthAmerican Fuzzy Information Processing Society (NAFIPS) held jointly with2015 5th World Conference on Soft Computing (WConSC), pages1–6. IEEE,2015. [Google Scholar]

71. Karami A, Gangopadhyay A, Zhou B, and Kharrazi H. Fuzzy approach topic discovery in health andmedical corpora. International Journal of Fuzzy Systems,20(4):1334–1345,2018. [Google Scholar]

72. Kiros R, Zhu Y, Salakhutdinov R, Zemel RS, Torralba A, Urtasun R, and Fidler S. Skip-thought vectors. In Proceedingsof the 28th International Conference on Neural Information ProcessingSystems - Volume 2, NIPS’15, page3294–3302, Cambridge, MA,USA, 2015. MITPress. [Google Scholar]

73. Krawczyk B. Learning from imbalanced data: open challengesand future directions. Progress in Artificial Intelligence,5(4):221–232,2016. [Google Scholar]

74. Krawczyk B, Minku LL, Gama J, Stefanowski J, and Woźniak M. Ensemble learning for data stream analysis: Asurvey. Information Fusion,37:132–156,2017. [Google Scholar]

75. Kumar J, Shao J, Uddin S, and Ali W. An online semantic-enhanced Dirichlet model for short text stream clustering. In Proceedings of the 58thAnnual Meeting of the Association for Computational Linguistics,pages 766–776, Online,July 2020. Association for ComputationalLinguistics. [Google Scholar]

76. Lau JH, Collier N, and Baldwin T. On-line trend analysis with topic models: twitter trends detection topic model online. In Proceedings of the 24thInternational Conference on Computational Linguistics, COLING‘12, pages1519–1534,2012. [Google Scholar]

77. Le Q and Mikolov T. Distributed representations of sentences and documents. In Proceedings of the 31st InternationalConference on International Conference on Machine Learning - Volume 32,ICML’14, pageII–1188–II–1196.JMLR.org, 2014. [Google Scholar]

78. Lee DD and Seung HS. Algorithms for non-negative matrix factorization. In Proceedings of the 13th InternationalConference on Neural Information Processing Systems,NIPS’00, page535–541, Cambridge, MA,USA, 2000. MITPress. [Google Scholar]

79. Li C, Duan Y, Wang H, Zhang Z, Sun A, and Ma Z. Enhancing topic modeling for short texts withauxiliary word embeddings. ACM Trans. Inf. Syst., 36(2), Aug.2017. [Google Scholar]

80. Li C, Wang H, Zhang Z, Sun A, and Ma Z. Topic modeling for short texts with auxiliary word embeddings. In Proceedings of the 39th International ACMSIGIR Conference on Research and Development in Information Retrieval, SIGIR‘16, page 165–174,New York, NY, USA, 2016.Association for Computing Machinery. [Google Scholar]

81. Liu B, Liu L, Tsykin A, Goodall GJ, Green JE, Zhu M, Kim CH, and Li J. Identifying functional mirna–mrnaregulatory modules with correspondence latent dirichletallocation. Bioinformatics,26(24):3105–3111,2010. [PMC free article] [PubMed] [Google Scholar]

82. Lo SL, Chiong R, and Cornforth D. An unsupervised multilingual approach for onlinesocial media topic identification. Expert Systems with Applications,81:282–298,2017. [Google Scholar]

83. Lossio-Ventura JA and Bian J. An inside look at the opioid crisis over twitter. In 2018 IEEE International Conference onBioinformatics and Biomedicine (BIBM), pages1496–1499. IEEE,2018. [Google Scholar]

84. Lossio-Ventura JA, Bian J, Jonquet C, Roche M, and Teisseire M. A novel framework for biomedical entity senseinduction. Journal of biomedical informatics,84:31–41,2018. [PMC free article] [PubMed] [Google Scholar]

85. Lossio Ventura JA, Hacid H, Ansiaux A, and Maag ML. Conversations reconstruction in the social web.In Proceedings of the 21st International Conference on World WideWeb, WWW ‘12 Companion, pages573–574, New York, NY,USA, 2012. ACM. [Google Scholar]

86. Lossio-Ventura JA, Hacid H, Roche M, and Poncelet P. Communication overload management through social interactions clustering. In Proceedings of the 31stAnnual ACM Symposium on Applied Computing, SAC ‘16, page1166–1169, New York, NY,USA, 2016. Association for ComputingMachinery. [Google Scholar]

87. Lossio-Ventura JA, Morzan J, Alatrista-Salas H, Hernandez-Boussard T, and Bian J. Clustering and topic modeling over tweets: A comparison over a health dataset. In 2019 IEEE InternationalConference on Bioinformatics and Biomedicine, BIBM’19.IEEE Computer Society, 2019 (inpress). [PMC free article] [PubMed] [Google Scholar]

88. Lu Y, Mei Q, and Zhai C. Investigating task performance of probabilistictopic models: An empirical study of plsa and lda.Information Retrieval,14(2):178–203,Apr. 2011. [Google Scholar]

89. Lu Y, Zhang P, Liu J, Li J, and Deng S. Health-related hot topic detection in onlinecommunities using text clustering. Plos one, 8(2):e56221,2013. [PMC free article] [PubMed] [Google Scholar]

90. Ma L, Wang Z, and Zhang Y. Extracting depression symptoms from social networks and web blogs via text mining. In International Symposium onBioinformatics Research and Applications, pages325–330. Springer,2017. [Google Scholar]

91. MacQueen Jet al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeleysymposium on mathematical statistics and probability, volume1, pages 281–297.Oakland, CA, USA, 1967. [Google Scholar]

92. MacQueen JB. Some methods for classification and analysis ofmultivariate observations. In Cam LML and Neyman J, editors, Proceedings of the fifth Berkeley Symposium on Mathematical Statistics and Probability, volume 1,pages 281–297. Universityof California Press, 1967. [Google Scholar]

93. Manaskasemsak B, Chinthanet B, and Rungsawang A. Graph clustering-based emerging event detection from twitter data stream. In Proceedings of the FifthInternational Conference on Network, Communication and Computing, ICNCC‘16, page 37–41,New York, NY, USA, 2016.Association for Computing Machinery. [Google Scholar]

94. Manning CD, Schütze H, and Raghavan P. Introduction to information retrieval.Cambridge university press,2008. [Google Scholar]

95. Maulik U and Bandyopadhyay S. Performance evaluation of some clusteringalgorithms and validity indices. IEEE Transactions on pattern analysis and machine intelligence,24(12):1650–1654,2002. [Google Scholar]

96. Mikolov T, Sutskever I, Chen K, Corrado G, and Dean J. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26thInternational Conference on Neural Information Processing Systems - Volume2, NIPS’13, page3111–3119, Red Hook, NY,USA, 2013. Curran AssociatesInc. [Google Scholar]

97. Millar JR, Peterson GL, and Mendenhall MJ. Document clustering and visualization with latent dirichlet allocation and self-organizing maps. InTwenty-Second International FLAIRS Conference,2009. [Google Scholar]

98. Milligan GW and Cooper MC. An examination of procedures for determining thenumber of clusters in a data set.Psychometrika,50(2):159–179,1985. [Google Scholar]

99. Myslín M, Zhu S-H, Chapman W, and Conway M. Using twitter to examine smoking behavior andperceptions of emerging tobacco products. Journal of medical Internet research,15(8):e174,2013. [PMC free article] [PubMed] [Google Scholar]

100. Nguwi Y-Y and Cho S-Y. An unsupervised self-organizing learning withsupport vector ranking for imbalanced datasets.Expert Systems with Applications,37(12):8303–8312,2010. [Google Scholar]

101. Nguyen DQ. jLDADMM: A Java package for the LDA and DMM topicmodels. arXiv preprint arXiv:1808.03835,2018. [Google Scholar]

102. Nguyen DQ, Billingsley R, Du L, and Johnson M. Improving topic models with latent feature wordrepresentations. Transactions of the Association for Computational Linguistics,3:299–313,2015. [Google Scholar]

103. Nigam K, McCallum AK, Thrun S, and Mitchell T. Text classification from labeled and unlabeleddocuments using em. Machine learning,39(2–3):103–134,2000. [Google Scholar]

104. Ofoghi B, Mann M, and Verspoor K. Towards early discovery of salient health threats: A social media emotion classification technique. InBiocomputing 2016: Proceedings of the PacificSymposium, pages 504–515.World Scientific, 2016. [PubMed] [Google Scholar]

105. co/>Olariu A. Hierarchical clustering in improving microblog stream summarization. In International Conference onIntelligent Text Processing and Computational Linguistics, pages424–435. Springer,2013. [Google Scholar]

106. Pappas Y, Atherton H, Sawmynaden P, and Car J. Email for clinical communication betweenhealthcare professionals. Cochrane Database of Systematic Reviews, (9),2012. [PubMed] [Google Scholar]

107. Paul MJ and Dredze M. Discovering health topics in social media usingtopic models. PloS one,9(8), 2014. [PMC free article] [PubMed] [Google Scholar]

108. Paul MJ and Dredze M. Social monitoring for publichealth. Synthesis Lectures on Information Concepts, Retrieval, and Services,9(5):1–183,2017. [Google Scholar]

109. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, and duch*esnay E. Scikit-learn: Machine learning inpython. Journal of Machine Learning Research,12:2825–2830,Nov. 2011. [Google Scholar]

110. Pennington J, Socher R, and Manning CD. Glove: Global vectors for word representation.In Proceedings of the 2014 conference on empirical methods in naturallanguage processing (EMNLP), pages1532–1543,2014. [Google Scholar]

111. Pestian J, Nasrallah H, Matykiewicz P, Bennett A, and Leenaars A. Suicide note classification using naturallanguage processing: A content analysis. Biomedical informatics insights,3:BII–S4706,2010. [PMC free article] [PubMed] [Google Scholar]

112. Pivovarov R, Perotte AJ, Grave E, Angiolillo J, Wiggins CH, and Elhadad N. Learning probabilistic phenotypes fromheterogeneous ehr data. Journal of biomedical informatics,58:156–165,2015. [PMC free article] [PubMed] [Google Scholar]

113. Prasad KR, Mohammed M, and Noorullah R. Visual topic models for healthcare dataclustering. Evolutionary Intelligence,pages 1–18,2019. [Google Scholar]

114. Prasad KR, Mohammed M, and Noorullah RM. Hybrid topic cluster models for social healthcaredata. International Journal of Advanced Computer Science and Applications, 10(11),2019. [Google Scholar]

115. Preo D¸tiuc-Pietro P. Srijith, Hepple M, and Cohn T. Studying the temporal dynamics of word co-occurrences: An application to event detection. In Proceedings of theTenth International Conference on Language Resources and Evaluation(LREC’16), pages4380–4387,2016. [Google Scholar]

116. Qiang J, Li Y, Yuan Y, and Wu X. Short text clustering based on pitman-yor processmixture model. Applied Intelligence,48(7):1802–1812,2018. [Google Scholar]

117. Qiang J, Qian Z, Li Y, Yuan Y, and Wu X. Short text topic modeling techniques,applications, and performance: a survey. IEEE Transactions on Knowledge and Data Engineering,2020. [Google Scholar]

118. Quan X, Kit C, Ge Y, and Pan SJ. Short and sparse text topic modeling via self-aggregation. In Proceedings of the 24thInternational Conference on Artificial Intelligence,IJCAI’15, page2270–2276. AAAI Press,2015. [Google Scholar]

119. Rand WM. Objective criteria for the evaluation ofclustering methods. Journal of the American Statistical association,66(336):846–850,1971. [Google Scholar]

120. Rangrej A, Kulkarni S, and Tendulkar AV. Comparative study of clustering techniques for short text documents. In Proceedings of the 20th InternationalConference Companion on World Wide Web, WWW ‘11, page111–112, New York, NY,USA, 2011. Association for ComputingMachinery. [Google Scholar]

121. Rehurek R and Sojka P. Software framework for topic modelling with large corpora. In In Proceedings of the LREC 2010 workshop onnew challenges for NLP frameworks. Citeseer,2010. [Google Scholar]

122. Rosenberg A and Hirschberg J. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 JointConference on Empirical Methods in Natural Language Processing andComputational Natural Language Learning (EMNLP-CoNLL), pages410–420, Prague, CzechRepublic, June 2007. Associationfor Computational Linguistics. [Google Scholar]

123. Rousseeuw PJ. Silhouettes: a graphical aid to theinterpretation and validation of cluster analysis.Journal of computational and applied mathematics,20:53–65,1987. [Google Scholar]

124. Rude S, Gortner E-M, and Pennebaker J. Language use of depressed anddepression-vulnerable college students. Cognition & Emotion,18(8):1121–1133,2004. [Google Scholar]

125. Salton G and Buckley C. Term-weighting approaches in automatic textretrieval. Information processing & management,24(5):513–523,1988. [Google Scholar]

126. Sawmynaden P, Atherton H, Majeed A, and Car J. Email for the provision of information on diseaseprevention and health promotion. Cochrane Database of Systematic Reviews, (11),2012. [PubMed] [Google Scholar]

127. Seneviratne MG, Seto T, Blayney DW, Brooks JD, and Hernandez-Boussard T. Architecture and implementation of a clinicalresearch data warehouse for prostate cancer.eGEMs, 6(1),2018. [PMC free article] [PubMed] [Google Scholar]

128. Shou L, Wang Z, Chen K, and Chen G. Sumblr: continuous summarization of evolving tweet streams. In Proceedings of the 36th international ACMSIGIR conference on Research and development in informationretrieval, pages 533–542,2013. [Google Scholar]

129. Shou L, Wang Z, Chen K, and Chen G. Sumblr: Continuous summarization of evolving tweet streams. In Proceedings of the 36th International ACMSIGIR Conference on Research and Development in Information Retrieval, SIGIR‘13, page 533–542,New York, NY, USA, 2013.Association for Computing Machinery. [Google Scholar]

130. Sinnenberg L, Buttenheim AM, Padrez K, Mancheno C, Ungar L, and Merchant RM. Twitter as a tool for health research: asystematic review. American J. of public health,107(1):e1–e8,2017. [PMC free article] [PubMed] [Google Scholar]

131. Strehl A and Ghosh J. Cluster ensembles—a knowledge reuseframework for combining multiple partitions. Journal of machine learning research,3(Dec):583–617,2002. [Google Scholar]

132. Sulieman L, Gilmore D, French C, Cronin RM, Jackson GP, Russell M, and Fabbri D. Classifying patient portal messages usingconvolutional neural networks. Journal of biomedical informatics,74:59–70,2017. [PubMed] [Google Scholar]

133. Sun A. Short text classification using very few words.In Proceedings of the 35th International ACM SIGIR Conference onResearch and Development in Information Retrieval, SIGIR‘12, page 1145–1146,New York, NY, USA, 2012.Association for Computing Machinery. [Google Scholar]

134. Surian D, Nguyen DQ, Kennedy G, Johnson M, Coiera E, and Dunn AG. Characterizing twitter discussions about hpvvaccines using topic modeling and community detection.Journal of Medical Internet Research,18(8):e232,2016. [PMC free article] [PubMed] [Google Scholar]

135. Tian F, Gao B, Cui Q, Chen E, and Liu T-Y. Learning deep representations for graph clustering. In Proceedings of the Twenty-Eighth AAAIConference on Artificial Intelligence, AAAI’14, page1293–1299. AAAI Press,2014. [Google Scholar]

136. Tibshirani R, Walther G, and Hastie T. Estimating the number of clusters in a data setvia the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology),63(2):411–423,2001. [Google Scholar]

137. Van der Zanden R, Curie K, Van Londen M, Kramer J, Steen G, and Cuijpers P. Web-based depression treatment: Associations ofclientsẃord use with adherence and outcome.Journal of affective disorders,160:10–13,2014. [PubMed] [Google Scholar]

138. Ventola CL. Social media and health care professionals:benefits, risks, and best practices. Pharmacy and Therapeutics,39(7):491,2014. [PMC free article] [PubMed] [Google Scholar]

139. Vraga EK, Stefanidis A, Lamprianidis G, Croitoru A, Crooks AT, Delamater PL, Pfoser D, Radzikowski JR, and Jacobsen KH. Cancer and social media: A comparison of trafficabout breast cancer, prostate cancer, and other reproductive cancers ontwitter and instagram. Journal of health communication,23(2):181–189,2018. [PubMed] [Google Scholar]

140. Wang Y and Chen L. Multi-exemplar based clustering for imbalanced data. In 2014 13th International Conference on ControlAutomation Robotics & Vision (ICARCV), pages1068–1073. IEEE,2014. [Google Scholar]

141. Wei T, Lu Y, Chang H, Zhou Q, and Bao X. A semantic approach for text clustering usingwordnet and lexical chains. Expert Syst. Appl.,42(4):2264–2275,Mar. 2015. [Google Scholar]

142. Wei X and Croft WB. Lda-based document models for ad-hoc retrieval.In Proceedings of the 29th annual international ACM SIGIR conferenceon Research and development in information retrieval, pages178–185. ACM,2006. [Google Scholar]

143. Wu Y, Liu M, Zheng WJ, Zhao Z, and Xu H. Ranking gene-drug relationships in biomedicalliterature using latent dirichlet allocation. InBiocomputing 2012, pages422–433. WorldScientific, 2012. [PMC free article] [PubMed] [Google Scholar]

144. Xie P and Xing EP. Integrating document clustering and topic modeling. In Proceedings of the Twenty-Ninth Conferenceon Uncertainty in Artificial Intelligence, UAI’13, page694–703, Arlington, Virginia,USA, 2013. AUAIPress. [Google Scholar]

145. Xu J, Xu B, Wang P, Zheng S, Tian G, and Zhao J. Self-taught convolutional neural networks forshort text clustering. Neural Networks,88:22–31,2017. [PubMed] [Google Scholar]

146. Xu T and Oard DW. Wikipedia-based topic clustering formicroblogs. Proceedings of the American Society for Information Science and Technology,48(1):1–10,2011. [Google Scholar]

147. Yan X, Guo J, Lan Y, and Cheng X. A biterm topic model for short texts. InProc of the 22nd Int Conference on World Wide Web, WWW‘13, pages1445–1456, New York, NY,USA, 2013. ACM. [Google Scholar]

148. Yin J, Chao D, Liu Z, Zhang W, Yu X, and Wang J. Model-based clustering of short text streams.In Proc of the 24th ACM SIGKDD Int Conference on Knowledge Discovery& Data Mining, KDD ‘18, pages2634–2642, New York, NY,USA, 2018. ACM. [Google Scholar]

149. Yin J and Wang J. A dirichlet multinomial mixture model-based approach for short text clustering. In Proceedings of the 20th ACMSIGKDD International Conference on Knowledge Discovery and Data Mining, KDD‘14, page 233–242,New York, NY, USA, 2014.Association for Computing Machinery. [Google Scholar]

150. Yin J and Wang J. A model-based approach for text clustering with outlier detection. In 2016 IEEE 32nd International Conference onData Engineering (ICDE), pages625–636. IEEE,2016. [Google Scholar]

151. Yin J and Wang J. A text clustering algorithm using an online clustering scheme for initialization. In Proceedings of the 22ndACM SIGKDD International Conference on Knowledge Discovery and Data Mining,KDD ‘16, page1995–2004, New York, NY,USA, 2016. Association for ComputingMachinery. [Google Scholar]

152. Žalik KR and Žalik B. Validity index for clusters of different sizesand densities. Pattern Recognition Letters,32(2):221–234,2011. [Google Scholar]

153. Zhang H, Wheldon C, Dunn AG, Tao C, Huo J, Zhang R, Prosperi M, Guo Y, and Bian J. Mining twitter to assess the determinants ofhealth behavior toward human papillomavirus vaccination in the unitedstates. Journal of the American Medical Informatics Association,27(2):225–235,2020. [PMC free article] [PubMed] [Google Scholar]

154. Zhang L, Hall M, and Bastola D. Utilizing twitter data for analysis ofchemotherapy. International journal of medical informatics,120:92–100,2018. [PubMed] [Google Scholar]

155. Zhao Y, Guo Y, He X, Huo J, Wu Y, Yang X, and Bian J. Assessing mental health signals among sexual and gender minorities using twitter data. In 2018 IEEEInternational Conference on Healthcare Informatics Workshop(ICHI-W), pages 51–52.IEEE, 2018. [PMC free article] [PubMed] [Google Scholar]

156. Zheng CT, Liu C, and San Wong H. Corpus-based topic diffusion for short textclustering. Neurocomputing,275:2444–2458,2018. [Google Scholar]

157. Zuo Y, Wu J, Zhang H, Lin H, Wang F, Xu K, and Xiong H. Topic modeling of short texts: A pseudo-document view. In Proceedings of the 22nd ACM SIGKDDInternational Conference on Knowledge Discovery and Data Mining, KDD‘16, page 2105–2114,New York, NY, USA, 2016.Association for Computing Machinery. [Google Scholar]

158. Zuo Y, Zhao J, and Xu K. Word network topic model: A simple but generalsolution for short and imbalanced texts. Knowl. Inf. Syst.,48(2):379–398,Aug. 2016. [Google Scholar]