Authors: Behnam Rahdari, Peter Brusilovsky, Dmitriy Babichenko
This Seed edition’s formatting was converted from the ACM version of record under the supplied ACM publication authorization.
Abstract
Over the past two decades, several information exploration approaches were suggested to support a special category of search tasks known as exploratory search. These approaches creatively combined search, browsing, and information analysis steps shifting user efforts from recall (formulating a query) to recognition (i.e., selecting a link) and helping them to gradually learn more about the explored domain. More recently, a few projects demonstrated that personalising the process of information exploration with models of user interests can add value to information exploration systems. However, the current model-based information exploration interfaces are very sophisticated and focus on highly experienced users. The project presented in this paper attempted to assess the value of open user modeling in supporting personalized information exploration by novice users. We present an information exploration system with an open and controllable user model, which supports undergraduate students in finding research advisors. A controlled study of this system with target users demonstrated its advantage over a traditional search interface and revealed interesting aspects of user behavior in a model-based interface.
CCS Concepts
Information systems → Graph-based database models; In-
formation integration; Web searching and information dis- covery.
Keywords
Exploratory search; Information exploration; Open user model; Recommender system; Intelligent interface; Knowledge graph
ACM Reference Format: Behnam Rahdari, Peter Brusilovsky, and Dmitriy Babichenko. 2020. Personalizing Information Exploration with an Open User Model. In Proceedings of the 31st ACM Conference on Hypertext and Social Media (HT ’20), July 13–15, 2020, Virtual Event, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3372923.3404797
1 INTRODUCTION
Over the past decades searching for information has become an inseparable part of our daily lives. Every day, Google alone receives
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. HT ’20, July 13–15, 2020, Virtual Event, USA © 2020 Association for Computing Machinery. ACM ISBN 978-1-4503-7098-1/20/07...$15.00 https://doi.org/10.1145/3372923.3404797
more than 5.4 billion search queries [34] ranging from simple questions such as “what day is today?” to more complex one, for instance, “where is the closest four-star Italian restaurant near me?” How ever the difference between searches is not always limited to the complexity of the query. In some cases, users do not have a clear idea of what they are looking for and need more information to reach or even formulate their final goal. Furthermore, the outcome of the search can not be simply described in a single best result. For example, when a novice user is looking for information about buying a new computer, many details need to be clarified first. In what way this computer is going to be used? How much computational power and memory is needed? Should it be a desktop or a laptop computer? In this case, the final results are uncertain and there are some technical details that the user needs to learn before making an informed decision. In contrast to simple lookup search with clear input and output, this kind of complex search is referred to as exploratory search [36].
Over the last 20 years, a range of researchers investigated the key issues and user needs of the exploratory search process and suggested a number of valuable recognition-focused technologies (such as faceted browsing and tag-based navigation) that offer better support of information exploration than traditional recall-focused keyword search [4, 10, 15, 21–23, 30, 37]. Within this stream of work, there were a few attempts to use personalization and, more specifically, open user models [1, 2, 5, 12] to support information exploration process. In contrast to the traditional user modeling with hidden user models [6], open user modeling approach offers users of personalized systems the ability to explore and even tune their user models. Open user models appeared to be a good match to the nature of information exploration and its use brought positive results. At the same time, existing attempts to integrate open user models into information exploration process suggested very complicated graphical designs with dozens of visual elements. These designs were focused on and evaluated with highly advanced users, usually, PhD students in computing and information fields.
The work presented in this paper attempts to expand the research on supporting information exploration with open user models. Our main goal was to suggest and assess a design that allows less advanced users, such as undergraduate students, to benefit from this technology. Our secondary goal was to design and assess a knowledge representation that could efficiently support personalized information exploration needs.
This paper explores the value of a relatively simple open user model design in the context of a typical exploratory search task of finding a research advisor. Finding an advisor for an undergraduate research project or a follow-up graduate study is a good example of an information exploration task. First, undergraduate students with no prior research experience rarely understand or express
their potential interests in clear scientific terms that experienced researchers use to describe their work. For example, a simple lookup search will hardly help someone interested in working on “big data in medicine” to find and recognize an advisor who applies machine learning approaches to health records. Second, students are frequently not aware of what their true interests are and might need to examine a few potentially interesting options to find their research passion. Third, the final result is usually not a single individual that satisfies all the student’s expectations but rather a group of individuals where each prospective advisor has several strong sides.
Focusing on the target users and their needs, we developed an experimental information exploration interface for finding research advisors. The interface integrated ideas and design elements from the areas of interactive recommender systems [14], open user modeling [2, 12], and exploratory search [9, 30]. To provide an efficient support for personalized information exploration, we used linked data integrated into a knowledge graph and supported by a native graph database. We then evaluated the effectiveness of our experimental system against a traditional search system with target users. Our results demonstrate that the target user extensively used novel information exploration and profile building features. The experimental system caused a major shift in their information exploration behavior and lead to a better user experience. A deeper analysis of user behavior allowed us to connect user satisfaction with the use of exploratory features and revealed some interesting patterns of system use. The paper is structured as follows. In the next section, we will review most relevant work on information exploration and open user modeling in this context. Section three describes the interface and the internal knowledge graph-based engine of Grapevine, an information exploration system for finding research advisors. The following sections report the design and the results of our user study. We conclude with a summary of main results followed by a discussion of the future works and limitations.
2 RELATED WORK
Exploratory search received its broader recognition as a special type of search in the early years of the 21[th] century. As defined by Wikipedia [36] “Exploratory search is a specialization of information exploration which represents the activities carried out by searchers who are: unfamiliar with the domain of their goal or unsure about the ways to achieve their goals or unsure about their goals in the first place.” A number of experts argued that the special nature of exploratory search calls for specialized information exploration interfaces that could better support less experienced users in this process. A range of these interfaces have been demonstrated at the first workshop on the topic “Exploratory Search Interfaces, Clustering, Categorization and Beyond”, which brought together experts in both Hypertext and Information Retrieval areas [20]. The follow-up publications demonstrated and evaluated several important types of exploratory search interfaces [11, 15, 19, 29]. This pioneer research along with two workshops focused on design and evaluation of exploratory search interfaces at SIGIR 2006 [32] and SIGCHI 2007 [31] helped to define exploratory search as a sub-field on the crossroads of information retrieval and HCI. Nowadays, this field is very active with a range of authors and project exploring
and assessing different ways to support users. A survey of these works could be found in [17, 24, 30].
The idea to apply open user models to better support information exploration process was among the early ideas explored in this field. Open user models allow users to examine and possibly change the content of user models applied to personalize their search or browsing process. Since the open user models increase interactivity, transparency, and controllability of the information exploration process, their application was a good match to the nature of exploratory search. Despite a recognized success of open user models in the field of personalized learning (where these models are usually called as open learner models [7]), the fist attempts to introduce open user models to exploratory search in the form of a keyword vector profile were not successful [2, 28]. However, a switch to semantic-level profiles that represent user interests over semantic entities such as domain concepts [5, 12, 25] or named entities
[3] allowed several research teams to develop highly successful information exploration systems with open user models.
One problem of existing information exploration systems with open user models is their complexity. All these system are based on relatively complex visualizations, which usually present many dozens of visual objects to their users, and expect relatively complex manipulations for profile-tuning. For example, Introspective Views [5] and SciNet [12] visualize open user models as a circular “radar” where more relevant concept points are located closer to
the center. To tune these models, the users are allowed to move these points towards the center or away from it. Adaptive Vibe [1] presents terms and named entities forming the user model as points of interests (POI) in a complex relevance-based visualization. The fine-tuning could be done by docking and undocking POI and moving it within the visualization space. These actions lead to useradapted and user-controlled visual presentation of search results. While all these interfaces demonstrated their efficiency with advanced users, such as graduate students looking for relevant research papers, it has not yet been demonstrated that these complex interfaces could support a more typical information exploration process where users are relative novices.
Our own experience with open user models focused on undergraduate students demonstrated that coarse-grain open models composed of 10-20 elements [27] are very efficient and could support relatively complex activities such as content navigation and social comparison. However, we also have strong evidence that fine-grained open models, which include over 100 visual elements might be too complex for this category of users to interpret [13].
3 THE GRAPEVINE SYSTEM
This section presents the interface and the underlying personalization mechanism of an information exploration system, Grapevine
[26], designed to help undergraduate students with no prior research experience in finding advisors – for an undergraduate research project or a graduate study – matching their research interests. This task is known to be challenging for students since they frequently cannot express their research interests in terms that their prospective advisors use to describe their work on their home pages and academic papers. To support this task, Grapevine helps every students to gradually form a model of her interests by
discovering new research topics and keywords that match their not-yet-well-formulated interests. Following the nature of the exploratory search, it provides users with different opportunities to recognize (rather than recall) relevant research topics in different contexts [8, 12, 15]. This model is then used to generate a list of recommendations. It also remains visible to the students who could fine-tune it in the context of their exploration. The system’s intelligent user interface is driven by a knowledge graph, a tightly connected network of research topics and prospective advisors.
3.1 Interface Design
During the information exploration, Grapevine engages users in building and refining their profile of interests, which is visualized as a list of preferred research topics (Figure 1C). The system offers users several ways to add a topic of interest to the profile. First, users are able to formulate a topics of interest directly using a search box (Figure 1A). To better engage recognition (vs. recall) and to expand the user’s vocabulary, the system uses the knowledge graph to recommend semantically similar topics (Figure 1B). The users could also find potentially relevant topics of interest by clicking on any recommended advisor and examining his interests in a detailed view (Figure 1: Grapevine system interface. Features of the interface include A: Search box, B: Recommended keywords,C: Interest profile and sliders, D: Results list, E: Heat bar and F: Final list.
Figure 2) where the full set of advisor’s interests is generated from the knowledge graph.
Once a user’s profile of interest is created, the system morphs into a controllable recommender system [14]. To allow greater user control in finding a relevant advisor, each specific topic (represented as a keyword or keyword phrase in the user’s profile) can be weighed separately using sliders (Figure 1C). The open and controllable model of user interests allows the user to achieve two objectives that are important for an information exploration system. First, at each point in time, the user is able to modify his/her preferences based on the newly acquired knowledge. This modification can be performed either by changing the importance and impact of a profile topic on the final results using the sliders or by adding or removing topics from the profile. Second, the dynamic profile enables users to combine multiple keywords from different areas of research to identify advisors who conduct research across multiple disciplines.
3.1.1 Search Box. The search box (Figure 1A) is the gateway to the system. Using an instant search approach, it allows users to
discover relevant topics without a fully formulated query. When a user starts typing a query, a series of lexically similar keywords appears, which helps the user to discover a range of matching topics (e.g., Data Mining and Educational Data Mining). All keywords in the knowledge graph are considered when generating the list of instant search matches.
3.1.2 Recommended Keywords. When at least one keyword is added to the user’s profile, a series of five semantically similar topics appear in the Similar Keywords area of the interface (Figure 1B). Users can add these recommended keywords to their interest profiles by clicking on the plus button to the right of each keyword. As the user’s profile grows and refines, the set of recommended keywords is updated since the system recommends instances similar to all keywords in the user’s profile. Each recommended keyword also provides users with a short description of the topic. Clicking on the question mark button next to the add button, opens up a separate window containing the abstract of that keyword’s Wikipedia entry. This information is crucial when the user is not familiar with the recommended keyword and needs more knowledge to decide whether the keyword must be added to the profile of interests.
ipfs://bafybeibqwplwdgkjs5i6fg5dwfkwty7fycf3ydck4mm3grv4cqzxczxas4)
3.1.3 Open User Profile. The user profile of interest area (Figure 1C) is an open model of user interests. As a model of interests, it defines the system-generated list of recommended advisors (Figure 1F). As an open model, it is visible and directly editable by the end users. To edit the model the user can add relevant topics as explained above as well as remove less relevant keywords (using the red x) as they discover more relevant topics or explore different interests. Sliders associated with each keyword enable users to control the relative importance of a topic compared to others in their profile, ranging from 1 (least important) to 10 (most important). The use of sliders for fine-tuning of user profile was motivated by keyword tuning approach in uRank design [9], which was confirmed as a user-friendly and efficient in an exploratory search context. The initial value of the sliders is set to five but can be changed at any time. All actions within the profile (adding, removing, or adjusting sliders) immediately affects the list of recommended advisors.
3.1.4 Recommended Advisors. As soon as the user adds the first keyword to the profile of interests, a list of the 20 most relevant advisors is generated as individual cards (Figure 1D). Each card contains a photo and brief information about the advisor. There is also a relevance bar (Figure 1E) to reflect the advisor’s relevance to the user profile of interests and two action buttons. The button Details opens the Details view (Figure 2) that provides more details about the advisor. The button Select adds the advisor to the final list of results (Figure 1F).
3.1.5 Details View. Details view (Figure 2) provides users with additional information about advisors such as affiliation, research impact, research interests (shown as a list of topics) and external links to their research page and Google Scholar profile. To stress an advisor’s relevance to the user profile, the research interests that match the user profile are shown in green. Since faculty usually explore a set of related topics, we expected that the research interests or the prospective advisor not yet added to the user profile (shown in blue) might also be of relevance to the given user. To support the “interest discovery” process, these blue topic keywords can be added to the user’s profile of interests and the short description of the topic can be reviewed with one click.
3.1.6 Results List. Throughout the search process, users can add multiple candidate advisors to the results list as they seem adequate with respect to the user’s need at that point in time. Later, when the user’s level of knowledge has changed by exploring the other options and learned about previously unknown topics, the results list provides users with the opportunity to review his/her previous choices and check whether or not they still satisfy the new and updated criteria. The detailed view with information about the advisor is still accessible for advisors in the results by clicking on their names. If the advisor in the list is no longer meets the user’s requirements it can be simply removed using the x button.
3.2 The Knowledge Graph
The knowledge graph consists of multiple entries extracted from Google Scholar [33], enriched by Wikipedia and hosted in a native graph database Neo4j[35]. The knowledge graph is an underlying knowledge layer of the system, which supplies the interface with real-time responses to the user interactions. Using a native graph
database to store the knowledge graph, as opposed to traditional flat databases, enabled us to perform complex relational queries with the near-zero waiting time. This was particularly important to provide users with a smooth and pleasant experience while using the system. Furthermore, the rich and interconnected nature of the graph enabled the system to extract complex semantic relationships between data points faster and simpler than the traditional approaches. Building the knowledge graph included the following stages.
HasLink
Link
Faculty HasKey Keyword
Category
3.2.1 Verification and Enrichment. We used the Wikipedia API to filter all extracted topic keywords; only keywords with an entry in Wikipedia were kept in the knowledge graph. This approach assured keyword quality and made it easier to weight each link between a keyword and an advisor using cosine similarity between the bags-of-words extracted from the Wikipedia page and the advisors’ publications. Furthermore, we added the top 10 most mentioned Links and all the Categories of the original Wikipedia entry to the knowledge graph and connect those to the corresponding keywords. “Links” are Wikipedia entities (keywords) that are not connected directly to advisors. The Link is connected to keywords if it has been mentioned in the keywords’ respective Wikipedia pages. The connections between keywords added a semantic layer to the knowledge graph and were used to recommend semantically similar keywords to the users.
3.2.2 Graph Schema. Figure 3: Graph Schema representing the entities of the knowledge graph and the relationship between them
Figure 3 presents the schematic representation of the knowledge graph. Advisors are interconnected by the relation WorksWith (based on co-authorship) and connected to Wikipedia-verified keywords by the relation HasKey. The latter carries a weight that determines the strength of the relationship between each keyword and the advisor’s research. Keywords are connected to Links and Categories (HasLink and HasCat).
3.2.3 Recommendation Method. We used the Cypher Querying Language to generate both advisor and keyword recommendations. For advisor recommendations, at each instance of user interaction with the system (e.g., adding/removing keywords), the system considers all advisors connected to at least one of the topics of interest in the user profile. Then, a relevance score is assigned to each candidate considering candidate’s similarity to each of profile topics and the value of the sliders (Equation 1). Finally, the system ranks the candidates by their relevance scores and presents the top 20 as results. If fewer than 20 candidates to recommend, similar keywords are used to find more advisors.
4.2 Data Source
To fill the system with data for our target population, we used Google Scholar information about the top 1000 (based on citations) researchers in two specific areas of computer science: Artificial intelligence and Computer architecture. We extracted academic information, including name, affiliation, number of citations, h-index, i10-index, self-defined areas of research (represented by keywords), and a list of the 20 most recent publications. We only included individuals with at least one publication within the last two years to ensure all individuals are still active and could realistically serve as prospective advisors. Next, we used the list of recent publications to extract keywords that represent each advisor’s research topics. Finally, we captured the top 20 co-authors for each prospective advisor to represent the social connections between researchers. Note that the use of top researchers worldwide simulates two realistic scenario for our cohort of junior and senior undergraduate students: finding an advisor for a Summer undergraduate project or finding a prospective advisor for a graduate school. This scenario also has two advantages over a possible alternative scenario (finding an advisor in one’s home department): first, it exposes students to a much larger set of candidates where support from an information exploration system is more valuable, second it allows the use of two comparable data sets for within-subject comparison of two designs.
4.3 The Baseline System
As the baseline system, we developed a version of our interface with all exploratory features removed. In essence, it is a Google-style search system where the search process is performed through the search box. No instant search is offered and user must rely on their knowledge to construct the query. Since the open user model is not part of the baseline interface, we allow users to concatenate multiple keywords, similar to Google Search, to capture the results with more than one keywords. When the final query is submitted, the system extract the keyword(s) from the query and follows the same procedure as the experimental system to generate the results. The list of results looks similar in both systems and the details about the advisor could be equally examined. While the baseline offers no clickable “research interests”, the users still can explore more information about a prospective advisor using external link to advisor’s research page and Google Scholar profile.
4.4 Procedure
To enable a direct comparison between the systems, we employed a within-subject design where all participants can experience both the baseline and the experimental systems. To minimize the order impact for both the systems and the research fields, we built four different versions of the study procedure, wherein each one, the type of systems (Grapevine and baseline) and the research field of advisors (Artificial Intelligence and Computer architecture) have been alternated. We labeled each of the versions with a color (Blue, Green, Red, and Yellow) and distributed color-coded cards with a URL links to the corresponding version to students in a random order. We ensured that there was the same number of participants for each version of the order.
The study procedure included a sequence of steps. In the first step, participants completed an electronic consent form. Next, a
RelevanceScore(f,A) =
size(A) �
i=0
Sim(ai, f ) ∗ wi (1)
1: Calculation of relevance score for each candidate advisor
In equation 1, A is a set of tuples {(a1, w1), (a2, w2), ...(an, wn )} that represent the current state of the user’s profile (topics and weights) and f is a given advisor in the graph. ai and wi correspond for i[th] keyword and its slider value at the moment. Sim(ai,f ) shows the value of relevance between a given keyword and a candidate advisor in our knowledge graph. We pre-calculated this value by computing the cosine similarity between the bag of words of advisor’s publications and Wikipedia entry. To generate recommended keywords, for each set of keywords in the user’s profile, the system generates three sets of candidate keywords. These sets were created using the co-occurrence of seed keywords and advisors’ research interests, links, and categories (using collaborative filtering [16]). Then, the system combines the number of co-occurred keywords in all three sets and uses it as a ranking mechanism. The system presents the top five results to the user.
4 THE STUDY
To evaluate our information exploration system, we performed a controlled user study with our target populations, undergraduate students. While the task of finding advisors could be beneficial for students on all levels (from undergraduate, to master, to PhD) we focused on undergraduate students to assess to what extent a personalized information exploration interface could be used by less advanced users. To make the study context as realistic as possible, we filled the system with data about prospective advisors that matched students’ background. To assess the value of several innovative features implemented in our system, we also developed a baseline system that supported a more traditional way of advisor search. This section presents the details of the study design. The results of the study are reviewed in the next section.
4.1 Participants
For our study we recruited 42 undergraduate students taking a data science class. We selected this class for our recruitment to make the context of research advisor search more realistic. As most students in this course were in their junior and senior years, they were more likely to be truly interested in finding a research advisor for their senior project or future graduate studies. The data confirmed that the majority of participants were in the senior year of their undergraduate degree (N = 38). Most students were completing their degree in Information Science (N = 33) and the rest in Computer Science (N = 3) and other fields of study. The average age of participants was 21.30 (SD: 2.00). None of the participants were involved in an active research project at the time of the study and they did not have any prior experience working with the Grapevine or baseline systems. Participants were not compensated for participation, however, they were motivated to invest their time in working with the system since the data collected by the system were offered to them, in anonymized form, for a class data analysis exercise.
pre-study survey gathered demographic information as well as the participant experience and opinions regarding the usage of common search and recommendation tools in their everyday lives. In the next step, we introduced the first system and task. A short tutorial video (with subtitles) has been provided to familiarize participants with the basic functionalities of the system and then the task presented to them. Depending on the study group color the participants were asked to explore either Artificial Intelligence or Computer Architecture, but otherwise, the tasks were equivalent: “You have been awarded a grant to conduct research in the field of
Artificial Intelligence (Computer Architecture) and related topics. You are trying to find a faculty member to act as your research advisor for the project. Use the system to identify FIVE faculty members with similar interests as you, whom you would ask to be your advisor for the grant”. We also asked participants to provide a short note of why they think each selected advisor is a suitable match for them, before adding them to their results list.
After finishing the task, participants completed a post-task survey about their experience with the system (fifteen questions, 5point Likert scale). In the case of working with the experimental system, the survey included five extra questions, which asked participants to rate the usefulness of each provided exploratory feature. In the next two steps, the user followed a similar exploration-feedback procedure with the alternative system and data set.
In the final step, we asked participants to rank their experience of working with both systems on the scale of 1 to 100 using sliders. The post-task survey contained six questions with a focus on three factors – general satisfaction, feeling of control, and the difficulty of working with the systems. Each factor had two questions in the final survey. We decided to use sliders to make this evaluation more distinct from post-task survey and encourage the students to re-think system benefits rather then to rely on their past answers. We also hoped that a broader scale could offer a more fine-grained understanding of users’ experience compared to the Likert-scale option use in the post-task surveys.
4.5 Main Results
In this section we review the main results of our experiment. In order to assess the value of the experimental system, we checked the following key factors: (1) whether the new exploratory and profiletuning features were embraced by the target users, (2) how the presence of these features changed their exploration behavior, and (3) whether the new design lead to better exploration experience.
Grapevine System Baseline Adding Keywords Tuning Queries direct similar details remove sliders Mean 3.83 5.74 3.19 2.47 4.86 7.8 SD 1.43 1.92 1.37 0.96 2.04 4.7 SUM 179 277 121 147 242 341 Total 527 389 341 Table 1: Querying and profile building summary.
4.5.1 Profile and Query Building Analysis. It is important to stress that the Grapevine system extended, but not replaced the traditional
query-based search. While the users could leverage recognitionbased keyword discovery provided by similar keyword recommendation and faculty details view, they could also complete the task using recall-based direct keyword addition, just as they do in a traditional search. However, our log analysis indicated that participants extensively used the provided exploratory and profile-tuning features. As it can be seen in Table 1, all three ways of adding keywords were used with a comparable frequency. This table provides several other important insights. First, the total number of directly added keywords (using the search-box) in the Grapevine system was less than the half of the number of queries in the baseline. Moreover, within the Grapevine system, less than a third of keyword was added by direct search. It indicates a major shift from traditional recall-based query building to mostly recognition-based profile building. Since recall-based activities lead to higher cognitive load, we could hypothesize that Grapevine could be a more enjoyable system to use. At the same time, the total number of added keywords in the Grapevine was almost twice as larger than in the baseline system. It points that exploration-bases features focused in topic recognition helped the users to assemble a much larger number of relevant topics. It could potentially increase user chances to find relevant advisors. The table also shows some considerable engagement in profile tuning: the participants used the opportunity to remove less relevant topics and also tuned the weight for more than a third of added keywords. At the same time, a reasonably small fraction of removed keywords, indicated that the participants were engaged in incremental building of their interest profile rather then using it as a a query replacing one set of keywords with another.
Figure 4 displays data on using model-tuning sliders in the Grapevine system. In total, sliders have been used more than 600 times (each participant interacted with sliders 15 times on average). This extensive use indicates that the complexity of slider-based approach to model-tuning was a good match for our target users as they frequently used sliders to modify their profile in order to receive better results. An interesting observation is that participants had a tendency to increase the value of sliders from the default position of 5 rather than decrease it. It could be an indication that participants tended to use sliders to stress the importance of specific keywords in their profile.
4.5.3 Exploration of Results. The process of finding advisors has two main activities – bringing in keywords to rank the results (either by issuing queries or building a profile of interests) and exploring the results in the ranked list. Table 2 compares the amount of result exploration between the systems. As the numbers show, users of the baseline systems had to examine details of almost 50% more prospective faculty to achieve the same goal – finding five best prospective advisors. Moreover, the use of external information through system–provided links to advisors’ home page and Google Scholar profile was about 5 times (and significantly) higher in the baseline system. It points that Grapevine offered better ranking helping the users to focus on most relevant advisors as well as offered more confidence for selecting the advisor so that the need to use information outside of the system was significantly reduced.
Grapevine System | Col2 | Col3 | Col4 | Col5 | Col6 | Baseline |
|---|---|---|---|---|---|---|
Adding Keywords | Adding Keywords | Adding Keywords | Adding Keywords | Adding Keywords | Adding Keywords | Queries |
Adding Keywords | Adding Keywords | similar | details | remove | sliders | sliders |
Mean | 3.83 | 3.83 | 3.83 | 3.83 | 3.83 | 7.8 |
Mean | 1.43 | 1.43 | 1.43 | 1.43 | 1.43 | 4.7 |
Mean | 179 | 179 | 179 | 179 | 179 | 341 |
Mean | 527 | 527 | 527 | 527 | 527 | 341 |
Baseline Grapevine significance Details View 956 692 Homepage Link 77 21 Google Scholar Link 110 13 Total 1143 747 Table 2: The number of clicks on the provided external links (= p < 0.05. = p < 0.01. = p < 0.001 – Independent T–Test)
Further insight about each system’s ability to help users confidently select an advisor is provided by the analysis of participants’ criteria that they have to specify each time they add an advisor to their final results list (Table 3). As it can be seen, the top three reasons for the Grapevine system are all related to system’s internal functionality (ranking, tuning, displaying a match). The value of all these internal factors is lower in the baseline system. In contrast, baseline systems users got less confidence for advisor selection from the system and have to rely more frequently on external factors. This is another evidence in the higher quality of recommendation and more trust in the results by the participants while using the Grapevine system.
Grapevine Baseline
Ranking/Compatibility Score 163 140 Based on Interest Match 149 135 Frequency of Appearance 66 25 Demographics of Researcher 37 41 University Ranking/Location 57 69
Table 3: The reported criteria for selecting an advisor.
The observed increase of profile building activities and decrease of result exploration overhead encouraged us to compare the full balance of query and profile building reviewed in Table 1 and exploration of results reviewed in Table 2. The results shown in Table 4 demonstrate that there is a significant shift of users’ activities
between the systems. While using the baseline system, participants spent a relatively small portion of their effort and time exploring different queries and had to invest much more efforts on exploring the results. In contrast, in Grapevine system, participants spent roughly the same amount of time and effort on profile building results exploration activities. This data clearly shows that the provision of exploration, profile building, and personalization lead to a major shift in the nature of user exploration behavior. However, to see to what extent this shift caused a better experience, we need to relate it to the user feedback, which is analyzed in the next section.
Baseline Experimental
4.00
3.65 3.56
3.32
3.00 3.19
3.06
2.67
2.00
1.00
0.00
Satisfaction Control Easiness
4.5.4 Post–Task Survey Results. Figure 5: The results of Post–Task survey: Grapevine system outperform the baseline in satisfaction, control, and easi- ness. (= p < 0.01. = p < 0.001 - Independent T–Test)
Figure 5 shows the results of the post–task survey for both Grapevine and baseline systems. The results show that in all three categories the Grapevine system performed better than the baseline. Participants were more satisfied, felt more in control, and completed the task with less cognitive load (the higher value means more positive feedback). The last result is especially encouraging. Although the Grapevine system offered a more complicated interface and with more features to explore, still it outperformed the baseline indicating that we achieve a reasonable balance of power and complexity in our interface.
Baseline Grapevine Significance
Search 20.9% 51.2% Actions Results 79.1% 48.8%
Search 24.3% 61.6% Time spent Results 75.7% 38.4% Table 4: The balance of actions and time spent on different types of activities in baseline and Grapevine systems. ( = p < 0.01. = p < 0.001 - Independent T–Test)
4.5.5 Post–Study Survey Results. The analysis of direct comparison of two systems provided by the participants in the post–study survey are presented in Figure 6: The post–study direct comparison between Grapevine and baseline system (= p < 0.05. = p < 0.01. = p < 0.001 - Independent T–Test)
Figure 6. These results indicate that the Grapevine system outperformed the baseline in several criteria, including Satisfaction, Feeling of Enjoyment, Feeling of being in control and Ease of learn. We can hypothesize that higher satisfaction and enjoyment might be the result of a decreased cognitive
Col1 | Baseline | Grapevine | significance |
|---|---|---|---|
Details View | 956 | 956 | |
Homepage Link | 77 | 77 | |
Google Scholar Link | 110 | 110 | |
Total | 1143 | 1143 |
Col1 | Baseline Grapevine Significance |
|---|---|
Actions | 20.9% |
Actions | 79.1% |
Time spent | 24.3% |
Time spent | 75.7% |
load, better ability to discover relevant topic and faculty, and higher trust in the results observed in the log analysis. The higher ability to control the system provided the expected increase of feeling in control and likely also contributed to the overall satisfaction and enjoyment [18]. It was also expected that baseline system was easier to use, which is likely due to resemblance of the baseline system to search engines that are being used by the users on a daily basis. It was, however, surprising that Grapevine was also found to be significantly less restrictive and easier to learn. Altogether, it shows that the Grapevine provided a considerably better match to the exploratory advisor search task for our target users.
Baseline Experimental
39.07
Satisfaction 59.17
47.90
Enjoyment 60.43
39.36
Control 58.02
71.69
Restriction 53.52
61.05
Ease of Use 42.90
39.83
Ease of Learn 58.52
0.00 20.00 40.00 60.00 80.00
4.6 A Deeper Analysis of Results
This section attempts to take a deeper look at some of the data reported above. We are comparing the use of the Grapevine between users and between different stages of working on the task.
4.6.1 Searchers vs. Explorers. As explained above, Grapevine system supported two ways of adding keywords to the profile - using a recall–based direct search and two recognition–based exploration ways (Table 1). As that table shows, at average, Grapevine users exhibited a large shift from direct search to exploration, as compared with the baseline system. However, it was up to the individual users how to balance the search and exploration within Grapevine. In order to observe differences in search–exploration balance between participants and connect the preferred balance to user satisfaction, we grouped the participants based on the balance of their activities attempting to form groups of comparable size. The first group (Searchers) was formed by participants who used search–box to add new keywords to their profile in more that half of all cases (64.1%). This group used the Grapevine more as a lookup than exploratory search system taking less advantage of the provided exploratory features of the system. The second group (Neutrals) was formed by participants who used direct search and exploratory features about equally to construct their profile of interests. The third group (Explorers) was formed by participants who extensively used the exploratory features such as recommended keywords and advisors’
list of research interests to build their profile of interests. The average percentage of user activities in these three groups is reported in Table 5.
Profile Size R² = 0.15 Total Keywords R² = 0.324
1.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
20 40 60 80 100
Feeling in Control
To connect the exhibited balance of search and exploration to the user productivity and satisfaction, we compared the user opinion about the system, as well as the total number of added keywords and the size of the profile among participants in these three clusters. As the results in Table 5 indicate, the Explorers have higher average satisfaction, enjoyment and feeling of being in control and less restricted as compared to the two other groups. They were also able to discover more relevant keywords on average and build larger profiles. Here and in the following analysis the “profile size” is measured as the total number of topic keywords in the user profile at the time of adding an advisor to the final result list; it averaged it between all added advisors for each user. We also see that all these parameters gradually improve as the fraction of exploration increases from searchers to neutrals to explorers. Combined with the the difference between the two systems (Figure 6), this data stresses the value of exploratory features provided by the Grapevine. It also demonstrates that the provision of novel features does not immediately assure its use by the target population and the expected benefit, the users had to understand and embrace these features. In this context, the learnability of the system becomes an important factor.
4.6.2 The Value of Finding more Keywords and Building Larger Profiles. In the analysis above, we considered that adding more keywords and building larger profiles are a positive factors. We expected that larger profiles could support better personalization offer more help in finding and recognizing relevant advisors. The variability of profile–building activities between users of the Grapevine (as observed above) allowed us to find more evidence in favor of these assumptions by investigating the effect of keyword discovery and profile size on users’ opinion about the Grapevine system. As it shows in Figure 7: The normalized profile size and total added key- words vs. feeling of being in control.
Figure 7, there is a direct positive correlations between both the number of added keywords and the size of user profile and feelings of being in control.
39.07 | 59.1 |
|---|
Explorers | Col2 | Neutrals | Searchers | Sig. |
|---|---|---|---|---|
Group Size | 11 | 11 | 11 | 11 |
Direct Search | 29.7% | 29.7% | 29.7% | 29.7% |
Exploration | Exploration | Exploration | Exploration | Exploration |
Satisfaction | Satisfaction | Satisfaction | Satisfaction | Satisfaction |
Enjoyment | 70.2 | 70.2 | 70.2 | |
Control | 73.5 | 73.5 | 73.5 | |
Restriction | 40.4 | 40.4 | 40.4 | |
Ease of Use | 65.4 | 65.4 | 65.4 | – |
Ease of Learn | 62.5 | 62.5 | 62.5 | – |
Total Keywords | 12.31 | 12.31 | 12.31 | |
Profile Size | 4.38 | 4.38 | 4.38 |
Table 5: User feedback and productivity for clusters with dif- ference balance of search–exploration behavior (= p < 0.05. = p < 0.01. = p < 0.001 - Independent T–Test)
Direct Search Exploration Sliders Removal Results
0.75
0.67
0.58
0.50
0.42
0.33
0.25
0.17
0.08
0.00
Phase 1 Phase 2 Phase 3 Phase 4
4.6.3 Different Phases of Exploration. To better understand the information exploration process in the Grapevine system, we divided the total of time spent by each participant into four equal parts. We then investigated the interaction the balance between main types of user actions (adding keywords by search and exploration, tuning results by removal and sliders, and examination of results) in any of those phases of exploration. The duration of each phase was similar for each individual, but could vary between different users. The average duration of phases between all participant was 5 minutes and 39 seconds. Figure 8: The balance of user activities at different phases of exploration process.
Figure 8 shows the average proportion of each type of interaction in different phases.
As the figure shows, the essence of Phase 1 is incremental profile building using the search box and exploratory features. Almost no exploration or tuning is done at that stage. It is also at the first stage when the lion share of search–based keyword addition is
performed, which is natural since no keyword recommendation or result exploration could be provided until at least one topic is added to the profile using search. In the next step (Phase 2), participants began to explore the results but keep actively adding keywords to find better matches. Starting from this stage the majority of keywords are added through exploratory options and the use of search rapidly drops. The users also started to tune their profiles, doing it at that stage mostly through keyword removal rather than weighting. The next phase (Phase 3) is where the majority of tuning actions are done - addition and removal of keywords here is almost balanced and the use of sliders increases. The participants hold the majority of their exploration activities until tuning is finished. Finally, the last phase (Phase 4) focuses predominantly on the exploration. The profile built at stages 1–3 apparently produces enough good recommendation to examine then in a quick succession. The participants still invest some fraction of time in profile tuning, but at that stage it is predominantly fine–tuning with sliders. The slider use is at its peak in this phase while the addition and removal of keywords is minimal. The balance of activities over time provide a good evidence that the profile–based personalization in Grapevine was used as intended - from profile building to tuning to examination of results. The large fraction of exploration at the last stage stresses the ability to leverage the established profile for finding relevant results.
5 CONCLUSIONS
In this paper, we presented a personalized information exploration system Grapevine with an open user model. To make personalization efficient, we supported the information exploration process with linked data organized into a semantic knowledge graph and implemented with native graph database. We evaluated the Grapevine against a baseline that resembled a regular search system. The data shows that the novel exploration and profile–building tools were extensively used by the target group. It also revealed a major shift of user exploration activity in the Grapevine with the increased investment into profile building balanced by more efficient and confident exploration of results. User feedback indicated that this shift lead to a considerably higher user experience: the Grapevine system outperformed the baseline in satisfaction, enjoyment, feeling of control and ease of learning. A more detailed analysis of user behavior in Grapevine indicated that higher investment into exploratory profile building lead to better user experience. It also revealed the role of each group of tools in the exploration process.
6 LIMITATIONS AND FUTURE WORK
The presented results are subject to limitations in the experiment design and implementation. While we performed the study within a target group of undergraduate students, the majority of participants were information and computer science students who might have more experience with complex interactive systems than average undergraduate population. Furthermore, the voluntary nature of the study might influence the engagement rate of the students during the experiment. We are planning to conduct more user studies with more diverse participants and specifically include participants with limited knowledge about the subject of the search.
7 ACKNOWLEDGEMENTS
This work has been partially supported by Personalized Education Grant program of the University of Pittsburgh.
REFERENCES
[1] Jae-Wook Ahn and Peter Brusilovsky. 2013. Adaptive Visualization for Exploratory Information Retrieval. Information Processing and Management 49, 5 (Sept. 2013), 1139–1164.
[2] Jae-wook Ahn, Peter Brusilovsky, Jonathan Grady, Daqing He, and Sue Yeon
Syn. 2007. Open user profiles for adaptive news systems: help or harm?. In the 16th international conference on World Wide Web, WWW ’07. Association for Computing Machinery, New York, NY, USA, 11–20.
[3] Jae-wook Ahn, Peter Brusilovsky, and Shuguang Han. 2015. Personalized Search:
Reconsidering the Value of Open User Models. In Proceedings of the 20th International Conference on Intelligent User Interfaces. Association for Computing Ma- chinery, New York, NY, USA, 202–212. https://doi.org/10.1145/2678025.2701410
[4] Einat Amitay, David Carmel, Nadav Harél, Shila Koifman, Aya Soffer, Sivan
Yogev, and Nadav Golbandi. 2009. Social search and discovery using a unified approach. In The 20th ACM conference on Hypertext and hypermedia. Association for Computing Machinery, New York, NY, USA, 199–208. https://doi.org/10. 1145/1557914.1557950
[5] Fedor Bakalov, Birgitta König-Ries, Andreas Nauerz, and Martin Welsch. 2010.
IntrospectiveViews: An Interface for Scrutinizing Semantic User Models. In 18th International Conference on User Modeling, Adaptation, and Personalization (UMAP 2010). Springer, Big Island of Hawaii,USA, 219–230. https://doi.org/10.1007/978- 3-642-13470-821
[6] Peter Brusilovsky and Eva Millán. 2007. User models for adaptive hypermedia
and adaptive educational systems. Lecture Notes in Computer Science, Vol. 4321. Springer-Verlag, Berlin Heidelberg New York, 3–53. http://dx.doi.org/10.1007/ 978-3-540-72079-91
[7] Susan Bull. 2004. Supporting learning with open learner models. In 4th Hel-
lenic Conference on Information and Communication Technologies in Education, Vol. A. Hellenic Conference in Information and Communication Technologies in Education, Athens, 47–61. http://ile2005-t3.tripod.com/text/support-OLM.pdf
[8] Robert G Capra and Gary Marchionini. 2008. The relation browser tool for faceted
exploratory search. In Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries. Association for Computing Machinery, New York, NY, USA, 420–420.
[9] Cecilia di Sciascio, Vedran Sabol, and Eduardo E. Veas. 2016. Rank as you go:
User-driven exploration of search results. In Proceedings of the 21st International Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, 118–129.
[10] Justin Donaldson, Michael Conover, Benjamin Markines, Heather Roinestad, and
Filippo Menczer. 2008. Visualizing Social Links in Exploratory Search. In The 19th ACM Conference on Hypertext and Hypermedia. Association for Computing Machinery, New York, NY, USA, 213–217.
[11] John Gersh, Bessie Lewis, Jaime Montemayor, Christine Piatko, and Russell
Turner. 2006. Supporting insight-based information exploration in intelligence analysis. Commun. ACM 49, 4 (2006), 63–68.
[12] Dorota Glowacka, Tuukka Ruotsalo, Ksenia Konuyshkova, Kumaripaba Athuko rala, Samuel Kaski, and Giulio Jacucci. 2013. Directing Exploratory Search: Reinforcement Learning from User Interactions with Keywords. In International conference on Intelligent User Interfaces, IUI ’2013. Association for Computing Machinery, New York, NY, USA, 117–127.
[13] Julio Guerra Hollstein, Jordan Barria Pineda, Christian Schunn, Susan Bull, and
Peter Brusilovsky. 2017. Fine-Grained Open Learner Models: Complexity Versus Support. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization. Association for Computing Machinery, New York, NY, USA, 41–49.
[14] Chen He, Denis Parra, and Katrien Verbert. 2016. Interactive recommender
systems: A survey of the state of the art and future research challenges and opportunities. Expert Systems with Applications 56, C (2016), 59–27.
[15] Marti A. Hearst. 2006. Clustering versus Faceted Categories for Information
Exploration. Commun. ACM 49, 4 (April 2006), 59–61. https://doi.org/10.1145/ 1121949.1121983
[16] Jonathan L Herlocker, Joseph A Konstan, Al Borchers, and John Riedl. 1999. An
algorithmic framework for performing collaborative filtering. In 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 1999. Association for Computing Machinery, New York, NY, USA, 230–237.
[17] Tingting Jiang. 2014. Exploratory search: a critical analysis of the theoretical
foundations, system features, and research trends. In Library and Information
Sciences: Trends and Research. Springer, Berlin, Heidelberg, 79–103.
[18] Bart P. Knijnenburg, Svetlin Bostandjiev, John O’Donovan, and Alfred Kobsa.
Inspectability and Control in Social Recommenders. In 6th ACM Conference
on Recommender System. Association for Computing Machinery, New York, NY, USA, 43–50.
[19] Bill Kules, Jack Kustanowitz, and Ben Shneiderman. 2006. Categorizing web
search results into meaningful and stable categories using fast-feature techniques. In Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries. Association for Computing Machinery, New York, NY, USA, 210–219.
[20] Kules, Bill and White, Ryen and Bederson, Ben. 2005. HCIL SOH 2005 - Ex ploratory Search Interfaces Clustering, Categorization and Beyond. http: //ryenwhite.com/xsi/index.html [Online; accessed 9-January-2020].
[21] Nicolas Marie, Olivier Corby, Fabien Gandon, and Myriam Ribière. 2013. Com posite Interests’ Exploration Thanks to on-the-Fly Linked Data Spreading Activation. In Proceedings of the 24th ACM Conference on Hypertext and Social Media (HT ’13). Association for Computing Machinery, New York, NY, USA, 31–40. https://doi.org/10.1145/2481492.2481496
[22] David R. Millen, Meng Yang, Steven Whittaker, and Jonathan Feinberg. 2007.
Social bookmarking and exploratory search. In Tenth European Conference on Computer Supported Cooperative Work, ECSCW’07. Springer, Berlin, Germany, 21–40.
[23] Paul Mulholland, Trevor Collins, and Zdenek Zdrahal. 2005. Spotlight browsing of
resource archives. In The Sixteenth ACM conference on Hypertext and Hypermedia. Association for Computing Machinery, New York, NY, USA, 23–31.
[24] Emilie Palagi, Fabien Gandon, Alain Giboin, and Raphaël Troncy. 2017. A Survey
of Definitions and Models of Exploratory Search. In Proceedings of the 2017 ACM Workshop on Exploratory Search and Interactive Data Analytics. Association for Computing Machinery, New York, NY, USA, 3–8. https://doi.org/10.1145/3038462. 3038465
[25] Behnam Rahdari and Peter Brusilovsky. 2019. Building a Knowledge Graph
for Recommending Experts. In Proceedings of the 1st International Workshop on Challenges and Experiences from Data Integration to Knowledge Graphs co-located with the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2019), Anchorage, Alaska, August 5, 2019 (CEUR Workshop Proceedings), Vol. 2512. CEUR-WS.org, Aachen, 1. http://ceur-ws.org/Vol-2512/ paper3.pdf
[26] Behnam Rahdari, Peter Brusilovsky, Dmitriy Babichenko, Eliza Beth Littleton,
Ravi Patel, Jaime Fawcett, and Zara Blum. 2020. Grapevine: A Profile-Based Exploratory Search and Recommendation System for Finding Research Advisors. In Proceedings of the 2020 ASIS&T Virtual Annual Meeting (AM2020). Association for Computing Machinery, New York, NY, USA, 0. InPress
[27] Sergey Sosnovsky and Peter Brusilovsky. 2015. Evaluation of Topic-based Adap tation and Student Modeling in QuizGuide. User Modeling and User-Adapted Interaction 25, 4 (2015), 371–424. https://doi.org/10.1007/s11257-015-9164-4
[28] Annika Wærn. 2004. User Involvement in Automatic Filtering: An Experimental
Study. User Modeling and User-Adapted Interaction 14, 2–3 (June 2004), 201–237. https://doi.org/10.1023/B:USER.0000028984.13876.9b
[29] Ryen W White, Bill Kules, Steven M Drucker, et al. 2006. Supporting exploratory
search. Commun. ACM 49, 4 (2006), 36–39.
[30] Ryen W White and Resa A Roth. 2009. Exploratory search: Beyond the query response paradigm. Synthesis lectures on information concepts, retrieval, and services 1, 1 (2009), 1–98.
[31] White, Ryen W. and Drucker, Steven M. and Marchionini, Gary and Hearst, Marti
and schraefel, m.c. 2007. SIGCHI 2007 Workshop - Exploratory Search and HCI Designing and Evaluating Interfaces to Support Exploratory Search Interaction. http://ryenwhite.com/esi/organizers.html [Online; accessed 9-January-2020].
[32] White, Ryen W. and Muresan, Gheorghe and Marchionini, Gary. 2006. SIGIR
2006 Workshop - First Workshop on Evaluating Exploratory Search Systems. http://ryenwhite.com/eess/index.html [Online; accessed 9-January-2020].
[33] Wikipedia contributors. 2019. Google Scholar — Wikipedia, The Free Ency clopedia. https://en.wikipedia.org/w/index.php?title=GoogleScholar&oldid= 933379317 [Online; accessed 9-January-2020].
[34] Wikipedia contributors. 2019. Google Search — Wikipedia, The Free Ency clopedia. https://en.wikipedia.org/w/index.php?title=GoogleSearch&oldid= 933379560 [Online; accessed 9-January-2020].
[35] Wikipedia contributors. 2019. Neo4j — Wikipedia, The Free Encyclopedia. https:
//en.wikipedia.org/w/index.php?title=Neo4j&oldid=929999540 [Online; accessed 9-January-2020].
[36] Wikipedia contributors. 2020. Exploratory Search — Wikipedia, The Free Ency clopedia. https://en.wikipedia.org/wiki/Exploratorysearch [Online; accessed 9-January-2020].
[37] Barbara M Wildemuth and Luanne Freund. 2012. Assigning search tasks de signed to elicit exploratory search behaviors. In Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval. Association for Comput- ing Machinery, New York, NY, USA, 4.
Do you like what you are reading? Subscribe to receive updates.
Unsubscribe anytime