URLs in this document have been updated. Links enclosed in {curly brackets} have been changed. If a replacement link was located, the new URL was added and the link is active; if a new site could not be identified, the broken link was removed.

Chat Reference Service: An Analysis of one Semester's Data

JoAnn Sears
Science & Technology Reference Librarian
Chemical, Mathematical, & General Science Subject Specialist
Auburn University Libraries
searsjo@auburn.edu

Abstract:

Four months of chat reference questions received at a centralized reference services desk of an ARL library are analyzed in this study. Types of questions and types of users (when identified) are investigated. This study examines whether the questions were localized to the specific library's resources/services as well as whether the person responding to the chat question gave any evidence of consulting resources during the course of the chat session. Suggestions for further areas of research are also included.

Introduction and Literature Review:

There has been a considerable amount of literature published on digital or electronic reference service in its various forms; most of this has dealt primarily with e-mail service (Goetsch 1999; Gray 2000; Lankes 2000). A recent article discussed a two-year study of e-mail reference questions received at an academic library in which 450 e-mail questions were analyzed by the authors into categories (Diamond & Pease 2001). Chat reference services have also begun to emerge recently in the literature. The articles that focus on this technology in libraries generally fall into one or more of the following types:

1. survey articles that examine either what libraries are doing or what chat technologies are available (Francoeur 2001; Gray 2000; Breeding 2001),
2. implementation articles that describe projects done at a specific library/consortia (Broughton 2001; Eichler & Halperin 2000; Saunders 2001), and
3. forecast articles that discuss the possibilities that chat technology will offer for the future of reference services (Coffman 2001).

We can observe emerging patterns of cooperative digital reference service ventures that cross between the second and third categories above; these projects--which vary greatly along the implementation-planning spectrum--include: the Metropolitan Cooperative Library System's 24/7 service, the Collaborative Digital Reference Services initiative, and even newer projects like the Association of Southeastern Libraries' Virtual Reference Project (Saunders 2001; ASERL 2001). Breeding (2001) observed: "One of the key issues associated with the cooperative approach is whether the questions posed by remote users are general enough that they can be answered by a reference librarian from another library, or if they require the expertise of local librarians."

Beyond the categories listed above, there seems to be little represented in the literature at this time, probably due to the relative newness of this technology to libraries. After completing an updated reference services survey of ARL libraries, Tenopir and Ennis (2001) noted "A few of these libraries report that they now offer online chat versions of reference-many report they are thinking about it and planning to offer it soon." One recent article includes some quantitative data of chat reference service at the U.S. Department of Energy Library; the statistics presented there appear to have been generated by the software that is used at the library. The author of that study also gave some general impressions of the types of questions received over a period of time (Patterson 2001). It is difficult to find any research that provides an in-depth analysis of chat reference service for a period of time, which is the objective of this study.

Chat reference services may be physically located in librarians' offices, remote facilities such as branch libraries, or even homes. Stemper and Butler (2001) observed: "Interactive modes of digital reference (e.g., chat, videoconferencing) present staffing requirements that more closely resemble staffing models of the traditional reference desk." The Reference Desk itself is also a logical location that libraries contemplating chat reference may consider, and that is where the library in this study houses it.

Auburn University Libraries implemented its chat reference service, InfoChat, in September 2000. One month prior to that, the four subject Reference Desks were consolidated into one centralized reference services desk in the main library, and the software was installed at one of four computer terminals at that centralized service point. While there are two small branch libraries on campus, the main library performs a high percentage of overall reference service at the university. Because of that, two to four people--including librarians, staff, and some graduate students--staff the main Reference Desk. These people respond to the chat reference service queries.

During the Fall 2000 semester, the newly implemented InfoChat service was publicized through a variety of avenues. These included, but were not limited to: departmental liaisons announcing the service to their constituents, instruction librarians demonstrating the service in classes, an article in the student newspaper, inclusion on the Libraries' "What's New" listserv, and appearance in the faculty newsletter. During that Fall 2000 semester, the service was limited to afternoon and evening availability, usually operating from 1:00-9:00 p.m. on Monday-Thursday and 1:00-5:00 p.m. on Friday; this was a total of approximately 36 hours per week. While reference librarians were adjusting to the changing needs of a newly centralized reference desk and desired a limited timeframe for this new service, it was agreeable that the limited hours would include the hours that were perceived to be the peak hours of the new service--thus offering the most service possible during those limited hours.

In January 2001, the hours of InfoChat were expanded to include any time in which there were two or more reference personnel at the Reference Desk; this is essentially all but the first and last hours that the main library is open to the public (approximately 77 hours per week). Because the software for this service is installed at the Reference Desk--where telephone, walk-in, and chat reference questions are all initiated--the department found that at least two people were necessary to manage all of the service queries.

Since its inception, InfoChat has been open to the general public, requiring no authentication to prove university affiliation of any kind; requests about users' affiliation to the university are not made. The chat software used is HumanClick(TM); it is a text-based chat system (HumanClick 2000). When the library initially offered its chat reference service, it was a free version of the software. In April 2001, that changed to a fee version that allowed automatic archiving, searchable transcripts, and canned message options. Because these features were not available for the complete duration of this study, manual archiving was continued throughout the time period of the study for consistent analysis and comparison of the data.

To illustrate the users' perspective of the InfoChat service, a sample is shown below of the web entry page where patrons can initiate a chat session; the alternative "leave us a message" icon that patrons see when no personnel are available is also shown in that image. Below that is a sample of the HumanClick(TM) interface that patrons see during the course of an InfoChat session.

It is important to clarify a vocabulary distinction used in this study before proceeding, that of: chat session vs. chat question. A chat session is defined as a chat connection between the user and library personnel; either the user or librarian could initiate the chat session. Visitors who look at an entry page but do not connect are not considered as participating in a chat session. Chat sessions may have 0, 1, 2, or more chat questions in a single chat session. A chat question is defined like any other question; there may be several questions asked to communicate a single information need.

Methods:

This study took place during the Spring 2001 semester; it consisted of fifteen full weeks of classes plus the week of Spring Break. This time-frame was chosen because: 1) it gave a reasonable amount of data to analyze, 2) it provided a benchmark to which future studies could be compared, 3) it seemed to be an appropriate period of time for an academic library--that is to say that some questions cycle with each academic term, 4) enough time had passed from our initial implementation of the chat service that librarians and staff felt comfortable with the interface, and 5) some months had passed from the initial marketing of the service, so it was felt that public relations efforts would not unduly affect statistics during any portion of the study.

Manual archiving of chat sessions was done at the conclusion of each session. The librarian or staff member responding to the session would simply copy and paste the chat session transcript into a word processing file and save the session with the filename indicating the date of the session. On days where more than one chat session occurred, the final digit of the filename noted whether it was the second, third, etc. session of the day. No data was collected about the length of the chat session, time of day the session occurred, or the IP address of the patron. Any chat service technology system that includes automatic archiving would likely make this data available and might be considered for future use.

Because manual archiving was done, it is possible that some sessions were not archived; it is to be assumed that the data included here represent the minimum number of sessions and questions during the examined time period, not the exact number. Sessions were not archived if no question was asked; that is to say if a chat session connection was established but there was no question, it was not archived or counted in the study.

No data was recorded about "repeat users." If a patron initiated a chat session, closed the session, pursued the suggestion made by the librarian, and then returned to initiate another chat session, it is recorded in the same manner as if it there had been two separate users in two separate chat sessions.

After the data was collected, each chat session was read by the author and several codes were recorded for each chat question; the categories identified by the codes included: 1) the chat session number of the day, 2) the chat question number within that chat session, 3) the user's affiliation with the university, if identified, 4) whether or not the librarian gave any evidence of consulting resources/colleagues, 5) whether or not the question was localized to our specific library's services/resources, 6) whether the question was answered during the course of the chat session or if it was referred to someone else, 7) which type of question it was, and 8) whether the question was general or subject-specific as outlined by the library's list of subject specialties. An initial effort was made to evaluate the success/failure of the chat reference transaction; that process was soon abandoned because it was unclear as to how that would be measured. Category number 8 was also later dropped because many questions could not be categorized using the library's list of subject specialties, even if the queries seemed to be specific in nature. Category number 6 was also removed from the study because it was found that in many of the sessions the responders appeared to both answer the question completely and give an additional resource for further inquiries; the two categories were not mutually exclusive of one another.

Results:

Below is a chart representing the number of chat questions and chat sessions received during each week of the sixteen-week study period. On average, there were 9.6 chat questions per week and 7.4 chat sessions per week, with a high degree of variability between weeks. Week 11 represents Spring Break when no classes were in session. In later weeks there seemed to be more questions asked per chat session than there were in the first few weeks, as Weeks 8 through 10 and 15 illustrate. Also, the peaks at Weeks 4, 10, and 15 may represent due dates of mid-term projects.

Following is a chart that represents the average number of chat questions and chat sessions per hour of service availability. The data is presented by the day of week with the total number of service availability hours noted below each day. On average, there were few questions asked per hour, with more questions asked during weekdays than on weekends. The weekend data is interesting because the nature of this service technology enables libraries to extend hours on weekends, even when the physical library is closed. A further area of study might be to investigate how weekend use changes if that is the only reference service option available to users during some hours.

It is important to note that InfoChat does not require or request any information regarding users' affiliation with the university. It is commonplace for the answering librarian to state "If you are a faculty, staff, or student, you have access to X database (or Y resource); if you are not, I can suggest another tool." Throughout this process, patrons frequently volunteer their affiliation or lack thereof; below is a breakdown of types of users as identified by the data.

There were 112 days included in the study involving 153 chat questions and 118 chat sessions. That represents an average of approximately 1.4 chat questions per day and about 1.1 chat sessions per day. The use of this service is not overwhelming our ability to answer chat questions at this time.

Types of Questions:

Efforts were made to follow the categories outlined by Katz (2002); directional questions are segmented out of this grouping as suggested by the ANSI/NISO Library Statistics Standard (NISO 1997). A separate category was also included to describe the policy and procedural questions, and some categories were further broken down as trends emerged. Below is a chart illustrating the number of chat questions in each category and some sample chat questions that were asked.

Note: Exact quotes are taken from chat questions for the following samples, including spelling/grammar errors, to illustrate the nature of questions received in this study.

Type of question:	Number of Chat Questions:	Sample chat questions:
Reference questions:
Ready-reference questions (Katz 2002):	34	"I'm trying to site more than one chapter...every chapter is by the author of the book. I just wasn't sure if I had to document the book title every time or if there was a way to abbreviate it or something." (Note: Further clarification was made regarding the necessary citation style.) "I need to find out the full name of this journal title Kiel. Meeresforsh; its german" "how do i find out about I/O psycholigists programs M.S. and Ph.D.?" (Note: The chat discussion that ensued determined that quick data from a standard reference book was sufficient, so this is classed as ready-reference.)
Specific-search questions (Katz 2002):	50	"I'm interested in articles about labor costs in I/T" "how would i go about finding information on a court decision concerning public school dress codes?" "I need help finding critical analysis on a play. My theater teacher said that you guys had good like study guide things. Can I get them off this web page. The play I need is Lysistrata"
Research questions (Katz 2002):	1	"I need to find info on...I am a therapist about to testify..."
Total Reference Questions:	85 (55.6%)
Policy & procedural questions:
Database instructions, passwords, etc.: (Note: Web site navigation has been listed under directional questions.)	11	"I need to renew books. I cannot get the site to accept my name and barcode?" "Yes, I am an AU student trying to access some of the full-text databases on my home computer. I don't know which username & password the screen is asking for...My global user ID and password aren't working"
Library policy clarification; availability of a specific library service:	38	"Can faculty have electronic/print documents added to the electronic reserves?" "I am looking for a book which is not borrowed out. But it is not at the location indicated by the number. What can I do? The book is Spheromak by Bellan, QC 718.5 .C65 .B45 2000" "what is the bindery?" "how long does it typically take for an item on interlibrary loan to arrive?"
Total policy & procedural questions:	49 (32.0%))
Directional questions: (NISO 1997):
Location within the physical library environment:	14	"where do I get a copy card" "what floor is QL on?"
Location of resources on the Libraries Web site:	3	"Hello!! I came in last week to see a demonstration on 'My Library. 'But I've forgotten how to start or where to go first, can you help?"
Location of places within the city or state:	2	"where is the library?"
Total directional questions:	19 (12.4%)
Total of all questions:	153 (100.0%)

Other Results:

Of the 153 chat questions, there were 92 (60.1%) in which the responder needed to know something unique about the local library's policies, procedures, resources, and/or services.

The data was also examined to see whether there was evidence to suggest that librarians consulted books, databases, colleagues, or other resources while responding to chat questions. Below are sample statements taken from chat transcripts that illustrate consultation:

• "One second I'm checking Peterson's Graduate & professional programs",
• " According to aubiecat we have the March 2001 in current issues on the third floor",
• "One moment, while I do a search for 'vital records' offices in Alabama", and
• "Also, my colleague informs me that the..."

Of the 153 chat questions, 59 (38.6%) of the responses gave evidence of consulting books, databases, colleagues, or other resources. When consultation references were not made, it was generally in response to directional, policy, and procedural questions. This brief data regarding resource consultation is not by itself sufficient to draw any conclusions. It does not break down into categories what kinds of resources were consulted or whether an alternative, remote resource could have been substituted. It does show that libraries will want to consider what kinds of resources they make available to individuals responding to chat queries; these resources might include: a small print ready-reference collection, web access with IP or password authentication to library databases, or a means to contact colleagues.

Conclusions and Areas for Further Study:

Some of the major findings and implications include:

• Approximately one-half of the questions in this study were of the ready-reference type; about one-third were policy/procedural questions. Only one research level question was asked during this study; it's possible that patrons do not ask research questions in this type of medium, suspecting somehow that chat technologies represent quick question/answer communications. Further studies could investigate to see if these general breakdowns are typical at other academic libraries or libraries of other types. Libraries that require authentication (i.e., faculty, staff, or student status at an academic library) to prove affiliation may wish to see if more research-level questions are received at their sites.
• More than half of the time in this study, users volunteered some information about their affiliation with the university or lack thereof.
• For more than one-third of the questions, librarians in this study gave some indication that they consulted other resources in their response to chat questions, including books, databases, fellow colleagues or other resources.
• About 60% of the time, chat questions in this study were related to the local library's policies, procedures, collections, or resources; this data has significant implications for those exploring collaborative projects. Planning should especially be considered in regards to answering questions for other participating libraries' constituents.
• A relatively limited number of users initiated chat sessions over the course of the semester--resulting in an average of only about 1.4 chat questions per day. Since the time that this data was recorded, the specific library in the study has attempted to provide a more visible access point to the chat service; it is a bright yellow icon that appears on the library's home page and within the web catalog. It is important to note that although there were few questions in the studied period, the service is still in its infancy stages and more chat questions of all types are expected in the future.

In future studies we might want to consider more quantitative data regarding the chat sessions. These may include: the time of day questions are most asked, "repeat visitors" (a returning IP address or a returning patron identification number if authentication is required), or the average time of chat sessions. Other possibilities for study include some assessment or evaluation of chat reference service. Follow-up e-mails may be needed to elicit user satisfaction levels for assessment; some level of authentication or identification would be necessary for this kind of analysis as well.

An analysis of how "reference interviews" play out during chat sessions would also be an area for investigation. The aspect of electronic interviewing was discussed in a recent article that focused primarily on e-mail service (Straw 2000). To examine how this is exhibited in chat service--where real-time questioning and interviewing takes place--could yield some interesting results.

Another area for further study might be to examine the relationship of how chat reference service compares with in-person reference service. Are users more satisfied in one medium than another? Are patrons given more attention (or more correct answers) in one format than in another? Are the question types really different in each forum? Should service policies be changed to better accommodate these technologies in reference service?

As for the future of InfoChat, the AU Libraries are exploring the possibility of branch library participation in chat service as well as investigating improved feature capabilities such as VoIP (Voice over Internet Protocol), co-browsing capabilities, and application sharing. One system that we are currently reviewing is a product by Convey Systems (Convey Systems 2001).

Evolving trends in the area of digital reference services can be observed in a variety of locations; a few key resources include: McKiernan's LiveRef web site, the DIG_REF listserv, and the Virtual Reference Desk conference (McKiernan 2001; Information Institute of Syracuse 2001).

References:

ASERL (Association of Southeastern Research Libraries). 2001. ASERL Virtual Reference Project. [Online]. Available: {http://www.aserl.org/projects/vref/default.htm} [September 21, 2001].

Breeding, M. 2001. Providing virtual reference service. Information Today 18(4):42-43.

Broughton, Kelly. 2001. Our experiment in online, real-time reference. Computers in Libraries 21(4):26-31.

Coffman, S. 2001. Distance education and where we are headed. Computers in Libraries 21(4):20-25.

Convey Systems, Inc. 2001. Convey Systems: OnDemand(TM) interaction. [Online]. Available: {http://www.scientigo.com/} [September 21, 2001].

Diamond, W. & Pease, B. 2001. Digital reference: a case study of question types in an academic library. Reference Services Review 29 (3):210-218.

Eichler, L. & Halperin, M. 2000. LivePerson: keeping reference alive and clicking. EContent 23(3):63-66.

Francoeur, S. 2001. An analytical survey of chat reference services. Reference Services Review 29(3):189-203.

Goetsch, L., Sowers, L., & Todd, C. 1999. Electronic Reference Service. ARL SPEC Kit # 251. Washington, D.C.: Association of Research Libraries Office of Leadership & Management Services.

Gray, S.M. 2000. Virtual reference services: directions and agendas. Reference & User Services Quarterly 39(4):365-375.

HumanClick(TM) Ltd. 2000. HumanClick(TM): A Service from LivePerson(SM). [Online]. Available: {http://www.humanclick.com/} [September 21, 2001].

Information Institute of Syracuse. 2001. DIG_REF listserv. [Online]. Available: {https://groups.yahoo.com/neo/groups/dig_ref/info} [September 17, 2001]. Also: The Virtual Reference Desk(SM). [Online]. Available: http://www.vrd.org/ [September 17, 2001].

Katz, W.A. 2002. Questions and Searches. Introduction to Reference Work, Volume I: Basic Information Sources. 8th ed. Boston: McGraw-Hill, pp. 15-19.

Lankes, D., Collins, J., & Kasowitz, A.S. (eds.) 2000. Digital Reference Service in the New Millennium: Planning, Management, and Evaluation. New York: Neal Schuman.

McKiernan, G. LiveRef(SM): A Registry of Real-Time Digital Reference Services. [Online]. Available: http://www.public.iastate.edu/~CYBERSTACKS/LiveRef.htm [September 14, 2001].

NISO (National Information Standards Organization). 1997. Library Statistics. ANSI/NISO Z39.7-1995. Bethesda, MD: NISO Press.

Patterson, R. 2001. Live virtual reference: more work and more opportunity. Reference Services Review 29(3):204-209.

Saunders, L. 2001. Building the virtual reference desk. Information Today 18(3):25-27.

Stemper, J.A. & Butler, J.T. 2001. Developing a model to provide digital reference services. Reference Services Review 29(3):172-188.

Straw, J.E. 2000. A virtual understanding. Reference & User Services Quarterly 39(4):376-379.

Tenopir, C. & Ennis, L.A. 2001. Reference services in the new millennium. Online 25(4):40-45.

Previous

Contents

Next