Previous | Contents | Next | ||
Issues in Science and Technology Librarianship |
Spring 2006 | |||
DOI:10.5062/F4D50JX8 |
URLs in this document have been updated. Links enclosed in {curly brackets} have been changed. If a replacement link was located, the new URL was added and the link is active; if a new site could not be identified, the broken link was removed. |
Amy Paster Head, Life Sciences Library alp4@psu.edu
|
Kathy Fescemyer Life Sciences Librarian kaf12@psu.edu
|
Nancy Henry Health Sciences Librarian nih1@psu.edu
|
Janet Hughes Biological Sciences Librarian jah19@psu.edu
|
Helen Smith Agricultural Sciences Librarian hfs1@psu.edu
| |
Life Sciences Library Penn State University Park, PA |
The Pennsylvania State University Life Sciences Library conducted the Wisconsin-Ohio Reference Evaluation Program survey in October 2005. The results of the survey indicated that the desk personnel do an average job in fulfilling the library user's expectations. Known-item questions were handled well but non-known questions need additional training to ensure patrons are not receiving too much information. This project raised a number of questions concerning user satisfaction and library type. It is hoped that this article can serve as the baseline for future research in the evaluation of science reference question transactions and patron satisfaction.
Assessment in libraries is currently a topic of much research and discussion within the profession. The Association of Research Libraries has developed the LibQUAL+tm program and according to their home page (accessed April 13, 2006), it is used to solicit, track, understand, and act upon users' opinions of service quality. However the instrument used is quite broad in its definition of service and does not specifically relate to reference desk transactions.
Richardson (2002) defines quality reference service as user satisfaction, which is driven by library staff behavior. However this leaves out the accuracy and usability of the answer, which is another measure that is often considered important. The Wisconsin-Ohio Reference Evaluation Program (WOREP) has been used to measure reference performance in a number of general academic reference departments and academic music libraries, as well as public libraries. "WOREP is designed to assess the outcome of the reference transaction and to identify factors related to success or lack of success" (WOREP home page 2006). WOREP collects data from both the patron and the staff involved in the reference transaction, and evaluates transaction success as well as patron satisfaction. According to early aggregate WOREP data, factors which affect success at general reference desks in academic libraries include: staffing; time and effort spent on the question; effective communication; type and source of the question; and collection size (Bunge 1991).
The utilization of the WOREP instrument in the academic music library setting (Christensen et. al 2001) resulted in some debate as to whether the instrument’s validity and reliability translated effectively into a more specialized, complex reference transaction environment (Lasocki 2002). However the instrument’s validity in the general reference environment has been established (Murfin & Gugelchuk 1987). The decision to use this general reference survey instrument in the Life Sciences Library assessment process at Penn State also resulted in discussions regarding the effective translation of the instrument into a complex science reference transaction environment.
Throughout the literature, there are few articles addressing the uniqueness of reference desk service in a science library. Moysa (2004) discusses evaluating reference behavior, but not the success of the transaction, and although the article describes the situation in a sci/tech library, the instrument developed is general in nature, focusing on approachability, interest, and positive attitudes. Robbins and Daniels (2001) reported on academic health science libraries using the SERVPERF instrument to measure the perceived quality of reference desk service and compare quality across libraries. However this instrument does not measure the success of the transaction, does not measure factors which might affect patron perceptions of success, and the low rate of return affected the possible comparative usefulness.
Is reference success in sci/tech libraries affected by the same factors as general academic libraries? Over the years WOREP has been available, other sci/tech libraries have used the survey (Radcliff 2006); however the data have not been published, and the libraries have not been aggregated as "sci/tech libraries" for comparative purposes. This article is the baseline for future research in the evaluation of science reference question transactions and patron satisfaction.
The University Libraries at the Pennsylvania State University is ranked 14th among 113 North American research libraries (ARL Statistics 2004). This includes 36 libraries at 24 locations throughout Pennsylvania, with the largest campus being at University Park. Overall, the Libraries' cataloged collection now approaches five million items. Approximately 100,000 volumes are added annually to the collection, and over 58,000 serials are received. The Life Sciences Library (LSL) is one of the subject-specific libraries at the University Park campus, and although it is housed in the main library building, it is a separate entity with its own reference desk, reference collection, current periodicals, and unique collections. The LSL collections are approximately 300,000 volumes and cover all aspects of agriculture, biology, biochemistry, nutrition, nursing, and public health. The LSL is a primary collection for the students, faculty and staff in the College of Agricultural Sciences, the College of Health and Human Development, and the Eberly College of Science. At the time of the study, the personnel staffing the Life Sciences Library reference desk consisted of five professional librarians, three full-time staff which were classified as "library assistant" for the purposes of this study, and five part-time staff (four of which were students) which were all classified as "student/other" for the purposes of this study.
The WOREP survey is a two-part form that is filled out by both the patron asking the question, and the library staff answering the question. The patron form asks for general demographic information on the patron (status, major), the satisfaction the patron had with the transaction, whether patrons received the information they needed, and any reasons for dissatisfaction. The library staff form asks for information about the type of question (known item vs. non-known item, subject area), number and type of sources used, question difficulty, and factors affecting the transaction (patron in a hurry, books off the shelf, etc.). Sample forms may be found at the {WOREP home page}.
To prepare all LSL personnel for the survey, the WOREP guidelines were discussed extensively at staff meetings. All questions on the survey were reviewed to ensure that personnel defined criteria in the same way. Because WOREP is designed only to evaluate reference transactions, the differences between a directional transaction and a reference transaction were explicitly defined. In addition, all personnel were asked to complete five sample surveys to become comfortable with the process before the actual survey was initiated. Surveys were distributed between October 24 and November 16, 2005.
In order to minimize increased workload on the reference personnel, only the first two reference (not directional) transactions in each hour, where the patron agreed to participate, were used for the study. All patrons were eligible for the survey whether they were PSU faculty, staff, students, or the general public. Patrons were asked if they would be willing to take a survey to help assess how the personnel at the Reference Desk performed. After securing agreement and ensuring the patron was over 18 years of age (for legal reasons) reference staff handed the patron a survey, a copy of the implied consent form, and a pencil. After the survey was filled out it was placed in the survey box which was housed in an area removed from the Reference Desk. The reference staff member answering the question would fill out their half of the survey form and then place it into the survey box. Following the completion of the minimum 100 surveys, they were returned to the WOREP office at Kent State University to be analyzed.
One hundred and two forms with usable results (out of the 104 handed out) were collected. This is a 98.1% rate of return. The results show how the LSL patron profiles, query profiles, satisfaction profiles compared with other academic libraries who also participated in WOREP surveys. At the time the LSL data were processed, there were 117 participating academic libraries, and 41 of these were classified as "comparable academic" because they were of comparable collection size to the LSL. The results showed actual numbers and percentages for the LSL, and then showed percentages for the comparable academic libraries, the academic library that had the best results, and the average of all participating academic libraries. Results primarily focus on the "non-known item" transactions, in which the patron is requesting "anything or something," in contrast to those "known item" transactions where a particular item is requested. Some of the following data were not explicitly provided by the WOREP report, but were easily calculated from the information provided.
In contrast to comparable academic libraries, the LSL surveyed more freshman, fewer sophomores, more juniors and fewer seniors. When combined, the LSL had a lower percentage of freshmen/sophomores asking questions than the comparable or all academic libraries, but a higher percentage of juniors/seniors. Graduate students were more highly represented than in comparable academic libraries, but less than the average for all academic libraries. Faculty, staff and alumni were more highly represented than in comparable and all libraries (Table 1). As would be expected in a life sciences subject-specific library, more surveys were completed by those in the medicine, health, agricultural and biological sciences disciplines (Table 2), a total of 67.1%.
TABLE 1: Patrons by Status | |||
---|---|---|---|
STATUS | LSL | Comparable Academic |
All Academic |
Freshmen | 20.8% | 18.7% | 16.5% |
Sophomores | 10.9% | 19.1% | 15.3% |
Juniors | 24.8% | 15.6% | 17.6% |
Seniors | 16.8% | 19.6% | 21.5% |
Graduates | 12.9% | 10.9% | 15.1% |
Continuing Educ. / Non-degree | 0.0% | 3.7% | 2.2% |
Faculty | 5.9% | 3.0% | 3.2% |
Staff | 4.0% | 1.9% | 2.1% |
Alumni | 3.0% | 1.8% | 1.8% |
Non affiliated | 1.0% | 7.0% | 5.9% |
TABLE 2: Patrons by Major | |||
---|---|---|---|
MAJOR | LSL | Comparable Academic |
All Academic |
Agricultural / Biological Sciences | 38.2% | 4.2% | 4.8% |
Medicine / Health | 28.9% | 8.3% | 8.6% |
Technical / Engineering | 9.2% | 6.6% | 7.0% |
Other Social Sciences | 7.9% | 10.2% | 13.0% |
Arts & Humanities | 5.3% | 14.6% | 19.1% |
Education | 3.9% | 14.2% | 11.5% |
Business / Management | 2.6% | 23.3% | 19.2% |
Undeclared | 1.3% | 7.5% | 6.3% |
Interdisciplinary/ Other | 1.3% | 3.7% | 4.8% |
Math / Physical Sciences | 1.3% | 3.3% | 2.8% |
Law | 0.0% | 4.6% | 3.3% |
The LSL survey respondents had more known-item questions than the comparable, or all, academic libraries (Table 3). Other than the "specific book" category, more questions fall into the need "something/anything" category than any other type of question (Table 4), and this is a greater percentage than in comparable or all academic libraries. Online tools such as the OPAC, indexes, full-text articles and online reference tools and web sites comprised over 80% of the sources used to answer the queries (Table 5). The LSL staff also tended to use multiple sources to answer a question more often than comparable or all libraries in the study (Table 6), and were more likely to spend more time on non-known item reference transactions (Table 7). The LSL had a higher percentage of "student/other" answering questions than comparable or all academic libraries (Table 8).
TABLE 3: Query Type - known item (particular text or specific resource) versus non-known item (something or anything on a topic). | |||
---|---|---|---|
QUERY TYPE | LSL | Comparable Academic |
All Academic |
Known Item | 26.5% | 12.0% | 15.7% |
Non-known Item | 73.5% | 88.0%% | 84.3% |
TABLE 4: Types of questions asked. | |||
---|---|---|---|
TYPE OF QUESTION | LSL | Comparable Academic |
All Academic |
Specific book, serial, score, recording etc. | 26.9% | 12.0% | 15.4% |
Something, Anything | 18.3% | 16.3% | 13.7% |
Certain type of source or materials | 9.6% | 11.5% | 11.6% |
Smaller item in larger publication | 7.7% | 2.5% | 2.6% |
Short answer facts | 5.8% | 9.0% | 9.5% |
Facts and statistics | 5.8% | 11.3% | 10.7% |
Focus on aspect: Biog, Hist, other | 4.8% | 6.9% | 6.1% |
Explanation of particular source or library | 2.9% | 8.9% | 9.5% |
Analysis, trends, pro/con, cause/effect. Etc | 2.9% | 5.4% | 4.9% |
Anything by particular author | 1.9% | 0.9% | 0.9% |
Criticism, Reviews, Interpretations | 0.00 | 4.9% | 5.6% |
TABLE 5: Frequency of sources used in LSL for non-known item queries | |
---|---|
SOURCE | Frequency used (%) |
Library Catalog | 25.95 |
Research Databases | 21.62 |
Full-text Articles (print or online) | 12.97 |
Online Reference | 9.73 |
External Free Web | 8.65 |
Print Reference | 6.49 |
Own Knowledge | 5.41 |
Library-Made Web Sites/Guides | 4.86 |
Print Indexes | 1.62 |
Consult | 1.08 |
Refer | 1.08 |
Circulating Books (print or online) | 0.54 |
Vertical File | 0.00 |
Other | 0.00 |
TABLE 6: Number of sources used in LSL for non-known item queries | |||
---|---|---|---|
# SOURCES USED/RECOMMENDED | LSL | Comparable Academic |
All Academic |
1 source | 21.6% | 35.2% | 40.9% |
2 sources | 32.4% | 30.1% | 29.8% |
3 sources | 28.4% | 17.6% | 16.3% |
4 sources | 13.5% | 8.7% | 6.6% |
5 sources | 4.1% | 8.4% | 6.5% |
TABLE 7: Time spent on question for non-known item queries | |||
---|---|---|---|
TIME SPENT | LSL | Comparable Academic |
All Academic |
0-3 minutes | 9.5% | 32.7% | 33.9% |
3-5 minutes | 41.9% | 42.2% | 40.9% |
5-15 minutes | 37.8% | 21.6% | 21.0% |
over 15 minutes | 10.8% | 4.0% | 4.6% |
TABLE 8: Staff types answering non-known item questions | |||
---|---|---|---|
Staff type | LSL | Comparable Academic |
All Academic |
Librarian | 46.7% | 73.5% | 71.1% |
Library Assistant | 25.3% | 14.3% | 16.7% |
Student/Other | 28.0% | 7.6% | 7.2% |
Staff not specified | 0.0% | 4.7% | 5.0% |
TABLE 9: Patron found exactly what was wanted and was satisfied, by query type | |||
---|---|---|---|
% "found exactly" and satisfied | |||
QUERY TYPE | LSL | Comparable Academic |
All Academic |
Known Item | 92.6 | 56.6 | 59.5 |
Non-known Item | 61.3 | 57.0 | 58.8 |
TABLE 10: Patron found exactly what was wanted and was satisfied, for non-known item queries, by staff type | |||
---|---|---|---|
% "found exactly" and satisfied | |||
STAFF TYPE | LSL | Comparable Academic |
All Academic |
Librarian | 60.0 | 58.1 | 60.1 |
Library Assistant | 78.9 | 62.0 | --- |
Student/Other | 50.0 | 54.8 | 60.3 |
Staff not reported | --- | 50.2 | 52.7 |
TABLE 11: Patrons who found exactly what was wanted and were satisfied, by type of question. | |||
---|---|---|---|
% "found exactly" and satisfied | |||
QUESTION TYPE | LSL | Comparable Academic |
All Academic |
Facts & Statistics | 100 | 57.3 | 53.8 |
Analysis, trends, pro/con, cause/effect, etc | 100 | 50.3 | 49.3 |
Focus on specific aspect: Biog, Hist, Other | 80 | 50.8 | 55.7 |
Certain type of source or materials | 78.6 | 55.8 | 58.8 |
Explanation of particular source or library | 66.7 | 62.7 | 66.1 |
Something, anything | 50 | 50.5 | 53.4 |
Short Answer Facts | 33.3 | 62.9 | 66.4 |
Criticism, Reviews | -- | 56.0 | 55.0 |
TABLE 12: Most frequent patron reported problems, for non-known item queries. | |||
---|---|---|---|
% reported problems | |||
Patron reported problems | LSL | Comparable Academic |
All Academic |
Too much information found | 14.7 | 1.5 | 1.5 |
Staff only partly, or not at all, knowledgeable | 13.3 | 8.9 | 9.1 |
Couldn't find information in source | 9.3 | 4.5 | 4.2 |
Staff only partly gave enough, or did not give enough at all, explanation or help | 6.7 | 7.3 | 7.6 |
Staff only partly, or did not at all, understand query | 6.7 | 5.7 | 5.2 |
Need more in-depth information | 5.3 | 10.3 | 8.5 |
Want more simple information | 5.3 | 2.9 | 2.9 |
Information not relevant enough | 4.0 | 4.1 | 4.2 |
Need information with different viewpoint | 2.7 | 1.9 | 2.1 |
Not Enough information | 1.3 | 13.0 | 11.7 |
Staff explanations were only partly, or not at all, clear | 1.3 | 6.0 | 6.1 |
Found Nothing | 1.3 | 3.2 | 2.9 |
Staff only partly, or did not at all, spend enough time | 0.0 | 4.3 | 5.0 |
Staff were only partly, or not at all, courteous | 0.0 | 2.0 | 2.0 |
Not sure if information is correct | 0.0 | 1.5 | 1.5 |
TABLE 13: Percent agreement between staff and patrons on perception of success | |||
---|---|---|---|
Success perception | LSL | Comparable Academic |
All Academic |
Staff and patron agree on success | 60.0 | 65.4 | 65.2 |
TABLE 14: Percent of patrons reporting learning new sources as result of transaction (non-known item queries). | |||
---|---|---|---|
# of new sources learned | LSL | Comparable Academic |
All Academic |
2 or more new sources | 29.7 | 37.4 | 35.5 |
1 new source | 54.1 | 47.2 | 48.7 |
no new sources | 18.2 | 15.3 | 15.7 |
TABLE 15: Percent of patrons reporting learning something about the library as a result of transaction (non-known item queries). | |||
---|---|---|---|
Learned something? | LSL | Comparable Academic |
All Academic |
Yes | 85.3 | 78.7 | 78.2 |
Yes and Partly | 94.7 | 93.6 | 93.4 |
No | 5.3 | 6.4 | 6.6 |
The higher percentages of upper level patrons may indicate that the LSL has more demanding clientele with more difficult questions but this may be similar to other science and technology libraries, or even indeed to other subject-specific libraries in a University setting. The majors most represented by the comparable academic libraries are business/management, arts and humanities, education and other social sciences. What is unknown is if these libraries were subject-specific libraries within a larger library, or were general libraries serving all subjects.
Like the LSL, music libraries had a higher percentage of known item queries than general libraries (Christensen et. al 2001). Although the reasons for this have not been determined from the data, perhaps this is common in all specialized libraries. Another area that may be common across science and technology libraries is the fact that there was a higher percentage of the "something, anything" type of question than in comparable or all libraries, and conversely a lower percentages in more general types of questions such as "short answer facts" and "facts & statistics," as well as a logically lower percentage in some questions more germane to humanities libraries such as "criticism, reviews, interpretations," and "focus on aspect: biog, hist, other."
The tendency of the LSL to use multiple sources, and spend more time on questions, might be a positive indication that there are more sources available to answer any question, or a negative indication that the LSL does not have sources completely suitable to that specific information need of the patrons. However, the strength of the collection at Penn State is indicated in several areas. First, for the known item questions at the LSL, an extremely high percentage "found exactly" what was wanted, and the result was better than at all other participating libraries. According to Christensen et.al (2001), larger libraries generally score higher on this measure. The strength of the collection is indicated in another area as well. In the report of problems by patrons, the LSL had a lower percentage of problems in "found nothing" and the "not enough information" categories than both comparable and all academic libraries.
The LSL distributed the reference load more evenly among librarians, assistants and students, and has students covering the desk at a much higher percentage of time than the comparable and all other libraries. The data indicate that the LSL patrons were more satisfied when working with library assistants than with librarians and less satisfied when served by students/others. This situation was similar to comparable academic libraries, but the LSL library assistants scored appreciably higher than assistants in comparable academic libraries and the students lower. This may be reflective of several long-term staff members with significant experience, and a need to train the students in better customer service and subject expertise.
For patrons' reports of problems most percentages or actual numbers were quite low, and in several areas were less than both comparable and all academic libraries. The most frequent single issue was the report of getting "too much information." This falls into the category of the information received (as opposed to the service), and it could be related to the time spent on queries, the type of query, the number of sources used, the staffing level, whether the patron was in a hurry, or even the patron's perception of the amount of information available. The category of "want more simple information" was also a problem a greater percentage of the time than for comparable or all academic libraries, and could be a similar, or related, issue. This may be a reflection of librarians' tendencies towards trying to instruct the patrons (in support of the instructional mission of the PSU University Libraries) instead of simply giving the patrons the requested information. The instructional inclination is evident in that patrons reported learning new sources, and/or something about the library, in a large majority of the transactions.
The two most frequent complaints under the service category were reports that the staff appeared only partly or not knowledgeable, or only partly or did not understand the query. Although these two complaints are also problems in comparable and all academic libraries, they are worthy of being addressed. It is unknown if these complaints were related to the staff type, and it may be a reflection of the LSL higher percentage of students serving the desk and their possible lack of the specialized subject knowledge important in a science library. Further training for all staff on specialized databases and subjects may be one way to decrease this type of issue.
Overall, patron and staff perceptions of the success of the transaction were in agreement 60% of the time, and although this was less than comparable or all academic libraries, it does indicate that the LSL perceptions of the reference process were not overly optimistic or overly pessimistic. However, further staff training to ensure that all staff members are able to recognize when communication difficulties occur, might increase this percentage.
Additional correlations would be helpful in identifying more exact issues that should be addressed in the PSU Life Sciences Library. Specifically it will be useful to determine the how various factors such as the type of question, staffing level, time spent on question, and number of sources used, correlate with the specific type of problem (such as "too much information" or "staff not knowledgeable"). And information about how the staff perception of some of the special conditions, such as "patron in a hurry" and "communication difficult," as well as the staff perception of the difficulty of the question, correlates with specific patron reported problems would be informative. General information that might also be useful includes correlations between the type of query (known item or non-known item) and the type of question, with the patron status or major.
WOREP shows great promise in comparative situations. Some advantages are that it is easy to administer, inexpensive to use, and the return rate is relatively high (Bunge 1991). Its primary use is in looking at reference success, but it also gives an indication of the types of reference being done by describing the types of patrons served, the types of questions asked, and the ways the questions are answered. Perhaps it can be determined if reference questions and patrons are different in science and technology libraries. Is our patron success affected by the same factors as those of a general reference department, or even of other subject specialty libraries such as music? Are patrons served by science and technology libraries more or less satisfied than those served by arts and humanities or social science libraries? Or are these factors independent of the subject served, and affected more by the type of patron and the type of question?
To be more relevant to science and technology libraries, it is possible that the WOREP survey instrument should be altered to reflect better the situation in those libraries. As was done in the music library situation, indication of the location of the library (branch or within main library) may be a measure of interest. And perhaps a closer look at the specific types, and possibly even the exact titles, of information resources utilized in specialized science libraries could provide richer comparative data and lead to greater knowledge of customer reference service needs in the science library environment. Alteration to increase the subject category specificity might also contribute to more specialized quantitative data collection about science library reference transactions. This benchmarking data could translate into useful information for comparative analysis of science library reference services nationally.
It is hoped that this article will encourage reference assessment in science and technology libraries. In order to truly compare ourselves, we all need to use the same instrument. It is possible that the WOREP survey, with some modification, is the appropriate vehicle for determining the answers to some of these questions.
Bunge, C.A. 1991. Gathering and using patron and librarian perceptions of question-answering success. In: Evaluation of Public Services and Public Services Personnel (ed. by B. Allen), pp.59-83. Champaign, Ill.: University of Illinois Graduate School of Library and Information Science.
Christensen, B.E., Du Mont, M.J. & Green, A. 2001. Taking note: assessing the performance of reference service in academic music libraries: a progress report. Notes 58(1):39-54.
Lasocki, D. 2002. A critique and response to the use of the instrument in the published study "Taking Note: Assessing the performance of Reference Service in Academic Music Libraries" (Notes, September 2001, pp.39-54). Notes 58(3): 699-703.
LibQUAL Home Page. n.d. [Online]. Available: http://www.libqual.org/ [Accessed April 13, 2006].
Moysa, S. 2004. Evaluation of customer service behaviour at the reference desk in an academic library. Feliciter 50(2):60-63.
Murfin, M. E. and Gugelchuk, G. M. 1987. Development and testing of a reference transaction assessment instrument. College & Research Libraries 48:314-38.
Radcliff, C. 2006. List of WOREP participating libraries. E-mail correspondence April 25, 2006.
Richardson, J. V. 2002. Reference is better than we thought. Library Journal 127(7): 41-42.
Robbins, K. & Daniels, K. 2001. Benchmarking reference desk service in academic health science libraries: a preliminary survey. College & Research Libraries 62(4):348-353.
Stalker, J.C., & Murfin, M.E. 1996. Quality reference service: a preliminary case study. The Journal of Academic Librarianship 22(6): 423-429.
Wisconsin-Ohio Reference Evaluation Program. n.d. [Online]. Available: {http://worep.library.kent.edu/} [Accessed April 13, 2006].
Previous | Contents | Next |