Evaluating Digital Libraries: A User-Friendly Guide

Arms, W. (2000). Digital libraries. Cambridge, MA: MIT Press. Available at: http://www.cs.cornell.edu/wya/DigLib/index.html.

Bias, R. G. (1994). The pluralistic usability walkthrough: Coordinated empathies. In J. Nielsen & R. L. Mack (Eds.), Usability inspection methods (pp. 63-76). New York: John Wiley & Sons.

Bollen, J., & Luce, R. (2002). Evaluation of digital library impact and user communities by analysis of usage patterns, D-Lib Magazine, 8(6). Available at: (http://www.dlib.org/dlib/june02/bollen/06bollen.html).

Bollen, J., Luce, R., Vemulapalli, S. S., & Xu, W. (2003). Usage analysis for the identifi-cation of research trends in digital libraries, D-Lib Magazine, 9(5). Available at: (http://www.dlib.org/dlib/may03/bollen/05bollen.html).

Borgman, C. L. (Ed.) (1990). Scholarly communication and bibliometrics. Newbury Park, CA: Sage Publications.

Budhu, M., & Coleman, A. (2002) The design and evaluation of interactivities in a digi-tal library. D-Lib Magazine, 8(11). Available at: http://www.dlib.org/dlib/november02/coleman/11coleman.html.

Bush, V. (1945) As we may think. Atlantic Monthly, 176, 101-108. Available at: http://www.theatlantic.com/unbound/flashbks/computer/bushf.htm.

Campbell, N. (2001). Usability assessment of library-related web sites: Methods and case studies (Guide # 7). Chicago: American Library Association.

Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for re-search. Chicago: Rand-McNally.

Casti, J. L. (1989). Paradigms lost: Images of man in the mirror of science. New York: William Morrow.

Chen, H. (1990). Theory-driven evaluations. Newbury park, CA: Sage.

Cherry, J. M., & Duff, W. M. (2002). Studying digital library users over time: A follow-up survey of Early Canadiana Online. Information Research, 7(2). Available at: http://InformationR.net/ir/paper123.html.

Choudhury, S., Hobbs, B., Lorie, M., & Flores, N. (2002). A framework for evaluating digital library services. D-Lib Magazine, 8(7/8). Available at: (http://www.dlib.org/dlib/july02/choudhury/07choudhury.html).

Clark, R. E. (Ed.). (2001). Learning from media: Arguments, analysis, and evidence. Greenwich, CT: Information Age Publishing.

Cook, T. D., & Gruder, C. L. (1978). Metaevaluation research. Evaluation Quarterly, 2(1), 5-51.

Covey, D. T. (2002). Usage and usability assessment: Library practices and concerns. Washing-ton, DC: Council on Library and Information Resources.

Dilevko, J., & Dolan, E. (1999). Government documents reference service in Canada: Implications for electronic access. Available at:

(http://dsp-psd.pwgsc.gc.ca/Rapports/Dilevko_Dolan/dilevko-e.html).

Durrance, J. C., & Fisher, K. E. (2003). Determining how libraries and librarians help. Library Trends, 51(4), 305-334. Available at:

(http://www.si.umich.edu/libhelp/DURRAFISH.pdf).

Fink, A. (2002). The survey kit. Thousand Oaks, CA: Sage Publications.

Fink, A. (1995). The survey handbook. Thousand Oaks, CA: Sage Publications.

Fitz-Gibbon, C. T., & Morris, L. L. (1987). How to design a program evaluation. Newbury Park, CA: Sage.

Hackos J. T., & Redish J. C. (1998). User and task analysis for interface design. New York: John Wiley & Sons.

Harter, S., & Hert, C. (1997). Evaluation of information retrieval systems: Approaches, issues, and methods. In M. Williams (Ed.), Annual review of information science and technology (pp. 3-94), Medford, NY: Information Today Inc.

Heath, F., Kyrillidou, M., Webster, D., Choudhury, S., Hobbs, B., Lorie, M., & Flores, N. (2003). Emerging tools for evaluating digital library services: Conceptual adapta-tions of LibQUAL+ and CAPM. Journal of Digital Information, 4(2). Available at: (http://jodi.ecs.soton.ac.uk/Articles/v04/i02/Heath/).

Hernon, P. (2002). Quality: New directions in the research. The Journal of Academic Li-brarianship, 28(4), 224-231.

Hovater, M., Krot, M., Kiskis, D. L., Holland, I., & Altman., M. (2002). Usability testing of the Virtual Data Center, Usability Workshop at Second ACM+IEEE Joint Con-ference on Digital Libraries. Available at: http://www.uclic.ucl.ac.uk/annb/DLUsability/Hovater7.pdf.

Jones, S., Cunningham, S. J., McNab, R., & Boddie, S. (2000). A transaction log analy-sis of a digital library. International Journal on Digital Libraries, 3, 152-169.

Kahle, B., Prelinger, R., & Jackson, M. E. (2001). Public access to digital material. D-Lib Magazine, 7(10). Available at: http://www.dlib.org/dlib/october01/kahle/10kahle.html.

Kahn, M. J., & Prail, A. (1994). Formal usability inspections. In J. Nielson and R. L. Mack (Eds.), Usability inspection methods (pp. 141-171). New York: John Wiley & Sons.

Keith, S., Blandford, A., Fields, B., & Theng, Y. L. (2002). An investigation into the application of Claims Analysis to evaluate usability of a digital library interface. Technical report available at:: http://www.cs.mdx.ac.uk/ridl/UET/.

Krueger, R. A., & Casey, M. A. (2000). Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage Publications.

Kyrillidou, M. (2002). From input and output measures to quality and outcome meas-ures, or, from the user in the life of the library to the library in the life of the user. Journal of Academic Librarianship, 28(1/2), 42-46.

Lagemann, E. C. (2000). An elusive science: The troubling history of educational research. Chicago: The University of Chicago Press.

Maeda, A. (2002, September). Multilingual information processing for digital libraries. In Proceedings of the PNC Annual Conference and Joint Meetings 2002, pp. 80-81 (ab-stract), Osaka, Japan. Available at: http://pnclink.org/annual/annual2002/pdf/0921/12/c211206-2.pdf.

Marchionini, G. (2000). Evaluating digital libraries: A longitudinal and multifaceted view. Library Trends, 49(2), 304-333. Preprint available online at: http://ils.unc.edu/~march/perseus/lib-trends-final.pdf.

Marchionini, G., Plaisant, C., & Komlodi, A.(in press) The people in digital libraries: Multifaceted approaches to assessing needs and impact. A. Bishop, B. Buttenfield, &

N. Van House (Eds.), Digital library use: Social practice in design and evaluation. MIT Press. Draft available at: (http://ils.unc.edu/~march/revision.pdf).

Mark, M. M., & Shotland, R. L. (Eds.). (1987). Multiple methods in program evaluation. San Francisco: Jossey-Bass.

Mead, J. P., & Gay, G. (1995). Concept mapping: An innovative approach to digital library design and evaluation. ACM SIGOIS Bulletin, 16(2), 10-14.

Melucci, M. (1999). An evaluation of automatically constructed hypertexts for informa-tion retrieval. Journal of Information Retrieval, 1(1-2), 91-114.

Miles, M. B., &Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage Publications.

Nielsen, J. (2001). First rule of usability? Don’t listen to users. Available at: http://www.useit.com/alertbox/20010805.html.

Nielsen, J. (2000). Designing Web usability. Indianapolis, IN: New Riders.

Nielsen, J. (1993). Usability engineering. San Francisco: Morgan Kaufmann.

Norlin, E. (2000). Reference evaluation: A three-step approach-surveys, unobtru-sive observations, and focus groups. College & Research Libraries, 61(6), 546-553.

Norlin, E., & Winters, C. (2002) Usability testing for library web sites: A hands-on guide. Chi-cago: American Library Association.

Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

Patton, M. Q. (1997). Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage.

Peters, T.A. (1993). The history and development of transaction log analysis. Library Hi Tech, 11(2), 41-66.

Peterson, E., & York, V. (2003). User evaluation of the Montana Natural Resource Information System (NRIS): In-depth evaluation of digital collections using snow-ball sampling and interviews, D-Lib Magazine, 9(7/8). Available at: http://www.dlib.org/dlib/july03/peterson/07peterson.html.

Pettigrew, K. E., & Durrance, J. C. (2001). Public use of digital community information systems: Findings from a recent study with implications for system design. International Conference on Digital Libraries Proceedings of the first ACM/IEEE-CS joint conference on Digital Librar-ies, 136-143.

Reeves, T. C., & Hedberg, J. G. (2003). Interactive learning systems evaluation. Engle-wood Cliffs, NJ: Educational Technology Publications.

Reid, J. (2000). A task-oriented non-interactive evaluation methodology for informa-tion retrieval systems. Journal of Information Retrieval, 2(1), 115-129.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2003). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.

Salamapsis, M., & Diamantaras, K. (2002). Experimental user-centered evaluation of an open hypermedia system and web information-seeking environments. Journal of Digital Information, 2(4). Available at: http://jodi.ecs.soton.ac.uk/Articles/v02/i04/Salampasis/.

Saracevic, T., & Dalbello, M. (2001). A survey of digital library education. Proceedings of the American Society for Information Science and Technology, Vol. 38 (pp. 209-223). Silver Spring, MD: American Society for Information Science and Technology. Available at: http://www.scils.rutgers.edu/~tefko/ProcASIST2001.doc.

Saracevic, T. (2000). Digital library evaluation: Toward evolution of concepts. Library Trends, 49(2), 350-369.

Saracevic, T. (1995). Evaluation of evaluation in information retrieval. Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Informa-tion Retrieval (pp. 138-148). New York: Association for Computing Machinery.

Schutt, R. K. (2003). Investigating the social world: The process and practice of research (3rd ed.) Thousand Oaks, CA: Pine Forge Press.

Schwab, J. J. (1970). The practical: A language for curriculum. Washington, DC: National Education Association, Center for the Study of Instruction.

Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage Publications.

Shavelson, R. J., Towne, L., & the Committee on Scientific Principles for Education Research. (Eds.). (2002). Scientific research in education. Washington, DC: National Academy Press. Available at: http://www.nap.edu/books/0309082919/html/R1.html.

Shneiderman, B. (1987). Designing the user interface: Strategies for effective human-computer inter-action. Reading, MA: Addison-Wesley.

Shulman, L. (2000). Inventing the future. In P. Hutchings (Ed). Opening lines: Approaches to the scholarship of teaching and learning. Menlo Park, CA: Carnegie Publications. Avail-able at: (http://www.carnegiefoundation.org/elibrary/docs/inventing.htm).

Spradley, J. P. (1997). Participant observation. New York: Holt Rinehart & Winston

Suchman, E. A. (1967). Evaluative research. New York: Russell Sage.

Tate, R. (1990). Experimental design. In H. J. Walberg & G. D. Haertel (Eds.), The in-ternational encyclopedia of educational evaluation (pp. 553-561). New York: Pergamon Press.

Tenopir, C. (2003). Use and users of electronic library resources: An overview and analysis of recent research studies. Washington, DC: Council on Library and Information Resources. Available at: http://www.clir.org/pubs/reports/pub120/pub120.pdf.

Wallace, D. P., & Van Fleet, C. (2000). (Eds.). Library evaluation: A casebook and can-do guide. Englewood, CO: Libraries Unlimited.

Wildemuth, B. M., Marchionini, G., Yang, M., Geisler, G., Wilkens, T., Hughes, A. & Gruss, R. (2003, June) How fast is too fast? Evaluating fast forward surrogates for digital video. Paper presented at the 2003 Joint Conference on Digital Libraries (JCDL ’03). Available at: (http://www.jcdl.org/awards.shtml).

Wragg, E. C. (1999). An introduction to classroom observation (2nd ed.). London: Routledge.

Wu, M., & Sonnenwald, D.H. (1999). Reflections on information retrieval evaluation. Proceedings of the 1999 EBTI, ECAI, SEER & PNC Joint Meeting. Academia Sinica, Taipei, Taiwan.

Zhang, Z., Basili, V., & Shneiderman, B. (1999). Perspective-based usability inspection: An empirical validation of efficacy. Empirical Software Engineering, 4(1), 43-69.