Search
2008 Volume 23
Article Contents
RESEARCH ARTICLE   Open Access    

User-centered evaluation of adaptive and adaptable systems: a literature review

More Information
  • Abstract: This literature review focuses on user-centered evaluation (UCE) studies of adaptive and adaptable systems. Usability, perceived usefulness and appropriateness of adaptation are the three most commonly assessed variables. Questionnaires appeared to be the most popular method, followed by interviews and data log analysis. The quality of most questionnaires was questionable, and the reporting of interviews and think-aloud protocols was found to be shallow. Furthermore, data logs need triangulation in order to be useful. The reports encountered lack empirical value. The article models the iterative design process for adaptive and adaptable systems, linked to the goals of UCE: supporting decisions, detecting problems and verifying quality. This model summarizes the variables to be assessed at each stage of the process and the relevant methods to assess them.
  • 加载中
  • Akoumianakis D., Grammenos D. & Stephanidis C.2001. User interface adaptation: Evaluation perspectives. In User interfaces for All, Stephanidis C.(ed.). Erlbaum, 339–352.

    Google Scholar

    Alpert S. R. & Vergo J. G.2007. User-centered evaluation of personalized web sites: What’s unique? In Human Computer Interaction Research in Web Design and Evaluation, Zaphiris P. & Kurniawan S.(eds). Idea Group, 257–272.

    Google Scholar

    Ammenwerth E., Iller C. & Mansmann U.2003. Can evaluation studies benefit from triangulation? A case study. International Journal of Medical Informatics70(2/3),237–248.

    Google Scholar

    Benbunan-Fich R.2001. Using protocol analysis to evaluate the usability of a commercial web site. Information & Management39(2), 151–163.

    Google Scholar

    Benyon D. & Murray D.1993. Adaptive systems: From intelligent tutoring to autonomous agents. Knowledge-Based Systems6(4), 197–219.

    Google Scholar

    Bohnenberger T., Jameson A., Krüger A. & Butz A.2002. Location-aware shopping assistance: Evaluation of a decision-theoretic approach. Lecture Notes in Computer Science2411, 155–169.

    Google Scholar

    BSI.1998. ISO 9241-11: Ergonomic Requirements for Office Work with Visual Display Terminals. British Standards Institution.

    Google Scholar

    Buchauer A., Pohl U., Kurzel N. & Haux R.1999. Mobilizing a health professional’s Workstation: Results of an evaluation study. International Journal of Medical Informatics54, 105–114.

    Google Scholar

    Cawsey A. J., Jones R. B. & Pearson J.2000. The evaluation of a personalised health information system for patients with cancer. User Modeling and User-Adapted Interaction10, 47–72.

    Google Scholar

    Chesnais P. R., Mucklo M. J. & Sheena J. A.1995. The Fishwrap personalized news system. Paper Presented at the Second International Workshop on Community Networking ‘Integrated Multimedia Services to the Home’,Princeton, USA.

    Google Scholar

    Cheverst K., Byun H. E., Fitton D., Sas C., Kray C. & Villar N.2005. Exploring issues of user model transparency and proactive behaviour in an office environment control system. User Modeling and User-Adapted Interaction15, 235–273.

    Google Scholar

    Chin D. N.2001. Empirical evaluation of user models and user-adapted systems. User Modeling and User-Adapted Interaction11, 181–194.

    Google Scholar

    Cronbach L. J.1951. Coefficient alpha and the internal structure of tests. Psychometrika16(3),297–334.

    Google Scholar

    Davis F. D., Bagozzi R. P. & Warshaw P. R.1989. User acceptance of computer technology: A comparison of two theoretical models. Management Science35, 982–1003.

    Google Scholar

    De Jong M. D. T. & Schellens P. J.1997. Reader-focused text evaluation. An overview of goals and methods. Journal of Business and Technical Communication11(4), 402–432.

    Google Scholar

    Dicks R. S.2002. Mis-usability: On the uses and misuses of usability testing. Paper Presented at the 20th Annual International Conference on Computer Documentation, Toronto, Canada.

    Google Scholar

    Ericsson K. A. & Simon H. A.1993. Protocol Analysis. MIT Press.

    Google Scholar

    Field A., Hartel P. & Mooij W.2001. Personal DJ, an open architecture for personalised content delivery. Paper Presented at the Tenth International Conference World Wide Web, Hong Kong.

    Google Scholar

    Gates K. F., Lawhead P. B. & Wilkins D. E.1998. Toward an adaptive www: A case study in customised hypermedia. New Review of Hypermedia and Multimedia4, 89–113.

    Google Scholar

    Gena C. & Torre I.2004. The importance of adaptivity to provide onboard services: A preliminary evaluation of an adaptive tourist information service onboard vehicles. Applied Artificial Intelligence18(6),549–580.

    Google Scholar

    Gena C.2005. Methods and techniques for the evaluation of user-adaptive systems. Knowledge Engineering Review20(1), 1–37.

    Google Scholar

    Gena C. & Weibelzahl S.2007. Usability engineering for the adaptive web. In The Adaptive Web, Brusilovsky P.,Kobsa A. & Nejdl W. (eds). Springer-Verlag, 720–762.

    Google Scholar

    Goren-Bar D., Graziola I., Kuflik T., Pianesi F., Rocchi C., Stock O. & Zancanaro M.2005. I like it: An affective interface for a multimodal museum guide. Retrieved on 20 April 2006 from http://peach.itc.it/papers/gorenbar2005.pdf

    Google Scholar

    Gregor P., Dickinson A., Macaffer A. & Andreasen P.2003. Seeword: A personal word processing environment for dyslexic computer users. British Journal of Educational Technology34(3),341–355.

    Google Scholar

    Hartson H. R., Andre T. S. & Williges R. C.2003. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction15(1), 145–181.

    Google Scholar

    Henderson R., Rickwood D. & Roberts P.1998. The beta test of an electronic supermarket. Interacting with Computers10, 385–399.

    Google Scholar

    Herder E.2006. Forward, Back and Home Again. Analyzing User Behaviour on the Web. University of Twente.

    Google Scholar

    Hertzum M. & Jacobsen N. E.2003. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction15(1), 183–204.

    Google Scholar

    Höök K.1997. Evaluating the utility and usability of an adaptive hypermedia system. Paper Presented at IUI '97, Orlando, USA.

    Google Scholar

    Höök K.2000. Steps to take before intelligent user interfaces become real. Interacting with Computers12(4), 409–426.

    Google Scholar

    Hvannberg E. T., Law E. L., Lárusdóttir M. K.2007. Heuristic evaluation: Comparing ways of finding and reporting usability problems. Interacting with Computers19, 225–240.

    Google Scholar

    Hyldegaard J. & Seiden P.2004. My E-journal: Exploring the usefulness of personalized access to scholarly articles and services. Information Research9(3), paper 181. Available at http://InformationR.net/ir/9–3/paper181.html

    Google Scholar

    ISO.1999. ISO 13407: Human-Centered Design Processes for Interactive Systems, International Standard Organizations.

    Google Scholar

    Jameson A.2003. Adaptive interfaces and agents. In Human-Computer Interaction Handbook, Jacko J. A. & Sears A. (eds). Erlbaum, 305–330.

    Google Scholar

    Jameson A.2006. Adaptive interfaces and agents. In Human-Computer Interaction Handbook (2nd edn), Jacko J. A. & Sears A. (eds). Erlbaum, 433–458.

    Google Scholar

    Jensen A. L., Boll P. S., Thysen I. & Pathak B. K.2000. Pl@nteInfo®: A web-based system for personalised decision support in crop management. Computers and Electronics in Agriculture25, 271–293.

    Google Scholar

    Kaasinen E.2003. User needs for location aware mobile services. Personal Ubiquitous Computing7(1), 70–79.

    Google Scholar

    Karat C. M., Brodie C., Karat J., Vergo J. & Alpert S. R.2003. Personalizing the user experience on ibm.Com. IBM Systems Journal42(4), 686–701.

    Google Scholar

    Ketamo H.2003. Xtask: An adaptable learning environment. Journal of Computer Assisted Learning19(3), 360–370.

    Google Scholar

    Kitchenham B. A., Pfleeger S. L., Pickard L. M., Jones P. W., Hoaglin D. C., El Emam K. & Rosenberg J.2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering28(8), 721–734.

    Google Scholar

    Kjeldskov J., Graham C., Pedell S., Vetere F., Howard S., Balbo S. & Davies J.2005. Evaluating the usability of a mobile guide: The influence of location, participants and resources. Behaviour & Information Technology24(1), 51–65.

    Google Scholar

    Kolari J., Laakko T., Hiltunen T., Ikonen V., Kulju M., Suihkonen R., Toivonen S., & Virtanen T.2004. Context-Aware Services for Mobile Users (Technology and User Experiences), VTT publications 539.

    Google Scholar

    Kramer J., Noronha S. & Vergo J.2000. A user-centered design approach to personalization. Communications of the ACM43(8), 45–48.

    Google Scholar

    Lentz L. & De Jong M. D. T.1997. The evaluation of text quality: Expert-focused and reader-focused methods compared. IEEE Transactions on Professional Communication40(3), 224–234.

    Google Scholar

    Magoulas G. D., Chen S. Y. & Papanikolaou K. A.2003. Integrating layered and heuristic evaluation for adaptive learning environments. Paper Presented at the Second Workshop on Empirical Evaluation of Adaptive Systems Held in Conjunction with User Modelling 2003, Pittsburg, USA.

    Google Scholar

    Maguire M.2001. Methods to support human-centred design. International Journal of Human-Computer Studies55(4), 587–634.

    Google Scholar

    Miles M. B. & Huberman A. M.1994. Qualitative Data Analysis (2nd edn), Sage.

    Google Scholar

    Morgan D. L.1996. Focus groups. Annual Review of Sociology, 22, 129–152.

    Google Scholar

    Muntean C. H. & McManis J.2006. The value of QoE-based adaptation approach in educational hypermedia: Empirical evaluation. Lecture Notes in Computer Science4018, 121–130.

    Google Scholar

    Nahl D.1998. Ethnography of novices’ first use of web search engines: Affective control in cognitive processing. Internet Reference Services Quarterly3(2), 51–72.

    Google Scholar

    NHS Centre for Reviews and Dissemination. 2001. Undertaking Systematic Reviews of Research on Effectiveness, University of York.

    Google Scholar

    Pateli A. G., Giaglis G. M. & Spinellis D. D.2005. Trial evaluation of wireless info-communication and indoor location-based services in exhibition shows. Advances in Informatics3746, 199–210.

    Google Scholar

    Patton M. Q.2002. Qualitative Research & Evaluation Methods, (3rd edn), Sage.

    Google Scholar

    Savage P.1996. User interface evaluation in an iterative design process: A comparison of three techniques. Paper presented at CHI'96, Vancouver, Canada.

    Google Scholar

    Schmidt-Belz B. & Poslad S.2003. User validation of a mobile tourism service. Paper Presented at the Workshop on HCI in Mobile Guides, Udine, Italy.

    Google Scholar

    Smith H., Fitzpatrick G. & Rogers Y.2004. Eliciting reactive and reflective feedback for a social communication tool: A multi-session approach. Paper Presented at Designing Interactive Systems: Across the Spectrum, Cambridge, USA.

    Google Scholar

    Sodergard C., Aaltonen M., Hagman S., Hiirsalmi M., Jarvinen T., Kaasinen E., Kinnunen T., Kolari J., Kunnas J. & Tammela A.1999. Integrated multimedia publishing: Combining TV and newspaper content on personal channels. Computer Networks: the International Journal of Computer and Telecommunications Networking31, 1111–1128.

    Google Scholar

    Spector P. E.1992. Summated Rating Scale Construction, Sage.

    Google Scholar

    Stary C. & Totter A.2003. Measuring the adaptability of universal accessible systems. Behaviour & Information Technology22(2), 101–116.

    Google Scholar

    Stein A.1997. Usability and assessments of multimodal interaction in the SPEAK! System: An experimental case study. The New Review of Hypermedia and Multimedia3, 159–180.

    Google Scholar

    Su L. T.1992. Evaluation measures for interactive information retrieval. Information Processing and Management28(4), 503–516.

    Google Scholar

    Van den Haak M. J., De Jong M. D. T. & Schellens P. J.2003. Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology22(5), 339–351.

    Google Scholar

    Van den Haak M. J., De Jong M. D. T. & Schellens P. J.2004. Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison. Interacting with Computers16, 1153–1170.

    Google Scholar

    Van der Geest T.2004. Beyond accessibility: Comparing three website usability test methods for people with impairments. Paper Presented at HCI. 2004: Design for Life, Leeds, England.

    Google Scholar

    Van Velsen L., Van der Geest T. & Klaassen R.2007. Testing the usability of a personalized system: Comparing the use of interviews, questionnaires and thinking-aloud. Paper Presented at the IEEE Professional Communication Conference, Seattle, USA.

    Google Scholar

    Venkatesh V., Morris M. G., Davis G. B. & Davis F. D.2003. User acceptance of information technology: Towards a unified view. MIS Quarterly27(3), 425–478.

    Google Scholar

    Virzi R. A., Sokolov J. L. & Karis D.1996. Usability problem identification using both low- and high-fidelity prototypes. Paper Presented at the SIGHI Conference on Human Factors in Computing Systems: Common Ground, Vancouver, Canada.

    Google Scholar

    Vredenburg K., Mao J., Smith P. W. & Carey T.2002. A survey of user-centered design practice. CHI Letters4(1), 471–478.

    Google Scholar

    Walker M., Takayama L. & Landay J. A.2002. High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. Paper Presented at the 46th Annual Meeting of the Human Factors and Ergonomics Society, Baltimore, USA.

    Google Scholar

    Weibelzahl S., Lippitsch S. & Weber G.2002. Advantages, opportunities, and limits of empirical evaluations: Evaluating adaptive systems. Künstliche Intelligenz3(2), 17–20.

    Google Scholar

    Weibelzahl S.2003. Evaluation of Adaptive Systems. PhD thesis, University of Trier.

    Google Scholar

    Weibelzahl S.2005. Problems and pitfalls in the evaluation of adaptive systems. In Adaptable and Adaptive Hypermedia Systems, Chen S. Y. & Magoulas G. D. (eds). IRM Press, 285–299.

    Google Scholar

    Weibelzahl S., Jedlitschka A. & Ayari B.2006. Eliciting requirements for a adaptive decision support system through structured user interviews. Paper Presented at the User Centered Design and Evaluation Workshop Held in Conjunction with Adaptive Hypermedia '06, Dublin, Ireland.

    Google Scholar

  • Cite this article

    LEX VAN VELSEN, THEA VAN DER GEEST, ROB KLAASSEN, MICHAËL STEEHOUDER. 2008. User-centered evaluation of adaptive and adaptable systems: a literature review. The Knowledge Engineering Review. 23: doi: 10.1017/S0269888908001379
    LEX VAN VELSEN, THEA VAN DER GEEST, ROB KLAASSEN, MICHAËL STEEHOUDER. 2008. User-centered evaluation of adaptive and adaptable systems: a literature review. The Knowledge Engineering Review. 23: doi: 10.1017/S0269888908001379

Article Metrics

Article views(29) PDF downloads(237)

RESEARCH ARTICLE   Open Access    

User-centered evaluation of adaptive and adaptable systems: a literature review

The Knowledge Engineering Review  23 Article number: 10.1017/S0269888908001379  (2008)  |  Cite this article

Abstract: Abstract: This literature review focuses on user-centered evaluation (UCE) studies of adaptive and adaptable systems. Usability, perceived usefulness and appropriateness of adaptation are the three most commonly assessed variables. Questionnaires appeared to be the most popular method, followed by interviews and data log analysis. The quality of most questionnaires was questionable, and the reporting of interviews and think-aloud protocols was found to be shallow. Furthermore, data logs need triangulation in order to be useful. The reports encountered lack empirical value. The article models the iterative design process for adaptive and adaptable systems, linked to the goals of UCE: supporting decisions, detecting problems and verifying quality. This model summarizes the variables to be assessed at each stage of the process and the relevant methods to assess them.

    • Copyright © Cambridge University Press 20082008Cambridge University Press
References (73)
  • About this article
    Cite this article
    LEX VAN VELSEN, THEA VAN DER GEEST, ROB KLAASSEN, MICHAËL STEEHOUDER. 2008. User-centered evaluation of adaptive and adaptable systems: a literature review. The Knowledge Engineering Review. 23: doi: 10.1017/S0269888908001379
    LEX VAN VELSEN, THEA VAN DER GEEST, ROB KLAASSEN, MICHAËL STEEHOUDER. 2008. User-centered evaluation of adaptive and adaptable systems: a literature review. The Knowledge Engineering Review. 23: doi: 10.1017/S0269888908001379
  • Catalog

      /

      DownLoad:  Full-Size Img  PowerPoint
      Return
      Return