April 29, 2003
Learner Profile | Assessment Criteria | Procedures
Results & Discussion | Revisions Completed
Overview This is the formative evaluation report of the tutorial Using PsycINFO to Find Articles. The formative evaluation was conducted April 24-28, 2003 on five UGA undergraduate potential users.
|
Learner
|
Gender
|
Age/Race
|
Year
in school
|
Used PsycINFO before | Use Internet to find information |
| Learner #1 | Female | 20 white | Junior | Never | Frequently |
| Learner #2 | Female | 21 white | Sophomore | Never | Frequently |
| Learner #3 | Female | 22 white | Senior | Never | Frequently |
| Learner #4 | Male | 19 white | Freshman | Never | Frequently |
| Learner #5 | Male | 21 white | Senior | Occasionally | Frequently |
All learners are current UGA students who have taken or will be shortly taking college Psychology classes. Unfortunately no minority undergraduates were available.
The usability of the tutorial was determined by observation of the five learners while they interacted with and completed the tutorial and self quiz. The quiz was based on the lesson objectives and main points of the tutorial. Further evaluation was provided by a post evaluation interview consisting of the following questions. (Special thanks to Tiffany Hsu for allowing me to use and modify her evaluation questions).
1. Please rate
this experience using the tutorial:
Frustrating Pleasant
1 2 3
4 5
Please Explain...
2. How difficult
was the tutorial?
____Too easy ____About right _____ Too difficult
Please Explain...
3. How was the
length of the tutorial?
____Too short ____About right _____ Too long
Please Explain...
4. How was the
vocabulary in the tutorial?
____Too easy ____About right _____ Too difficult
Please Explain...
5. How much help
was provided in the tutorial?
____ Not enough ____About right _____ Too much
Please Explain...
6. What did you
think about the graphics?
____Too few ____About right _____ Too many
____ Too simple ____ Fit instruction _____ Too complicated
Please Explain...
7. What parts of the tutorial did you have difficulty understanding?
8. Was the organization
of information on the tutorial clear?
Strongly Disagree Strongly Agree
1 2 3
4 5
Please explain…
9. What did you like most about the tutorial?
10 What did you like least about the tutorial?
11. How would you improve this tutorial?
12. Would you recommend
this tutorial to your friends?
____No ____Maybe _____ Yes
Please Explain...
The evaluations were conducted on a computer in the Science Library with Internet connection to the tutorial. Each learner was introduced to the basic concept of Research Central and told the goal was to test the tutorial and not them. The learners completed a learner profile questionnaire than went through the entire tutorial while "thinking out loud" about what was useful and what was difficult. As a part of the tutorial the learners also did the three live searches and the interactive quiz at the end. After completing the tutorial the learners were asked the post evaluation interview questions above.
I observed that
each learner quickly saw the side menu options and green next button at the
bottom of the home page. Since they were going through the entire tutorial most
used the bottom next button to navigate, but when a question arose they would
quickly get to a section using the side menu bar. Two of the five students did
not see the "Next" text link from a live search section to the next
section.
Most students used the feedback provided in the live search sections and were
able to modify search strategies effectively. Several commented positively on
the relevancy and usefulness of the content and its arrangement. The learners
appeared to enjoy themselves while using the tutorial and all completed the
tutorial in 25-35 minutes. Four learners got all quiz questions correct and
one learner got 6 of 7 correct.
In the post evaluation questionnaire the learners responded to all questions with a rating of 4 or 5 (on a 1-5 scale) or rating of "About right" with the exception of:
|
Problem
|
Revision
Completed
|
| Learners felt that the advanced search graphic was too complicated with too many circles. | Circles were replaced with underlines and arrows. |
| Learners had problems seeing the "Next" text link in the live search sections. | A small navigational next and back button were added to the live search sections. |
| Learners felt that the "Finding too much" page was too long. | The section was broken in two with a separate"Advanced Search" page created. |
| Learner did not see the links to get feedback on live search. | "Check yourself" in red text was added beside the feedback links. |
| Several offered minor changes to text for clarification. | Changes to text were made. |
The above changes were made after having four learners evaluate the product. The improved product was then tested on the fifth learner to make sure the changes solved the problems. The changes did solve the problems identified.
Learner Profile | Assessment Criteria | Procedures
Results & Discussion | Revisions Completed
Last updated 04/30/2003