BrendleFormativeEvaluation

Instructional Design Unit

Formative Evaluation This project’s objective was to teach eleventh grade students how to evaluate a web site for academic use. When given a research assignment, many students go directly to Google and use the first hits, assuming they are the best. I identified that the students’ need is to learn a process for evaluating a web site. Thus, the evaluation was based on if a student could independently determine if a web site is acceptable for academic (high school) research. I originally planned to get three students from my study hall to do the try-outs. I wanted to keep this assignment out of my classroom in order to maintain objectivity. However, I got no takers. The kids were too busy or disinterested. So I conducted the try-outs during my third period class while we were in the lab working on an essay. I solicited them and had seven kids who conceded to work through all steps of the process (without extra credit or candy despite their pleas). I could not give them extra credit as this would go against our school policies. I also did not want to make it a whole class assignment as my sample would have been too large. The setting was a high school computer lab as the students needed to conduct the process online. It was quiet and comfortable. The students who did not participate worked quietly on an assignment and did not disturb the others. I was able to mingle and observe student reactions to the process. While this was not the preferred one-on-one where the student talked through the process, I was able to gather much observational feedback and later asked for verbal feedback from the group. My seven learners were typical of the students described in my Learner Analysis. As my class is heavy on boys, I got five boys and two girls. They are typical eleventh grade English students in a regular education class. This group was made of B-C students who are college bound. They do what they need to do to get the work done but are not interested in perfection as an honors student may be. They are all very social and verbal as I mentioned in the analysis. Four have fairly short attention spans; one has documented ADHD. One has a 504 plan for a documented learning disability in reading. As a sample, I think they represent my three classes fairly well. The try-out was conducted over two days. The prerequisite test and pre-test were given on May 5 and the lesson (parts 1 and 2) and the final assessment were given on May 6. I wasn’t sure exactly how long the kids would need and I did not want to squeeze everything into one 50 minute period. I administered all parts of the instruction myself although it was created with the intention of having someone else do this. I tried not to intervene. Day 1 The prerequisite assessment consisted of 1. a matching vocabulary section with 30 words and a word bank, and 2. a hands-on computer skills assessment that had six steps. The scores were a lot lower on this than I had predicted. Shared terminology is a very important factor to communication, and this assessment reiterated the need to clarify terms for students. We had a chance to talk over the terms before the lesson. 1. Authorial source: I want the kids to see that many types of sources exist beyond a human source (author). The concept of “source” did not resonate with them. No kids matched this term accurately, so I knew I had to clarify it prior to the lesson. A discussion of how the authorial source may or may not be a person or publisher ensued. 2. Citation: Even though they have been taught in-text citations since 9th grade, it was painfully clear that they didn’t understand this concept as only one student matched the term. 3. Documented expertise: This concept was confused with “credibility.” The students weren’t sophisticated enough to see that documented expertise results in credibility. 4. Field-specific terminology: Only two matched this concept. Students need to be aware of how the usage of field-specific terminology helps one to determine a target audience. 5. Subjectivity: This was confused with “bias”; some discussion ensued about the fine line between them. 6. Works Cited vs. Bibliography: These were the most mismatched terms although the definition for WC had the word “Work” in it. The computer skills evaluation was easy for all the kids, so I knew they could find their way around a web site easily. This assessment was a series of questions about HOW to evaluate a web site. I asked specific questions as to what to look for and then to explain one’s answer. Four of the seven students got 85% or above, so overall, the group knew the material. The students could articulate what they needed to do; the one weakness was explaining their answers. Some left the explanation section blank and lost partial credit. My hunch is that this was not completed due to lack of motivation rather than lack of knowledge. Based on my observations, the students who left gaps were rushing through or didn’t read the whole question. This pre-unit assessment showed me that most students had a decent sense of what they should do to evaluate a web site. They would need to apply these skills in the lesson. Day 2 ** The Lesson ** (The lesson itself was broken into two parts.) TUTORIAL The students were given a paper tutorial that moved through the entire evaluation process. It included screen-saved web pages to model the concepts. The students were instructed to read the tutorial carefully, taking notes as they read. I originally planned to make it interactive, where they had to record answers as they went along. However, as I was writing it, I changed to straight expository mode so that I could explain the steps and the web pages instead of having them answer questions about them. It seemed more directive and instructive this way. The tutorial ended up being a 16 page document due to the large images. The text was squeezed in between images. Overall, the feedback was positive; the students later told me that the tutorial was easy to follow and made sense. It defined unfamiliar words. The one main comment was the format and length were unfriendly; I should have made it into a booklet or even a PowerPoint presentation. WEB SITE EVALUATION ON YOUR OWN and SHARING The students then had to find and evaluate a random web site using the web site evaluation form. After everyone was done, the students told their peers about their web sites and why they were or were not credible. The students did a great job using the terminology and process (three parts: source, timeliness, content). One student misunderstood the concept of academic sites; she analyzed the school’s web site! While it is timely, well-authored, and accurate, it is not something one would use for academic research. I realized that this concept should have been stressed more from the beginning. As there were only seven students, this process was quick. They really enjoyed talking about their chosen web sites. This was an excellent activity for my group; they are social and chatty, so this gave them an outlet after sitting and reading the tutorial. The lesson itself was broken into three parts of about 10 minutes each, so the pacing and variety suited my learners as described in the Learner Analysis. The post-unit test involved evaluating a web site I gave them. I wanted to cut out any variables and see if they could follow the process independently and accurately with a web site I knew well. The results were favorable: all students scored above 85% and five scored in the 90’s. The answers were complete and accurate. The only weakness was locating the author’s documented expertise as it was on the site’s Home link. The site I chose was a section of a larger site, and some of the kids did not link to the Home page to see his credentials, which were numerous. Many of the students jumped to the author’s “bibliography and links” section to find information about the author. They did not understand that this section would tell them about the sources he used, not about him. The confusion over sources, bibliographies, work cited, etc., showed itself again, so these concepts need to be taught more. Overall, I was pleased with the growth. The students commented that the assessment was easy to follow and helped them go through the process of evaluating the site.
 * Introduction **
 * Methods **
 * Results and Discussion **
 * Student ** ||
 * PreReq Vocab **
 * Score (30) **
 * PreReq **
 * Comp **
 * Skills **
 * (6) **
 * PreTest **
 * Score **
 * (16) **
 * Own **
 * Web Site Eval **
 * (formative) ** ||
 * Post Test **
 * Score **
 * (34) **
 * Target Skill Met? 85% + **
 * Y =Yes **
 * No=No ** ||
 * 1 ** ||
 * 15, 50% ** ||
 * 6, **
 * 100% ** ||
 * 12, **
 * 75% ** ||
 * Ok ** ||
 * 29, **
 * 85% ** ||
 * Y ** ||
 * 2 ** ||
 * 25, **
 * 83% ** ||
 * 6, **
 * 100% ** ||
 * 16, **
 * 100% ** ||
 * Ok ** ||
 * 32, **
 * 94% ** ||
 * Y ** ||
 * 3 ** ||
 * 20, **
 * 67% ** ||
 * 6, **
 * 100% ** ||
 * 11, **
 * 69% ** ||
 * Ok ** ||
 * 33, **
 * 97% ** ||
 * Y ** ||
 * 4 ** ||
 * 15, **
 * 50% ** ||
 * 5, **
 * 83% ** ||
 * 10, **
 * 62% ** ||
 * Bad web site choice ** ||
 * 26, **
 * 76% ** ||
 * Y ** ||
 * 5 ** ||
 * 16, **
 * 53% ** ||
 * 5, **
 * 83% ** ||
 * 15, **
 * 94% ** ||
 * Ok ** ||
 * 33, **
 * 97% ** ||
 * Y ** ||
 * 6 ** ||
 * 24, **
 * 80% ** ||
 * 6, **
 * 100% ** ||
 * 15, **
 * 94% ** ||
 * Ok ** ||
 * 33, **
 * 97% ** ||
 * Y ** ||
 * 7 ** ||
 * 28, **
 * 93% ** ||
 * 6, **
 * 100% ** ||
 * 15, **
 * 94% ** ||
 * Ok ** ||
 * 29, **
 * 85% ** ||
 * Y ** ||
 * 7 ** ||
 * 28, **
 * 93% ** ||
 * 6, **
 * 100% ** ||
 * 15, **
 * 94% ** ||
 * Ok ** ||
 * 29, **
 * 85% ** ||
 * Y ** ||
 * Y ** ||
 * __ Prerequisite Assessment  __**
 * Some problem terms and concepts **
 * __ Pre-unit Assessment  __**
 * Problem areas **
 * 1) Several students indicated that you find an author at the bottom of the page. They did not mention the byline, the “About the Author” or the links. This showed me that students expect to see the author on the exact page they are using and may not know to check for the home page of the entire site. If they Google a term and pull up a page from a larger site, they may not be aware of the larger site. This is a problem of awareness of how the Web works. They need to see a page of a site as a page in a book; you can usually find the source in the book somewhere although it may not be on the page you are using.
 * 1) The biggest confusion is around the issue of when a site is outdated. How old is too old? Student answers ranged from six months to “it doesn’t matter.” This issue definitely needs clarification.
 * Post-unit Assessment **