As an educational institution, our purpose is to help students learn. We do this by creating educational experiences within which students learn the information, skills and attitudes we believe are important. Students may learn in formal (e.g., classroom) and informal (e.g., residence halls) spaces. Unfortunately, learning is invisible—hidden within the students’ minds. So how do we know if students learned? We ask students to exhibit observable behaviors we think reasonably approximate the learning that has happened in their minds. This evidence—this proxy for learning—is the basis of assessment.  For maximum effectiveness, we seek evidence that approximates the learning as closely as possible.

Assessment is a fundamental part of the learning process. It occurs when we compare our actual outcome against the outcome we had hoped to accomplish. Discrepancies give us opportunities to change. Formal assessments include classroom exams, a concert, and a competitive sporting events. Informal assessments include seeing student’s puzzled faces during lecture, recognizing off-key notes during practice, and recording steps on our pedometer. When shifting from personal to organization level assessment, the basic principles remain the constant and the goals change to reflect the different context.

Assessment Process[1]

 

When we reflect on our initiatives, we may ask “Was it successful?” We can answer that question by gathering evidence in many different ways, depending on the context.

Using a modified Assessment Loop Model (Figure 2), the choices we make in each step influence our approach in the next step. We start with our end goal in mind. Then we select the most appropriate ways to collect and interpret evidence to help us accomplish our goal.

Figure 2: Modified Assessment Loop Model

 

Step 1: Set goals, identify issues, and ask questions

Choose a goal based on a decision you need to make. Consider the following questions as you choose a target: What is important to me? What am I trying to achieve? What problems or pain points am I trying to resolve? What am I hoping would happen due to my intervention? Did students achieve the desired learning outcomes?

Write your goal or issue as a question which you will answer in the following steps.

• Example 1.1: Did students achieve mastery of the X technique?
• Example 1.2: Are students socially responsible?

Step 2: Gather evidence

Choose qualitative or quantitative methods to collect evidence that illustrates how successfully you achieved your goals. Each approach has strengths and weakness, so you need to select ones that best fit your situation. (See Evidence-Gathering Options on page 14) The strongest evidence will answer the question as directly as possible. Multiple indirect indicators may also provide useful direction when lacking direct evidence.

Describe your evidence collection approaches and explain why you feel these methods are most appropriate for providing useful information which will address the concern in step 1.

• Example 2.1: We will have an expert in X technique observe the students perform the technique and rate the performance level using a rubric, verbal description and holistic score. We chose this approach because observing the student’s performance is the most direct measure we considered.
• Example 2.2: We defined social responsibility as Z, which we inferred from students’ respectful interactions with a variety of people and from their descriptions of what social responsibility means to them. Thus, we used 360 degree surveys to gather the opinions of peers, supervisors, and colleagues who had observed the student interacting with others in various contexts. We also asked the students to write a short essay on what social responsibility means and how they practice it.

Step 3: Interpret findings

Choose appropriate data-analysis approaches for the evidence collected in step 2. Summarize the findings and draw conclusions based on the evidence.

• Example 3.1: We concluded that some of our students are not achieving the learning outcome to the desired degree. The expert’s rubrics showed most students within the “developing” level and we had expected students to be at the “mastering” level by this point. The expert’s descriptive summaries suggest that the major weakness seems to be in the X & Y parts of the technique.
• Example 3.2: We concluded that our students are socially responsible. According to the 360 degree surveys, the students interacted respectfully with each person they met. Students’ essays indicated that they felt social responsibility was related to making sure that everyone is included. These perceptions and actions are consistent with our definitions of social responsibility.

Step 4: Decide and act

Make a decision about your goal based on the conclusions in step 3. Take appropriate action based on your decisions. Consider asking yourself: Are we satisfied with the outcome? If so, what do we continue doing in the future? If not, what steps might we take to more closely meet our target?

• Example 4.1: We are not satisfied with the current student performance level. To help more students achieve the “Mastery” level, we plan to do [A] to help students practice the X & Y components and we will do [B] within our classrooms. After implementing these changes, we will re-assess to see if the student performance has improved.
• Example 4.2: Since our students are adequately demonstrating the learning outcome related to social responsibility, we feel the best course of action is to continue our current approach. We will periodically assess to make sure we’re still on target.

[1] Assessment Loop Model is modified from Leskes, A., & Wright, B. D. (2005). The art & science of assessing general education outcomes: A practical guide. Washington, D.C.: Association of American Colleges and Universities.

[PV1]As a proxy, we might run into trouble when the assessment we use is not actually closely representing the real “learning” inside the brain but we don’t realize that. For example, if someone hands a scan-tron test to someone with a motor disability. It is possible the person will score poorly not because that person didn’t know the answers but because they couldn’t fill in the little circles correctly. In this case, the assessment is testing the physical ability, not the mental skills we think it is.  –This may be useful for discussion, but I’m not sure that it is a useful point in this document.

Table 5: Rubric demonstrating the  levels of assessment reporting

Criterion	Infant	Developing	Mature
Step 1: Set goals, ask questions	Lacks goal, issue, or purpose for the inquiry	Unclear goal, issue, or purpose for inquiry; or these must be inferred	Clearly stated goal, issue or purpose for inquiry
Step 2: Gather evidence	Unclear evidence-collection approach Unclear rationale for why the evidence would address the goal.	Vaugely describes evidence-collection approaches Limited rationale partially explains how the evidence would address the goal	Explicitly describes evidence-collection approaches for each goal Rationale explains how and why the evidence would address each goal
Step 3: Interpret findings	Lacks summary of findings Lacks conclusions about the findings	Limited or unclear summary of findings Vauge conclusions or evidence does not fully support the conclusions	Summarizes the relevant findings for each goal Clear conclusions are supported by evidence
Step 4: Decide and act	Lacks decision statement Lacks clear plan for enacting decisions Lacks rationale for the decisions	Vauge decisions or the evidence does not fully support the action plan for each goal Unclear rationale for why the evidence supports the decision and action plan	States decisions and action plans for each goal Rationale justifies the decision and action plan based on the conclusion

 

Evidence-Gathering Options

A few types of assessments include the following:

• Questionnaires: Typically used to record individuals’ perceptions on a topic. Closed-ended questions might permit statistical analysis. Open-ended questions might require qualitative analysis. These may be completed as a self-report or by a 3^rd party observer or expert.
• Performance: A physical manifestation of individuals’ knowledge or skills. These include, but are not limited to concerts, presentations, documents, exams, and artwork. These can be very strong evidence to indicate how well the individual achieved a learning outcome.
• Written examination: Typically used as a way to have individuals exhibit their knowledge or cognitive skills. The exam design can influences how individuals to reprent their learning. Exams may be quantitative, qualitative, or both.
• Interviews: Typically used to obtain participants’ perceptions on a topic. These require qualitative analysis, which can be time-consuming but might provide “deeper” insight on the reasons why participants think as they do about a topic.
• End-of-event evaluation: Typically used to obtain participants’ perceptions about the event. Depending on the format, these might inform us about participant satisfaction with the event and provide useful feedback for improvement.
• Expert rating: One or more subject-matter experts evaluate the quality of something to determine how well it matches the standards in the field.
• Rubrics: With training for inter-rater reliability, rubrics can allow raters to consistently evaluate the quality of performances, such as writing.

[PV1]What should we name the levels?