Frequently Asked Questions about Assessment
MARQUETTE’S ASSESSMENT PROCEDURES AND TIMELINE
What is program assessment?
Assessment at the program level systematically collects information about student learning across courses and experiences in an academic or co-curricular program. Its purpose is determine how well students as a whole are meeting expected learning goals, and inform decisions about how to improve program content, delivery and pedagogy. Assessment is action research, and is intended to inform individual and local action, instead of leading to generalizable results.
How is program assessment different from student assessment?
Student assessment uses the results of assessment to evaluate and grade the individual student. Program assessment uses the results of assessment to understand and improve student learning. Program assessment typically makes judgments about aspects of the program using samples of students or samples of their work.
How is program assessment different from program review?
Program review is a comprehensive evaluation of a program’s past performance, challenges and opportunities in areas that encompass quality, service, efficiency and resources. Assessment of student learning is one component of program review.
I’m confused about the terminology – what’s the difference between goals, objectives, outcomes, performance indicators?
Generally, goals and objectives refer to what programs hope to achieve. Goals are broad statements that provide a framework for more specific objectives. Outcomes refer to what the student will know or be able to do upon completion of a learning experience. Performance indicators are quantitative measures of student achievement. For example, a political science program may have a goal to create an intellectual environment in which students can acquire the knowledge of the discipline. An objective within that goal might be for students to understand the various types of political organizations in the world. One learning outcome is that students are able to compare and contrast the processes for selection of rulers in the U.S. with other nations. The performance indicator is the percentage of students who can successfully demonstrate that outcome. A glossary of assessment terms used at Marquette is found in the procedures document.
What is a PAL?
PALs (Program Assessment Leaders) are responsible for coordination of the assessment processes in their designated programs. At Marquette, every program has a PAL, and a particular PAL might be responsible for multiple programs. PALs submit completed program reports to the Assessment Reporting Management System each fall, and represent their programs at the annual Peer Review Seminar. There is training for new PALs (or anyone in need of a refresher) at the beginning of the academic year, and workshops on topics relevant to program assessment during the year.
Why are we required to do assessment?
Faculty at Marquette are characterized by their passionate dedication to students, and assessment is about helping students learn. Program assessment provides faculty with evidence about how well they are achieving the goals of their programs, and direction for how to create an environment for improved learning. Assessment is not an end in itself, something that is completed once the report is filed; it is a means for inspiring action to achieve what faculty most value.
In addition, assessment is a means for responding to accreditors’ and other external stakeholders’ concerns for accountability. The Higher Learning Commission of the North Central Association requires member institutions to demonstrate a commitment to educational achievement and improvement through ongoing assessment of student learning. Standards for discipline or program accreditation include assessment.
Why aren’t grades assessment?
Course grades are usually insufficient evidence for program assessment because: 1) They are a composite of a student’s achievement in a single course; 2) They reflect the criteria and course material of individual faculty and may not be consistent across faculty teaching the same course; 3) They may include student behaviors not related to learning, such as class attendance; and 4) They are not diagnostic - faculty are unable to determine what aspects of learning students have or have not mastered. Graded course assignments, on the other hand, can be useful for assessment if they are tied to a specific learning outcome. In this instance, grades may be derived from rubric scores that are diagnostic of the level of learning achieved in a particular area.
Our assessment results are fine. Must we keep doing this every year?
While celebrating positive assessment results, program faculty should be aware that assessment is more about ferreting out the areas where improvements can be made. If assessment results routinely meet or exceed standards, it’s possible that the standards are set too low, or that measures are not sufficiently diagnostic. Programs should not repeat the same assessments every year if they are satisfied that student learning in a particular area is well established; they should move on to assess different outcomes. A good way to start this conversation is for faculty to ask, “What aspects of learning in our program are we really interested in knowing more about?”
What are the benefits of program assessment?
The process of assessment brings faculty together to discuss, clarify and agree upon their expectations and standards for student learning. Program assessment results provide the feedback that faculty need to understand what is not working, and what can be done to improve the delivery of the program curriculum. Data collected for assessment can be used for other purposes, such as program review, annual reports, or discipline accreditation reports. Assessment can benefit students by making expectations for program-level learning explicit. When students are aware of a program’s learning outcomes, they are better able to focus their time and energy on what learning is most important.
What is the connection between assessment and the scholarship of teaching and learning?
As Pat Hutchings of the Carnegie Foundation for the Advancement of Teaching points out, assessment is not simply a service or good campus citizenship, but an important intellectual enterprise – a form of scholarship reflecting faculty’s professional judgment about the nature of deep understanding of their field and about how such understanding is developed. Both assessment and the scholarship of teaching and learning involve systematic inquiry into the conditions that promote learning. Using research questions that arise from assessment data, faculty can extend the parameters of assessment by delving deeper into pedagogy and classroom practices, and documenting and sharing their work in ways that can be reviewed, built on, and rewarded.
How many learning outcomes should we have?
Programs should start with identifying the 3 – 5 most important knowledge, abilities, values and/or attitudes that students are expected to acquire as a result of their educational experience. Once sufficient evidence is collected to inform improvement in those aspects of the program, faculty can retire those and move on to assess other areas that they may be interested in exploring further. See this link for guidance on writing student learning outcomes.
Should we assess all outcomes every year?
Programs do not need to assess every learning outcome every year, but the program must have a plan for periodically assessing all outcomes. A program might choose to adopt a process of rotating through the outcomes on a regular schedule. The schedule should be noted in the program’s assessment plan. Assessment reports should include all learning outcomes, with a brief explanation for those not assessed in that particular year.
Can we use exams for assessment?
Comprehensive exam scores are usually insufficient evidence for program assessment unless they include sub scores indicating which areas of the curriculum students have or have not mastered.
Can we use student survey data for assessment?
Surveys capture valuable information about students’ perceptions of their learning experiences and attitudes towards the learning process. They do not provide direct evidence of what students are able to demonstrate, represent or produce because of what and how they have learned in the program. Surveys can supplement information gathered through more direct measures, but cannot substitute for them.
How does curriculum mapping benefit student learning and assessment?
A curriculum map is a communication tool that visually aligns learning outcomes with the content of individual courses or experiences in the program. It is typically designed as a matrix, with the learning outcomes listed in the first column and the individual courses or experiences listed on the top row (or the other way around.) A curriculum map improves understanding of how the curriculum fits together; helps with course sequencing; reveals strengths, gaps, redundancies the curriculum; and leads to more effective assessment. Each learning outcome should be introduced, reinforced and mastered at least once across multiple courses.
Must every measure use a rubric?
A scoring guide (rubric) is the best option for scoring assignments, projects, performances or presentations. Analytic rubrics, which score multiple criteria or components of a given product, provide more information than holistic rubrics, which produce a single rating. That said, it is not the only instrument in the toolbox; well-designed exam items can yield useful diagnostic information on where students are strong or weak.
Do we need pre-and posttests to do a valid assessment of our program?
Pre- and post- measures are intended to document the increase in learning that occurs during a course, program, or educational experience. These “value-added” assessments may be useful for measuring growth in core learning outcomes (“Do senior papers demonstrate more sophisticated writing skills than freshmen papers?”) but may be less relevant to discipline-specific programs, where there is no assumption of pre-existing knowledge. Student portfolios are the most authentic instruments for demonstrating growth over time, not pre-post surveys or multiple-choice tests.
Should we assess a program with fewer than five students?
You should collect information from programs with small numbers of students, and record those results in ARMS. However, you may want to wait several years before making any conclusions about the extent to which students are meeting learning outcomes. Note in the assessment report that you intend to collect more data for a specified period of time before reflecting on results. Sometimes a discussion about student work among faculty in the program can substitute for data in the report.
How do we assess a program with a large number of students?
It is acceptable to sample student work, especially if classes are large and/or multiple assignments are used for assessment.
Should assessment always occur when the student is preparing to complete the program?
Although program assessments are usually summative – they provide evidence of students’ final mastery levels – don’t overlook opportunities for formative assessment. Feedback at different points in the program helps students learn and helps faculty improve their instruction.
Can we assess aspects of our program other than student learning?
Student satisfaction, destinations after graduation, and other indicators of program effectiveness may be used to supplement more direct evidence of student learning and are appropriate for program assessment. Metrics associated with the program’s processes, resources, efficiencies, and service are more properly reported within the program review structure. However, since the ARMS system can be used to store ancillary reports, feel free to add any contextual information about the program that will help faculty interpret and act on results of student learning assessment.
MARQUETTE’S ASSESSMENT PROCEDURES AND TIMELINE
Where do I find my program’s assessment plan?
Program assessment plans are located on the assessment website at http://www.marquette.edu/assessment/
They are publicly available.
Where do I find my program’s assessment reports?
Archived assessment reports from 2008-09 through 2010-11 are found on the assessment website. They are restricted to campus.
Assessment reports from 2011-12 onward are located in the Assessment Reporting Management System (ARMS). They are available to all full-time faculty and others with approved access into the system.
What is ARMS and how do I learn to use it?
The Assessment Reporting Management System was launched in 2012 as a tool to document, track and report the results of academic and co-curricular program assessments. It is a Campus Labs’ product. The link to ARMS is located on the assessment website. All full-time faculty have view access to all reports in ARMS. PALs and designated others have the ability to edit reports. The announcement page of ARMS includes power point training slides for users. The Assessment Director is available to assist you in navigating the ARMS system.
How often should we update our assessment plan?
Faculty should review the plan annually at the same time that assessment results are reviewed and discussed. Assessment plans should be updated any time that the faculty determine that the learning outcomes or methods and measures are no longer appropriate to the program. If faculty are satisfied with results after several years of assessing the same learning outcomes, different outcomes should be chosen for assessment. Send any plan modifications to the assessment director.
When are assessment reports due?
Assessment reports for the previous academic year are due at the end of October. AY2012-13 assessment reports are due on October 25th, 2013.
Can we go back and edit our assessment reports after they are submitted?
Yes, assessment reports are always ‘open’ and able to be edited at any time.
Are we expected to “improve” our program every year?
No. After discussing assessment results, program faculty might decide that the best course of action is to continue collecting information in the same way for another year before making any programmatic decisions. Faculty may also determine that there is a need to make changes to the assessment process and instruments. Changes to curriculum, pedagogy, resources and other aspects of program delivery should be made when faculty are comfortable that assessment results substantiate or suggest those changes.
Can we skip a year of assessing?
Under certain circumstances, it might be beneficial for a program to suspend its current assessment plan. This could occur if the program needs to spend the year doing a complete overhaul of the assessment process, or if the program is interested in undertaking an alternative assessment approach, such as problem-based assessment. Requests to suspend the current assessment plan should be made no later than the spring semester preceding the suspension. The University Assessment Committee will review and approve these requests.
What happens to our assessment reports?
Individual program assessment reports are always available for viewing in ARMS. Administrators and PALs can request aggregate reports (across programs) from the assessment director. The director aggregates information across programs on types of measures used and follow-up actions for the Annual Institutional Assessment Report. Reports are shared among PALs at the annual Peer Review Assessment Seminar in November of each year. The director may select examples of exemplary assessment practices to share with the campus as a whole.
What can the Assessment Director do for me?
The Assessment Director can provide your program with resources, advice, and support. The Director will consult on developing and implementing assessment plans and reports, including writing effective student learning outcomes and choosing appropriate measures, analyzing data and reporting results in ARMS. The Director has some limited financial support for costs associated with attending workshops and conferences centered on assessment. In addition to the resources available online, the Director hosts a number of workshops on topics relevant to program assessment. Please watch the assessment webpage for upcoming workshops and events. If you are interested in a one-on-one consultation or a specialized workshop for your program or department, contact Sharron Ronco at 8-6390 or email@example.com.
What resources does the University Assessment Committee offer?
The University Assessment Committee (UAC) includes representatives from all Colleges, as well as Student Affairs and the Core Curriculum. Contact your UAC rep if you have assessment topics or concerns that you would like the Committee to address. The UAC reviews new or substantially-revised assessment plans and provides expert feedback. See the University Assessment Committee for further information on Committee responsibilities, membership, and meeting dates.
Shouldn’t faculty get extra compensation for doing assessment?
Course and program-level assessment is considered part of a faculty member’s regular teaching and service duties. Each department determines how to handle the responsibilities of coordinating assessment or serving on the University Assessment Committee. This may involve course release time or other compensation or recognition. This is standard procedure across most universities.
Where can I find best practices for assessment in my discipline?
Many institutions post their program assessment plans online, and a few also make their assessment results public. Assessment publications include some best practice case studies in various disciplines. Please contact the Assessment Director for help locating examples in your discipline.