College Assessment Plan

College Educational Assessment Plan

The basic elements of the engineering educational assessment plan are as follows:

College assessment team

This team provides leadership for student outcomes assessment activities within the College. Their activities include guidance for: the definition of educational objectives, the measurement of learning outcomes, and continuous educational improvement activities. This team provides a forum for sharing and testing ideas and the results of departmental assessment activities.

This team has met continuously since 2001, first developing the College Educational Assessment Plan, then monitoring and assisting assessment activities, and in 2003-4 assisting programs in developing their self-study reports to ABET. The committee has representatives from each program in the College. Meetings are held as needed, but always several times per semester.

Since the University requires that each program has educational objectives, defines learning outcomes, and assesses results, a representative of each of the College’s programs should be represented on the team. Whether or not the program is subject to ABET review is not relevant.

The University is committed to student outcomes assessment. North Central Association requires it. ABET requires it. The Arizona Board of Regents requires it. It makes sense that an educational program should have defined objectives, and as engineers we should be comfortable with the idea of measurement to see how we are doing, and putting a feedback loop in place to continuously improve what we are doing.

College educational mission

Provide an excellent education for students entering the engineering profession.

College educational strategy

To accomplish the College educational mission each academic program will:

Define the professional expertise that graduates in their discipline should have upon successful completion of the program
Provide the educational activities that will:
- allow students to experience the benefits of interacting with experienced faculty and staff
- prepare them to understand the practice of the profession from political, ethical, economic and technological viewpoints
- enable graduates to fully participate in the state-of-the-art profession of the field at graduation and throughout their careers
Continuously improve its academic programs

Departmental strategic planning

The College’s educational strategy must be implemented on a program-by program basis by defining sets of program educational objectives and the learning outcomes students should demonstrate prior to graduation to assure that the educational objectives are accomplished. The resulting learning outcomes must be measurable and reflect what the program wants to be known for: its ‘brand’ identity. It is best if each learning outcome is stated as a measurable verb followed by the expected outcome. By measurable it is meant that one should think about how the learning outcome will be measured and use a verb appropriate to the planned assessment.

Each program should:

Define its educational mission and educational objectives, as well as its expected learning outcomes.
Determine what can be measured from the various elements of its educational program to verify to the outside world that its graduates have accomplished each of its expected learning outcomes. Two or more indicators should be used to measure each learning outcome. The elements of the educational program include coursework, laboratory experience, the capstone design experience, student-faculty interaction, student-student interaction, student services provided, work experiences, and others.
Determine the level of achievement (metric) that is expected for each learning outcome to assure that the desired educational objectives are achieved.
Measure its prescribed learning outcomes.
Compare these measurements with the chosen metrics.
Survey its constituents to evaluate the appropriateness of its educational objectives.
Make measurements as frequently as necessary to ensure that continuous improvement is being accomplished.

Measurements may take the form of surveys, quantitative interviews, industrial feedback, student portfolios, external exam results, external reviews, etc. There should be evidence that the measurements are not merely self-serving, but are useful for the purpose of program improvement. This usually requires separating the role of evaluator from that of instructor.

Evaluation

Figure 1 indicates how the educational objectives and constituent surveys are mapped to the learning outcomes. The elements of the educational objective matrix indicate the extent to which each learning outcome supports the educational objectives, and the elements of the outcome evaluation matrix indicate which learning outcomes are evaluated by each constituent survey.

Surveys of constituents measure the importance of each of the learning outcomes and evaluate the appropriateness of the educational objectives.

Figure 1. The educational objective and evaluation matrices

Assessment

In addition to their mapping to the educational objectives, each of the learning outcomes needs to be mapped to ABET criteria, the curriculum, and the assessment plan. These mappings can be done with a set of matrices like those suggested in Figure 2 for undergraduate programs. Graduate programs are better described in Figure 3.

The elements of the program’s educational objective matrix and the ABET criteria matrix indicate the extent to which each learning outcome supports the specified objectives and criteria. The elements of the curriculum matrix indicate the impact each course or other learning activity has on each learning outcome. The elements of the assessment matrix indicate which learning outcomes each assessment tool measures.

Indicated in Table 1 are the constituent surveys and other measurement tools currently used to evaluate educational objectives and assess learning outcomes.

Figure 2. For undergraduate programs, educational objectives map into the learning outcomes, which in turn map into ABET criteria, the curriculum, and the outcomes assessment through a set of carefully designed matrices.

Figure 3. For graduate programs, educational objectives map into the learning outcomes, which in turn map into the learning activities, and the outcomes assessment through a set of carefully designed matrices.

Table 1. Assessment/evaluation tools currently used within the College

Constituent surveys:

Students (I,S), freshmen (multi school)
Exiting seniors (I,S) and EBI (multi school)
Alumni (I,S)
Faculty (I)
Industrial Advisory Council, other employers

Other tools used by various departments:

Student academic progress reports
Longitudinal student portfolios (industry can help w/writing)
Course portfolios
Pre-requisite examinations in follow-on courses
Senior design exam
Student assessment of team participation
Senior design project presentation review by industry
Senior design course requirements
FE exam, GRE exam (multi school)
Placement data

S = Satisfaction measurements of the learning objectives to verify that the courses and other learning activities are appropriate to accomplish the learning outcome metrics or if improvement is needed.

I = Importance measurements of the learning outcomes to verify that the educational objectives are appropriate for constituency needs.

Each program’s assessment plan needs to identify a step-by-step description of how the program:

collects the data (transcripts, surveys, portfolios, design projects, placement data, etc.)
analyzes the data that is collected (various rubrics may be useful for this)
compares the results with the expected achievement levels (metrics)
identifies competencies (learning outcomes that meet or exceed expected achievement levels and can be used to brand the program)
identifies gaps (learning outcomes that do not meet expected achievement levels)
implements changes to improve its educational processes and remove the gaps
reassesses to see if the changes were sufficient to remove the gaps

Since the learning outcomes are mapped to the educational objectives, assessment of the learning outcomes also provides an assessment of the educational objectives.

Continuous improvement

Assessment steps 6 and 7 are referred to as ‘closing the loop’ and each program needs to do this each year. The overall planning, assessment, evaluation, and improvement activities are represented schematically in the flow chart of Figure 4.

Figure 4. Overall flow chart for the improvement process

Reporting of assessment results

Each academic program is expected to report on its assessment and continuous improvement activities annually. This will be done by the program maintaining a web page that presents its educational mission, educational objectives, expected learning outcomes, assessment program, and appropriate outcomes data; and by the department head including the following in each annual report to the dean:

measurement data for each learning outcome specified by its academic programs
a statement of the extent to which its learning outcomes are being accomplished at the expected levels
academic program change activities that have been implemented in response to gaps in outcome data (including documentation of the gaps)
data verifying that previously implemented change activities have removed the gaps that identified their need
descriptions of the one or two most significant competencies of graduates from each program as identified by outcomes measurements
verification that each program has reviewed its educational objectives to determine if they continue to be appropriate

Note

ABET’s training materials say, "Achievement of program outcomes by each student should be verified before certification for graduation." The actual printed ABET criteria says, "Engineering programs must demonstrate that their graduates have"...accomplished a-k. The training material says each student should, the criteria says graduates must. Neither says each student must. It appears that they are trying to get programs to check every student, but they only require that programs assess students in general.

If it is decided that every student should accomplish a specific outcome, the secret is to couple that outcome to completion of a course so degree audits (or senior checks) will do the verification automatically. Coupling it to a course means making sure that a course exists in which the expected outcome is a focus, and then requiring students to verify the expected level of accomplishment to pass the course. Students either fail (sounds harsh) or get incomplete grades until they verify that it is accomplished. The actual grade they get for the activity that verifies compliance can be used as part of the assessment data.

Warning

Requiring too much data collection will overwhelm the assessment system. It has been recommended by experienced leaders in another engineering college that no more than about five different measurement tools be used for assessing outcomes. No data should be collected unless there is a plan to use it.

For programs where there are common educational objectives or learning outcomes, similar assessment methodologies and tools can be used. Though evaluated on a program-by-program basis, it is often possible to coordinate some assessment activities at the college level to save time and energy as well as to provide programs with college-wide data for comparison.

Definitions

Educational Mission Statement: An overarching long-range statement (about 15 words) describing the purpose for which the educational program exists.

Educational objectives: A set of 3-5 objectives that describe what the program plans to do to ensure that its mission is accomplished with respect to the students it is designed to serve.

Evaluation: The process of measuring the attitudes of constituents relative to the appropriateness of the program’s educational objectives.

Constituents: The constituents of the program are: continuing students, graduates, faculty, and employers. Employers can be interpreted as either those who hire the graduate for work in the field of the profession or graduate schools that admit the graduate for further study in the field.

Learning outcomes: A set of measurable learning results that, if satisfactorily achieved by the program’s students, will verify that the program has accomplished its educational objectives. These describe what the program wants to be known for by its constituents: its brand.

Assessment: The process of using quantitative measurements to determine if students are accomplishing the desired learning outcomes.

Continuous Improvement: A formal system is put in place to insure that the results of assessment are compared to expected learning outcome metrics and used to improve the program’s educational processes.