Measuring student growth using student learning objectives

From: http://ies.ed.gov/ncee/edlabs/projects/project.asp?ProjectID=369
(If you haven't signed up for e-mails from ies.ed.gov, you should!)

Student learning objectives (SLOs)—classroom-specific growth targets chosen by individual teachers and approved by principals—are becoming popular as alternative measures of student growth because they can be used to evaluate teachers in any grade or subject.  Although very little of the literature on SLOs addresses their statistical properties, key findings show that:

• SLOs have the potential to better distinguish teachers based on performance than traditional evaluation metrics do, but no studies have looked at SLO reliability.  Most of the limited evidence on the statistical properties of SLOs is on the proportion of teachers achieving SLO objectives. Whether that differentiation represents true differences in teacher performance or random statistical noise is unknown.
• Little is known about whether SLOs can yield ratings that correlate with other measures of teacher performance. Only three studies have explored the relationship between SLO ratings and standardized assessment-based (value-a dded) growth measures. These studies found small but positive correlations. More research is needed as states and districts roll out SLOs as teacher evaluation measures and instructional planning tools.
• Until some of the research gaps are filled, districts that intend to use SLOs may want to roll them out for instructional planning before using them in high-stakes teacher evaluations. Several studies found teacher concerns about fairness in SLO implementation. This is no surprise, because SLOs are difficult to make valid and reliable. They are by definition customized to individual teachers and based on the professional judgments of teachers and principals. Making SLOs an important component of high-stakes evaluation could undermine their validity, because it means that teachers are in essence grading themselves.
• Studies of teacher experiences with SLOs indicate that SLOs can require substantial training and technology infrastructure and that they can be time-consuming for teachers and evaluators alike.

TKES: SLO, Day 2-Timing

Note: these comments are for the benefit of those who still have time to plan and or make adjustments to their pre-tests.  I thought ours were going to be great, flawless, and easy...that has not been the case.

Legislators should take note about what they have required us to do.  The teachers who were involved in creating our SLO (Student Learning Object) test as a part of the TKES (Teacher Keys Effectiveness System) evaluation did the best they could with the information they were given (I was one of those teachers).  In hindsight, we needed more and better information at the beginning and more time to develop and test for a truly valid and reliable instrument.

For example: I think some teachers had created a written portion of the SLO that had over 100 questions, some with 35 questions, some with 65+; apparently some of the SLOs cover portions of the entire curriculum, some that cover only the major parts (where the most time is spent during the year).  Again, it seems as if there are so many variables, the validity and reliability could come into question (IES and REL have already spoken on this matter).  While I think that my giving my post-test to the same students will reveal valid results, I don't think that it would reliable across the subject matter in my county; there are certain performance aspects that each teacher is to develop for their class.  This performance aspect should be well thought out, prepared, organized, and an effective measurement of the performance task.

Timing:

• In our county, we are supposed to administer the SLO during the first 2 weeks of school.  This is the same time period where the students are still shifting in their Connections classes, students are entering the school and moving to another school.  The rosters are changing, being printed, put in teachers' boxes for the next day; but, the computer system doesn't update until after 12 midnight.  Therefore, the rosters in our boxes are not correct for the students who should be in the class.  We had to individually tell students what classes to go to - as they're walking in the door.  This took time out of our testing window.
• It is possible that the written portion of a SLO may be given for at least 45 minutes.  The Connections class period is 45 minutes - which includes the transition to the class, calling roll, setting up for the test, etc.  Therefore, unless the Connections classes are only meeting the A block for 90 minutes, a regular Connection class of 45 minutes can't meet the requirements for the test.  Some schools are extending the A class (and B class the next day) for the 90 minutes, others are not.  This timing issue would never be allowed in the CRCT (which is now gone away for Georgia).  It appears that there was an assumption that each Connections class would be extended for the entire 90 minute block: Class A on Day 1, Class B on Day 2.
• Some of the performance aspects for some of the Connections classes are given 5 minutes per student to demonstrate mastery.  This creates 2 huge issues:
• There is a teacher in the county that has 70 students in one class....Do the math.  How many class periods will that take to administer just the performance aspect of the SLO?  14 days - if everything works perfectly.
• Some teachers have completed the written portion of the SLO but now have to complete the performance (or spoke/language) portion next week.  That's fine except we have ITBS testing Monday - Thursday AND CogAT testing next week.  The entire school schedule is adjusted and Connections classes will not be more than 35 minutes each - and that does not include the transition to class, calling roll, distributing material, etc.  So, if a teacher has 30 students and has to give 5 minutes to an individual student demonstrating their performance, but only has 25 minutes of class time, how many DAYS will it take to pre-assess the class?  I count 6 DAYS.  6 days for performance, 1 day for written this 9 weeks; then 7 days again at the end of the year: that is almost 3 instructional weeks missed due to testing....

I recommend that you get with your administrator(s), TKES evaluator, or assessment coordinator and try to work out as many bugs as you can before the day of the test.  If you would like advice or other thoughts, please contact me....

Student Learning Objectives (SLOs)

Georgia Department of Education. (2014). Teacher Keys Effectiveness System Handbook.  Atlanta:  Retrieved from http://legisweb.state.wy.us/InterimCommittee/2012/TKESHandbook.pdf

“All Rights Reserved”

p.27

Student Learning Objectives: District-determined SLOs are content specific, grade level learning objectives that are measureable, focused on growth in student learning, and aligned to curriculum standards. As a measure of teachers’ impact on student learning, SLOs give educators, school systems, and state leaders an additional means by which to understand, value, and recognize success in the classroom.

The primary purpose of SLOs is to improve student achievement at the classroom level. An equally important purpose of SLOs is to provide evidence of each teacher’s instructional impact on student learning. The process of setting and using SLOs requires teachers to use assessments to measure student growth. This allows teachers to plan for student success by ensuring that every minute of instruction is moving students, teachers, and schools toward the common vision of exemplary instruction and high levels of student academic growth. The Student Learning Objectives Operations Manual which is located on SharePoint has detailed information and forms regarding SLO development.