Measuring student growth using student learning objectives

From: http://ies.ed.gov/ncee/edlabs/projects/project.asp?ProjectID=369
(If you haven't signed up for e-mails from ies.ed.gov, you should!)

Student learning objectives (SLOs)—classroom-specific growth targets chosen by individual teachers and approved by principals—are becoming popular as alternative measures of student growth because they can be used to evaluate teachers in any grade or subject.  Although very little of the literature on SLOs addresses their statistical properties, key findings show that:

• SLOs have the potential to better distinguish teachers based on performance than traditional evaluation metrics do, but no studies have looked at SLO reliability.  Most of the limited evidence on the statistical properties of SLOs is on the proportion of teachers achieving SLO objectives. Whether that differentiation represents true differences in teacher performance or random statistical noise is unknown.
• Little is known about whether SLOs can yield ratings that correlate with other measures of teacher performance. Only three studies have explored the relationship between SLO ratings and standardized assessment-based (value-a dded) growth measures. These studies found small but positive correlations. More research is needed as states and districts roll out SLOs as teacher evaluation measures and instructional planning tools.
• Until some of the research gaps are filled, districts that intend to use SLOs may want to roll them out for instructional planning before using them in high-stakes teacher evaluations. Several studies found teacher concerns about fairness in SLO implementation. This is no surprise, because SLOs are difficult to make valid and reliable. They are by definition customized to individual teachers and based on the professional judgments of teachers and principals. Making SLOs an important component of high-stakes evaluation could undermine their validity, because it means that teachers are in essence grading themselves.
• Studies of teacher experiences with SLOs indicate that SLOs can require substantial training and technology infrastructure and that they can be time-consuming for teachers and evaluators alike.

Teacher Pay for Performance Results

New: from the Institute of Education Sciences

To obtain hard copy of many IES products as well as hard copy and electronic versions of hundreds of other U.S. Department of Education products please visit http://www.edpubs.org or call 1-877-433-7827 (877-4-EDPUBS).

New from NCEE: Teacher Incentive Fund Implementation and Impacts of Pay-for-Performance After Two Years

Today, NCEE released the second of four evaluation reports that studies Teacher Incentive Fund (TIF) grants that were awarded in 2010 to support performance-based compensation systems for teachers and principals in high-need schools. The report provides basic implementation information for all 2010 TIF grantees, and more in-depth implementation and impact information for the subset of 10 districts that agreed to participate in a random assignment study. The main findings among all TIF districts with 2010 awards are: * Ninety percent of all TIF districts in 2012–2013 reported implementing at least 3 of the 4 required components for teachers, and only about one-half (52 percent) reported implementing all four. This was a slight improvement from the first year of implementation. * Near the end of the second year of implementation, 65 percent of TIF districts reported that sustainability of the program was a major challenge. In contrast, no more than one-third of districts reported that other activities related to their program were a major challenge. For the subset of 10 districts that agreed to participate in a random assignment study, key findings on the effect of pay-for-performance on educators include the following: * Few evaluation districts structured pay-for-performance bonuses to align well with TIF guidance in the grant competition notice. * Educators’ understanding of key program components improved from the first to the second year, but many teachers still did not understand that they were eligible for a bonus. They also continued to underestimate how much they could earn from performance bonuses. * Offering educators pay-for-performance had small, positive impacts on their students’ reading achievement; impacts on students’ math achievement were not statistically significant but similar in magnitude. To read the report, visit http://ies.ed.gov/ncee/pubs/20154020/index.asp To learn more about the study research design and the impact evaluation of TIF, visithttp://ies.ed.gov/ncee/projects/evaluation/tq_incentive.asp.

This is important, but it won't matter...

If you have read the previous blogs, you should notice that there are practical, real concerns in developing and administering the S.L.O.s (student learning objectives) as part of the new Georgia TKES (Teacher Keys Effectiveness System).  If you haven't read them, please take a moment to do that now....  I'll wait....  Researchers have now compiled evidence that should be taken notice by those in charge (although it is too late for that).  The following statements are from a recent article released by the IES (Institute of Education Sciences); if you haven't signed up for their automatic e-mail notices, you should - it's good stuff - it's the real deal.  I have put a copy of this article in my TKES/TAPS notebook (a picture of my notebook can be seen in a previous blog).

So, from: Gill, B., English, B., Furgeson, J., & McCullough, M. (2014).  Alternative student growth measures for teacher evaluation: Profiles of early-adoption districts.  (REL 2014-016).  Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Mid-Atlantic.  Retrieved from http://ies.ed.gov/ncee/edlabs:

SLOs can be used for teacher evaluation in any grade or subject, but require substantial effort by teachers and principals, and ensuring consistency is challenging [emphasis added].

Use of alternative growth measures that do not depend on state assessments is recent, and little is known about their validity and reliability [emphasis added] or about how they are being used. 

I have subtly raised concerns about the reliability of our SLOs - it is assumed that the teacher will provide a substantial performance task(s) (based on the end of the year expectations) that will show student growth, but because of time limitations, lack of preparation, lack of knowledge, etc., it is conceivable that it may not have happened - it is an unknown variable.  Thus, to me, the reliability of our SLOs is in question....

I have an excellent article, but it is in my TKES/TAPS notebook at school; I'll bring it home and give you the information soon.  The researchers show that SLOs really shouldn't be used in teacher evaluations at all....  It is from the same research institute.

If you have comments or questions, let me know....  gcason123@gmail.com; online portfolio is: https://sites.google.com/site/gcason123/