Student Survey Questions: valid? reliable?

As the TKES year draws to a close, it appears "the system" has some significant flaws.  There are some good elements to the new system (the TAPS in general are a good start), but other aspects are woefully unscientific, unreliable, even unprofessional.  This post focuses on the Student Survey Questions of Instructional Practice for grades 6-8.

1. It is possible that the student survey questions will not and can not lead to valid and reliable results (but we will use the results to evaluate teachers anyway).
2. The questions themselves can not apply to all classes or all teachers or all grade levels.
3. Most of the questions themselves are impossible to score a "strongly agree" due to the fact that a 12 year old is answering the question.  (I typically do not "strongly agree" or "strongly disagree" to anything.  Better choices would be: Most of the time, Usually, Sometimes, Not very often, etc.)
4. Because the questions can not apply to each teacher, to determine teacher effectiveness, the teachers will be compared to other teachers in the school, the district, and the state (thus the "School, District, and State Mean" and "State Median" columns).  Meaning, our effectiveness is not being graded against a teaching standard (the survey question), but against other teachers.  (What if we graded students against each other and not the curriculum standard?  That would be nonsense.)

Question #6: "My teacher chooses activities and assignments based on what students need to learn."  Yes, all the time, during class, for homework, for the day, the week, nine weeks, the year, and for all three years in middle school.  What else do teachers do?  How could a student possibly answer anything other than "strongly agree"?  Because they do not know; they could not know.  Anyone without strong pedagogical training, knowledge, and practice is not even qualified to answer that question.  Teachers even collaborate across grade levels so that there is continuity of learning in middle school.  We are not having students fill out word search puzzles.  87% of my students answered positively to that; however, at least one student answered strongly disagree to all questions.

Question #7: "My teacher gives students as much individual attention as they need [emphasis added] to be successful."  Laughable survey question.  58% of my students answered positively to that.  Think of what a classroom would look like to have 90% of the students answer "strongly agree" to that (I can not even imagine); remember most classrooms are overcrowded....

Question #10:  "My teacher allows me to work with different groups of students depending on the activity we are doing."  Of course we do, any time, all the time, every time - depending on the activity that is assigned to accomplish the learning goal as determined by the teacher so that the students will learn the curriculum.  But, it does not mean everyday or whenever the students want to.  The long-believed, and growing perception that collaborative learning is as effective as direct instruction is wrong.  The conditions of effective collaborative learning to mean direct instruction are so high, no one meets them.  Stacks of high-quality research show that over and over again (I can send you sources if you want because it was an influence in my doctoral study).  I received 47% positive response on that question.  Again, anyone without strong pedagogical training, knowledge, and practice is not even qualified to answer that question.

It appears to me that if I want higher scores on my survey results, I will need to create (another) system for next year to teach middle school students not only the state curriculum of concepts and skills, but also how to identify and understand the Teacher Performance Standards, pedagogy on something they have no knowledge of (so I can get better survey results), and master the Student Learning Objectives.  If only I could see my students for a full class period each day.

Student Surveys of Teacher Instructional Practices: absolutely laughable, if it weren't true

1. The use of student surveys (for middle school: 11-13 year olds) to impact teacher evaluation, TEM score, effectiveness, retention, and certificate, is absolutely laughable - if it weren't true.  Because it is true, it is horrifying and ridiculous.  Now teachers need to teach the students how to evaluate their teacher; here's what summary looks like, here's what differentiation looks like, here's what asking questions looks like, etc.  More later.
2. It seems reasonable that if a child - students - could take a survey that would impact a teachers' -professional educator's - job, why couldn't we create a survey of our students that could count toward their grade (i.e., a summative test grade).  More later.
3. A friend of mine, who has a birds eye view of many things, has the opinion that this entire TKES process was developed to create an evaluation system which has the ability to create a sense of outrage and distrust in the Georgia public education system so that public sentiment would move toward supporting Georgia's elected politician's push toward more charter schools (which was just voted on and approved), allowing for schools, teachers, and benefits to move out of the GA DOE funding into privatized companies allowing schools to function outside of the GA DOE which moved from the lack of governmental funding public schools years ago which, in its fullness, will create a disparity of high functioning charter schools and low functioning public schools creating a reinvented system of segregation--this time, between the haves and the have nots.  TKES isn't about providing an improved valid teacher evaluation, it's about creating political movement toward charter schools.  The opinion is hard to argue against; time will tell.
4. A teachers, as with all other evaluation systems, is in the hands of the interpreting evaluator.  A teacher with less than 5 years experience has seven IVs and three IIIs on his/her summative evaluation.  Another teacher with minimal experience has five IVs and five IIIs.  Another teacher with less than 5 years with experience who provides differentiated experiences for his/her students every day receives IIs.  One evaluator interprets by the letter of the rubric, another interprets: "if you could teach a class on this Standard, you should receive a IV."  ...completely different interpretations, different grades, and different summative outcomes for the teacher--connected to their certificate.  Word is that if you receive a II in any Standard, you may not be eligible for interview or hire in some counties.
5. The survey questions do not/can not apply to all subject matters; this creates validity/reliability problems (yet, we are going to evaluate and judge teachers anyway.  See #3.)

For example:  Survey Question #4: "My teacher takes time each day to summarize what we have learned."  My score: 1.95 (17% strongly agree, 37% agree, 29% disagree, 15% strongly disagree).  Clearly 12-year-old students are clueless (because they are only 12), I summarize more than one time per class period, and now it seems teachers need to teach students how to answer survey questions (in lieu of time spent on standards) to keep their jobs - or at least avoid uncomfortable summative TKES conferences regarding their student survey results.  I graduated with High Honors (high school), cum laude (college), 3.8 (masters), and a 4.0 (doctorate) in my degrees and apply what I know, and constantly use formative assessments during class so my students will learn efficiently, and now I have to figure out a way for 12 year old students to recognize and approve of my teaching strategies and methods to validate my abilities.  Something seems out of place here (see #3).

So, 44% of my students do not think I summarize the concept each day.  Hummmm.  Let's see.  Suppose I teach music.  I teach a new note, I give the location on the staff, I give the name, I give the fingering, I let the students see it and finger it, I give how to play it, I give how it is used, I show them why the note is the way it is, I have the students play the note, I double check (through formative assessments) that the students understand and can perform the note, I have them play the note, and then I have the students play the note in new music.  If that is not summarizing of the new note, I do not know what is, but 44% of my students do not think so.  The same would go for rhythms, dribble a basketball, sing pitch, sew a straight line, or draw a perspective.  In many performance-based classes (Connections classes in particular), concept is about 20% and performance/skill is 80%; therefore, playing/singing/drawing/performing the new concept is summarizing.  So, if teachers are going to improve their student survey averages and TEM score in the coming years, it appears we will need to find new ways of teaching the standards, documenting performance for evaluators, and teaching students how to answer the survey.

TLE Electronic Platform

This information is quoted from a document from the GaDOE TLE Electronic Platform dated August 5, 104 [sic] entitled, "Just for Teachers!  August"  All Rights Reserved.

1. Why do I need to upload documentation for every standard?  Documentation is not required for every standard.  Evaluators may request documentation from a teacher when a standard is not observed during a formative observation, walkthrough, or when the consistency of a teacher's practice cannot be established [emphasis added].  The teacher is responsible for submitting requested documentation in a timely manner.
2. Why will my TEM (Teacher Effectiveness Measure) be based on last year's student growth?  Student Growth (Student Growth Percentiles) and/or Student Learning Objectives data are finalized during the summer of each year, and after the completion of the school year and the evaluation cycle.  Therefore, student growth is lagging data and is used in the following school year's evaluation cycle and will contribute at least 50% towards the Teacher Effectiveness Measure (TEM).  [emphasis added]
3. Do the student surveys count a percentage toward my TEM?  No, these results do not count a percentage toward a TEM.  Results of the Surveys of Instructional Practice (student surveys) are used as a source of documentation to assist evaluators in rating Performance Standards 3, 4, 7, and 8 on the Summative Assessment [emphasis added].
4. How much training did my administrator receive to evaluate teachers?  All administrators who perform evaluations on teachers must participate in a GaDOE approved TKES training and pass a credentialing assessment at the end of the training.
5. No one in my school receives Level IV ratings.  Are they impossible to achieve?  No, ratings of Level IV are not impossible to achieve.  Teachers who perform at Level III and go beyond could certainly earn Level IV ratings.  In addition to going above and beyond a Level III rating, teachers must be a role model or teacher leader [emphasis added].

Student Surveys of Instructional Practice

Georgia Department of Education. (2014). Teacher Keys Effectiveness System Handbook.  Atlanta:  Retrieved from http://legisweb.state.wy.us/InterimCommittee/2012/TKESHandbook.pdf

“All Rights Reserved”

p. 40

PART III: Surveys of Instructional Practice

Another measure of the Teacher Keys Effectiveness System consists of student surveys of instructional practice. Surveys are an important data collection tool used to gather client (in this instance, student) data from individuals regarding the clients’ perceptions of teacher performance. Among the advantages of using a survey design include the rapid turnaround in data collection, the limited cost in gathering the data, and the ability to infer perceptions of a larger population from smaller groups of individuals. In the Teacher Keys Effectiveness System, surveys will be used as a measure of teacher effectiveness and documentation to support four of the TAPS standards. These four standards: Standard 3- Instructional Strategies, Standard 4-Differentiated Instruction, Standard 7- Positive Learning Environment, & Standard 8-Academically Challenging Environment reflect the direct experience of students in classrooms.

Multiple data sources enable the evaluator to obtain a more accurate picture of performance and assist the teacher in increasing student success. These data sources do not stand-alone but are complementary to each other and should be integrated into the process of evaluation to provide a richer portrait of teacher performance. The flaws of one data source are often the strengths of another, and by combining multiple methods, evaluators can make more solid judgments regarding teacher performance and make decisions that are supported by multiple types of data.  Student surveys may help the teacher set goals for continuous improvement (i.e., for formative evaluation) — in other words, to provide feedback directly to the teacher for professional growth and development. Student surveys also may be used to provide information to evaluators that may not be accurately obtained during observation or through other types of documentation.

The surveys ask students to report on items they have directly experienced. Three different versions of the student survey (grades 3-5, 6-8, and 9-12) will be provided. The versions are designed to reflect developmental differences in students’ ability to provide useful feedback regarding their teacher. All surveys are to be completed anonymously to promote honest feedback.