Does Anyone Know What’s Really In There?
There’s an old saying about the inmates running the asylum that comes to mind with this story from a science educator in Florida who looked into the science portion of Florida’s standardized assessment. Robert Krampf was developing some science test prep questions for 5th graders preparing to take that state’s high-stakes FCAT test. Using the FL DOE’s own site as a guide, he found many flaws in what was presented as science.
A few weeks ago, I started developing FCAT practice questions to help students review concepts and prepare for the test. To develop those questions I used FLDOE’s FCAT 2.0 Science Test Item Specifications. These documents are used as:
“a resource that defines the content and format of the test and test items for item writers and reviewers.”
I expected the Test Item Specifications to be a tremendous help in writing simulated FCAT questions. What I found was a collection of poorly written examples, multiple-choice questions where one or more of the wrong responses were actually scientifically correct answers, and definitions that ranged from misleading to totally wrong.
On his blog – the Happy Scientist, he sited these examples:
A glossary of definitions (Appendix C) is provided for test item writers to indicate the level of understanding expected of fifth grade students. Included in that list is the following definition:
Predator—An organism that obtains nutrients from other organisms.
By that definition, cows are predators because they obtain nutrients from plants. The plants are predators too, since they obtain nutrients from decaying remains of other organisms. I have yet to find anyone who thinks that this is a proper definition of a predator.
This sample question offers the following observations, and asks which is scientifically testable.
- The petals of red roses are softer than the petals of yellow roses.
- The song of a mockingbird is prettier than the song of a cardinal.
- Orange blossoms give off a sweeter smell than gardenia flowers.
- Sunflowers with larger petals attract more bees than sunflowers with smaller petals.
The document indicates that 4 is the correct answer, but nswers 1 and 3 are also scientifically testable.
For answer 1, the Sunshine State Standards list texture as a scientifically testable property in the third grade (SC.3.P.8.3), fourth grade (SC.4.P.8.1), and fifth grade (SC.5.P.8.1), so even the State Standards say it is a scientifically correct answer..
.For answer 3, smell is a matter of chemistry. Give a decent chemist the chemical makeup of the scent of two different flowers, and she will be able to tell you which smells sweeter without ever smelling them.
While this question has three correct answers, any student that answered 1 or 3 would be graded as getting the question wrong. Why use scientifically correct “wrong” answers instead of using responses that were actually incorrect? Surely someone on the Content Advisory Committee knew enough science to spot this problem.
The teacher/proctor is not permitted to read the question, only to assist with students and computer operating issues. Student works through problem on scratch paper, and finds that his/her answer doesn’t match with any choices given. Teacher looks at problem worked out on scratch paper, and determines that the child has correctly answered the problem, but the correct answer is not one listed. Teacher can do nothing about it since he/she is not permitted to read the test, only the test prompts. School therefore does nothing. Child has no defense, and since testing by computer is graded by computer, the testing company is not held accountable.
- Florida Thinks Their Students Are Too Stupid To Know the Right Answers (science.slashdot.org)
- Science Blogger Finds Problems With FCAT Questions (grumpyelder.com)
- FCAT stress ramps up after state raises passing score for students, ties teacher evaluations to kids’ performance (tcpalm.com)