Tag Archives: assessment

Scaling up SBG for the New Year

In my new school, the mean size of my classes has doubled. The maximum size is now 22 students, a fact about which I am not complaining. I've missed the ease of getting students to interact with simple proximity as the major factor.

I have also been given the freedom to continue with the standards based grading system that I've used over the past four years. The reality of needing to adapt my systems of assessment to these larger sizes has required me to reflect upon which aspects of my system need to be scaled, and what (if anything) needs to change.

The end result of that reflection has identified these three elements that need to remain in my system:

  • Students need to be assessed frequently through quizzes relating to one to two standards maximum.
  • These quizzes need to be graded and returned within the class period to ensure a short feedback cycle.
  • There must still be a tie between work done preparing for a reassessment and signing up for one.

Including the first element requires planning ahead. If quizzes are going to take up fifteen to twenty minutes of a class block, the rest of the block needs to be appropriately planned to ensure a balance between activities that respond to student learning needs, encourage reinforcement of old concepts, and allow interaction with new material. The second element dictates that those activities need to provide me time to grade the quizzes and enter them as standards grades before returning them to students. The third happens a bit later in the cycle as students act on their individualized needs to reassess on individual standards.

The major realization this year has been a refined need for standards that can be assessed within a twenty minute block. In the past, I've believed that a quiz that hits one or two aspects of the topic is good enough, and that an end of unit assessment will allow complete assessment on the whole topic. Now I see that a standard that has needs to have one component assessed on a quiz, and another component assessed on a test, really should be broken up into multiple standards. This has also meant that single standard quizzes are the way to go. I gave one quiz this week that tested a previously assessed standard, and then also assessed two new ones. Given how frantic I was in assessing mastery levels on three standards, I won't be doing that again.

The other part of this first element is the importance of writing efficiently targeted assessment questions. I need students to arrive at a right answer by applying their knowledge, not by accident or application of an algorithm. I need mistakes to be evidence of misunderstanding, not management of computational complexity. In short, I need assessment questions that assess what they are designed to assess. That takes time, but with my simplified schedule this year, I'm finding the time to do this important work.

My last post was about my excitement over using the Numbas web site to create and generate the quizzes. A major bottleneck in grading these quizzes quickly in the past has been not necessarily having answers to the questions I give. Numbas allows me to program and display calculated answers based on the randomized values used to generate the questions.

Numbas has a feature that allows students to take the exam entirely online and enter their answers to be graded automatically. In this situation, I have students pass in their work as well. While I like the speed this offers, that advantage primarily exists in cases where students answer questions correctly. If they make mistakes, I look at the written work and figure out what went wrong, and individual values require that I recalculate along the way. This isn't a huge problem, but it brings into question the need for individualized values which are (as far as I know right now) the only option for the fully online assessment. The option I like more is the printed worksheet theme that allows generation of printable quizzes. I make four versions and pass these out, and then there are only four sets of answers to have to compare student work against.

With the answers, I can grade the quizzes and give feedback where needed on wrong answers in no more than ten or fifteen minutes total. This time is divided into short intervals throughout the class block while students are working individually. The lesson and class activities need to be designed to provide this time so I can focus on grading.

The third element is still under development, but my credit system from previous years is going to make an appearance. Construction is still underway on that one. Please pardon the dust.


P.S:

If you're an ed-tech company that wants to impress me, make it easy for me to (a) generate different versions of good assessment questions with answers, (b) distribute those questions to students, (c) capture the student thinking and writing that goes with that question so that I can adjust my instruction accordingly, and (d) make it super easy to share that thinking in different ways.

That step of capturing student work is the roughest element of the UX experience of the four. At this time, nothing beats looking at a student's paper for evidence of their thinking, and then deciding what comes next based on experience. Snapping a picture with a phone is the best I've got right now. Please don't bring up using tablets and a stylus. We aren't there yet.

Right now there are solutions that hit two or three, but I'm greedy. Let me know if you know about a tool that might be what I'm looking for.

Numbas and Randomized Assessment

At the beginning of my summer vacation, I shared the results of a project I had created to fill a need of mine to generate randomized questions. I subsequently got a link from Andrew Knauft (@aknauft) about another project called Numbas that had similar goals. The project is out of Newcastle University and the team is quite interested in getting more use and feedback on the site.

You can find out more at http://www.numbas.org.uk/. The actual question editor site is at https://numbas.mathcentre.ac.uk/.

Screen Shot 2016-08-22 at 8.23.01 PM

I've used the site for a couple of weeks now for generating assessments for my students. I feel pretty comfortable saying that you should be using it too, and in place of my own QuestionBuilder solution. I've taken the site down and am putting time into developing my own questions on Numbas. Why am I so excited about it?

  • It has all of the randomization capabilities of my site, along with robust variable browsing and grouping, conditions for variable constraints, and error management in the interface that I put on the back burner for another day. Numbas has these features right now
  • LaTEX formatting is built in along with some great simplification functions for cleaning up polynomial expressions.
  • Paper and online versions (including SCORM modules that work with learning management sites like Moodle) are generated right out of the box.
  • It's easy to create, share, and copy questions that others have created and adapt them to your own uses.
  • Visualization libraries, including Geogebra and Viz.js, are built in and ready to go.
  • The code is open sourced and available to install locally if you want to do so.

I have never planned to be a one-person software company. I will gladly take the output of a team of creative folks that know what they are doing with code over my own pride, particularly when I am energized and focused on what my classroom activities will look like tomorrow. The site makes it easy to generate assessments that I can use with my students with a minimal amount of friction in the process.

I'll get more into the details of how I've been using Numbas shortly. Check out what they've put together - I'm sure you'll find a way to include it in part of your workflow this year.

QuestionBuilder: Create and Share Randomized Questions

I've written previously about my desire to write randomized questions for the purpose of assessment. The goal was never to make a worksheet generator - those exist on the web already. Instead, I wanted to make it easy to create assessment questions that are similar in form, but different enough from each other that the answers or procedures to solve them are not necessarily identical.

Since January, I've been working on a project called QuestionBuilder. It's a web application that does the following:

  • Allows the creation of assessment questions that contain randomized elements, values, and structures.
  • Uses regular Javascript, HTML, and the KaTEX math rendering library to create and display the questions
  • Makes it easy to share questions you create with community members and build upon the work of others to make questions that work for you.

example1

Here's a video in which I convert a question from the June 2016 New York State Regents exam for Algebra 2 Common Core into a randomized question. Without all of my talking, this is a quick process.

I've put a number of questions on the site already to demonstrate what I've been using this to do. These range from simple algebra to physics questions. Some other folks I appreciate and respect have also added questions in their spare time.

For now, you'll need to create an account and log in to see these questions in action. Go to http://question-builder.evanweinberg.org, make an account, and check out the project as it exists at this point.

My hope is to use some time this summer to continue working on it to make it more useful for the fall. I'll also be making some other videos to show how to use the features I've added thus far. Feel free to contact me here, through Twitter (@emwdx), or by email (evan at evanweinberg.com) if you have questions or suggestions.

Choosing the Next Question

If a student can solve 3x - 1 = 5 for x, how convinced are we of that student's ability to solve two step equations?

If that same student can also solve 14 = 3x + 2 , how does our assessment of their ability change, if at all?

What about -2-3x= 5 ?

Ideally, our class activities push students toward ever increasing levels of generalization and robustness. If a student's method for solving a problem is so algorithmic that it fails when a slight change is made to the original problem, that method is clearly not robust enough. We need sufficiently different problems for assessing students so that we know their method works in all cases we might through their way.

In solving 3x-1 = 5 , for example, we might suggest to a student to first add the constant to both sides, and then divide both sides by the coefficient. If the student is not sure what 'constant' or 'coefficient' mean, he or she might conclude that the constant is the number to the right of the x, and the coefficient is the number to the left. This student might do fine with 10 =2x-4 , but would run into trouble solving -2-3x = 5 . Each additional question gives more information.

The three equations look different. The operation that is done as a first step to solving all three is the same, though the position of the constant is different in all three. Students that are able to solve all three are obviously proficient. What does it mean that a student can solve the first and last equations, but not the middle one? Or just the first two? If a student answers a given question correctly, what does that reveal about the student's skills related to that question?

It's the norm to consider these issues in choosing questions for an assessment. The more interesting question to me theses days is that if we've seen what a student does on one question, what should the next question be? Adaptive learning software tries to do this based on having a large data set that maps student abilities to right/wrong answers. I'm not sure that it succeeds yet. I still think the human mind has the advantage in this task.

Often this next step involves scanning a textbook or thinking up a new question on the spot. We often know the next question we want when we see it. The key then is having those questions readily available or easy to generate so we can get them in front of students.

Standards Based Grading(SBG) and The SUMPRODUCT Command

I could be very late to the party finding this out. If so, excuse my excitement.

I gave a multiple choice test for my IB Physics course last week. Since I am using standards based grading (SBG), I wanted a quick way to see how students did on each standard. I made a manually coded spreadsheet eight years or so ago to do this. It involved multiple columns comparing answers, multiple logical expressions, and then a final column that could be tallied for one standard. Multiply that by the total number of standards...you get the drill.

I was about to start piecing together an updated one together using that same exhausting methodology when I asked myself that same question that always gets me going: is there a better way?

Of course there is. There pretty much always is, folks.

For those of you that don't know, the SUMPRODUCT command in Excel does exactly what I was looking for here. It allows you to add together quantities in one range that match a set of criteria in another. Check out the example below:

Screen Shot 2014-10-14 at 3.28.09 PM

The column labeled 'Response Code' contains the formula '=1*(B6=E6)', which tests to see if the answer is correct. I wanted to add together the cells in F6 to F25 that were correct (Response Code = 1) and had the same standard as the cell in H6. The command in cell I6 is '=SUMPRODUCT((F6:F25)*(E6:E25=H6))'. This command is equivalent to the sum F6*(E6=H6) + F7*(E7=H6)+F8*(E8=H6)+...and so on.

If I had known about this before, I would've been doing this in some way for all of my classes in some way since moving to standards based grading. I've evaluated students for SBG after unit exams in the past by looking at a student's paper, and then one-by-one looking at questions related to each standard and evaluating them. The problem has been in communicating my rationale to students.

This doesn't solve the problem for the really great problems that are combinations of different standards, but for students that need a bit more to go on, I think this is a nice tool that (now) doesn't require much clerical work on my part. I gave a print out of this page (with column F hidden) to each student today.

Here is a sample spreadsheet with the formulas all built in so you can see how it works. Let me know what you think.
Exam Results Calculator

The perils of playing cards and probability. What do you assume your students know?

One of the topics taught in the first semester of my first semester teaching was probability. Flipping coins and rolling dice both serve to bring the kids to understand how it is used in games, but the first thing a couple teachers told me to do once they got the basics was to go to playing cards. This seemed like a natural fit to get the students excited - I figured they had seen people playing cards on the street as I had seen countless times wandering around the city. There are also so many opportunities to talk about intersection and union of sets. How many cards are hearts or face cards? How many are hearts and face cards? Sounded like a good idea to me.

When I actually did this with my class the first time, there were a couple really big issues that came up. Being a new teacher, I wasn't as strong in terms of preventing students from calling out answers. When I did write up some fairly simple questions on the board (such as find P(red card) if a single card is selected) the enthusiasm for three or four students in answering these led me to believe that this small sample was representative of the class. If these four knew it (or so assumed my naive first year teacher brain), the rest probably knew it, but just didn't feel comfortable answering. This was a ridiculously inaccurate assumption. In fact, I think it's a painfully clear example of self-selection bias that all teachers should consider when asking any question of an entire class. Who is going to raise their hand for the purposes of establishing that he or she does not know what I am talking about?

Another issue appeared when I started walking around the room during independent work. I saw that the students were struggling both with the idea of probability AND with the details of the different types of cards. It was hard separating the two bodies of knowledge because I had framed the topic only in terms of these concepts. Students that didn't understand what the various cards were couldn't answer the questions because they couldn't figure out which cards were desired outcomes. Students that didn't get probability in general didn't understand how the sample space and the desired outcomes fit together to calculate theoretical probability. Some didn't understand either idea.

After the class, I talked to a few teachers about it. One said a phrase that makes my blood boil every time I hear it: "Come on - they really should know _______". In this case, the phrase in the blank was "the types of playing cards." The assertion that there is something wrong with a student because he or she doesn't know an arbitrary fact is not an argument we should be making. The biggest reason it is a problem is this: if your lesson predicates itself on students knowing a fact, and you haven't made any effort to establish as part of your lesson whether or not students actually know that fact, your lesson is going to backfire. Hard. It will be like pulling your own teeth while simultaneously telling your students "look, you can do it too!"

I understood more about this in talking to my mentor teacher. He pointed out that using playing cards is one of the worst ways teachers could teach probability because of the cultural bias inherent in assuming students have the required background knowledge. Reasons why:

  • Alright kids, we're going to do some probability, but make sure you know what these words mean first, because I'm going to be using them all with the assumption you do: suit, hearts, diamonds, clubs, spades, face card, king, queen, jack, ace, joker. Don't forget that there are red cards and black cards.
  • Wait, English isn't your first language? OK, so spend your time learning these words in addition to the math content terms I really want you to learn: probability, sample space, and outcome. Uh...I guess that learning this esoteric set of words will be good for you because it will help you understand spoken English better. The more words you know, the better your English, right?
  • What's that? How can you not understand that something can be a face card and a club? Face cards are jacks, queens, kings, and aces - get it? And there are four different suits, so there have to be four face cards that are also clubs - get it?. Well, yes, spades are also black, but clubs are black and have the little clover shape. Yes, the symbol tells you the suit. No, the card doesn't actually say "spades" or "hearts". But it's easy because the heart is for hearts, the diamond is for diamond, and well, you might just have to remember the others. Oh wait, the spade is shaped like a shovel - did you know shovels are sometimes called spades? That will help you remember it. Get it? [By now, the student is nodding to get you out of his face.]
  • So now that we've covered all the vocabulary, what is the probability of randomly picking a card that has a value of 10 or greater? Oh, you don't know about the value of cards? Sure, well that's just fine. Obviously the jack is above the 10 because it has a guy on the front. It has the lowest value of the face cards, because the queen and king are higher. The king is of higher value than the queen because of the patriarchal culture that has dominated the globe for, well, forever. And then there's the ace. Sometimes the ace is the highest card. Other times it has the lowest value. That's life. Who has an answer?

How much math content has actually been explored during this entire (imagined) dialogue? Furthermore, if we assume that playing cards is an engaging and authentic application of probability, shouldn't understanding the math content be made easier by the presence of all of this extra knowledge? Think about the reverse situation - should a student that knows her probability, but does not know the details of the card system, get a 50% on a quiz of this topic in a math class?

I don't know about you, but I didn't actually play cards that much as a kid. It's a dangerous assumption to make that all kids have. If you don't know if your students have this knowledge or not, and don't want to guess from looking at them (which is always good policy not to do), and don't want to spend class time reviewing, it probably isn't a good idea.

One of the other teachers with whom I discussed this issue gave out a reference sheet with all of the vocabulary, pictures, and cards in order of value, and let them use it for quizzes and tests that included this topic. I think that's fine. An even better solution though? Find a topic that doesn't require so much background knowledge. Flip a coin and roll a 20 sided die. Put numbers on index cards, throw them in a bag, and ask for probabilities of drawing a card that is even or prime. At least in that case, students need to use mathematical knowledge to classify the outcomes. That's what you want to assess anyway, right?

Making connections to background knowledge is one of the most powerful ways to help students learn. Making assumptions about what background knowledge students have is an easy way to make a lesson a dud. Assess, don't assume.

Having conversations about and through homework

I've been collecting homework and checking individual problems this year. I grade it on completion, though if students tell me directly that they had trouble with a question before class (and it is obvious it isn't a case of not being able to do ANY of it because they waited until the last minute to try) I don't mind if they leave some things blank. I did this in the beginning since I had heard there were students that tried to skip out on doing homework if it wasn't checked. We do occasionally go over assigned problems during class, but I tend not to unless students are really perplexed by something.

I have lots of opinions on homework and its value. Some can use the extra practice and review of ideas developed in class. Some need to use homework time to make the material their own. In some cases, it gives students a chance to develop a skill, but in those cases I insist that students have a reliable resource nearby that they know how to use (textbook, Wolfram Alpha, Geogebra) to check their work. I don't think it is necessary to assign it just to "build character" or discipline. I read Alfie Kohn's The Homework Myth, and while I did find myself disagreeing with some aspects of his arguments, it did make me think about why I assign it and what it is really good for. I do not assign busy work, nor do I assign 1 - 89 - each problem I assign is deliberately chosen.
Among the many ways I try to assess my students, I admit that homework doesn't actually tell me that much about the skill level of a student. Why do I do it then?

My reason for assessing homework is for one selfish reason, and I make no secret of it with my students:

The more work I see from students relating to a concept, the better I get at developing that concept with students.

I would love to say that I know every mistake students are going to make. I know many of them. If I can proactively create activities that catch these misconceptions before they even start (and even better, get students talking about them) then the richness of our work together increases astronomically. You might ask why I can't get this during conversation or circulation with students during the class period. I always do get some insight this way. The difference is that I can have a conversation with the student at that point about their thinking because he or she is in the room with me. I can push them in the right direction in that situation if the understanding is off. The key is that most of my students are alone when they do their work, or at least, have only online contact with their classmates. In that situation, I can really see what students do when they are faced with a written challenge. The more I see this work, the better I get.

I am not worried about students copying - if they do it, it always sticks out like a sore thumb. Maybe they just aren't good at copying. Either way, I don't have any cases of students that say 'I could do it in the homework, but can't do it when it comes to quizzes or tests.' Since I can see clearly when the students can/can't do it in the homework, I can immediately address the issue during the next class.

The other thing I have started doing is changing the type of feedback I give students on homework. I still fall into the habit of marking things that are wrong with an 'x' when I am not careful. I now try to make all feedback a question or statement, as if I am starting a conversation with a student about their work through my comments, whether positive or negative:

  • Great explanation using definition here.
  • Does x = 7 check in the original equation? (This rather than marking an x when a solution is clearly wrong.)
  • (pointing out two correct steps and then third with an error) - mistake is in here somewhere.
  • You can call "angle CPK"  "angle P" here.
  • Good use of quotient rule - can you use power rule and get the same answer?

The students that get papers back with ink on them don't necessarily have wrong answers - they just have more I can chat with them about on paper. The more I can get the students to understand that the homework is NOT about being right or wrong, but about the quality of their mathematical thinking, I think we are all better off.

This does take time, but it is so valuable to me, and I think the students not only benefit from the feedback, but appreciate the effort on my part. I don't check every problem, just key ones that I know might cause trouble. If a student has everything right on the questions I am checking, it's a chance to give feedback on one of the others. If there's nothing to say because the paper is perfect (which is rare), I can praise the student for both their clear written solutions, hard work, and attention to detail.

I decided at the beginning of this year to look at more student work, and checking homework in this way is letting me do this. I am lucky to have prep time in the morning, and I have committed to using morning time for looking at student work almost exclusively. I have had to force myself to do this on many mornings because it's so easy to use the time for other things. Some of my best ideas and modifications to lessons come after seeing ten students make the same mistake - it feels good to custom fit my lessons to the group of students I have in front of me.

In the end, it's just one more way the students benefit from having a real teacher working with them instead of a computer. Every mark I make on the paper is another chance to connect with my students and conversation that can help make them better thinkers and learners. I don't think I really need to justify my presence in the classroom, but it feels good to say that this is one of the reasons it's good I'm there.