There is a common misconception when building assessments where quantitative questions are thought to have ‘only 1 right answer’ and are therefore more difficult to deliver online in a manner that limits the opportunity for collaboration. It is true that specific calculations will result in unique correct answers, however the scenario behind the calculation as well as the variables themselves can vary in many ways. The key is to design questions with variation opportunities in mind while maintaining the Learning Outcome (LO) goal of the assessment. If you’re already investing time to make your assessment compatible with online delivery, consider also investing in designing questions so they can be more easily updated year-to-year.
Build Variation into Your Questions
A single question can have a variety of unique answers by including opportunities for scenario and variable variation. Consider framing the assessment in alternative ways that still satisfy the key LO. For example, a learned skill can be applied in several different, yet similar, scenarios; Calculation steps to resolve a challenge would still apply regardless if the initial variables differ. For specific examples, review the below questions and potential variations to see how certain elements can be altered to achieve equally unique questions and answers. Note that developing assessments with built in opportunities for variation is an iterative process that will improve with time and practice, but also sets a great design foundation for future terms.
Keep Your Questions Current & Encourage Academic Integrity
When designing any assessment, whether online or in person, it is not only good practice to keep your course content fresh, but doing so can discourage potential academic dishonesty from referencing passed assessments. Consider the many resources students may have access to, including the internet (e.g., Course Hero, reused textbook Q/As, etc.). Don’t rely on a physical testing environment or one that can be proctored to ensure academic integrity. The reality is that any course content can be copied or made available online eventually, regardless of the learning environment. As a result, avoid or alter assignments from previous terms and textbook Q/As. Also consider students searching for passed exams with honest intentions, like those looking for opportunities to review and practice their understanding of course content. Rather than fighting an up-hill battle to prevent online sharing or collaboration, design assessments in ways that will ensure academic integrity regardless.
Use Question Banks & Other Canvas Features to Generate Unique Assessments for All Students
Take advantage of your online learning environment and its helpful built-in tools, such as Canvas and the ‘New Quizzes’ feature. With New Quizzes, you can set a variety of question types ranging from short answer to multiple choice and pre set response matches. You can also build question banks where students complete a set of randomly selected questions. You can take advantage of question banks by creating variations of the same question, and randomly selecting one to complete. This will maintain the key LO of the question while minimizing the potential for unauthorized collaboration. New Quizzes also includes a ‘stimulus’ option, allowing you to group a multi-part question and set auto grading for parts of the assessment to save on grading time. See Optimizing Canvas Quizzes for Open Book Exams for more quiz settings and related suggestions.
Assessment Design Examples
The following examples will help highlight some of the ways quantitative questions can adapt from the face-to-face classroom to the online environment while maintaining key learning outcomes and academic integrity. They will also review elements of Canvas’ New Quizzes tool, building assessments with a variety of questions styles ranging from multiple choice and auto matching numerical answers to short answer texts. These examples focus on the ‘question banks’ feature and related settings such as question / answer ‘scrambling’.
Special thank you to Schulich lecturer, Hjalmar Turesson, Master of Business Analytics and Master of Management in Artificial Intelligence, for kindly providing the example questions.
Hjalmar Turesson
Option 1: Revise the Scenario, Not the Final Answer
Variations of a question involving different scenarios can give the illusion of a unique challenge, making it more difficult to collaborate or compare the initial question text, especially in a timed scenario. The goal of these assessments is to identify key information regardless of the scenario. Review the following examples assessing LOs related to Natural Language Processing (NLP) and Discrete Events Simulation (DES). Notice the red text in each example can be altered easily to create a variety of similar questions.
Example Question #1: Natural Language Processing (NLP)
Version 1: A textual data set is stored in a folder containing 2,367 text files. Each text file holds a movie review from Rotten Tomatoes. What are the terms used in NLP for the whole data set as well as the individual text files? Select the correct answer:
A: document, words | B: corpora, tokens | C: corpus, documents | D: documents, tokens
Version 2: A textual data set is stored as a CSV file. Each row holds a 2-3 sentence long restaurant review from Yelp! What are the terms used in NLP for the whole data set as well as the individual reviews? Select the correct answer:
A: document, words | B: corpora, tokens | C: corpus, documents | D: documents, tokens
Notice: The answer options (A-D) and correct answer (C) are the same for both versions of Example Question #1. However, the version scenarios differ enough that you would need to understand the question and key LO to still arrive at the correct answer.
Example Question #2: Discrete Events Simulation (DES)
Version 1: You are working with a Discrete Events Simulation (DES) to optimize the customer experience at a fast-food chain. The main events in your simulation are the arrival of a customer, taking an order, serving a meal and a customer leaving before ordering because the line is too long. When running the simulation, how often is the system state updated? Select the correct answer:
- Every 1 second. [Distraction option]
- Every 1 minute. [Distraction option]
- Every 1 hour. [Distraction option]
- At every instance of the four types of events. [Correct]
- Only on the arrival of a customer. [Content specific]
- Insufficient information to answer this. [‘I don’t know’]
Version 2: You are working with a Discrete Events Simulation (DES) to optimize the customer experience at a train station. The main events in your simulation are the arrival of a traveler to the ticket booth, the sale of tickets, and the departure of a traveler by train. When running the simulation, how often is the system state updated? Select the correct answer:
- Every 1 minute. [Distraction option]
- Every 15 minutes. [Distraction option]
- At some constant interval dt, set to best capture the dynamics of a train station. [Content specific]
- Only on the sale of a ticket. [Content specific]
- At every instance of the three types of events. [Correct]
- Insufficient information to answer this. [‘I don’t know’]
Quick Design Tip: When building a question with multiple answer options, such as multiple choice, you can make the question more challenging by including distraction options and content specific options. See the provided options in the DES question above for examples. Where able, you can also include an ‘insufficient information’ option which can double as an ‘I don’t know’ selection, even when all required information is available. If this option is selected, it can help identify students who may not understand the assessment rather than selecting another option at random. This can also help identify where to focus more attention in lesson materials to help solve similar future assessments.
Quick Time Saver Tip: When building variations on a question in Canvas, you can save time by duplicating and adjusting the initial question. First, fully set up the initial question, including all answer options and settings (ex: any auto feedback, order scrambling, etc.). Then, duplicate the question. Revise the scenario details and answer options (as needed) of the duplicated question. Repeat for the number of desired variations. Ensure all variations share the same question bank. You can then randomly select 1 question from that bank. The more variations available in the bank, the less likely students will receive the same question for the related learning outcome. Note, be sure to preview your quizzes to ensure they behave as expected!
Option 2: Revise the Variables (and optionally the Scenario) for Unique Answers
Different approaches to the same scenario can reinforce one’s understanding and practiced skills to achieve the related LO. Review the following versions of the Markov Decision Process (MDP) example and related diagram. This question requires students to estimate the value function for any given state where the provided variables and paths may vary. By using the same diagram, but solving for unique states, you can quickly create a variety of questions with unique answers to assess the same LO. Notice the variables in the yellow highlighted text can be easily altered in each version to create a variety of similar questions.
Example Question #1: Markov Decision Process (MDP) & Diagram
Image: Markov Decision Process Diagram
(select to enlarge) © Hjalmar Turesson 2020
|
Version 1: Given the Markov Decision Process (MDP) provided, estimate the value function, |
|
Version 2: Given the Markov Decision Process (MDP) provided, estimate the value function, |
|
Version 3: Given the Markov Decision Process (MDP) provided, estimate the value function, |
Notice: Optionally updating the initial diagram in addition to the question variations will give a new set of unique answers!
Quick Canvas Quiz Tip: When versions of a question have a unique answer, such as the numerical answers above, you can save grading time by setting them to auto grade / award points if the submitted answer matches with the preset answer.
Example Question #2: Byte-Pair Encoding
Similar to the example ‘MDP & Diagram’ question above where you can vary items in the requested action, you can also vary the initial information provided. Review the following versions of the Byte-Pair Encoding example. Both versions require students to apply a random number of iterations (same LO), and also provides a random set of information to ensure unique correct responses. Notice the red text in each version (i.e. any number of iterations, a variety of words, and different counts) can be altered easily to create a variety of similar questions while maintaining the same key LO.
Version 1: Apply 4 iterations of byte-pair encoding to the following dictionary (i.e. vocabulary with frequency counts). What vocabulary did you produce?
| Word | Count |
| dark | 2 |
| darkest | 3 |
| test | 2 |
| newer | 6 |
| old | 2 |
Version 2: Apply 3 iterations of byte-pair encoding to the following dictionary (i.e. vocabulary with frequency counts). What vocabulary did you produce?
| Word | Count |
| light | 4 |
| lighter | 3 |
| live | 4 |
| older | 6 |
| new | 1 |
Quick Canvas Quiz Tip: If using Canvas ‘New Quizzes’, this type of question can be set up in both manual and auto grading formats. Ex: You can manually grade responses with the ‘Essay’ format, where users type a response. Alternatively, Canvas can help auto grade with predetermined formats, such as ‘Multiple Choice’. For auto graded formats, along with the correct answer, you can optionally including ‘distraction’ options such as common calculation mistakes, and give related feedback if selected.
Final Thoughts
Design with variation opportunities in mind, and don’t lose sight of your key learning outcomes. It is important to keep your questions current. Variety in variables and scenarios can deter collaboration while encouraging the need for skill building to understand and complete given assessments. Note that building larger question banks can decrease academic honesty concerns, but also add to instructor workload to both build and review. However, doing so may also be necessary to achieve your LOs. Find an appropriate balance between student and instructor workloads to ensure no one is overwhelmed, and work around the ‘only 1 right answer’ mentality. Don’t lose sight of your LOs as the key goal and continue to think creatively with your assessments. Good luck!