Annotation options
There are a few options that are common to all annotations; in the last module, you learned about the success_message and failure_message options. In this module, we discuss three more options that can be applied to annotations and how they can be used.
name
The name option is used to group different instances of annotation classes that represent the same annotation together. This option is used to prevent messages from being displayed multiple times when they don't need to be. Let's consider the following example: the maximum function calls Python's max function to check that the student correctly identified the maximum, but one success message is printed for each input the function is tested on.
>>> max_ref = []
>>> def maximum(l, track=False):
... m = max(l)
... if track:
... max_ref.append(pybryt.Value(
... m,
... success_message="Found the max!",
... failure_message="Did not find the max",
... ))
... return m
>>> test_lists = [[1, 2, 3], [-1, 0, 1], [10, -4, 2, 0], [1]]
>>> for test_list in test_lists:
... maximum(test_list, track=True)
>>> max_ref = pybryt.ReferenceImplementation("maximum", max_ref)
>>> with pybryt.check(max_ref):
... for test_list in test_lists:
... maximum(test_list)
REFERENCE: maximum
SATISFIED: True
MESSAGES:
- Found the max!
- Found the max!
- Found the max!
- Found the max!
The problem with this function is simple: the annotation being created in each test is fundamentally checking the same thing, whether the student returned the correct value. Having the same message printed multiple times makes it seem like the annotations are testing different conditions and clutters the report generated by PyBryt. We can collapse all of these messages together by naming the annotation created in the maximum function:
>>> max_ref = []
>>> def maximum(l, track=False):
... m = max(l)
... if track:
... max_ref.append(pybryt.Value(
... m,
... name="list-maximum",
... success_message="Found the max!",
... failure_message="Did not find the max",
... ))
... return m
>>> test_lists = [[1, 2, 3], [-1, 0, 1], [10, -4, 2, 0], [1]]
>>> for test_list in test_lists:
... maximum(test_list, track=True)
>>> max_ref = pybryt.ReferenceImplementation("maximum", max_ref)
>>> with pybryt.check(max_ref):
... for test_list in test_lists:
... maximum(test_list)
REFERENCE: maximum
SATISFIED: True
MESSAGES:
- Found the max!
Now, we can see that the message is only printed once.
When PyBryt collapses the annotations into a single message, it only displays the success message if all of the annotations in the name group are satisfied. If any test in the group fails, it displays the failure message instead. Let's introduce a bug into maximum to demonstrate:
>>> def maximum(l):
... if len(l) % 2 == 0:
... m = min(l)
... else:
... m = max(l)
... return m
>>> with pybryt.check(max_ref):
... for test_list in test_lists:
... maximum(test_list)
REFERENCE: maximum
SATISFIED: False
MESSAGES:
- Did not find the max
limit
The limit option allows you to control how many copies of named annotations are included in the reference implementation. This option helps for cases in which the functions constructing the annotations are reused many times throughout an assignment. Cases where a few initial tests are sufficient for checking the validity of the implementation by reducing the size of the reference implementation itself.
Let's illustrate this using our maximum function. Here, we use a similar implementation to the previous reference but set limit to five annotations and test it on several input lists.
>>> max_ref = []
>>> def maximum(l, track=False):
... m = max(l)
... if track:
... max_ref.append(pybryt.Value(
... m,
... name="list-maximum",
... limit=5,
... success_message="Found the max!",
... failure_message="Did not find the max",
... ))
... return m
>>> for _ in range(1000):
... test_list = np.random.normal(size=100)
... maximum(test_list, track=True)
>>> print(f"Annotations created: {len(max_ref)}")
>>> max_ref = pybryt.ReferenceImplementation("maximum", max_ref)
>>> print(f"Annotations in reference: {len(max_ref.annotations)}")
Annotations created: 1000
Annotations in reference: 5
As you can see, the length of max_ref.annotations is 5 even though 1,000 annotations were included in the list passed to the constructor.
group
The group option is similar to the name option in that it's used to group annotations together, but these annotations don't necessarily represent the "same annotation"; instead, they're grouped into meaningful chunks so that specific portions of references can be checked one at a time instead of all at once. This option can be useful in constructing assignments with multiple questions in PyBryt.
For example, consider an assignment that asks students to implement a mean and median function. You may divide it up into two questions like so:
# Question 1
mean_ref = []
def mean(l, track=False):
size = len(l)
if track:
mean_ref.append(pybryt.Value(
size,
name="len",
group="mean",
success_message="Determined the length of the list",
))
m = sum(l) / size
if track:
mean_ref.append(pybryt.Value(
m,
name="mean",
group="mean",
success_message="Calculated the correct mean of the list",
failure_message="Did not find the correct mean of the list",
))
return m
# Question 2
median_ref = []
def median(l, track=True):
sorted_l = sorted(l)
if track:
median_ref.append(pybryt.Value(
sorted_l,
name="sorted",
group="median",
success_message="Sorted the list",
))
size = len(l)
if track:
mean_ref.append(pybryt.Value(
size,
name="len",
group="median",
success_message="Determined the length of the list",
))
middle = size // 2
is_set_size_even = size % 2 == 0
if is_set_size_even:
m = (sorted_l[middle - 1] + sorted_l[middle]) / 2
else:
m = sorted_l[middle]
if track:
mean_ref.append(pybryt.Value(
m,
name="mean",
group="mean",
success_message="Calculated the correct mean of the list",
failure_message="Did not find the correct mean of the list",
))
return m
test_lists = [[1, 2, 3], [-1, 0, 1], [10, -4, 2, 0], [1]]
for test_list in test_lists:
mean(test_list, track=True)
median(test_list, track=True)
assignment_ref = pybryt.ReferenceImplementation("mean-median", [*mean_ref, *median_ref])
With a reference constructed as in the previous example, we can give students a chance to check their work on each individual question before moving on to the next by telling PyBryt which group of annotations to consider:
>>> with pybryt.check(assignment_ref, group="mean"):
... for test_list in test_lists:
... mean(test_list)
REFERENCE: mean-median
SATISFIED: True
MESSAGES:
- Determined the length of the list
- Calculated the correct mean of the list
>>> with pybryt.check(assignment_ref, group="median"):
... for test_list in test_lists:
... median(test_list)
REFERENCE: mean-median
SATISFIED: True
MESSAGES:
- Determined the length of the list
- Sorted the list
>>> with pybryt.check(assignment_ref):
... for test_list in test_lists:
... mean(test_list)
... median(test_list)
REFERENCE: mean-median
SATISFIED: True
MESSAGES:
- Determined the length of the list
- Calculated the correct mean of the list
- Sorted the list