You’ve been there, right? You spend hours, maybe even days, writing dozens of unit tests for a new function. You cover the happy path, the sad path, and every weird edge case you can think of. You hit 100% code coverage, push to production, and then… a bug report comes in for a scenario you never even considered.
It’s frustrating. It feels like you’re playing a game of whack-a-mole, and the moles have a PhD in hiding.
For years, I thought this was just the reality of software development. But then I discovered a different way of thinking about testing, a technique that felt less like manual labor and more like teaching a computer how to be the world's most creative (and destructive) QA engineer. It’s called property-based testing, and the tool that makes it a joy to use in Python is called Hypothesis.
Let's talk about this testing superpower. Instead of writing a test that says, assert my_function(5) == 10, you write a test that describes a property of your function, like, "for any integer x, the output of my_function(x) should always be an even number." Then, you let Hypothesis loose. It will intelligently generate hundreds or even thousands of inputs—positives, negatives, zeros, huge numbers, tiny numbers—to try and prove you wrong.
When it finds an input that breaks your property, it does something magical. It doesn't just show you the giant, complex input that failed. It simplifies it, shrinking it down to the absolute smallest, simplest example that still causes the bug. It’s like having a brilliant assistant who not only finds the problem but also hands you the exact clue you need to solve it.
So, How Does This Actually Work?
Alright, let's get our hands dirty. The core idea is to move from "testing examples" to "testing rules" or "properties."
Think about a simple function that sorts a list. A traditional unit test might look like this:
assert sort_my_list([3, 1, 2]) == [1, 2, 3]
A property-based test would think about the rules of a sorted list:
- The output list should have the same length as the input list.
- The output list should contain the exact same elements as the input list.
- For any element in the output list, it should be less than or equal to the element that comes after it.
We can teach Hypothesis these rules, and it will test them against every kind of list it can dream up: empty lists, lists with one item, lists with duplicate items, lists with negative numbers, you name it.
Let's look at some real code. Imagine we have a simple clamp function that keeps a number within a certain range.
def clamp(x: int, lo: int, hi: int) -> int:
if x < lo: return lo
if x > hi: return hi
return x
How would we test this with Hypothesis? We wouldn't just check clamp(5, 0, 10). We’d define the properties.
One obvious property is that the output should always be within the bounds lo and hi. Here's what that test looks like:
from hypothesis import given
from hypothesis import strategies as st
# A strategy to generate valid (low, high) bounds
bounds = st.tuples(st.integers(), st.integers()).map(sorted)
@given(x=st.integers(), b=bounds)
def test_clamp_within_bounds(x, b):
lo, hi = b
y = clamp(x, lo, hi)
assert lo <= y <= hi
See what's happening here? We’re not picking the numbers. We’re telling Hypothesis what kind of numbers to pick. st.integers() tells it to generate any integer for x, and our bounds strategy generates pairs of integers for the low and high bounds. The @given decorator then runs our test function over and over with different generated values.
Another cool property of clamp is that it’s idempotent. That’s a fancy word that just means if you apply the function twice, you get the same result as applying it once. Clamping a number that's already clamped shouldn't change it.
@given(x=st.integers(), b=bounds)
def test_clamp_idempotent(x, b):
lo, hi = b
y = clamp(x, lo, hi)
assert clamp(y, lo, hi) == y
With just these two small tests, we have way more confidence in our clamp function than we would with a dozen manually written examples.
Pitting Your Code Against Itself: Differential Testing
This is one of my favorite techniques. It’s perfect for when you’re refactoring code or writing a new, optimized version of an old function.
The idea is simple: you have two versions of a function that are supposed to do the same thing. One might be a simple, slow, but obviously correct version (the "reference"), and the other is your new, fast, and complex version. Differential testing just generates the same random input for both and asserts that they always produce the same output.
Let's say we wrote a fancy merge_sorted function to merge two sorted lists.
# Our new, clever implementation
def merge_sorted(a, b):
# ... some clever merging logic ...
# A simple, obviously correct reference implementation
def merge_sorted_reference(a, b):
return sorted(list(a) + list(b))
The test is beautifully straightforward:
# A strategy to generate sorted lists of integers
sorted_lists = st.lists(st.integers()).map(sorted)
@given(a=sorted_lists, b=sorted_lists)
def test_merge_sorted_matches_reference(a, b):
out = merge_sorted(a, b)
ref = merge_sorted_reference(a, b)
assert out == ref
If Hypothesis can find any pair of sorted lists where your clever implementation gives a different result from the simple one, you've found a bug. This is an incredibly powerful way to refactor with confidence.
Testing for Relationships: Metamorphic Testing
Okay, this one sounds a bit academic, but the idea is actually very intuitive. Sometimes, you don't know what the exact output of a function should be, but you do know how the output should change when you change the input. This relationship is a "metamorphic property."
Let’s take a statistical function like calculating variance. The exact value can be a messy float, which is hard to assert directly. But we know some properties about variance. For example, if we take a list of numbers and add 5 to every single number, the spread of the data (the variance) shouldn't change at all. The whole dataset just shifted, but its shape is the same.
That’s a metamorphic property! Here’s how we can test it:
from hypothesis import given
from hypothesis import strategies as st
def variance(xs):
# ... implementation of variance ...
@given(xs=st.lists(st.integers(-1000, 1000), min_size=2))
def test_variance_is_translation_invariant(xs):
v1 = variance(xs)
# Let's create a translated list
k = 7
translated_xs = [x + k for x in xs]
v2 = variance(translated_xs)
# The variance should be the same (or very, very close for floats)
assert math.isclose(v1, v2)
We’re not checking if variance([1, 2, 3]) equals a specific number. We’re checking a much more fundamental truth about the function's behavior. This is fantastic for testing scientific computing, machine learning models, or any complex algorithm where the relationships are more important than specific output values.
What About Code That Remembers Things? Stateful Testing
This is where things get really, really cool. So far, we've only talked about pure functions—functions that take an input and produce an output without any side effects or memory. But what about real-world objects, like a bank account, a shopping cart, or a user session? Their behavior depends on a sequence of actions.
This is where traditional testing really struggles. The number of possible sequences of "deposit, withdraw, deposit, deposit, withdraw" is basically infinite.
Hypothesis gives us a tool for this called RuleBasedStateMachine. It lets you define a "machine" that represents your object, along with the rules for how you can interact with it.
Let's imagine a simple Bank class:
class Bank:
def __init__(self):
self.balance = 0
def deposit(self, amt):
# ... adds to balance ...
def withdraw(self, amt):
# ... subtracts from balance ...
We can create a state machine that tells Hypothesis how to call these methods. We define rules for depositing and withdrawing, and we can also define invariants—properties that must always be true, no matter what sequence of actions Hypothesis takes.
from hypothesis.stateful import RuleBasedStateMachine, rule, invariant
class BankMachine(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.bank = Bank()
@rule(amt=st.integers(min_value=1, max_value=1000))
def deposit(self, amt):
self.bank.deposit(amt)
@rule(amt=st.integers(min_value=1, max_value=1000))
def withdraw(self, amt):
# We can tell Hypothesis to skip this rule if there isn't enough money
assume(amt <= self.bank.balance)
self.bank.withdraw(amt)
@invariant()
def balance_never_negative(self):
assert self.bank.balance >= 0
TestBankMachine = BankMachine.TestCase
Now, Hypothesis will generate a random sequence of deposits and withdrawals, checking after every single step that the balance is never negative. It’s like having a thousand monkeys randomly using your bank account, but in a controlled way that finds bugs for you. If it ever finds a sequence of actions that makes the balance go negative, it will report that exact sequence to you.
It's a Mindset Shift, Not Just a Tool
Look, property-based testing won't replace all of your other tests. You'll still want simple, clear unit tests for your core business logic. But adding Hypothesis to your toolbox changes how you think.
You stop focusing on "which specific inputs should I test?" and start asking a much more powerful question: "What are the fundamental truths about my code?"
By describing those truths and letting a clever tool try to break them, you uncover bugs you never would have thought to look for. You build a deep, foundational confidence in your code's behavior, not just its output on a few hand-picked examples. It's about working smarter, not harder, and letting the computer do what it does best: explore a vast sea of possibilities to find the one that you missed.
