Chapter 6. Getting to the Minimum Viable Site
In this chapter we’re going to address the problems we discovered at the end of the last chapter. In the immediate, the problem of cleaning up after functional test runs. Later, the more general problem, which is that our design only allows for one global list. I’ll demonstrate a critical TDD technique: how to adapt existing code using an incremental, step-by-step process which takes you from working code to working code. Testing Goat, not Refactoring Cat.
Ensuring Test Isolation in Functional Tests
We ended the last chapter with a classic testing problem: how to ensure isolation between tests. Each run of our functional tests was leaving list items lying around in the database, and that would interfere with the test results when you next ran the tests.
When we run unit tests, the Django test runner automatically creates a brand new test database (separate from the real one), which it can safely reset before each individual test is run, and then throw away at the end. But our functional tests currently run against the “real” database, db.sqlite3.
One way to tackle this would be to “roll our own” solution, and add some code
to functional_tests.py which would do the cleaning up. The setUp and
tearDown methods are perfect for this sort of thing.
Since Django 1.4 though, there’s a new class called LiveServerTestCase which
can do this work for you. It will automatically create a test database (just
like in a unit test run), and start up a development server for the functional
tests to run against. Although as a tool it has some limitations which we’ll
need to work around later, it’s dead useful at this stage, so let’s check it
out.
LiveServerTestCase expects to be run by the Django test runner using
manage.py. As of Django 1.6, the test runner will find any files whose name
begins with test. To keep things neat and tidy, let’s make a folder for
our functional tests, so that it looks a bit like an app. All Django needs is
for it to be a valid Python package directory (i.e., one with a __init__.py in
it):
$ mkdir functional_tests $ touch functional_tests/__init__.py
Then we move our functional tests, from being a standalone file called
functional_tests.py, to being the tests.py of the functional_tests app.
We use git mv so that Git notices that we’ve moved the file:
$ git mv functional_tests.py functional_tests/tests.py $ git status # shows the rename to functional_tests/tests.py and __init__.py
At this point your directory tree should look like this:
.
├── db.sqlite3
├── functional_tests
│ ├── __init__.py
│ └── tests.py
├── lists
│ ├── admin.py
│ ├── __init__.py
│ ├── migrations
│ │ ├── 0001_initial.py
│ │ ├── 0002_item_text.py
│ │ ├── __init__.py
│ │ └── __pycache__
│ ├── models.py
│ ├── __pycache__
│ ├── templates
│ │ └── home.html
│ ├── tests.py
│ └── views.py
├── manage.py
└── superlists
├── __init__.py
├── __pycache__
├── settings.py
├── urls.py
└── wsgi.pyfunctional_tests.py is gone, and has turned into functional_tests/tests.py.
Now, whenever we want to run our functional tests, instead of running python3
functional_tests.py, we will use python3 manage.py test functional_tests.
Note
You could mix your functional tests into the tests for the lists app.
I tend to prefer to keep them separate, because functional tests usually
have cross-cutting concerns that run across different apps. FTs are meant
to see things from the point of view of your users, and your users don’t
care about how you’ve split work between different apps!
Now let’s edit functional_tests/tests.py and change our NewVisitorTest
class to make it use LiveServerTestCase:
functional_tests/tests.py (ch06l001).
fromdjango.testimportLiveServerTestCasefromseleniumimportwebdriverfromselenium.webdriver.common.keysimportKeysclassNewVisitorTest(LiveServerTestCase):defsetUp(self):[...]
Next,[9]
instead of hardcoding the visit to localhost port 8000, LiveServerTestCase
gives us an attribute called live_server_url:
functional_tests/tests.py (ch06l002).
deftest_can_start_a_list_and_retrieve_it_later(self):# Edith has heard about a cool new online to-do app. She goes# to check out its homepageself.browser.get(self.live_server_url)
We can also remove the if __name__ == '__main__' from the end if we want,
since we’ll be using the Django test runner to launch the FT.
Now we are able to run our functional tests using the Django test runner, by
telling it to run just the tests for our new functional_tests app:
$ python3 manage.py test functional_tests
Creating test database for alias 'default'...
F
======================================================================
FAIL: test_can_start_a_list_and_retrieve_it_later
(functional_tests.tests.NewVisitorTest)
---------------------------------------------------------------------
Traceback (most recent call last):
File "/workspace/superlists/functional_tests/tests.py", line 61, in
test_can_start_a_list_and_retrieve_it_later
self.fail('Finish the test!')
AssertionError: Finish the test!
---------------------------------------------------------------------
Ran 1 test in 6.378s
FAILED (failures=1)
Destroying test database for alias 'default'...The FT gets through to the self.fail, just like it did before the refactor.
You’ll also notice that if you run the tests a second time, there aren’t any
old list items lying around from the previous test—it has cleaned up after
itself. Success! We should commit it as an atomic change:
$ git status # functional_tests.py renamed + modified, new __init__.py $ git add functional_tests $ git diff --staged -M $ git commit # msg eg "make functional_tests an app, use LiveServerTestCase"
The -M flag on the git diff is a useful one. It means “detect moves”, so it
will notice that functional_tests.py and functional_tests/tests.py are the
same file, and show you a more sensible diff (try it without the flag!).
Running Just the Unit Tests
Now if we run manage.py test, Django will run both the functional and the
unit tests:
$ python3 manage.py test
Creating test database for alias 'default'...
.......F
======================================================================
FAIL: test_can_start_a_list_and_retrieve_it_later
[...]
AssertionError: Finish the test!
---------------------------------------------------------------------
Ran 8 tests in 3.132s
FAILED (failures=1)
Destroying test database for alias 'default'...In order to run just the unit tests, we can specify that we want to
only run the tests for the lists app:
$ python3 manage.py test lists
Creating test database for alias 'default'...
.......
---------------------------------------------------------------------
Ran 7 tests in 0.009s
OK
Destroying test database for alias 'default'...Now let’s move on to thinking about how we want support for multiple lists to work. Currently the FT (which is the closest we have to a design document) says this:
functional_tests/tests.py.
# Edith wonders whether the site will remember her list. Then she sees# that the site has generate a unique URL for her -- there is some# explanatory text to that effect.self.fail('Finish the test!')# She visits that URL - her to-do list is still there.# Satisfied, she goes back to sleep
But really we want to expand on this, by saying that different users don’t see each other’s lists, and each get their own URL as a way of going back to their saved lists. Let’s think about this a bit more.
Small Design When Necessary
TDD is closely associated with the agile movement in software development, which includes a reaction against Big Design Up Front the traditional software engineering practice whereby, after a lengthy requirements gathering exercise, there is an equally lengthy design stage where the software is planned out on paper. The agile philosophy is that you learn more from solving problems in practice than in theory, especially when you confront your application with real users as soon as possible. Instead of a long up-front design phase, we try and put a minimum viable application out there early, and let the design evolve gradually based on feedback from real-world usage.
But that doesn’t mean that thinking about design is outright banned! In the last chapter we saw how just blundering ahead without thinking can eventually get us to the right answer, but often a little thinking about design can help us get there faster. So, let’s think about our minimum viable lists app, and what kind of design we’ll need to deliver it.
- We want each user to be able to store their own list—at least one, for now.
- A list is made up of several items, whose primary attribute is a bit of descriptive text.
- We need to save lists from one visit to the next. For now, we can give each user a unique URL for their list. Later on we may want some way of automatically recognising users and showing them their lists.
To deliver the “for now” items, it sounds like we’re going to store lists and their items in a database. Each list will have a unique URL, and each list item will be a bit of descriptive text, associated with a particular list.
YAGNI!
Once you start thinking about design, it can be hard to stop. All sorts of other thoughts are occurring to us—we might want to give each list a name or title, we might want to recognise users using usernames and passwords, we might want to add a longer notes field as well as short descriptions to our list, we might want to store some kind of ordering, and so on. But we obey another tenet of the agile gospel: “YAGNI” (pronounced yag-knee), which stands for “You aint gonna need it!” As software developers, we have fun creating things, and sometimes it’s hard to resist the urge to build things just because an idea occurred to us and we might need it. The trouble is that more often than not, no matter how cool the idea was, you won’t end up using it. Instead you have a load of unused code, adding to the complexity of your application. YAGNI is the mantra we use to resist our overenthusiastic creative urges.
REST
We have an idea of the data structure we want—the Model part of Model-View-Controller (MVC). What about the View and Controller parts? How should the user interact with Lists and their Items using a web browser?
Representational State Transfer (REST) is an approach to web design that’s usually used to guide the design of web-based APIs. When designing a user-facing site, it’s not possible to stick strictly to the REST rules, but they still provide some useful inspiration.
REST suggests that we have a URL structure that matches our data structure, in this case lists and list items. Each list can have its own URL:
/lists/<list identifier>/
That will fulfill the requirement we’ve specified in our FT. To view a list, we use a GET request (a normal browser visit to the page).
To create a brand new list, we’ll have a special URL that accepts POST requests:
/lists/new
To add a new item to an existing list, we’ll have a separate URL, to which we can send POST requests:
/lists/<list identifier>/add_item
(Again, we’re not trying to perfectly follow the rules of REST, which would use a PUT request here—we’re just using REST for inspiration.)
In summary, our scratchpad for this chapter looks something like this:
Implementing the New Design Using TDD
How do we use TDD to implement the new design? Let’s take another look at the flowchart for the TDD process in Figure 6-1.
At the top level, we’re going to use a combination of adding new functionality (by extending the FT and writing new application code), and refactoring our application—i.e., rewriting some of the existing implementation so that it delivers the same functionality to the user but using aspects of our new design. At the unit test level, we’ll be adding new tests or modifying existing ones to test for the changes we want, and we’ll be able to use the untouched unit tests to make sure we don’t break anything in the process.
Let’s translate our scratchpad into our functional test. As soon as Edith
submits a first list item, we’ll want to create a new list, adding one
item to it, and take her to the URL for her list. Look for the point
at which we say inputbox.send_keys('Buy peacock feathers'), and amend
the next block of code like this:
functional_tests/tests.py.
inputbox.send_keys('Buy peacock feathers')# When she hits enter, she is taken to a new URL,# and now the page lists "1: Buy peacock feathers" as an item in a# to-do list tableinputbox.send_keys(Keys.ENTER)edith_list_url=self.browser.current_urlself.assertRegex(edith_list_url,'/lists/.+')#self.check_for_row_in_list_table('1: Buy peacock feathers')# There is still a text box inviting her to add another item. She[...]
assertRegexis a helper function fromunittestthat checks whether a string matches a regular expression. We use it to check that our new REST-ish design has been implemented. Find out more in theunittestdocumentation.
Let’s also change the end of the test and imagine a new user coming along. We want to check that they don’t see any of Edith’s items when they visit the home page, and that they get their own unique URL for their list.
Delete everything from the comments just before the self.fail (they say
“Edith wonders whether the site will remember her list …”) and replace
them with a new ending to our FT:
functional_tests/tests.py.
[...]# The page updates again, and now shows both items on her listself.check_for_row_in_list_table('2: Use peacock feathers to make a fly')self.check_for_row_in_list_table('1: Buy peacock feathers')# Now a new user, Francis, comes along to the site.## We use a new browser session to make sure that no information## of Edith's is coming through from cookies etc #self.browser.quit()self.browser=webdriver.Firefox()# Francis visits the home page. There is no sign of Edith's# listself.browser.get(self.live_server_url)page_text=self.browser.find_element_by_tag_name('body').textself.assertNotIn('Buy peacock feathers',page_text)self.assertNotIn('make a fly',page_text)# Francis starts a new list by entering a new item. He# is less interesting than Edith...inputbox=self.browser.find_element_by_id('id_new_item')inputbox.send_keys('Buy milk')inputbox.send_keys(Keys.ENTER)# Francis gets his own unique URLfrancis_list_url=self.browser.current_urlself.assertRegex(francis_list_url,'/lists/.+')self.assertNotEqual(francis_list_url,edith_list_url)# Again, there is no trace of Edith's listpage_text=self.browser.find_element_by_tag_name('body').textself.assertNotIn('Buy peacock feathers',page_text)self.assertIn('Buy milk',page_text)# Satisfied, they both go back to sleep
-
I’m using the convention of double-hashes (
##) to indicate “meta-comments”—comments about how the test is working and why—so that we can distinguish them from regular comments in FTs which explain the User Story. They’re a message to our future selves, which might otherwise be wondering why the heck we’re quitting the browser and starting a new one…
Other than that, the changes are fairly self-explanatory. Let’s see how they do when we run our FTs:
[...]
self.assertRegex(edith_list_url, '/lists/.+')
AssertionError: Regex didn't match: '/lists/.+' not found in
'http://localhost:8081/'As expected. Let’s do a commit, and then go and build some new models and views:
$ git commit -aNote
I found the FTs hung when I tried to run them today. It turns out I
needed to upgrade Selenium, with a pip3 install --upgrade selenium. You
may remember from the preface that it’s important to have the latest version
of Selenium installed—it’s only been a couple of months since I last
upgraded, and Selenium had gone up by six point versions. If something weird is
happening, always try upgrading Selenium!
Iterating Towards the New Design
Being all excited about our new design, I had an overwhelming urge to dive in at this point and start changing models.py, which would have broken half the unit tests, and then pile in and change almost every single line of code, all in one go. That’s a natural urge, and TDD, as a discipline, is a constant fight against it. Obey the Testing Goat, not Refactoring Cat! We don’t need to implement our new, shiny design in a single big bang. Let’s make small changes that take us from a working state to a working state, with our design guiding us gently at each stage.
There are four items on our to-do list. The FT, with its Regexp didn't
match, is telling us that the second item—giving lists their own URL and
identifier—is the one we should work on next. Let’s have a go at fixing
that, and only that.
The URL comes from the redirect after POST. In lists/tests.py, find
test_home_page_redirects_after_POST, and change the expected redirect
location:
lists/tests.py.
self.assertEqual(response.status_code,302)self.assertEqual(response['location'],'/lists/the-only-list-in-the-world/')
Does that seem slightly strange? Clearly, /lists/the-only-list-in-the-world isn’t a URL that’s going to feature in the final design of our application. But we’re committed to changing one thing at a time. While our application only supports one list, this is the only URL that makes sense. We’re still moving forwards, in that we’ll have a different URL for our list and our home page, which is a step along the way to a more REST-ful design. Later, when we have multiple lists, it will be easy to change.
Note
Another way of thinking about it is as a problem-solving technique: our new URL design is currently not implemented, so it works for 0 items. Ultimately, we want to solve for n items, but solving for 1 item is a good step along the way.
Running the unit tests gives us an expected fail:
$ python3 manage.py test lists
[...]
AssertionError: '/' != '/lists/the-only-list-in-the-world/'We can go adjust our home_page view in lists/views.py:
lists/views.py.
defhome_page(request):ifrequest.method=='POST':Item.objects.create(text=request.POST['item_text'])returnredirect('/lists/the-only-list-in-the-world/')items=Item.objects.all()returnrender(request,'home.html',{'items':items})
Of course, that will now totally break the functional tests, because there is no such URL on our site yet. Sure enough, if you run them, you’ll find they fail just after trying to submit the first item, saying that they can’t find the list table; it’s because URL /the-only-list-in-the-world/ doesn’t exist yet!
self.check_for_row_in_list_table('1: Buy peacock feathers')
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: {"method":"id","selector":"id_list_table"}So, let’s build a special URL for our one and only list.
Testing Views, Templates, and URLs Together with the Django Test Client
In previous chapters we’ve used unit tests that check the URL resolution explicitly, that test view functions by actually calling them, and that check that views render templates correctly too. Django actually provides us with a little tool that can do all three at once, which we’ll use now.
I wanted to show you how to “roll your own” first, partially because it’s a better introduction to how Django works, but also because those techniques are portable—you may not always use Django, but you’ll almost always have view functions, templates, and URL mappings, and now you know how to test them.
A New Test Class
So let’s use the Django test client. Open up lists/tests.py, and add a new
test class called ListViewTest. Then copy the method called
test_home_page_displays_all_ list_items across from HomePageTest into our
new class, rename it, and adapt it slightly:
lists/tests.py (ch06l009).
classListViewTest(TestCase):deftest_displays_all_items(self):Item.objects.create(text='itemey 1')Item.objects.create(text='itemey 2')response=self.client.get('/lists/the-only-list-in-the-world/')#self.assertContains(response,'itemey 1')#self.assertContains(response,'itemey 2')#
-
Instead of calling the view function directly, we use the Django test client, which is an attribute of the Django
TestCasecalledself.client. We tell it to.getthe URL we’re testing—it’s actually a very similar API to the one that Selenium uses.
Instead of using the slightly annoying
assertIn/response.content.decode()dance, Django provides theassertContainsmethod which knows how to deal with responses and the bytes of their content.
Note
Some people really don’t like the Django test client. They say it provides too much magic, and involves too much of the stack to be used in a real “unit” test—you end up writing what are more properly called integrated tests. They also complain that it is relatively slow (and relatively is measured in milliseconds). We’ll explore this argument further in a later chapter. For now we’ll use it because it’s extremely convenient!
Let’s try running the test now:
AssertionError: 404 != 200 : Couldn't retrieve content: Response code was 404
A New URL
Our singleton list URL doesn’t exist yet. We fix that in superlists/urls.py.
Tip
Watch out for trailing slashes in URLs, both here in the tests and in urls.py—They’re a common source of bugs.
superlists/urls.py.
urlpatterns=[url(r'^$',views.home_page,name='home'),url(r'^lists/the-only-list-in-the-world/$',views.view_list,name='view_list'),# url(r'^admin/', include(admin.site.urls)),]
Running the tests again, we get:
AttributeError: 'module' object has no attribute 'view_list' [...] FAILED (errors=4)
A New View Function
Nicely self-explanatory. Let’s create a dummy view function in lists/views.py:
lists/views.py.
defview_list(request):pass
Now we get:
ValueError: The view lists.views.view_list didn't return an HttpResponse object. It returned None instead.
Let’s copy the two last lines from the home_page view and see if they’ll do
the trick:
lists/views.py.
defview_list(request):items=Item.objects.all()returnrender(request,'home.html',{'items':items})
Rerun the tests and they should pass:
Ran 8 tests in 0.016s OK
And the FTs should get a little further on:
AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Buy peacock feathers']
Green? Refactor
In the Red/Green/Refactor dance, we’ve arrived at green, so we should see what needs a refactor. We now have two views, one for the home page, and one for an individual list. Both are currently using the same template, and passing it all the list items currently in the database. If we look through our unit test methods, we can see some stuff we probably want to change:
$ grep -E "class|def" lists/tests.py
class HomePageTest(TestCase):
def test_root_url_resolves_to_home_page_view(self):
def test_home_page_returns_correct_html(self):
def test_home_page_displays_all_list_items(self):
def test_home_page_can_save_a_POST_request(self):
def test_home_page_redirects_after_POST(self):
def test_home_page_only_saves_items_when_necessary(self):
class ListViewTest(TestCase):
def test_displays_all_items(self):
class ItemModelTest(TestCase):
def test_saving_and_retrieving_items(self):We can definitely delete the test_home_page_displays_all_list_items method,
it’s no longer needed. If you run manage.py test lists now, it should say
it ran 7 tests instead of 8:
Ran 7 tests in 0.016s OK
Next, we don’t actually need the home page to display all list items any more; it should just show a single input box inviting you to start a new list.
A Separate Template for Viewing Lists
Since the home page and the list view are now quite distinct pages, they should be using different HTML templates; home.html can have the single input box, whereas a new template, list.html, can take care of showing the table of existing items.
Let’s add a new test to check that it’s using a different template:
lists/tests.py.
classListViewTest(TestCase):deftest_uses_list_template(self):response=self.client.get('/lists/the-only-list-in-the-world/')self.assertTemplateUsed(response,'list.html')deftest_displays_all_items(self):[...]
assertTemplateUsed is one of the more useful functions that the Django test
client gives us. Let’s see what it says:
AssertionError: False is not true : Template 'list.html' was not a template used to render the response. Actual template(s) used: home.html
Great! Let’s change the view:
lists/views.py.
defview_list(request):items=Item.objects.all()returnrender(request,'list.html',{'items':items})
But, obviously, that template doesn’t exist yet. If we run the unit tests, we get:
django.template.base.TemplateDoesNotExist: list.html
Let’s create a new file at lists/templates/list.html:
$ touch lists/templates/list.htmlA blank template, which gives us this error—good to know the tests are there to make sure we fill it in:
AssertionError: False is not true : Couldn't find 'itemey 1' in response
The template for an individual list will reuse quite a lot of the stuff we currently have in home.html, so we can start by just copying that:
$ cp lists/templates/home.html lists/templates/list.htmlThat gets the tests back to passing (green). Now let’s do a little more
tidying up (refactoring). We said the home page doesn’t need to list items, it
only needs the new list input field, so we can remove some lines from
lists/templates/home.html, and maybe slightly tweak the h1 to say “Start a
new To-Do list”:
lists/templates/home.html.
<body><h1>Start a new To-Do list</h1><formmethod="POST"><inputname="item_text"id="id_new_item"placeholder="Enter a to-do item"/>{% csrf_token %}</form></body>
We rerun the unit tests to check that hasn’t broken anything—good…
There’s actually no need to pass all the items to the home.html template in
our home_page view, so we can simplify that:
lists/views.py.
defhome_page(request):ifrequest.method=='POST':Item.objects.create(text=request.POST['item_text'])returnredirect('/lists/the-only-list-in-the-world/')returnrender(request,'home.html')
Rerun the unit tests; they still pass. Let’s run the functional tests:
AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Buy peacock feathers']
We’re still failing to input the second item. What’s going on here? Well, it’s
not immediately obvious, but it looks like our POST requests aren’t working the
way they should. After a bit of head-scratching and digging through the various
views and templates, we will eventually uncover the problem: both our
forms are missing the action= attribute, which means that, by default, they
submit to the same URL they were rendered from. That works for the home page,
because it’s the only one that knows how to deal with POST requests currently,
but it won’t work for our view_list function, which is just ignoring the
POST.
We can fix that in lists/templates/list.html:
lists/templates/list.html (ch06l019).
<formmethod="POST"action="/">
And try running the FT again:
self.assertNotEqual(francis_list_url, edith_list_url) AssertionError: 'http://localhost:8081/lists/the-only-list-in-the-world/' == 'http://localhost:8081/lists/the-only-list-in-the-world/'
Hooray! We’re back to where we were earlier, which means our refactoring is complete—we now have a unique URL for our one list. It may feel like we haven’t made much headway since, functionally, the site still behaves almost exactly like it did when we started the chapter, but this really is progress. We’ve started on the road to our new design, and we’ve implemented a number of stepping stones without making anything worse than it was before. Let’s commit our progress so far:
$ git status # should show 4 changed files and 1 new file, list.html $ git add lists/templates/list.html $ git diff # should show we've simplified home.html, # moved one test to a new class in lists/tests.py added a new view # in views.py, and simplified home_page and made one addition to # urls.py $ git commit -a # add a message summarising the above, maybe something like # "new URL, view and template to display lists"
Another URL and View for Adding List Items
Where are we with our own to-do list?
We’ve sort of made progress on the third item, even if there’s still only one list in the world. Item 2 is a bit scary. Can we do something about items 4 or 5?
Let’s have a new URL for adding new list items. If nothing else, it’ll simplify the home page view.
A Test Class for New List Creation
Open up lists/tests.py, and move the
test_home_page_can_save_a_POST_request and
test_home_page_redirects_after_POST methods into a new class, then change
their names:
lists/tests.py (ch06l021-1).
classNewListTest(TestCase):deftest_saving_a_POST_request(self):request=HttpRequest()request.method='POST'[...]deftest_redirects_after_POST(self):[...]
Now let’s use the Django test client:
lists/tests.py (ch06l021-2).
classNewListTest(TestCase):deftest_saving_a_POST_request(self):self.client.post('/lists/new',data={'item_text':'A new list item'})self.assertEqual(Item.objects.count(),1)new_item=Item.objects.first()self.assertEqual(new_item.text,'A new list item')deftest_redirects_after_POST(self):response=self.client.post('/lists/new',data={'item_text':'A new list item'})self.assertEqual(response.status_code,302)self.assertEqual(response['location'],'/lists/the-only-list-in-the-world/')
This is another place to pay attention to trailing slashes, incidentally. It’s
/new, with no trailing slash. The convention I’m using is that URLs without
a trailing slash are “action” URLs which modify the database.
Try running that:
self.assertEqual(Item.objects.count(), 1)
AssertionError: 0 != 1
[...]
self.assertEqual(response.status_code, 302)
AssertionError: 404 != 302The first failure tells us we’re not saving a new item to the database, and the
second says that, instead of returning a 302 redirect, our view is returning a
404. That’s because we haven’t built a URL for /lists/new, so the
client.post is just getting a 404 response.
Note
Do you remember how we split this out into two tests in the last chapter?
If we only had one test that checked both the saving and the redirect, it would
have failed on the 0 != 1 failure, which would have been much harder to
debug. Ask me how I know this.
A URL and View for New List Creation
superlists/urls.py.
urlpatterns=[url(r'^$',views.home_page,name='home'),url(r'^lists/new$',views.new_list,name='new_list'),url(r'^lists/the-only-list-in-the-world/$',views.view_list,name='view_list'),# url(r'^admin/', include(admin.site.urls)),]
Next we get a no attribute 'new_list', so let’s fix that, in lists/views.py:
lists/views.py.
defnew_list(request):pass
Then we get “The view lists.views.new_list didn’t return an HttpResponse
object”. (This is getting rather familiar!) We could return a raw
HttpResponse, but since we know we’ll need a redirect, let’s borrow a line
from home_page:
lists/views.py.
defnew_list(request):returnredirect('/lists/the-only-list-in-the-world/')
That gives:
self.assertEqual(Item.objects.count(), 1) AssertionError: 0 != 1 [...] AssertionError: 'http://testserver/lists/the-only-list-in-the-world/' != '/lists/the-only-list-in-the-world/'
Let’s start with the first failure, because it’s reasonably straightforward. We
borrow another line from home_page:
lists/views.py.
defnew_list(request):Item.objects.create(text=request.POST['item_text'])returnredirect('/lists/the-only-list-in-the-world/')
And that takes us down to just the second, unexpected failure:
self.assertEqual(response['location'], '/lists/the-only-list-in-the-world/') AssertionError: 'http://testserver/lists/the-only-list-in-the-world/' != '/lists/the-only-list-in-the-world/'
It’s happening because the Django test client behaves slightly differently to our pure view function; it’s using the full Django stack which adds the domain to our relative URL. Let’s use another of Django’s test helper functions, instead of our two-step check for the redirect:
lists/tests.py.
deftest_redirects_after_POST(self):response=self.client.post('/lists/new',data={'item_text':'A new list item'})self.assertRedirects(response,'/lists/the-only-list-in-the-world/')
That now passes:
Ran 8 tests in 0.030s OK
Removing Now-Redundant Code and Tests
We’re looking good. Since our new views are now doing most of the work that
home_page used to do, we should be able to massively simplify it. Can we
remove the whole if request.method == 'POST' section, for example?
lists/views.py.
defhome_page(request):returnrender(request,'home.html')
Yep!
OK
And while we’re at it, we can remove the now-redundant
test_home_page_only_saves_ items_when_necessary test too!
Doesn’t that feel good? The view functions are looking much simpler. We rerun the tests to make sure…
Ran 7 tests in 0.016s OK
Pointing Our Forms at the New URL
Finally, let’s wire up our two forms to use this new URL. In both home.html and lists.html:
lists/templates/home.html, lists/templates/list.html.
<formmethod="POST"action="/lists/new">
And we rerun our FTs to make sure everything still works, or works at least as well as it did earlier…
AssertionError: 'http://localhost:8081/lists/the-only-list-in-the-world/' == 'http://localhost:8081/lists/the-only-list-in-the-world/'
Yup, we get to the same point we did before. That’s a nicely self-contained commit, in that we’ve made a bunch of changes to our URLs, our views.py is looking much neater and tidier, and we’re sure the application is still working as well as it did before. We’re getting good at this refactoring malarkey!
$ git status # 5 changed files $ git diff # URLs for forms x2, moved code in views + tests, new URL $ git commit -a
And we can cross out an item on the to-do list:
Adjusting Our Models
Enough housekeeping with our URLs. It’s time to bite the bullet and change our models. Let’s adjust the model unit test. Just for a change, I’ll present the changes in the form of a diff:
lists/tests.py.
@@ -3,7 +3,7 @@ from django.http import HttpRequestfrom django.template.loader import render_to_string from django.test import TestCase-from lists.models import Item+from lists.models import Item, Listfrom lists.views import home_page class HomePageTest(TestCase):@@ -60,22 +60,32 @@ class ListViewTest(TestCase):-class ItemModelTest(TestCase):+class ListAndItemModelsTest(TestCase):def test_saving_and_retrieving_items(self):+ list_ = List()+ list_.save()+first_item = Item() first_item.text = 'The first (ever) list item'+ first_item.list = list_first_item.save() second_item = Item() second_item.text = 'Item the second'+ second_item.list = list_second_item.save()+ saved_list = List.objects.first()+ self.assertEqual(saved_list, list_)+saved_items = Item.objects.all() self.assertEqual(saved_items.count(), 2) first_saved_item = saved_items[0] second_saved_item = saved_items[1] self.assertEqual(first_saved_item.text, 'The first (ever) list item')+ self.assertEqual(first_saved_item.list, list_)self.assertEqual(second_saved_item.text, 'Item the second')+ self.assertEqual(second_saved_item.list, list_)
We create a new List object, and then we assign each item to it
by assigning it as its .list property. We check the list is properly
saved, and we check that the two items have also saved their relationship
to the list. You’ll also notice that we can compare list objects with each
other directly (saved_list and list)—behind the scenes, these
will compare themselves by checking their primary key (the .id attribute)
is the same.
Note
I’m using the variable name list_ to avoid “shadowing” the Python
built-in list function. It’s ugly, but all the other options I tried were
equally ugly or worse (my_list, the_list, list1, listey…).
Time for another unit-test/code cycle.
For the first couple of iterations, rather than explicitly showing you what code to enter in between every test run, I’m only going to show you the expected error messages from running the tests. I’ll let you figure out what each minimal code change should be on your own:
Your first error should be:
ImportError: cannot import name 'List'
Fix that, then you should see:
AttributeError: 'List' object has no attribute 'save'
Next you should see:
django.db.utils.OperationalError: no such table: lists_list
So we run a makemigrations:
$ python3 manage.py makemigrations
Migrations for 'lists':
0003_list.py:
- Create model ListAnd then you should see:
self.assertEqual(first_saved_item.list, list_) AttributeError: 'Item' object has no attribute 'list'
A Foreign Key Relationship
How do we give our Item a list attribute? Let’s just try naively making it
like the text attribute:
lists/models.py.
fromdjango.dbimportmodelsclassList(models.Model):passclassItem(models.Model):text=models.TextField(default='')list=models.TextField(default='')
As usual, the tests tell us we need a migration:
$ python3 manage.py test lists [...] django.db.utils.OperationalError: no such column: lists_item.list $ python3 manage.py makemigrations Migrations for 'lists': 0004_item_list.py: - Add field list to item
Let’s see what that gives us:
AssertionError: 'List object' != <List: List object>
We’re not quite there. Look closely at each side of the !=. Django has only
saved the string representation of the List object. To save the relationship to
the object itself, we tell Django about the relationship between the two
classes using a ForeignKey:
lists/models.py.
fromdjango.dbimportmodelsclassList(models.Model):passclassItem(models.Model):text=models.TextField(default='')list=models.ForeignKey(List,default=None)
That’ll need a migration too. Since the last one was a red herring, let’s delete it and replace it with a new one:
$ rm lists/migrations/0004_item_list.py $ python3 manage.py makemigrations Migrations for 'lists': 0004_item_list.py: - Add field list to item
Warning
Deleting migrations is dangerous. If you delete a migration that’s already been applied to a database somewhere, Django will be confused about what state it’s in, and how to apply future migrations. You should only do it when you’re sure the migration hasn’t been used. A good rule of thumb is that you should never delete a migration that’s been committed to your VCS.
Adjusting the Rest of the World to Our New Models
Back in our tests, now what happens?
$ python3 manage.py test lists
[...]
ERROR: test_displays_all_items (lists.tests.ListViewTest)
django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id
[...]
ERROR: test_redirects_after_POST (lists.tests.NewListTest)
django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id
[...]
ERROR: test_saving_a_POST_request (lists.tests.NewListTest)
django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id
Ran 7 tests in 0.021s
FAILED (errors=3)Oh dear!
There is some good news. Although it’s hard to see, our model tests are passing. But three of our view tests are failing nastily.
The reason is because of the new relationship we’ve introduced between Items and Lists, which requires each item to have a parent list, which our old tests weren’t prepared for.
Still, this is exactly why we have tests. Let’s get them working again. The
easiest is the ListViewTest; we just create a parent list for our two test
items:
lists/tests.py (ch06l031).
classListViewTest(TestCase):deftest_displays_all_items(self):list_=List.objects.create()Item.objects.create(text='itemey 1',list=list_)Item.objects.create(text='itemey 2',list=list_)
That gets us down to two failing tests, both on tests that try to POST to our
new_list view. Decoding the tracebacks using our usual technique, working back
from error, to line of test code, to the line of our own code that caused the
failure, we identify:
File "/workspace/superlists/lists/views.py", line 14, in new_list Item.objects.create(text=request.POST['item_text'])
It’s when we try and create an item without a parent list. So we make a similar change in the view:
lists/views.py.
fromlists.modelsimportItem,List[...]defnew_list(request):list_=List.objects.create()Item.objects.create(text=request.POST['item_text'],list=list_)returnredirect('/lists/the-only-list-in-the-world/')
And that gets our tests passing again:
OK
Are you cringing internally at this point? Arg! This feels so wrong, we create a new list for every single new item submission, and we’re still just displaying all items as if they belong to the same list! I know, I feel the same. The step-by-step approach, in which you go from working code to working code, is counterintuitive. I always feel like just diving in and trying to fix everything all in one go, instead of going from one weird half-finished state to another. But remember the Testing Goat! When you’re up a mountain, you want to think very carefully about where you put each foot, and take one step at a time, checking at each stage that the place you’ve put it hasn’t caused you to fall off a cliff.
So just to reassure ourselves that things have worked, we rerun the FT. Sure enough, it gets all the way through to where we were before. We haven’t broken anything, and we’ve made a change to the database. That’s something to be pleased with! Let’s commit:
$ git status # 3 changed files, plus 2 migrations $ git add lists $ git diff --staged $ git commit
And we can cross out another item on the to-do list:
Each List Should Have Its Own URL
What shall we use as the unique identifier for our lists? Probably the
simplest thing, for now, is just to use the auto-generated id field from the
database. Let’s change ListViewTest so that the two tests point at new
URLs.
We’ll also change the old test_displays_all_items test and call it
test_displays_only_items_for_that_list instead, and make it check that
only the items for a specific list are displayed:
lists/tests.py (ch06l033-1).
classListViewTest(TestCase):deftest_uses_list_template(self):list_=List.objects.create()response=self.client.get('/lists/%d/'%(list_.id,))self.assertTemplateUsed(response,'list.html')deftest_displays_only_items_for_that_list(self):correct_list=List.objects.create()Item.objects.create(text='itemey 1',list=correct_list)Item.objects.create(text='itemey 2',list=correct_list)other_list=List.objects.create()Item.objects.create(text='other list item 1',list=other_list)Item.objects.create(text='other list item 2',list=other_list)response=self.client.get('/lists/%d/'%(correct_list.id,))self.assertContains(response,'itemey 1')self.assertContains(response,'itemey 2')self.assertNotContains(response,'other list item 1')self.assertNotContains(response,'other list item 2')
Note
If you’re not familiar with Python string substitutions, or the
printf function from C, maybe that %d is a little confusing?
Dive Into Python has a good overview, if you
want to go look them up quickly. We’ll see an alternative string substitution
syntax later in the book too.
Running the unit tests gives an expected 404, and another related error:
FAIL: test_displays_only_items_for_that_list (lists.tests.ListViewTest) AssertionError: 404 != 200 : Couldn't retrieve content: Response code was 404 (expected 200) [...] FAIL: test_uses_list_template (lists.tests.ListViewTest) AssertionError: No templates used to render the response
Capturing Parameters from URLs
It’s time to learn how we can pass parameters from URLs to views:
superlists/urls.py.
urlpatterns=[url(r'^$',views.home_page,name='home'),url(r'^lists/new$','lists.views.new_list',name='new_list'),url(r'^lists/(.+)/$','lists.views.view_list',name='view_list'),# url(r'^admin/', include(admin.site.urls)),]
We adjust the regular expression for our URL to include a capture group,
(.+), which will match any characters, up to the following /. The captured
text will get passed to the view as an argument.
In other words, if we go to the URL /lists/1/, view_list will get a second
argument after the normal request argument, namely the string "1".
If we go to /lists/foo/, we get view_list(request, "foo").
But our view doesn’t expect an argument yet! Sure enough, this causes problems:
ERROR: test_displays_only_items_for_that_list (lists.tests.ListViewTest) ERROR: test_uses_list_template (lists.tests.ListViewTest) ERROR: test_redirects_after_POST (lists.tests.NewListTest) [...] TypeError: view_list() takes 1 positional argument but 2 were given
We can fix that easily with a dummy parameter in views.py:
lists/views.py.
defview_list(request,list_id):[...]
Now we’re down to our expected failure:
FAIL: test_displays_only_items_for_that_list (lists.tests.ListViewTest) AssertionError: 1 != 0 : Response should not contain 'other list item 1'
Let’s make our view discriminate over which items it sends to the template:
lists/views.py.
defview_list(request,list_id):list_=List.objects.get(id=list_id)items=Item.objects.filter(list=list_)returnrender(request,'list.html',{'items':items})
Adjusting new_list to the New World
Now we get errors in another test:
ERROR: test_redirects_after_POST (lists.tests.NewListTest) ValueError: invalid literal for int() with base 10: 'the-only-list-in-the-world'
Let’s take a look at this test then, since it’s whining:
lists/tests.py.
classNewListTest(TestCase):[...]deftest_redirects_after_POST(self):response=self.client.post('/lists/new',data={'item_text':'A new list item'})self.assertRedirects(response,'/lists/the-only-list-in-the-world/')
It looks like it hasn’t been adjusted to the new world of Lists and Items. The test should be saying that this view redirects to the URL of the new list it just created:
lists/tests.py (ch06l036-1).
deftest_redirects_after_POST(self):response=self.client.post('/lists/new',data={'item_text':'A new list item'})new_list=List.objects.first()self.assertRedirects(response,'/lists/%d/'%(new_list.id,))
That still gives us the invalid literal error. We take a look at the view itself, and change it so it redirects to a valid place:
lists/views.py (ch06l036-2).
defnew_list(request):list_=List.objects.create()Item.objects.create(text=request.POST['item_text'],list=list_)returnredirect('/lists/%d/'%(list_.id,))
That gets us back to passing unit tests. What about the functional tests? We must be almost there?
AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Use peacock feathers to make a fly']
The functional tests have warned us of a regression in our application: because we’re now creating a new list for every single POST submission, we have broken the ability to add multiple items to a list. This is exactly what we have functional tests for!
And it correlates nicely with the last item on our to-do list:
One More View to Handle Adding Items to an Existing List
We need a URL and view to handle adding a new item to an existing list ( /lists/<list_id>/add_item). We’re getting pretty good at these now, so let’s knock one together quickly:
lists/tests.py.
classNewItemTest(TestCase):deftest_can_save_a_POST_request_to_an_existing_list(self):other_list=List.objects.create()correct_list=List.objects.create()self.client.post('/lists/%d/add_item'%(correct_list.id,),data={'item_text':'A new item for an existing list'})self.assertEqual(Item.objects.count(),1)new_item=Item.objects.first()self.assertEqual(new_item.text,'A new item for an existing list')self.assertEqual(new_item.list,correct_list)deftest_redirects_to_list_view(self):other_list=List.objects.create()correct_list=List.objects.create()response=self.client.post('/lists/%d/add_item'%(correct_list.id,),data={'item_text':'A new item for an existing list'})self.assertRedirects(response,'/lists/%d/'%(correct_list.id,))
We get:
AssertionError: 0 != 1 [...] AssertionError: 301 != 302 : Response didn't redirect as expected: Response code was 301 (expected 302)
Beware of Greedy Regular Expressions!
That’s a little strange. We haven’t actually specified a URL for
/lists/1/add_item yet, so our expected failure is 404 != 302. Why are we
getting a 301?
This was a bit of a puzzler, but it’s because we’ve used a very “greedy” regular expression in our URL:
url(r'^lists/(.+)/$','lists.views.view_list',name='view_list'),
Django has some built-in code to issue a permanent redirect (301) whenever
someone asks for a URL which is almost right, except for a missing slash.
In this case, /lists/1/add_item/ would be a match for lists/(.+)/, with
the (.+) capturing 1/add_item. So Django “helpfully” guesses that we
actually wanted the URL with a trailing slash.
We can fix that by making our URL pattern explicitly capture only numerical
digits, by using the regular expression \d:
superlists/urls.py.
url(r'^lists/(\d+)/$',views.view_list,name='view_list'),
That gives:
AssertionError: 0 != 1 [...] AssertionError: 404 != 302 : Response didn't redirect as expected: Response code was 404 (expected 302)
The Last New URL
Now we’ve got our expected 404, let’s add a new URL for adding new items to existing lists:
superlists/urls.py.
urlpatterns=[url(r'^$',views.home_page,name='home'),url(r'^lists/new$',views.new_list,name='new_list'),url(r'^lists/(\d+)/$',views.view_list,name='view_list'),url(r'^lists/(\d+)/add_item$',views.add_item,name='add_item'),# url(r'^admin/', include(admin.site.urls)),]
Three very similar-looking URLs there. Let’s make a note on our to-do list; they look like good candidates for a refactoring.
Back to the tests, we get the usual missing module objects:
AttributeError: 'module' object has no attribute 'add_item'
The Last New View
Let’s try:
lists/views.py.
defadd_item(request):pass
Aha:
TypeError: add_item() takes 1 positional argument but 2 were given
lists/views.py.
defadd_item(request,list_id):pass
And then:
ValueError: The view lists.views.add_item didn't return an HttpResponse object. It returned None instead.
We can copy the redirect from new_list and the List.objects.get from
view_list:
lists/views.py.
defadd_item(request,list_id):list_=List.objects.get(id=list_id)returnredirect('/lists/%d/'%(list_.id,))
That takes us to:
self.assertEqual(Item.objects.count(), 1) AssertionError: 0 != 1
Finally we make it save our new list item:
lists/views.py.
defadd_item(request,list_id):list_=List.objects.get(id=list_id)Item.objects.create(text=request.POST['item_text'],list=list_)returnredirect('/lists/%d/'%(list_.id,))
And we’re back to passing tests.
Ran 9 tests in 0.050s OK
But How to Use That URL in the Form?
Now we just need to use this URL in our list.html template. Open it up and adjust the form tag…
lists/templates/list.html.
<formmethod="POST"action="but what should we put here?">
... oh. To get the URL for adding to the current list, the template needs to know what list it’s rendering, as well as what the items are. We want to be able to do something like this:
lists/templates/list.html.
<formmethod="POST"action="/lists/{{ list.id }}/add_item">
For that to work, the view will have to pass the list to the template.
Let’s create a new unit test in ListViewTest:
lists/tests.py (ch06l041).
deftest_passes_correct_list_to_template(self):other_list=List.objects.create()correct_list=List.objects.create()response=self.client.get('/lists/%d/'%(correct_list.id,))self.assertEqual(response.context['list'],correct_list)
response.context represents the context we’re going to pass into
the render function—the Django test client puts it on the response
object for us, to help with testing. That gives us:
KeyError: 'list'
because we’re not passing list into the template. It actually gives us an
opportunity to simplify a little:
lists/views.py.
defview_list(request,list_id):list_=List.objects.get(id=list_id)returnrender(request,'list.html',{'list':list_})
That, of course, will break because the template is expecting items:
AssertionError: False is not true : Couldn't find 'itemey 1' in response
But we can fix it in list.html, as well as adjusting the form’s POST action:
lists/templates/list.html (ch06l043).
<formmethod="POST"action="/lists/{{ list.id }}/add_item">[...] {% for item in list.item_set.all %}<tr><td>{{ forloop.counter }}: {{ item.text }}</td></tr>{% endfor %}
.item_set is called a “reverse lookup”—it’s one of Django’s incredibly
useful bits of ORM that lets you look up an object’s related items from a
different table…
So that gets the unit tests to pass:
Ran 10 tests in 0.060s OK
How about the FT?
$ python3 manage.py test functional_tests
Creating test database for alias 'default'...
.
---------------------------------------------------------------------
Ran 1 test in 5.824s
OK
Destroying test database for alias 'default'...Yes! And a quick check on our to-do list:
Irritatingly, the Testing Goat is a stickler for tying up loose ends too, so we’ve got to do this one final thing.
Before we start, we’ll do a commit—always make sure you’ve got a commit of a working state before embarking on a refactor:
$ git diff $ git commit -am "new URL + view for adding to existing lists. FT passes :-)"
A Final Refactor Using URL includes
superlists/urls.py is really meant for URLs that apply to your
entire site. For URLs that only apply to the lists app, Django encourages us
to use a separate lists/urls.py, to make the app more self-contained. The
simplest way to make one is to use a copy of the existing urls.py:
$ cp superlists/urls.py lists/Then we replace three lines in superlists/urls.py with an include. Notice
that include can take a part of a URL regex as a prefix, which will be
applied to all the included URLs (this is the bit where we reduce duplication,
as well as giving our code a better structure):
superlists/urls.py.
fromdjango.conf.urlsimportinclude,urlfromlistsimportviewsaslist_views#fromlistsimporturlsaslist_urls#urlpatterns=[url(r'^$',list_views.home_page,name='home'),url(r'^lists/',include(list_urls)),# url(r'^admin/', include(admin.site.urls)),]
-
While we’re at it, we use the
import x as ysyntax to aliasviewsandurls. This is good practice in your top-level urls.py, because it will let us import views and urls from multiple apps if we need to—and indeed we will need to later on in the book.
Back in lists/urls.py we can trim down to only include the latter part of our three URLs, and none of the other stuff from the parent urls.py:
lists/urls.py (ch06l045).
fromdjango.conf.urlsimporturlfromlistsimportviewsurlpatterns=[url(r'^new$',views.new_list,name='new_list'),url(r'^(\d+)/$',views.view_list,name='view_list'),url(r'^(\d+)/add_item$',views.add_item,name='add_item'),]
Rerun the unit tests to check everything worked. When I did it, I couldn’t quite believe I did it correctly on the first go. It always pays to be skeptical of your own abilities, so I deliberately changed one of the URLs slightly, just to check if it broke a test. It did. We’re covered.
Feel free to try it yourself! Remember to change it back, check the tests all pass again, and then commit:
$ git status $ git add lists/urls.py $ git add superlists/urls.py $ git diff --staged $ git commit
Phew. A marathon chapter. But we covered a number of important topics, starting with test isolation, and then some thinking about design. We covered some rules of thumb like “YAGNI” and “three strikes then refactor”. But, most importantly, we saw how to adapt an existing site step by step, going from working state to working state, in order to iterate towards a new design.
I’d say we’re pretty close to being able to ship this site, as the very first beta of the superlists website that’s going to take over the world. Maybe it needs a little prettification first…let’s look at what we need to do to deploy it in the next couple of chapters.
[9] Are you unable to move on because you’re wondering what those ch06l0xx things are, next to some of the code listings? They refer to specific commits in the book’s example repo. It’s all to do with my book’s correctness tests. You know, the tests for the tests in the book about testing. They have tests of their own, incidentally.
Get Test-Driven Development with Python now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
