Table of Contents
Practices of the Python Pro
Return to Python Bibliography
by Dane Hillard
Professional developers know the many benefits of writing application code that’s clean, well-organized, and easy to maintain. By learning and following established Python design patterns and Python best practices, you can take your code and your career to a new level. With Practices of the Python Pro, you’ll learn to design professional-level, clean, easily-maintainable software at scale using the incredibly popular programming language, Python. You’ll find easy-to-grok examples that use pseudocode and Python to introduce software development best practices, along with dozens of instantly-useful techniques that will help you code like a pro.
Dane Hillard has spent the majority of professional software development career building web applications using Python. He's passionate about introducing professional software development techniques to the many data scientists, business pros, and other self-taught programmers working with Python.
- Paperback: 305 pages
Dane Hillard
M A N N I N G
Practices of the Python Pro
SHELTER ISLAND
For online information and ordering of this and other Manning books, please visit https://manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact Special Sales Department, Manning Publications Co., 20 Baldwin Road, PO Box 761, Shelter Island, NY 11964
Email: [email protected]
©2020 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.
Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.
Manning Publications Co.
Development editor: Toni Arritola
20 Baldwin Road, Technical development editor: Nick Watts, PO Box 761, Review editor: Aleks Dragosavljevic Śhelter Island, NY 11964
Production editor: Lori Weidert
Copy editor: Andy Carroll
Proofreader: Carl Quesnel
Technical proofreader: Jens Christian Bredahl Madson
Typesetter: Gordan Salinovic
Cover designer: Marija Tudor
ISBN 9781617296086
Printed in the United States of America
Brief contents
PART 1 - WHY IT ALL MATTERS
1 ■ The bigger picture
PART 2 - FOUNDATIONS OF DESIGN
2 ■ Separation of concerns - 19
3 ■ Abstraction and encapsulation - 41
4 ■ Designing for high performance - 58
5 ■ Testing your software - 77
PART 3 - NAILING DOWN LARGE SYSTEMS
6 ■ Separation of concerns in practice - 103
7 ■ Extensibility and flexibility - 127
8 ■ The rules (and exceptions) of inheritance - 143
9 ■ Keeping things lightweight - 160
10 ■ Achieving loose coupling - 177
PART 4 - WHAT’S NEXT
11 ■ Onward and upward - 199
Full Table of Contents
preface
xiii
acknowledgments
xv
about this book
xvii
about the author
xxi
about the cover illustration
xxii
PART 1 WHY IT ALL MATTERS
The bigger picture
1.1 - Python is an enterprise language
The times they are a-changin’
■ What I like about Python
1.2 - Python is a teaching language
1.3 - Design is a process
The user experience
■ You’ve been here before
1.4 - Design enables better software
Considerations in software design
■ Organically grown software
1.5 - When to invest in design
1.6 - New beginnings
1.7 - Design is democratic
Presence of mind
1.8 - How to use this book
PART 2 FOUNDATIONS OF DESIGN
2.1 - Python Namespacing
- The many masks of importing
2.2 - Python hierarchy of separation
3 Python Abstraction and Python encapsulation
3.1 - What is abstraction?
The “black box”
■ Abstraction is like an onion
Abstraction is a simplifier
■ Decomposition enables
abstraction
3.2 - Python Encapsulation
Encapsulation constructs in Python
■ Expectations of privacy in Python
3.3 - Try it out
3.4 - Programming styles are an abstraction too
3.5 - Python Typing, inheritance, and polymorphism
3.6 - Recognizing the wrong abstraction
Square pegs in round holes
Clever gets the cleaver
4 * Python Designing for high performance
4.1 - Hurtling through time and space
Complexity is a little . . . complex
4.2 - Python performance and data types
Space complexity of operations on data types
4.3 - Make it work, make it right, make it fast
Making it work
68 ■ Making it right
68 ■ Making it fast
4.4 - Python Tools
72 ■ CPU profiling
4.5 - Try it out
5 Python Testing your software 77
5.1 - What is software testing?
Does it do what it says on the tin?
■ The anatomy of a Python functional test
5.2 - Python Functional testing approaches
5.3 - Statements of fact
5.4 - Python Unit testing with unittest
■ Try it out
■ Writing interesting tests
5.5 - * Python Testing with pytest
5.6 - Python Beyond functional testing
5.7 - * Python Test-driven development: A primer
It’s a mindset
It’s a philosophy
PART 3 NAILING DOWN LARGE SYSTEMS
6 * Python Separation of concerns in practice
6.1 - A command-line bookmarking application
6.2 - A tour of Bark
The benefits of separation: Reprise
6.3 - An initial code structure, by concern
106
The persistence layer
107 ■ The business logic layer
115
The presentation layer
119
CONTENTS
x
7 Extensibility and flexibility 127
7.1
What is extensible code?
127
Adding new behaviors
128 ■ Modifying existing behaviors
130
Loose coupling
131
7.2
Solutions for rigidity
133
Letting go: Inversion of control
133 ■ The devil’s in the details:
Relying on interfaces
136 ■ Fighting entropy: The robustness
principle
137
7.3
An exercise in extension
138
8 The rules (and exceptions) of inheritance 143
8.1
The inheritance of programming past
143
The silver bullet
144 ■ The challenges of hierarchies
144
8.2
The inheritance of programming present
146
What is inheritance for, really?
146 ■ Substitutability
147
The ideal use case for inheritance
148
8.3
Inheritance in Python
150
Type inspection
150 ■ Superclass access
151 ■ Multiple
inheritance and method resolution order
152 ■ Abstract base
classes
155
8.4
Inheritance and composition in Bark
157
Refactoring to use an abstract base class
157 ■ A final check on
your inheritance work
159
9 Keeping things lightweight 160
9.1
How big should my class/function/module be?
161
Physical size
161 ■ Single responsibility
161 ■ Code
complexity
162
9.2
Breaking down complexity
166
Extracting configuration
166 ■ Extracting functions
168
9.3
Decomposing classes
170
Initialization complexity
171 ■ Extracting classes and forwarding
calls
173
10 Achieving loose coupling 177
10.1
Defining coupling
177
The connective tissue
178 ■ Tight coupling
178 ■ Loose
coupling
181
CONTENTS
xi
10.2
Recognizing coupling
184
Feature envy
184 ■ Shotgun surgery
184 ■ Leaky
abstractions
185
10.3
Coupling in Bark
186
10.4
Addressing coupling
188
User messaging
189 ■ Bookmark persistence
191 ■ Try it
out
192
PAR T 4 WHAT’S NEXT? ……………………………………………197
11 Onward and upward 199
11.1
What now?
199
Develop a plan
200 ■ Execute the plan
201 ■ Track your
progress
203
11.2
Design patterns
204
Ups and downs of design patterns in Python
206 ■ Terms to start
with
206
11.3
Distributed systems
206
Modes of failure in distributed systems
207 ■ Addressing
application state
208 ■ Terms to start with
208
11.4
Take a Python deep dive
208
Python code style
208 ■ Language features are patterns
209
Terms to start with
210
11.5
Where you’ve been
210
There and back again: A developer’s tale
210 ■ Signing off
212
appendix
Installing Python
213
index
217
preface
Python, like me, was born in December of 1989. Although I’ve accomplished a great
deal in the subsequent three decades, Python’s success is prolific. More people than
ever before are picking it up to accomplish fascinating things in data science, machine
learning, and more. Since I learned Python, this “second-best language for every-
thing” has in reality been my first choice for many endeavors.
I had a rather traditional path into programming through the Electrical Engineer-
ing and Computer Science Department at the University of Michigan. At that time,
the coursework focused mainly on C++ and MATLAB—languages I continued to use
in my first job out of school. I developed some shell scripting and SQL chops in my
next position, processing big data for bioinformatics. I also started using PHP to work
on a personal WordPress site from scratch.
Although I was getting results (and cool ones, in some cases), none of the lan-
guages I was using resonated with me. But I was oblivious. I assumed that programming
languages were purely means to an end, and they had little chance of being fun to
work with. Around this time, a friend invited me to join him in a hackathon project to
build a Ruby library.
The world exploded with color, fruits tasted sweeter, and all that. The ease of using
an interpreted language and the human-friendly syntax of Ruby really made me think
about the tools I’d been using. Although I didn’t stick with Ruby for too long, I
decided to give Python and the Django web framework a try for the next iteration of
my personal site. It gave me the same joy and shallow learning curve I’d seen with
Ruby, and I haven’t looked back since!
xiii
PREFACE
xiv
Now that Python is recognized widely as a language of choice for many tasks, folks
coming into software development don’t need to go through the trial and error pro-
cess I did. New and interesting pathways into a career in software are opening up all
around too. Despite these differences, I hope we can all share in the common experi-
ence of finding joy in programming with Python. I also hope this book can contribute
to that joy.
Come along on the wonderful Python journey I fell into somewhat haphazardly. I
want to see you build a website, a data pipeline, or an automated plant-watering sys-
tem. Whatever you fancy. Python’s got your back. Send photos and code samples of
your projects to [email protected].
acknowledgments
I didn’t write this book alone. My appreciation runs deep for everyone who helped me
along the way, at every stage and in every capacity. You are loved.
Most anyone who’s been involved in the production of a book can tell you that it’s
always more work than you think. I heard this many times throughout the process,
and it certainly was a lot of work. What’s not always clear is that the real struggle is bal-
ancing all that extra work with your existing life.
To my partner, Stefanie: your support, encouragement, and tolerance of my rant-
ing and raving were paramount in making this book a reality. Thank you for judging
my neglect lightly and extricating me from this project during the roughest times. I
could not have done this without you.
Thank you to my parents, Kim and Donna, for always funneling my energy toward
curiosity, creativity, and compassion.
Thanks to my dear friend Vincent Zhang for spending countless nights at the cof-
fee shop coding by my side. You were there when the concept for this book was born,
and your validation helped spur me to take on this endeavor.
Thank you to James Nguyen for persevering as you changed paths to become a
developer. You embody the audience for this book, and your input has been invalu-
able. I’m proud of your accomplishments.
My gratitude goes to all my colleagues at ITHAKA and beyond for your input and
support. I thank you for enduring what has undoubtedly been a flighty period for me.
xv
ACKNOWLEDGMENTS
xvi
To Toni Arritola, my editor: thank you for your determination in pushing me ever
toward higher-quality teaching. The writing process is fraught with many unexpected
snags, but you provided me consistency and stability. Thank you.
To Nick Watts, my technical editor: your feedback has pushed the content of this
book from frantic ramblings to plausible software teachings. Your candor and insight
are much appreciated.
Thank you to Mike Stephens and Marjan Bace at Manning for believing in this
idea and trusting me as its shepherd. Thank you to everyone at Manning for working
tirelessly to bring authors’ ideas to life.
To all the reviewers—Al Krinker, Bonnie Bailey, Burkhard Nestmann, Chris Way-
man, David Kerns, Davide Cadamuro, Eriks Zelenka, Graham Wheeler, Gregory
Matuszek, Jean-François Morin, Jens Christian Bredahl Madsen, Joseph Perenia, Mark
Thomas, Markus Maucher, Mike Stevens, Patrick Regan, Phil Sorensen, Rafael Cas-
semiro Freire, Richard Fieldsend, Robert Walsh, Steven Parr, Sven Stumpf, and Willis
Hampton—your suggestions helped make this a better book.
A final thank you to anyone and everyone else who has had a positive influence—
directly, intentionally, or otherwise—on my journey in programming and this book. I
cannot hope to produce an exhaustive list; names not appearing here are due
expressly to the limitations of my own mind. Thank you to Mark Brehob, Dr. Andrew
DeOrio, Jesse Sielaff, Trek Glowacki, everyone at SAIC (in our little Ann Arbor office),
everyone at Compendia Bioscience (and friends), Brandon Rhodes, Kenneth Love,
Trey Hunner, Jeff Triplett, Mariatta Wijaya, Ali Spittel, Chris Coyier, Sarah Drasner,
David Beazley, Dror Ayalon, Tim Allen, Sandi Metz, and Martin Fowler.
about this book
Practices of the Python Pro introduces several concepts that software developers in almost
any language can use to improve their work. This would be a great book to read after
learning the fundamentals of the Python language.
Who should read this book
Practices of the Python Pro is for anyone in the early stages of their programming jour-
ney. In fact, people outside the software industry altogether who use software to sup-
plement their work can find value in this book. The concepts contained in these pages
will help readers build software that’s more maintainable, which in turn makes their
software easier to collaborate on.
In the sciences, reproducibility and provenance are important aspects of the
research process. As more research comes to rely on software, code that people can
understand, update, and improve is a major consideration. But college curricula are
still catching up to this intersection of software with other disciplines. For those with
limited experience in formal software development, this book provides a set of princi-
ples for producing shareable, reusable software.
If you’re seasoned in object-oriented programming and domain-driven design, you
may find this book too introductory for your benefit. On the other hand, if you’re rel-
atively new to Python, software, or software design, give this book a try. There’s some-
thing in here for you.
xvii
ABOUT THIS BOOK
xviii
How this book is organized: A roadmap
Practices of the Python Pro consists of 11 chapters in 4 parts. Parts 1 and 2 provide discus-
sion along with short examples and an occasional exercise. Part 3 builds on what
you’ve learned in earlier chapters and contains a variety of exercises. Part 4 provides
strategies for learning more, along with recommendations about what to try after
reading this book.
Part 1, “Why it all matters,” sets the stage for Python’s rise to fame and why software
design is valuable.
■
Chapter 1 covers some recent history of Python and why I enjoy developing
Python programs. It goes on to explain software design, why it’s important, and
how it manifests in your day-to-day work.
Part 2, “Foundations of design,” covers the high-level concepts that underpin software
design and development.
■
Chapter 2 covers separation of concerns, a fundamental activity that provides a
basis for several others in the book.
■
Chapter 3 explains abstraction and encapsulation, showing you how hiding
information and providing simpler interfaces to more complex logic helps you
keep a handle on your code.
■
Chapter 4 prompts you to think about performance, covering different data
structures, approaches, and tools to help you build speedy programs.
■
Chapter 5 teaches you about testing your software, using a variety of approaches,
from unit testing to end-to-end testing.
Part 3, “Nailing down large systems,” walks you through building a real application
using the principles you’ve learned.
■
Chapter 6 introduces the application you’ll build in the book and provides
exercises for creating a program’s foundation.
■
Chapter 7 covers the concepts of extensibility and flexibility and includes exer-
cises that add extensibility to the application.
■
Chapter 8 helps you understand class inheritance, providing recommendations
about where and when it should be used. It continues on with exercises that
examine inheritance in the application you’re building.
■
Chapter 9 steps back a bit, introducing tools and an approach for keeping code
from growing too large as you go along.
■
Chapter 10 explains loose coupling, providing some final exercises to reduce
the coupling in the application you’re building.
Part 4, “What’s next?” gives you some recommendations for how and what to learn next.
■
Chapter 11 shows you how I map out new learning material and gives you a few
areas of study to try if you’re interested in going deeper into software
development.
ABOUT THIS BOOK
xix
I recommend reading Practices of the Python Pro from cover to cover, though you may
choose to skip chapters in parts 1 and 2 if you’re familiar with the material. Part 3 is
best read in order so you can go through the exercises in a linear fashion.
There’s an appendix that will help you install Python, should you need it:
■
The appendix covers which version of Python you should install, along with the
most common approaches folks use to install it on their systems.
About the code
You can get the full source code for the book’s examples and exercises in the book’s
repository on GitHub (https://github.com/daneah/practices-of-the-python-pro).
Alternatively, you can visit the book’s homepage (www.manning.com/books/prac-
tices-of-the-python-pro) and click Source Code to download the code.
This book contains many examples of source code, both in numbered listings and
in line with normal text. In both cases, source code is formatted in a fixed-width
font like this to separate it from ordinary text.
In many cases, the original source code has been reformatted; we’ve added line
breaks and reworked indentation to accommodate the available page space in the
book. In rare cases, even this was not enough, and listings include line-continuation
markers (➥). Additionally, comments in the source code have often been removed
from the listings when the code is described in the text. Code annotations accompany
many of the listings, highlighting important concepts.
For each chapter, the code is organized into Python modules that are referenced
in the text. In general, you’re expected to write your own version of the code and use
the provided source only to check your work. In part 3, the projects in each chapter
build on the code from previous chapters, but each chapter provides a full working
copy of the source.
All code in this book is written in Python 3, and more specifically is intended to
work with Python 3.7+. Most of the code could be made to work on earlier versions
without much fuss, but consider installing a relatively new version of Python for use
with this book.
liveBook discussion forum
Purchase of Practices of the Python Pro includes free access to a private web forum run by
Manning Publications where you can make comments about the book, ask technical
questions, and receive help from the author and from other users. To access the
forum, go to https://livebook.manning.com/#!/book/practices-of-the-python-pro/
discussion. You can also learn more about Manning’s forums and the rules of conduct at https://livebook.manning.com/#!/discussion.
Manning’s commitment to our readers is to provide a venue where a meaningful
dialogue between individual readers and between readers and the author can take
place. It is not a commitment to any specific amount of participation on the part of
the author, whose contribution to the forum remains voluntary (and unpaid). We
ABOUT THIS BOOK
xx
suggest you try asking the author some challenging questions lest his interest stray!
The forum and the archives of previous discussions will be accessible from the
publisher’s website as long as the book is in print.
about the author
Dane Hillard is currently a lead web application developer at ITHAKA, a nonprofit in
higher education. His prior experience includes building inference engines for telem-
etry data and ETL pipelines for bioinformatics applications.
Dane’s first forays into programming included creating custom styling for his
MySpace page, scripting for the Rhinoceros 3D modeling application, and making
custom skins and weapons for the MS-DOS game Liero. He enjoys creative coding and
is actively seeking ways to combine his loves of music, photography, food, and software.
Dane has spoken at Python and Django conferences internationally and plans to
continue until someone asks him to stop.
xxi
about the cover illustration
Saint-Sauver
The figure on the cover of Practices of the Python Pro is captioned “Homme Finnois,” or
“Finnish Man.” The illustration is taken from a collection of dress costumes from vari-
ous countries by Jacques Grasset de Saint-Sauveur (1757–1810), titled Costumes de Dif-
férents Pays, published in France in 1797. Each illustration is finely drawn and colored
by hand. The rich variety of Grasset de Saint-Sauveur’s collection reminds us vividly of
how culturally apart the world’s towns and regions were just 200 years ago. Isolated
from each other, people spoke different dialects and languages. In the streets or in
the countryside, it was easy to identify where they lived and what their trade or station
in life was just by their dress.
The way we dress has changed since then and the diversity by region, so rich at the
time, has faded away. It is now hard to tell apart the inhabitants of different conti-
nents, let alone different towns, regions, or countries. Perhaps we have traded cultural
diversity for a more varied personal life—certainly for a more varied and fast-paced
technological life.
At a time when it is hard to tell one computer book from another, Manning cele-
brates the inventiveness and initiative of the computer business with book covers
based on the rich diversity of regional life of two centuries ago, brought back to life by
Grasset de Saint-Sauveur’s pictures.
xxii
Part 1
Why it all matters
When you set out to learn new topics, it’s important to consider the big
picture, to frame and focus your thinking. The first part of this book will famil-
iarize you with Python’s importance in modern software development, and it will
provide a framework for understanding the value of software design principles
and practices in furthering your career in programming.
Whether you’re new to programming, looking for the next language you’d
like to learn, or trying to advance your skills to tackle bigger projects, this part of
the book should convince you that Python is a great choice.
The bigger picture
This chapter covers
Using Python in complex software projects
Getting familiar with the high-level process of
software design
Recognizing when you should invest in design
I’m glad you picked up this book; it means you’d like to take the next step with soft-
ware development. Maybe you’re looking to enter the software industry, or maybe
you’re looking to use software to supplement your work. Maybe you’ve even been
paid to write software before. Congratulations—you’re already a pro! Coding like a
pro just means learning the concepts and strategies that will help you build and
maintain big software for the long term.
By reading on, you’re committing yourself to learning how Python can help you
think big and go from writing utility scripts to writing complex software. I’ll help
you lay a foundation on which you can construct your software development skills.
Throughout your career, you will likely be exposed to ever-increasing software
complexity. That software could be something you build over time, or it could very
well be an existing heap of code thrust upon you at the most inopportune moment.
3
4
CHAPTER 1
The bigger picture
Whatever the case, you’ll want to have a suite of utilities at your disposal so you can be
prepared to make sense of it.
By reading this book, you’ll gain experience and familiarity with how complex
software systems work so that you can use that expertise to improve upon them. You’ll
be learning how to envision these kinds of systems before building them to minimize
surprises and risks. Once you’re through with this book, you should be able to dive
headlong into things that you’re confused or anxious about now with a newfound
enthusiasm.
You’ll learn about putting the complexities of your code into easy-to-understand,
reusable wrappers. You’ll make sure your code is neatly organized by its purpose so
you can remember what’s what. These tools will help you help yourself and become
more productive in your projects, both new and old!
I’m going to use Python as the vehicle for the examples in this book. Python has
been my favorite programming language for some time now, and I hope it’s one of
yours too. If you haven’t had a chance to get to know Python much yet, take the time
to do that first. The Quick Python Book, third edition, by Naomi Ceder (Manning, 2018),
is a great place to get started.
All examples in this book are written with a recent version of Python 3 in mind. I
strongly recommend you install Python 3 before proceeding. See the appendix if you
need some guidance on the installation process.
The great divide
Are you using Python 2 or Python 3? A sizable number of people are still using Python
2, even though Python 3 came onto the scene a while ago— quite a while ago, in
2008. To put that in perspective, Flo Rida’s “Low” and Alicia Keys’ “No One” were at
the top of the charts that year.
Python 3 brought with it several backward-incompatible changes whose effects are
still being felt today. Many of these changes have been backported to later versions
of Python 2 to ease the transition. Developers on large projects using Python 2 have
some hurdles to overcome, but some people seem to be taking their Python 2 soft-
ware to the grave with them.
If you need a bit of convincing about why Python is a good choice of language, read
on a bit further.
1.1
Python is an enterprise language
The Python programming language has been treated historically as a scripting lan-
guage. Developers perceived its performance and applicability negatively, choosing
other languages for their enterprise software needs. Python was used for small data-
processing jobs or personal tools, but enterprise software was still a job for languages
like Java, C, or SAS.
Python is a teaching language
5
1.1.1
The times they are a-changin’
Over the last few years, the notion that Python couldn’t stand up to enterprise use has
shifted dramatically. Python is now being applied to nearly every discipline out there,
from robotics to machine learning to chemistry. Python has powered some of the most
successful internet companies of the last decade and doesn’t show any signs of slowing.
1.1.2
What I like about Python
Python is a breath of fresh air. Like many of my friends and colleagues, I learned a
great deal of C++ in school, along with a bit of MATLAB, Perl, and PHP. I built my first
website in PHP and even tried a Java Spring version at one point. PHP and Java are, as
many successful companies will attest, perfectly capable languages in this arena, but
they didn’t click with me for some reason.
I found that Python excelled in its syntax; this is often cited as one reason for its
accelerating popularity. The syntax comes closer to written English than other lan-
guages, and as a result it can be more approachable for those new to programming, as
well as for people who don’t like the verbosity of other languages. I’ve seen people
light up with joy when asking Python to print('Hello world!') and seeing it do
exactly that. Even now I will occasionally have one of those moments when I uncover a
standard library module I didn’t know about before.
Python is readable. This translates to faster development even for fairly seasoned
developers. Hui Ding, an engineer at Instagram, astutely points out that “Perfor-
mance speed is no longer the primary worry. Time to market speed is.”1 Python
enables rapid prototyping and, as you’ll see later on, the ability to solidify software
into a robust, maintainable codebase. This is what I like about Python.
1.2
Python is a teaching language
In 2017, Stack Overflow revealed that, in high-income countries, questions related to Python
made up more than 10% of all questions on the platform, surpassing all other major pro-
gramming languages.2 Python is the fastest growing programming language today, which is
why it’s a handy teaching tool. The thriving developer community and wealth of information
available online mean that it will be a safe choice for the next several years.
Throughout this book, I’ll assume you have a foundational knowledge of Python
syntax, data types, and classes. You’ve seen it and played with it, but you don’t need to
have won awards with it. (Do they have those?). Anyone with a bit of programming
under their belt and a few hours of learning and using Python on their own should
have no problem with the code in this book. You’re going to go through this book
with Python as the conduit for designing bigger, better software. That being said, what
you learn here will, with any luck, be applicable to any language you choose to use.
You’ll find that many software design concepts transcend any particular technology.
1
Michelle Gienow, “Instagram Makes a Smooth Move to Python 3,” The New Stack, http://mng.bz/Ze0j. This is a great write-up on Instagram’s transition from Python 2 to Python 3.
2
See David Robinson, “The Incredible Growth of Python,” Stack Overflow Blog, http://mng.bz/m48n.
6
CHAPTER 1
The bigger picture
1.3
Design is a process
Although the word design often describes a tangible outcome, the value of design is in
the process of arriving at that outcome. Consider fashion designers. Their goal is ulti-
mately to create pieces that will end up in the hands of the people wearing them. For
the designer to reach customers with the next great trend, though, a lot of steps—and
people—are involved (see figure 1.1).
Each interaction the designer has
requires passing information about
the fabric, pattern, cost, etc.
Fabric supplier
Designer
Fulfillment
Retailer
Customer
Each person needs specific information to perform
Pattern maker
a specific task. They’re fairly independent, but it
takes them all to get the job done.
Figure 1.1
The workflow for a fashion designer. The designer works with a number of other people to get the job
done.
Designers usually work with a fabric supplier to source the right materials for the look,
fit, and texture they want. Once they’ve designed a piece, they work with a patterner to
get different sizes made. Once they produce the pieces, they’re sent through fulfillment
to retail stores where customers can finally buy the clothing. This can take months!
As in fashion, art, and architecture, design in software is the process of sketching
out the plans for a system so that it can be executed for maximum effect. In software,
these plans help us understand the flow of data and the pieces of the system operating
on that data. Figure 1.2 shows a high-level diagram of an e-commerce workflow, out-
lining how a user would progress through the steps.
Each interaction the customer has
requires passing information about
Each step involves a specific set of
who they are, where they live, their
tasks based on the information
credit card information, etc.
provided by the customer.
Authentication
Shipping
Payment
Order
Customer
Figure 1.2
The workflow for an e-commerce website. The system performs a number of activities to
get the job done.
Design is a process
7
A customer looking to buy something online usually logs in, enters their shipping
information, and pays for the item. This creates an order for the company to process
and ship. Workflows like these require a great deal of design to nail down. The soft-
ware that runs these systems tackles complex rules, error-state checking, and more.
And it has to do it all without missing a beat, because users are sensitive to errors.
They might abandon or even actively speak out against a product that isn’t working
well for them.
1.3.1
The user experience
Workflows that appear concise and clear often take a lot of work to create. Creating
software that works smoothly for all use cases requires market research, user testing,
and robust design. Some products work well for the intended use case, but companies
may find after release that users are doing something totally unexpected with the
product. The software may work for that use case, but it wasn’t optimized for it. There
may be gaps in the design that need to be considered.
When software works well, we hardly notice. People using software products like to
have a frictionless experience, and developers working on software like it too. Working
with code that hasn’t been maintained can lead to frustration, and not knowing how
to fix it can lead to anger! Take a deep breath.
Friction
Imagine ice skating at the local hockey rink. When you get on the ice right after the
Zamboni finishes smoothing it out, skating requires little effort. You can lean into
each step just a little, letting the skate do the work. After some time, everyone’s
skates start to cut up the ice. It gets more difficult to glide; you have to push hard
into each step.
Friction in a user experience is a lot like the rough ice. The user may still be able to
accomplish what they’re trying to do, but that doesn’t mean it’s fun. A frictionless
experience is one that guides users along lightly, to the point that they hardly notice
they’re doing work.
Say you’ve been tasked with updating the reporting software at your company. It’s cur-
rently using comma-separated values (CSV) in its export files, but users have been
talking about how much they like tab-separated values (TSV). You think, “I’ll just go
update the delimiter in the output function to a tab instead of a comma!” Now imag-
ine opening up the code to find that the lines of output are all being built up like so:
print(col1_name + ',' + col2_name + ',' + col3_name + ',' + col4_name)
print(first_val + ',' + second_val + ',' + third_val + ',' + fourth_val)
To change the output from CSV to TSV, you’d have to make sure you changed the
comma to a tab in six places. This leaves some room for human error; maybe you saw
8
CHAPTER 1
The bigger picture
the first line printing the header but missed the line printing the data rows. To make
this more friendly to the next developer who uses the code, you can store the delim-
iter value in a constant and make use of it where needed. You could also use a Python
function to make building the string easier on yourself. Then, when users decide they
like the commas better after all, the change could be made in just one place:
DELIMITER = '\t'
print(DELIMITER.join([col1_name, col2_name, col3_name, col4_name]))
print(DELIMITER.join([first_val, second_val, third_val, fourth_val]))
By sitting down and thinking through the system at a high level, you’ll start to notice
rough areas you didn’t see before, or realize that certain assumptions you had aren’t
accurate. You’ll surprise yourself more than once, and this kind of enlightenment can
motivate you to keep at it. Once you start seeing repeated patterns and common
mistakes, you can start recognizing which thorns can be pulled out. It can be quite
therapeutic.
1.3.2
You’ve been here before
Whether you realize it or not, you’ve almost certainly gone through a design process
in the past. Think of a time when you stopped writing code for a moment to revisit the
goal you were trying to achieve. Did you notice something that made you change
direction? Did you see a more efficient way of doing things?
These little moments are design processes in themselves. You take stock of the goal
and current state of your software and use them together to inform what you do next.
Generating these moments intentionally and early on in your software process will
have both short- and long-term benefits.
1.4
Design enables better software
I’ll level with you: good design requires time and effort. It’s not something you get for
free. Although embedding design thinking into the development work you do
everyday is ideal, an independent design step before writing (or rewriting) your code
is crucial.
Planning out a software system will help you uncover areas that present risk. You
can identify where sensitive user information might be exposed to a vulnerability. You
can also see which pieces of the system might be performance bottlenecks or single
points of failure.
You can save time and money by simplifying, combining, or splitting up pieces of
the system. Gains like this are difficult to identify when looking at a component in iso-
lation because it isn’t clear whether other components are doing similar jobs. Viewing
the system as a whole allows you to regroup and make informed decisions about the
path forward.
Design enables better software
9
1.4.1
Considerations in software design
We often think about writing software for “the user,” but software can often serve mul-
tiple audiences. Sometimes “the user” is a person using the product the software is a
part of, whereas other times “the user” is a person trying to develop additional fea-
tures of the software. Often, you’re the only user of your software! By looking at soft-
ware from these different points of view, you can better identify the qualities of the
software you want to build.
The following are some common aspects consumers use to assess software for their
use cases:
Speed—The software does its job as quickly as it can.
Integrity—Data used or created by the software is protected from corruption.
Resources—The software uses disk space and network bandwidth efficiently.
Security—Users of the software can read and write only data for which they’re
authorized.
In addition, these are some common outcomes you as a developer might want:
Loose coupling—Components of the software are not intricately dependent on
one another.
Intuitability—Developers can discover the nature of the software and how it
works by reading it.
Flexibility—Developers can adapt the software to related or similar tasks.
Extensibility—Developers can add or change one aspect of the software without
affecting other aspects.
The pursuit of these outcomes often involves real-world costs. As an example, commit-
ting to increasing security in your software likely means you’ll have to spend more
time in development. Because that development time may increase your expenses,
you may choose to sell your software at a higher price. Effective planning and an
understanding of the trade-offs between these outcomes will help you minimize the
costs to you and your consumers.
Programming languages don’t typically address most of these considerations head
on; they simply provide tools that will enable developers to cater to them. For exam-
ple, high-level languages like Python, which allow developers to write in something simi-
lar to human language instead of machine language, provide some protections in
terms of memory corruption. Python also encourages the use of efficient data types
through its syntax; you’ll learn more about this in chapter 4.
That being said, there’s still a lot of work we can do on our own, because even
Python can’t predict all the ways developers might screw things up. This is where care-
ful design and thinking about the system as a whole will help.
10
CHAPTER 1
The bigger picture
1.4.2
Organically grown software
Unlike the produce at your local farmers’ market, organically grown software is not
good for your health. In the context of software, a system that has grown organically
over time is likely a system ripe for refactoring. Refactoring code is the process of updat-
ing code so it’s better designed and reflects your latest best practices. It might involve
improving the performance, maintainability, or readability of code.
As the term suggests, organically grown software has become an organism, com-
plete with a nervous system and a mind of its own. Bits of other software may have
been plastered onto it (usually more than once), methods that haven’t been used in
years are in there somewhere, rotting, and maybe there’s one function that does
about 150% of the work. Choosing when to refactor a system like this can be difficult,
but it’s sometime before the moment that makes you yell, “It’s alive!”
An example of this phenomenon is shown in figure 1.3, which depicts the check-
out process for an e-commerce site. It involves several important steps:
1
Determine that the product is available in the inventory.
2
Based on the price of the product, calculate the subtotal.
3
Based on the region of purchase, calculate:
a
Tax
b
Shipping and handling
4
Based on the current promotions, calculate any discounts.
5
Calculate the final total.
6
Process the payment.
7
Fulfill the order.
In this system, some of the steps are separated clearly. Not bad! But there is a rough
patch in the middle. It looks like all of the price-related logic happens in one big chunk.
If there’s a bug in that process, it might be difficult to understand exactly which step con-
tains the bug. You may see that the price is wrong, but there will be a lot of code to sift
through to figure out why. The payment processing and fulfillment are also lumped
Check inventory
Calculate
cart subtotal,
This area handles a lot of logic!
tax, shipping,
Checkout
discounts,
final total
Customer
Process payment
What happens when the payment
and send to
succeeds but fulfillment fails?
fulfillment
Figure 1.3
An e-commerce system that grew organically
When to invest in design
11
together, so with an ill-timed error it’s possible you could process the payment success-
fully but never fulfill the order. That would make for a disgruntled customer.
A good start on the path to making this workflow more robust is to split its logical
steps up (figure 1.4). If each step is handled by its own service, the service for a partic-
ular step only needs to concern itself with one job. The inventory service keeps track of
how many items are in stock. The pricing service knows the cost and tax for each item.
This isolates each step from the others, making each one less likely to suffer from bugs.
Purchase info can be persisted here
so fulfillment can retry as needed.
Purchase
Checkout
complete
Trigger order
Purchase
Send to fulfillment
Customer
Inventory
Pricing
Shipping
Payment
Fulfillment
service
service
service
service
service
Fulfillment
Purchase info can be persisted here
service
so fulfillment can retry as needed.
Figure 1.4
What a thoughtfully planned e-commerce system might look like
Design often allows you to see where a system’s existing pieces can be broken down
into simpler ones. This idea of decomposition is just one of the tools we’ll explore more
thoroughly in the chapters to come. Keep in mind that this work is almost never done;
refactoring and redesigning code will happen constantly. By internalizing some of the
techniques you’ll learn in this book, though, you’ll find these tasks get easier and
quicker in a given project over time. Stay sharp and recognize opportunities for
improving your existing code!
1.5
When to invest in design
We tend to focus our efforts on creating new software to complete tasks. But as proj-
ects grow, we forget about the implementation of working code until it gets in our way.
Some code gets in the way so often that it creates more trouble than value. At this
point the project incurs technical debt, because additional work must be done to
remain productive.
The more frequently a gnarly piece of code gets in the way, and the more difficult
it is to deal with it when it gets in the way, the more time you should allot to getting in
there and sweeping up the mess. This is often based on a gut feeling after a system is
already built, but sometimes you can catch things early.
12
CHAPTER 1
The bigger picture
Intentional software design up front can save time and headaches down the road.
When software is flexible enough to be extended to new use cases, it can be a pleasure
to work with, so putting thought into the system before writing a line of code is a good
way to keep productivity up. I like to think of this as a technical investment because it’s
putting work in up front for a later return.
One place you may have encountered this is in a framework. Frameworks are large
libraries of code that act as guides to some goal. A framework might help you make your
website look wonderful, or it may help you build a neural network for detecting faces
in video. Regardless of its function, a framework seeks to provide the building blocks
that you can use to make something all your own. For a framework to be useful, it must
be flexible enough to handle a variety of use cases and extensible enough that you can
write new functionality that the original developers didn’t think of. Python developers
have created numerous frameworks: Requests, for making HTTP calls; Flask and
Django, for web development; and Pandas, for data analysis, to name a few. In a way,
much of the code you write is a framework. It provides some useful functionality that
you may need to use again and again or for different purposes along the way. Writing
your code with these facts in mind will keep you from putting hurdles in your own way.
The process of designing software, whether revisiting a project or starting a new
one, is an investment. The hope is that the return on this investment will be code that
adapts to the needs of developers and consumers without incurring a great deal of
overhead or frustration. There will be times when some code is in poor shape but may
not warrant the time and effort good design can require. How often the code is used
or updated is an important consideration, because spending weeks improving a script
that’s used once or twice in its lifetime isn’t economical.
1.6
New beginnings
When you set out to be more mindful of design, the opportunities for improvement
can become overwhelming. There is so much to learn and do that trying to manage it
all at once won’t be fun. Taking on design concepts little by little, until they become a
part of your mindset, is a more sustainable approach to success. In this book, I’ll intro-
duce small sets of concepts in each chapter, and you can revisit particular chapters at
any time to reinforce what you learned there.
1.7
Design is democratic
Up to now, it’s quite possible that you’ve worked on projects mostly by yourself. If you
did any coding as part of a class, you may have been required to write all the code your-
self. In the real world, this doesn’t happen often for large projects. In companies writ-
ing software for business uses, there may be tens of developers working on a single
product. Each developer has a unique set of experiences that can affect how they
choose to work. This diversity of viewpoints can lead to a more robust system because
experiences with previous bugs, failures, and successes all inform directions to take in
upcoming work.
Design is democratic
13
It’s to your benefit to get input from other developers, especially at the early stages.
There’s rarely one way of doing something, so learning many approaches, along with
their pros and cons, will empower you to make educated choices, or at least to choose
what feels best if all other things are equal. Some approaches will make sense for one
use case but not for another, so knowing several will increase your productivity.
If you don’t have the privilege of working with an active team of developers,
examining some open source projects is another way to get some exposure to the
collaborative nature of software. Look for discussions where developers disagreed
(constructively!) about how to achieve some task, and see what kinds of considerations
came into play on the way to a resolution. The thought process that leads to a solution
is often more important than the specific solution the developers choose. This kind of
reasoning and discussion capability will get you through more difficulties than knowing
a specific algorithm.
1.7.1
Presence of mind
It’s easy to get carried away when writing software. Think about a time when you were
excited to get something done. You were probably anxious to see your code work, and
it’s often difficult in that situation to sit still and be deliberate about writing perfect
code.
When working with a small script or doing some exploratory work, a quick feed-
back cycle can be valuable in staying productive. I often do this kind of work in
Python’s read-eval-print loop (REPL).
The REPL
The REPL—pronounced REH-pull—is what’s hiding behind the »> when you type
python at the terminal. It reads what you type, evaluates it, prints the result, and waits for it all to happen again (the loop). Many languages provide a REPL so developers can interactively test a few lines of code.
But beware: at some point, the back and forth of writing a quick line of code and see-
ing how it changes the program’s output becomes tedious. You’ll want to write length-
ier or longer-lived code in a file and run it with the interpreter. Each person has a
different threshold; I usually hit mine when I want to reuse a line of code I previously
wrote, and it’s 15 lines back in my history.
The example in listing 1.1 shows how you might work through transforming a dictio-
nary of data. Given a dictionary that maps states in the United States to their capital
cities, you want to produce a list of all capital cities in alphabetical order. The
approach is something like this:
1
Get the city values from the dictionary.
2
Sort the city values.
14
CHAPTER 1
The bigger picture
Listing 1.1
Getting the United States capitals in alphabetical order
us_capitals_by_state = {
A dictionary that maps state
'Alabama': 'Montgomery',
names to capital names
'Alaska': 'Juneau',
…
}
capitals = us_capitals_by_state.values()
Only the capital names
dict_values(['Montgomery', 'Juneau'])
capitals.sort()
Whoops! This isn’t a “list”, so
Traceback (most recent call last):
no “sort” method is available.
File “<stdin>”, line 1, in <module>
AttributeError: 'dict_values' object has no attribute 'sort'
sorted(capitals)
['Albany', 'Annapolis', …]
New (sorted) list using “sorted”,
which accepts any iterable
This task wasn’t too bad; there was only one fumble along the way. But as a project
grows and the scope of the change you’re making increases, taking a step back and
planning your actions in advance is helpful.
Some thoughtful planning will often save you time in the long run because you
won’t be going two steps forward and one step back as you develop. If you do this up
front, you can also get into a good habit of recognizing opportunities to refactor as
they happen, rather than when you’re further down the road. When I’m in this mode,
I typically shift to writing my code in a real Python module, even if I’m still writing a
pretty short script. This encourages me to slow down a bit and keep the bigger goal in
mind during development.
In the case of the state capitals code, imagine you’ll need the list of state capitals in
many contexts. You might need it on a registration form, a shipping form, or a billing
form. To avoid doing the same calculation over and over, you could wrap that calcula-
tion in a function and call it whenever you need it, as shown in the following listing.
Listing 1.2
Wrapping the state capital logic in a function
def get_united_states_capitals():
Same code as listing
us_capitals_by_state = {'Alabama': …}
1.1, in a function
capitals = us_capitals_by_state.values()
return sorted(capitals)
Now you have a reusable function. But looking at this function, you can see that it
operates on constant data but does a bit of calculation each time it’s called. If this
function is called frequently in the program, it can be refactored further to improve
its performance.
In fact, it turns out a function isn’t necessary at all. You can achieve the reusability
while still making only one set of calculations by storing the result in a constant for
later use, as shown in the following listing.
How to use this book
15
Listing 1.3
Refactored code reveals a more concise solution
US_CAPITALS_BY_STATE = {'Alabama': 'Montgomery', …}
Constant data,
US_CAPITALS = sorted(US_CAPITALS_BY_STATE.values())
defined once
Also constant, no need for a function;
just reference “US_CAPITALS”.
This has the added benefit of cutting the number of lines of code in half without sacri-
ficing readability.
The process we just went through from initial problem statement to a final solu-
tion is a design process. As you progress, you may find that you can identify areas for
improvement earlier and earlier. Eventually, you may even decide to start drawing
high-level diagrams that represent several complex pieces of software, using your dia-
grams to assess opportunities and risk before writing any code. Not everyone works
this way, of course, so you’ll need to use what you learn in this book where it gives you
the most value.
You might be feeling the urge to scrap everything and start your project anew at
this point, but hold on! As you go through this book, you’ll see that the processes of
designing and refactoring software are not only interrelated, but in fact are two sides
of the same coin. Doing one often means doing the other, and they’re both continu-
ous processes throughout the life of a project. Nothing and no one is perfect either, so
it’s valuable to revisit code early and often, especially when you start to feel friction.
With that in mind, take a deep breath and relax. There’s plenty more to cover.
1.8
How to use this book
Generally speaking, this book is best experienced from cover to cover. I’ve laid out the
parts of the book so they build on one another; later parts use concepts from earlier
parts. In part 3, each chapter builds upon a software project you’ll start in chapter 6.
But feel free to skim or skip chapters that cover things you already know, with the
caveat that you may need to thumb back to an earlier chapter from time to time.
Most chapters will leave you in a good place to incorporate a new concept or prac-
tice into your software development routine. If there’s a chapter whose concepts you
find particularly valuable, you might want to work on applying those concepts to your
projects until you’ve got the hang of them. Once you feel comfortable, you can come
back and read the next chapter.
Remember that the code for the examples and exercises is in this book’s GitHub
repository (https://github.com/daneah/practices-of-the-python-pro), and also remember that most of the source code is meant as a way to check your own work after complet-
ing an exercise. Use the provided code if you’re stuck or you want to compare solutions,
but give each exercise your own effort first.
Happy coding!
16
CHAPTER 1
The bigger picture
Summary
Python pulls as much weight in complex, enterprise projects as other major
programming languages.
Python has one of the fastest growing user bases of any programming language.
Design isn’t only a thing you draw on paper; it’s the process you follow to get
there.
Design up front is an investment that will reward you with clean and flexible code
later on.
You need to build software with a diverse audience in mind.