Exploring Python's stdlib

About me

Baptiste Mispelon

Online: bmispelon

Web developer

Django core dev since June 2013

Content of the talk

• Builtin functions
• Little-known usage of common builtins
• Little-known modules

Notes

• Nothing too advanced
• Lots of bite-sized examples
• Python 3

Table of content

• Builtin functions
• Little-known usage of common builtins
• Little-known modules

Part 1: Builtins

Python's builtins

Short Selection

dir

Quick and dirty introspection.

List of all the attributes/methods of an object (as strings).

Example

What are all the public attributes of this object?

obj = 'hello world'
[attr for attr in dir(obj)
    if not attr.startswith('_')]

Example

What are all the public attributes of this object?

obj = 'hello world'
[attr for attr in dir(obj)
    if not attr.startswith('_')]
# ['capitalize', 'casefold',
#  'center', 'count', ...]

divmod

Given two numbers, return a pair of quotient, remainder.

Example

a, b = 42, 5
q, r = a // b, a % b
q, r # ?

Example

a, b = 42, 5
q, r = a // b, a % b
q, r # 8, 2

Example

a, b = 42, 5
q, r = divmod(a, b)
q, r # 8, 2

slice

Create slice objects.

Example

s = 'abcdefghi'
s[ :8]   # ?
s[1:8]   # ?
s[1:8:2] # ?

Example

s = 'abcdefghi'
s[ :8]   # 'abcdefgh'
s[1:8]   # ?
s[1:8:2] # ?

Example

s = 'abcdefghi'
s[ :8]   # 'abcdefgh'
s[1:8]   # 'bcdefgh'
s[1:8:2] # ?

Example

s = 'abcdefghi'
s[ :8]   # 'abcdefgh'
s[1:8]   # 'bcdefgh'
s[1:8:2] # 'bdfh'

With slice

s = 'abcdefghi'
s[ :8]   == s[slice(8)]
s[1:8]   == s[slice(1, 8)]
s[1:8:2] == s[slice(1, 8, 2)]

set

Unordered collection of distinct objects.

Advantages

• Behave like lists: iter(set), len(set), x in set
• Elements are unique
• x in set is O(1)

Usage

>>> set('abcabcabc')

Usage

>>> set('abcabcabc')
{'c', 'b', 'a'}

Usage

>>> set('abcabcabc')
{'c', 'b', 'a'}
>>> len({1, 2, 3, 2, 1})

Usage

>>> set('abcabcabc')
{'c', 'b', 'a'}
>>> len({1, 2, 3, 2, 1})
3

Usage

>>> set('abcabcabc')
{'c', 'b', 'a'}
>>> len({1, 2, 3, 2, 1})
3
>>> sum({1, 2, 3, 2, 1})

Usage

>>> set('abcabcabc')
{'c', 'b', 'a'}
>>> len({1, 2, 3, 2, 1})
3
>>> sum({1, 2, 3, 2, 1})
6

Usage

>>> set('abcabcabc')
{'c', 'b', 'a'}
>>> len({1, 2, 3, 2, 1})
3
>>> sum({1, 2, 3, 2, 1})
6
>>> 3 in {1, 2, 3}

Usage

>>> set('abcabcabc')
{'c', 'b', 'a'}
>>> len({1, 2, 3, 2, 1})
3
>>> sum({1, 2, 3, 2, 1})
6
>>> 3 in {1, 2, 3}
True

Restrictions

• Elements must be hashable (like keys of a dict)
• Not indexable: some_set[0] does not work.
• No order (like a dict)

Set literals

Since python 2.7:

set([1, 2, 3])
# Is the same as doing
{1, 2, 3}

Set comprehensions

Since python 2.7 (too):

set(w.lower() for w in wordlist)
# Is the same as doing
{w.lower() for w in wordlist}

Operators

• intersection: s1 & s2
• union: s1 | s2
• difference: s1 - s2
• symmetric difference: s1 ^ s2
• Subset/Superset: s1 < s2

Examples

{1, 2} & {2, 3}
{1, 2} | {2, 3}
{1, 2} - {2, 3}
{1, 2} ^ {2, 3}
{1, 2} < {2, 3}
{1, 2} < {3, 2, 1}

Examples

{1, 2} & {2, 3} # {2}
{1, 2} | {2, 3}
{1, 2} - {2, 3}
{1, 2} ^ {2, 3}
{1, 2} < {2, 3}
{1, 2} < {3, 2, 1}

Examples

{1, 2} & {2, 3} # {2}
{1, 2} | {2, 3} # {1, 2, 3}
{1, 2} - {2, 3}
{1, 2} ^ {2, 3}
{1, 2} < {2, 3}
{1, 2} < {3, 2, 1}

Examples

{1, 2} & {2, 3} # {2}
{1, 2} | {2, 3} # {1, 2, 3}
{1, 2} - {2, 3} # {1}
{1, 2} ^ {2, 3}
{1, 2} < {2, 3}
{1, 2} < {3, 2, 1}

Examples

{1, 2} & {2, 3} # {2}
{1, 2} | {2, 3} # {1, 2, 3}
{1, 2} - {2, 3} # {1}
{1, 2} ^ {2, 3} # {1, 3}
{1, 2} < {2, 3}
{1, 2} < {3, 2, 1}

Examples

{1, 2} & {2, 3} # {2}
{1, 2} | {2, 3} # {1, 2, 3}
{1, 2} - {2, 3} # {1}
{1, 2} ^ {2, 3} # {1, 3}
{1, 2} < {2, 3}    # False
{1, 2} < {3, 2, 1}

Examples

{1, 2} & {2, 3} # {2}
{1, 2} | {2, 3} # {1, 2, 3}
{1, 2} - {2, 3} # {1}
{1, 2} ^ {2, 3} # {1, 3}
{1, 2} < {2, 3}    # False
{1, 2} < {3, 2, 1} # True

frozenset

Like a set, but immutable.

Why?

Making it immutable makes it hashable.

This means we can use it as the key of a dictionnary.

Or put it in another set/frozenset.

Why?

Making it immutable makes it hashable.

This means we can use it as the key of a dictionnary.

Or put it in another set/frozenset.

(Yo Dawg, I heard you like sets...)

Table of content

• Builtin functions
• Little-known usage of common builtins
• Little-known modules

Part 2: Builtins with a twist

iter

Make an iterator out of an iterable.

Iterator vs. Iterable

Iterable

An object that can be iterated over (for x in y)

Iterator

A type of iterable that can only go forward (no rewinding).

Example

>>> L = list('abc')

Example

>>> L = list('abc')
>>> ''.join(c.upper() for c in L)

Example

>>> L = list('abc')
>>> ''.join(c.upper() for c in L)
'ABC'

Example

>>> L = list('abc')
>>> ''.join(c.upper() for c in L)
'ABC'
>>> ''.join(c.upper() for c in L)

Example

>>> L = list('abc')
>>> ''.join(c.upper() for c in L)
'ABC'
>>> ''.join(c.upper() for c in L)
'ABC'

Example (2)

>>> I = iter('abc')

Example (2)

>>> I = iter('abc')
>>> ''.join(c.upper() for c in I)

Example (2)

>>> I = iter('abc')
>>> ''.join(c.upper() for c in I)
'ABC'

Example (2)

>>> I = iter('abc')
>>> ''.join(c.upper() for c in I)
'ABC'
>>> ''.join(c.upper() for c in I)

Example (2)

>>> I = iter('abc')
>>> ''.join(c.upper() for c in I)
'ABC'
>>> ''.join(c.upper() for c in I)
''

The Twist

If you pass a second argument, iter behaves quite differently.

iter(callable, sentinel)

iter(callable, sentinel)

Call the first argument yielding the return value and stopping when the sentinel value is reached (or when the call raises StopIteration).

Example

fp = open('mydata.txt')
S = 'STOP\n'
for line in iter(fp.readline, S):
    print(line)

Example

fp = open('mydata.txt')
S = 'STOP\n'
for line in iter(fp.readline, S):
    print(line)
# Print all lines and stop when
# reading the one containing "STOP"

Example (2)

from random import choice
def roll():
    return choice(range(1, 7))
list(iter(roll, 6)) # []
list(iter(roll, 6)) # [4, 4, 1]

ne×t

Advance an iterator and return the value.

Raise StopIteration if empty.

Example

i = iter('abc')
next(i) # ?
next(i) # ?
next(i) # ?
next(i) # ?

Example

i = iter('abc')
next(i) # 'a'
next(i) # ?
next(i) # ?
next(i) # ?

Example

i = iter('abc')
next(i) # 'a'
next(i) # 'b'
next(i) # ?
next(i) # ?

Example

i = iter('abc')
next(i) # 'a'
next(i) # 'b'
next(i) # 'c'
next(i) # ?

Example

i = iter('abc')
next(i) # 'a'
next(i) # 'b'
next(i) # 'c'
next(i) # StopIteration

The Twist

next takes a second argument.

Similar to dict.get, it's a value to return if the iterator is empty (instead of raising StopIteration).

Example (part 2)

i = iter('abc')
next(i, 'x') # 'a'
next(i, 'x') # 'b'
next(i, 'x') # 'c'
next(i, 'x') # ?

Example (part 2)

i = iter('abc')
next(i, 'x') # 'a'
next(i, 'x') # 'b'
next(i, 'x') # 'c'
next(i, 'x') # 'x'

Imaginary friends

I hope you've studied your maths lesson for this one...

Complex numbers

Did you know that python has first class support for complex numbers?

a = complex(real=4, imag=1)
b = complex(real=1, imag=2)
a + b     # ?

Complex numbers

Did you know that python has first class support for complex numbers?

a = complex(real=4, imag=1)
b = complex(real=1, imag=2)
a + b     # complex(5, 3)

Complex numbers

Did you know that python has first class support for complex numbers?

a = complex(real=4, imag=1)
b = complex(real=1, imag=2)
a + b     # complex(5, 3)
a + b + 1 # ?

Complex numbers

Did you know that python has first class support for complex numbers?

a = complex(real=4, imag=1)
b = complex(real=1, imag=2)
a + b     # complex(5, 3)
a + b + 1 # complex(6, 3)

The Twist

It also has imaginary number litterals:

1 + 2j # complex(1, 2)

The Twist

It also has imaginary number litterals:

1 + 2j  # complex(1, 2)
3j      # complex(0, 3)

The Twist

It also has imaginary number litterals:

1 + 2j  # complex(1, 2)
3j      # complex(0, 3)
1j ** 2 # ?

The Twist

It also has imaginary number litterals:

1 + 2j  # complex(1, 2)
3j      # complex(0, 3)
1j ** 2 # -1

Fun with maths

>>> from math import e, pi

Fun with maths

>>> from math import e, pi
>>> e ** (1j * pi)

Fun with maths

>>> from math import e, pi
>>> e ** (1j * pi)
-1

Fun with maths

>>> from math import e, pi
>>> e ** (1j * pi)
(-1+1.2246467991473532e-16j)
# Close enough :)

Table of content

• Builtin functions
• Little-known usage of common builtins
• Little-known modules

Part 3: Exotic Stdlib Modules

python -m FOO

Looks for a FOO.py file on your python path and executes it.

$ echo "print('Hello')" > FOO.py
$ python -m FOO
Hello

Easter eggs

python -m this

python -m __hello__

python -m antigravity

Utilities

Some standard modules are written in a way that they can be used as standalone programs (not just imported).

Uses the "trick":

if __name__ == '__main__':
    # Put code here

Share a directory over the network?

Share a directory over the network?

python -m http.server

Browse and dowload files from the current directory (on port 8000 by default).

Share a directory over the network?

python -m http.server
python2 -m SimpleHTTPServer

Browse and dowload files from the current directory (on port 8000 by default).

Let's play

class Foo(object):
    def foo(self):
        return 42

    def bar(self):
        return 43

Let's play

>>> f = Foo()

Let's play

>>> f = Foo()
>>> f.foo()

Let's play

>>> f = Foo()
>>> f.foo()
42

Let's play

>>> f = Foo()
>>> f.foo()
42
>>> f.bar()

Let's play

>>> f = Foo()
>>> f.foo()
42
>>> f.bar()
AttributeError: 'Foo' object has
 no attribute 'bar'

Let's play

>>> f = Foo()
>>> f.foo()
42
>>> f.bar()
AttributeError: 'Foo' object has
 no attribute 'bar'
# WAT!?

Let's play

class Foo(object):
    def foo(self):
        return 42

    def bar(self):
        return 43

Let's play

class Foo(object):
....def foo(self):
........return 42

--->def bar(self):
--->--->return 43

Let's play

class Foo(object):
    def foo(self):
        return 42

        def bar(self):
                return 43

Tab Nanny to the rescue!

python -m tabnanny your_file.py

Print a calendar of the current year?

Print a calendar of the current year?

python -m calendar

Print a calendar of the current year?

python -m calendar
python -m calendar 2015

Print a calendar of the current year?

python -m calendar
python -m calendar 2015 10

Encode a file in base64?

Encode a file in base64?

python -m base64 some_file

Open a URL in the browser?

Open a URL in the browser?

python -m webbrowser \
    "http://google.pl"

Get documentation for a function?

Get documentation for a function?

python -m pydoc str.find
python -m pydoc \
    yourmodule.yourfunction

List of functions in a file?

List of functions in a file?

python -m pyclbr your_file.py
python -m pyclbr modulename

List of functions in a file?

python -m pyclbr your_file.py
python -m pyclbr modulename

Safe to run on arbitrary files.

Batteries Included

An arbitrary selection of useful modules,

for use in python code.

Parsing CSV?

 with open('file.csv') as f:
     for line in f:
         row = line.split(',')
         print(row)

import csv

 import csv
 with open('file.csv') as f:
     for row in csv.reader(f):
         print(row)

Leap Years

 def is_leap(year):
     return (year % 4) == 0

Leap Years

 def is_leap(year):
     return (year % 4) == 0
 # Hmm... wait

Leap Years

 def is_leap(year):
     return (year % 4) == 0 \
         and (year % 100) != 0
 # Aah, that's better

Leap Years

 def is_leap(year):
     return (year % 4) == 0 \
         and (year % 100) != 0
 # ... or is it?

import calendar

 import calendar
 calendar.isleap(2013)

Prompting for a password

 password = input("Password: ")

import getpass

 from getpass import getpass
 password = getpass()

Alphabet

 s = 'abcdefghijklmnoprstuvwxyz'
 s = '0123456789'

import string

 import string
 string.ascii_lowercase
 string.ascii_uppercase
 string.digits
 string.punctuation
 string.whitespace

Temporary Files

 from random import choice as ch
 S = 'abcdefghjklmnopqrstuvwxyz'
 name = ''.join(
     ch(S) for _ in range(8))
 path = '/tmp/%s' % name
 temp = open(path, 'w')

import tempfile

 import tempfile
 temp = tempfile.TemporaryFile()

Passing files to a program

 import sys
 for fname in sys.argv[1:]:
     f = open(fname)
     for line in f:
         process(line)

import fileinput

 import fileinput
 lines = fileinput.fileinput()
 for line in lines:
     process(line)

import fileinput

 import fileinput
 lines = fileinput.fileinput()
 for line in lines:
     process(line)
 # It even supports - for STDIN

Wrapping text

 N = len(text) // 80
 for i in range(N):
     print(text[i*80:(i+1)*80])

import textwrap

 import textwrap
 lines = textwrap.wrap(text,
     width=80)
 print('\n'.join(lines))

import textwrap

 import textwrap
 print(textwrap.fill(text,
     width=80))

Python Keywords

 kw = ['class', 'for', 'if', ...]

import keyword

 import keyword
 keyword.kwlist

import keyword

 import keyword
 keyword.kwlist
 keyword.iskeyword('class')

Unicode

print('Kąt')

Unicode

print('Kąt')
# <insert joke about angular.js>

Unicode

print('Kąt')
print('K\x0105t')

Unicode

print('Kąt')
print('K\x0105t')
print('K\u0105t')

Unicode

print('Kąt')
print('K\x0105t')
print('K\u0105t')
print('K\U00000105t')

Unicode

print('Kąt')
print('K\x0105t')
print('K\u0105t')
print('K\U00000105t')
print('K\N{LATIN SMALL LETTER A\
 WITH OGONEK}t')

import unicodedata

>>> from unicodedata import (
    name, lookup)
>>> name('ą')
'LATIN SMALL LETTER A WITH OGONEK'
>>> lookup('LATIN SMALL LETTER A '
...        'WITH OGONEK')
'ą'

Listing Files

 import os
 for fname in os.listdir('.'):
     if fname.endswith('.txt'):
         print(fname)

import glob

 import glob
 for fname in glob.glob('*.txt'):
     print(fname)

And many more

colorsys, difflib, fractions, fnmatch, robotparser, wave, shlex, contextlib, inspect, ...

Table of content

• Builtin functions
• Little-known usage of common builtins
• Little-known modules

TLDL;

Dziękuję!