Templating in Python is a technique used in web development to dynamically generate static HTML pages using templates and data.
In this tutorial, we will explore the basics of templating in Python, including installation, creating templates, and rendering templates with data, with a focus on the Jinja2 templating engine.
String Templates in Python
String templates in Python is a simple way to perform string substitutions. Python’s string module includes the Template class, which provides an easy way to replace placeholders in a string with actual values.
The Template class in the string module is useful for dynamically forming a string object through a substitution technique described in PEP 292. Its simpler syntax and functionality make it easier to translate for internationalization purposes compared to other built-in string formatting facilities in Python.
Template strings use the $ symbol for substitution, immediately followed by an identifier that follows the rules of forming a valid Python identifier.
Creating a Template
To create a template, you instantiate the Template class with a string that contains placeholders prefixed with $ as shown below −
from string import Template
template = Template("Hello, $name!")
Substituting Values
You can substitute values into the template using the substitute() method, which takes a dictionary of key-value pairs.
The substitute() method replaces the placeholders (identifiers) in the template with actual values. You can provide these values using keyword arguments or a dictionary. The method then returns a new string with the placeholders filled in.
Example: Using Keyword Arguments
Following code substitute identifiers in a template string using keyword arguments −
from string import Template
tempStr = Template('Hello. My name is $name and my age is $age')
newStr = tempStr.substitute(name ='Pushpa', age =26)print(newStr)
It will produce the following output −
Hello. My name is Pushpa and my age is 26
Example: Using a Dictionary
In the following example, we use a dictionary object to map the substitution identifiers in the template string −
from string import Template
tempStr = Template('Hello. My name is $name and my age is $age')
dct ={'name':'Pushpalata','age':25}
newStr = tempStr.substitute(dct)print(newStr)
Following is the output of the above code −
Hello. My name is Pushpalata and my age is 25
Example: Missing Parameters Raises KeyError
If the substitute() method is not provided with sufficient parameters to be matched against the identifiers in the template string, Python raises KeyError −
from string import Template
tempStr = Template('Hello. My name is $name and my age is $age')
dct ={'name':'Pushpalata'}
newStr = tempStr.substitute(dct)print(newStr)
Following is the error produced −
Traceback (most recent call last):
File "/home/cg/root/667e441d9ebd5/main.py", line 5, in <module>
newStr = tempStr.substitute(dct)
File "/usr/lib/python3.10/string.py", line 121, in substitute
return self.pattern.sub(convert, self.template)
File "/usr/lib/python3.10/string.py", line 114, in convert
return str(mapping[named])
KeyError: 'age'
Substituting Values Using safe_substitute() Method
The safe_substitute() method behaves similarly to substitute() method, except for the fact that it doesn’t throw error if the keys are not sufficient or are not matching. Instead, the original placeholder will appear in the resulting string intact.
Example
In the following example, we are using the safe_substitue() method for substituting values −
from string import Template
tempStr = Template('Hello. My name is $name and my age is $age')
dct ={'name':'Pushpalata'}
newStr = tempStr.safe_substitute(dct)print(newStr)
It will produce the following output −
Hello. My name is Pushpalata and my age is $age
Installing Jinja2
To use Jinja2 for templating in Python, you first need to install the library. Jinja2 is a powerful templating engine that is widely used in web development for rendering HTML. It can be installed easily using pip, Python’s package installer −
pip install jinja2
Creating and Rendering Jinja2 Templates
Jinja2 is a powerful templating engine for Python that allows you to create dynamic content by blending static template files with data. This section explores how to create Jinja2 templates and render them with data.
Creating a Jinja2 Template
To create a Jinja2 template, you define a template string or load it from a file. Templates use double curly braces {{ … }} for placeholders and support control structures like “loops” and “conditionals” with {% … %}.
Example
Following is an example of a simple Jinja2 template stored in a file “template.html” −
<!DOCTYPE html><html><head><title>Hello,{{ name }}!</title></head><body><h1>Hello,{{ name }}!</h1><p>Welcome to our site.</p></body></html>
Rendering a Jinja2 Template
To render a Jinja2 template, follow these steps −
Loading the Template − Load the template from a file or create it from a string.
Creating a Template Object − Use “jinja2.Template” to create a template object.
Rendering − Use the render() method on the template object, passing data as arguments or a dictionary.
Example
In here, we are rendering Jinja2 template −
from jinja2 import Template, FileSystemLoader, Environment
# Loading a template from a file (template.html)
file_loader = FileSystemLoader('.')
env = Environment(loader=file_loader)
template = env.get_template('template.html')# Rendering the template with data
output = template.render(name='Alice')# Output the rendered templateprint(output)
The output of the rendered Jinja2 template would be an HTML document with the placeholders replaced by the actual data passed during rendering −
<!DOCTYPE html>
<html>
<head>
<title>Hello, Alice!</title>
</head>
<body>
<h1>Hello, Alice!</h1>
<p>Welcome to our site.</p>
</body>
</html>
Advanced Jinja2 Features
Jinja2 supports various advanced features such as loops, conditionals, and custom filters, making it a powerful tool for creating complex templates.
Template Inheritance
Jinja2 supports template inheritance, allowing you to create a base template with common elements (like headers, footers, navigation bars) and extend or override specific blocks in child templates. This promotes code reuse and maintainability in large projects.
Example
This HTML template file named “base.html” defines a basic structure for a web page using Jinja2 templating syntax.
It includes blocks “{% block title %}” and “{% block content %}” that can be overridden in derived templates to customize the title and main content of the page, respectively −
The following Jinja2 template file “child.html” extends the “base.html” template, overriding the title block to set it to “Child Page” and the content block to include an <h1> header with the text “Child Page Content”.
Jinja2 allows you to iterate over lists or other iterable objects using {% for %}loops. Following is an example of how you can use a loop to generate an unordered list (<ul>) in HTML −
<ul>{%for item in items %}<li>{{ item }}</li>{% endfor %}</ul>
Conditionals
Conditional statements in Jinja2 ({% if %} and {% else %}) is used to control the flow of your templates based on conditions. Here is an example where “Jinja2” checks if user exists and displays a personalized greeting if true; otherwise, it prompts to log in −
{%if user %}<p>Welcome,{{ user }}!</p>{%else%}<p>Please log in.</p>{% endif %}
Custom Filters
Custom filters in Jinja2 is used to define your own filters to manipulate data before displaying it in the template.
In the following example, a custom filter reverse is defined in Jinja2 to reverse the string “hello”, resulting in “olleh” when applied in the template −
# Define a custom filter functiondefreverse_string(s):return s[::-1]# Register the filter with the Jinja2 environment
env.filters['reverse']= reverse_string
In your template, you can then apply the “reverse” filter to any string −
Serialization refers to the process of converting an object into a format that can be easily stored, transmitted, or reconstructed later. In Python, this involves converting complex data structures, such as objects or dictionaries, into a byte stream.
Why Do We Use Serialization?
Serialization allows data to be easily saved to disk or transmitted over a network, and later reconstructed back into its original form. It is important for tasks like saving game states, storing user preferences, or exchanging data between different systems.
Serialization Libraries in Python
Python offers several libraries for serialization, each with its own advantages. Here is a detailed overview of some commonly used serialization libraries in Python −
Pickle − This is Python’s built-in module for serializing and deserializing Python objects. It is simple to use but specific to Python and may have security implications if used with untrusted data.
JSON − JSON (JavaScript Object Notation) is a lightweight data interchange format that is human-readable and easy to parse. It is ideal for web APIs and cross-platform communication.
YAML − YAML: YAML (YAML Ain’t Markup Language) is a human-readable data serialization standard that is also easy for both humans and machines to read and write. It supports complex data structures and is often used in configuration files.
Serialization Using Pickle Module
The pickle module in Python is used for serializing and deserializing objects. Serialization, also known as pickling, involves converting a Python object into a byte stream, which can then be stored in a file or transmitted over a network.
Deserialization, or unpickling, is the reverse process, converting the byte stream back into a Python object.
Serializing an Object
We can serialize an object using the dump() function and write it to a file. The file must be opened in binary write mode (‘wb’).
Example
In the following example, a dictionary is serialized and written to a file named “data.pkl” −
import pickle
data ={'name':'Alice','age':30,'city':'New York'}# Open a file in binary write modewithopen('data.pkl','wb')asfile:# Serialize the data and write it to the file
pickle.dump(data,file)print("File created!!")
When above code is executed, the dictionary object’s byte representation will be stored in data.pkl file.
Deserializing an Object
To deserialize or unpickle the object, you can use the load() function. The file must be opened in binary read mode (‘rb’) as shown below −
import pickle
# Open the file in binary read modewithopen('data.pkl','rb')asfile:# Deserialize the data
data = pickle.load(file)print(data)
This will read the byte stream from “data.pkl” and convert it back into the original dictionary as shown below −
{'name': 'Alice', 'age': 30, 'city': 'New York'}
Pickle Protocols
Protocols are the conventions used in constructing and deconstructing Python objects to/from binary data.
The pickle module supports different serialization protocols, with higher protocols generally offering more features and better performance. Currently pickle module defines 6 different protocols as listed below −
Sr.No.
Protocol & Description
1
Protocol version 0Original “human-readable” protocol backwards compatible with earlier versions.
2
Protocol version 1Old binary format also compatible with earlier versions of Python.
3
Protocol version 2Introduced in Python 2.3 provides efficient pickling of new-style classes.
4
Protocol version 3Added in Python 3.0. recommended when compatibility with other Python 3 versions is required.
5
Protocol version 4Introduced in Python 3.4. It adds support for very large objects.
6
Protocol version 5Introduced in Python 3.8. It adds support for out-of-band data.
You can specify the protocol by passing it as an argument to pickle.dump() function.
To know the highest and default protocol version of your Python installation, use the following constants defined in the pickle module −
The pickle module in Python also defines Pickler and Unpickler classes for more detailed control over the serialization and deserialization processes. The “Pickler” class writes pickle data to a file, while the “Unpickler” class reads binary data from a file and reconstructs the original Python object.
Using the Pickler Class
To serialize a Python object using the Pickler class, you can follow these steps −
from pickle import Pickler
# Open a file in binary write modewithopen("data.txt","wb")as f:# Create a dictionary
dct ={'name':'Ravi','age':23,'Gender':'M','marks':75}# Create a Pickler object and write the dictionary to the file
Pickler(f).dump(dct)print("Success!!")
After executing the above code, the dictionary object’s byte representation will be stored in “data.txt” file.
Using the Unpickler Class
To deserialize the data from a binary file using the Unpickler class, you can do the following −
from pickle import Unpickler
# Open the file in binary read modewithopen("data.txt","rb")as f:# Create an Unpickler object and load the dictionary from the file
dct = Unpickler(f).load()# Print the dictionaryprint(dct)
The pickle module can also serialize and deserialize custom classes. The class definition must be available at both the time of pickling and unpickling.
Example
In this example, an instance of the “Person” class is serialized and then deserialized, maintaining the state of the object −
# Create an instance of the Person class
person = Person('Alice',30,'New York')# Serialize the person objectwithopen('person.pkl','wb')asfile:
pickle.dump(person,file)# Deserialize the person objectwithopen('person.pkl','rb')asfile:
person = pickle.load(file)print(person.name, person.age, person.city)
After executing the above code, we get the following output −
Alice 30 New York
The Python standard library also includes the marshal module, which is used for internal serialization of Python objects. Unlike pickle, which is designed for general-purpose use, marshal is primarily intended for use by Python itself (e.g., for writing .pyc files).
It is generally not recommended for general-purpose serialization due to potential compatibility issues between Python versions.
Using JSON for Serialization
JSON (JavaScript Object Notation) is a popular format for data interchange. It is human-readable, easy to write, and language-independent, making it ideal for serialization.
Python provides built-in support for JSON through the json module, which allows you to serialize and deserialize data to and from JSON format.
Serialization
Serialization is the process of converting a Python object into a JSON string or writing it to a file.
Example: Serialize Data to a JSON String
In the example below, we use the json.dumps() function to convert a Python dictionary to a JSON string −
import json
# Create a dictionary
data ={"name":"Alice","age":25,"city":"San Francisco"}# Serialize the dictionary to a JSON string
json_string = json.dumps(data)print(json_string)
In here, we use the json.dump() function to write the serialized JSON data directly to a file −
import json
# Create a dictionary
data ={"name":"Alice","age":25,"city":"San Francisco"}# Serialize the dictionary and write it to a filewithopen("data.json","w")as f:
json.dump(data, f)print("Success!!")
Deserialization
Deserialization is the process of converting a JSON string back into a Python object or reading it from a file.
Example: Deserialize a JSON String
In the following example, we use the json.loads() function to convert a JSON string back into a Python dictionary −
import json
# JSON string
json_string ='{"name": "Alice", "age": 25, "city": "San Francisco"}'# Deserialize the JSON string into a Python dictionary
loaded_data = json.loads(json_string)print(loaded_data)
YAML (YAML Ain’t Markup Language) is a human-readable data serialization standard that is commonly used for configuration files and data interchange.
Python supports YAML serialization and deserialization through the pyyamlpackage, which needs to be installed first as shown below −
pip install pyyaml
Example: Serialize Data and Write to a YAML File
In the below example, yaml.dump() function converts the Python dictionary data into a YAML string and writes it to the file “data.yaml”.
The “default_flow_style” parameter ensures that the YAML output is more human-readable with expanded formatting −
import yaml
# Create a Python dictionary
data ={"name":"Emily","age":35,"city":"Seattle"}# Serialize the dictionary and write it to a YAML filewithopen("data.yaml","w")as f:
yaml.dump(data, f, default_flow_style=False)print("Success!!")
Example: Deserialize Data from a YAML File
Here, yaml.safe_load() function is used to safely load the YAML data from “data.yaml” and convert it into a Python dictionary (loaded_data) −
Using safe_load() is preferred for security reasons as it only allows basic Python data types and avoids executing arbitrary code from YAML files.
import yaml
# Deserialize data from a YAML filewithopen("data.yaml","r")as f:
loaded_data = yaml.safe_load(f)print(loaded_data)
Python uses reference counting mechanism while implementing garbage collection policy. Whenever an object in the memory is referred, the count is incremented by one. On the other hand, when the reference is removed, the count is decremented by 1. If the garbage collector running in the background finds any object with count as 0, it is removed and the memory occupied is reclaimed.
Weak reference is a reference that does not protect the object from getting garbage collected. It proves important when you need to implement caches for large objects, as well as in a situation where reduction of Pain from circular references is desired.
To create weak references, Python has provided us with a module named weakref.
The ref class in this module manages the weak reference to an object. When called, it retrieves the original object.
Calling the reference object after deleting the referent returns None.
It will produce the following output −
object: <__main__.Myclass object at 0x00000209D7173290>
reference: <weakref at 0x00000209D7175940; to 'Myclass' at
0x00000209D7173290>
call r(): <__main__.Myclass object at 0x00000209D7173290>
deleting obj
(Deleting <__main__.Myclass object at 0x00000209D7173290>)
r(): None
The callback Function
The constructor of ref class has an optional parameter called callback function, which gets called when the referred object is deleted.
import weakref
classMyclass:def__del__(self):print('(Deleting {})'.format(self))defmycallback(rfr):"""called when referenced object is deleted"""print('calling ({})'.format(rfr))
obj = Myclass()
r = weakref.ref(obj, mycallback)print('object:', obj)print('reference:', r)print('call r():', r())print('deleting obj')del obj
print('r():', r())
It will produce the following output −
object: <__main__.Myclass object at 0x000002A0499D3590>
reference: <weakref at 0x000002A0499D59E0; to 'Myclass' at
0x000002A0499D3590>
call r(): <__main__.Myclass object at 0x000002A0499D3590>
deleting obj
(Deleting <__main__.Myclass object at 0x000002A0499D3590>)
calling (<weakref at 0x000002A0499D59E0; dead>)
r(): None
Finalizing Objects
The weakref module provides finalize class. Its object is called when the garbage collector collects the object. The object survives until the reference object is called.
import weakref
class Myclass:
def __del__(self):
print('(Deleting {})'.format(self))
def finalizer(*args):
print('Finalizer{!r})'.format(args))
obj = Myclass()
r = weakref.finalize(obj, finalizer, "Call to finalizer")
print('object:', obj)
print('reference:', r)
print('call r():', r())
print('deleting obj')
del obj
print('r():', r())
It will produce the following output −
object: <__main__.Myclass object at 0x0000021015103590>
reference: <finalize object at 0x21014eabe80; for 'Myclass' at
0x21015103590>
Finalizer('Call to finalizer',))
call r(): None
deleting obj
(Deleting <__main__.Myclass object at 0x0000021015103590>)
r(): None
The weakref module provides WeakKeyDictionary and WeakValueDictionary classes. They don't keep the objects alive as they appear in the mapping objects. They are more appropriate for creating a cache of several objects.
WeakKeyDictionary
Mapping class that references keys weakly. Entries in the dictionary will be discarded when there is no longer a strong reference to the key.
An instance of WeakKeyDictionary class is created with an existing dictionary or without any argumentThe functionality is the same as a normal dictionary to add and remove mapping entries to it.
In the code given below three Person instances are created. It then creates an instance of WeakKeyDictionary with a dictionary where the key is the Person instance and the value is the Person's name.
We call the keyrefs() method to retrieve weak references. When the reference to Peron1 is deleted, dictionary keys are printed again. A new Person instance is added to a dictionary with weakly referenced keys. At last, we are printing keys of dictionary again.
Example
import weakref
class Person:
def __init__(self, person_id, name, age):
self.emp_id = person_id
self.name = name
self.age = age
Person1 = Person(101, "Jeevan", 30)
Person2 = Person(102, "Ramanna", 35)
Person3 = Person(103, "Simran", 28)
weak_dict = weakref.WeakKeyDictionary({Person1: Person1.name, Person2: Person2.name, Person3: Person3.name})
print("Weak Key Dictionary : {}\n".format(weak_dict.data))
print("Dictionary Keys : {}\n".format([key().name for key in weak_dict.keyrefs()]))
del Person1
print("Dictionary Keys : {}\n".format([key().name for key in weak_dict.keyrefs()]))
Person4 = Person(104, "Partho", 32)
weak_dict.update({Person4: Person4.name})
print("Dictionary Keys : {}\n".format([key().name for key in weak_dict.keyrefs()]))
It will produce the following output −
Weak Key Dictionary : {<weakref at 0x7f542b6d4180; to 'Person' at 0x7f542b8bbfd0>: 'Jeevan', <weakref at 0x7f542b6d5530; to 'Person' at 0x7f542b8bbeb0>: 'Ramanna', <weakref at 0x7f542b6d55d0; to 'Person' at 0x7f542b8bb7c0>: 'Simran'}
Dictionary Keys : ['Jeevan', 'Ramanna', 'Simran']
Dictionary Keys : ['Ramanna', 'Simran']
Dictionary Keys : ['Ramanna', 'Simran', 'Partho']
WeakValueDictionary
Mapping class that references values weakly. Entries in the dictionary will be discarded when no strong reference to the value exists any more.
We shall demonstrate how to create a dictionary with weakly referenced values using WeakValueDictionary.
The code is similar to previous example but this time we are using Person name as key and Person instance as values. We are using valuerefs() method to retrieve weakly referenced values of the dictionary.
Example
import weakref
class Person:
def __init__(self, person_id, name, age):
self.emp_id = person_id
self.name = name
self.age = age
Person1 = Person(101, "Jeevan", 30)
Person2 = Person(102, "Ramanna", 35)
Person3 = Person(103, "Simran", 28)
weak_dict = weakref.WeakValueDictionary({Person1.name:Person1, Person2.name:Person2, Person3.name:Person3})
print("Weak Value Dictionary : {}\n".format(weak_dict.data))
print("Dictionary Values : {}\n".format([value().name for value in weak_dict.valuerefs()]))
del Person1
print("Dictionary Values : {}\n".format([value().name for value in weak_dict.valuerefs()]))
Person4 = Person(104, "Partho", 32)
weak_dict.update({Person4.name: Person4})
print("Dictionary Values : {}\n".format([value().name for value in weak_dict.valuerefs()]))
It will produce the following output −
Weak Value Dictionary : {'Jeevan': <weakref at 0x7f3af9fe4180; to 'Person' at 0x7f3afa1c7fd0>, 'Ramanna': <weakref at 0x7f3af9fe5530; to 'Person' at 0x7f3afa1c7eb0>, 'Simran': <weakref at 0x7f3af9fe55d0; to 'Person' at 0x7f3afa1c77c0>}
Dictionary Values : ['Jeevan', 'Ramanna', 'Simran']
Dictionary Values : ['Ramanna', 'Simran']
Dictionary Values : ['Ramanna', 'Simran', 'Partho']
Database access in Python is used to interact with databases, allowing applications to store, retrieve, update, and manage data consistently. Various relational database management systems (RDBMS) are supported for these tasks, each requiring specific Python packages for connectivity −
GadFly
MySQL
PostgreSQL
Microsoft SQL Server
Informix
Oracle
Sybase
SQLite
and many more…
Data input and generated during execution of a program is stored in RAM. If it is to be stored persistently, it needs to be stored in database tables.
Relational databases use SQL (Structured Query Language) for performing INSERT/DELETE/UPDATE operations on the database tables. However, implementation of SQL varies from one type of database to other. This raises incompatibility issues. SQL instructions for one database do not match with other.
DB-API (Database API)
To address this issue of compatibility, Python Enhancement Proposal (PEP) 249 introduced a standardized interface known as DB-API. This interface provides a consistent framework for database drivers, ensuring uniform behavior across different database systems. It simplifies the process of transitioning between various databases by establishing a common set of rules and methods.
Using SQLite with Python
Python’s standard library includes sqlite3 module, a DB_API compatible driver for SQLite3 database. It serves as a reference implementation for DB-API. For other types of databases, you will have to install the relevant Python package −
Database
Python Package
Oracle
cx_oracle, pyodbc
SQL Server
pymssql, pyodbc
PostgreSQL
psycopg2
MySQL
MySQL Connector/Python, pymysql
Working with SQLite
Using SQLite with Python is very easy due to the built-in sqlite3 module. The process involves −
Connection Establishment − Create a connection object using sqlite3.connect(), providing necessary connection credentials such as server name, port, username, and password.
Transaction Management − The connection object manages database operations, including opening, closing, and transaction control (committing or rolling back transactions).
Cursor Object − Obtain a cursor object from the connection to execute SQL queries. The cursor serves as the gateway for CRUD (Create, Read, Update, Delete) operations on the database.
In this tutorial, we shall learn how to access database using Python, how to store data of Python objects in a SQLite database, and how to retrieve data from SQLite database and process it using Python program.
The sqlite3 Module
SQLite is a server-less, file-based lightweight transactional relational database. It doesn’t require any installation and no credentials such as username and password are needed to access the database.
Python’s sqlite3 module contains DB-API implementation for SQLite database. It is written by Gerhard Hring. Let us learn how to use sqlite3 module for database access with Python.
Let us start by importing sqlite3 and check its version.
A connection object is set up by connect() function in sqlite3 module. First positional argument to this function is a string representing path (relative or absolute) to a SQLite database file. The function returns a connection object referring to the database.
Various methods are defined in connection class. One of them is cursor() method that returns a cursor object, about which we shall know in next section. Transaction control is achieved by commit() and rollback() methods of connection object. Connection class has important methods to define custom functions and aggregates to be used in SQL queries.
The Cursor Object
Next, we need to get the cursor object from the connection object. It is your handle to the database when performing any CRUD operation on the database. The cursor() method on connection object returns the cursor object.
We can now perform all SQL query operations, with the help of its execute() method available to cursor object. This method needs a string argument which must be a valid SQL statement.
Creating a Database Table
We shall now add Employee table in our newly created ‘testdb.sqlite3’ database. In following script, we call execute() method of cursor object, giving it a string with CREATE TABLE statement inside.
import sqlite3
conn=sqlite3.connect('testdb.sqlite3')
cur=conn.cursor()
qry='''
CREATE TABLE Employee (
EmpID INTEGER PRIMARY KEY AUTOINCREMENT,
FIRST_NAME TEXT (20),
LAST_NAME TEXT(20),
AGE INTEGER,
SEX TEXT(1),
INCOME FLOAT
);
'''try:
cur.execute(qry)print('Table created successfully')except:print('error in creating table')
conn.close()
When the above program is run, the database with Employee table is created in the current working directory.
We can verify by listing out tables in this database in SQLite console.
sqlite>.open mydb.sqlite
sqlite>.tables
Employee
INSERT Operation
The INSERT Operation is required when you want to create your records into a database table.
Example
The following example, executes SQL INSERT statement to create a record in the EMPLOYEE table −
You can also use the parameter substitution technique to execute the INSERT query as follows −
import sqlite3
conn=sqlite3.connect('testdb.sqlite3')
cur=conn.cursor()
qry="""INSERT INTO EMPLOYEE(FIRST_NAME,
LAST_NAME, AGE, SEX, INCOME)
VALUES (?, ?, ?, ?, ?)"""try:
cur.execute(qry,('Makrand','Mohan',21,'M',5000))
conn.commit()print('Record inserted successfully')except Exception as e:
conn.rollback()print('error in INSERT operation')
conn.close()
READ Operation
READ Operation on any database means to fetch some useful information from the database.
Once the database connection is established, you are ready to make a query into this database. You can use either fetchone() method to fetch a single record or fetchall() method to fetch multiple values from a database table.
fetchone() − It fetches the next row of a query result set. A result set is an object that is returned when a cursor object is used to query a table.
fetchall() − It fetches all the rows in a result set. If some rows have already been extracted from the result set, then it retrieves the remaining rows from the result set.
rowcount − This is a read-only attribute and returns the number of rows that were affected by an execute() method.
Example
In the following code, the cursor object executes SELECT * FROM EMPLOYEE query. The resultset is obtained with fetchall() method. We print all the records in the resultset with a for loop.
import sqlite3
conn=sqlite3.connect('testdb.sqlite3')
cur=conn.cursor()
qry="SELECT * FROM EMPLOYEE"try:# Execute the SQL command
cur.execute(qry)# Fetch all the rows in a list of lists.
results = cur.fetchall()for row in results:
fname = row[1]
lname = row[2]
age = row[3]
sex = row[4]
income = row[5]# Now print fetched resultprint("fname={},lname={},age={},sex={},income={}".format(fname, lname, age, sex, income ))except Exception as e:print(e)print("Error: unable to fecth data")
UPDATE Operation on any database means to update one or more records, which are already available in the database.
The following procedure updates all the records having income=2000. Here, we increase the income by 1000.
import sqlite3
conn=sqlite3.connect('testdb.sqlite3')
cur=conn.cursor()
qry="UPDATE EMPLOYEE SET INCOME = INCOME+1000 WHERE INCOME=?"try:# Execute the SQL command
cur.execute(qry,(1000,))# Fetch all the rows in a list of lists.
conn.commit()print("Records updated")except Exception as e:print("Error: unable to update data")
conn.close()
DELETE Operation
DELETE operation is required when you want to delete some records from your database. Following is the procedure to delete all the records from EMPLOYEE where INCOME is less than 2000.
import sqlite3
conn=sqlite3.connect('testdb.sqlite3')
cur=conn.cursor()
qry="DELETE FROM EMPLOYEE WHERE INCOME<?"try:# Execute the SQL command
cur.execute(qry,(2000,))# Fetch all the rows in a list of lists.
conn.commit()print("Records deleted")except Exception as e:print("Error: unable to delete data")
conn.close()
Performing Transactions
Transactions are a mechanism that ensure data consistency. Transactions have the following four properties −
Atomicity − Either a transaction completes or nothing happens at all.
Consistency − A transaction must start in a consistent state and leave the system in a consistent state.
Isolation − Intermediate results of a transaction are not visible outside the current transaction.
Durability − Once a transaction was committed, the effects are persistent, even after a system failure.
The Python DB API 2.0 provides two methods to either commit or rollback a transaction.
Example
You already know how to implement transactions. Here is a similar example −
# Prepare SQL query to DELETE required records
sql ="DELETE FROM EMPLOYEE WHERE AGE > ?"try:# Execute the SQL command
cursor.execute(sql,(20,))# Commit your changes in the database
db.commit()except:# Rollback in case there is any error
db.rollback()
COMMIT Operation
Commit is an operation, which gives a green signal to the database to finalize the changes, and after this operation, no change can be reverted back.
Here is a simple example to call the commit method.
db.commit()
ROLLBACK Operation
If you are not satisfied with one or more of the changes and you want to revert back those changes completely, then use the rollback() method.
Here is a simple example to call the rollback() method.
db.rollback()
The PyMySQL Module
PyMySQL is an interface for connecting to a MySQL database server from Python. It implements the Python Database API v2.0 and contains a pure-Python MySQL client library. The goal of PyMySQL is to be a drop-in replacement for MySQLdb.
Installing PyMySQL
Before proceeding further, you make sure you have PyMySQL installed on your machine. Just type the following in your Python script and execute it −
import PyMySQL
If it produces the following result, then it means MySQLdb module is not installed −
Traceback (most recent call last):
File "test.py", line 3, in <module>
Import PyMySQL
ImportError: No module named PyMySQL
The last stable release is available on PyPI and can be installed with pip −
pip install PyMySQL
Note − Make sure you have root privilege to install the above module.
MySQL Database Connection
Before connecting to a MySQL database, make sure of the following points −
You have created a database TESTDB.
You have created a table EMPLOYEE in TESTDB.
This table has fields FIRST_NAME, LAST_NAME, AGE, SEX and INCOME.
User ID “testuser” and password “test123” are set to access TESTDB.
Python module PyMySQL is installed properly on your machine.
You have gone through MySQL tutorial to understand MySQL Basics.
Example
To use MySQL database instead of SQLite database in earlier examples, we need to change the connect() function as follows −
import PyMySQL
# Open database connection
db = PyMySQL.connect("localhost","testuser","test123","TESTDB")
Apart from this change, every database operation can be performed without difficulty.
Handling Errors
There are many sources of errors. A few examples are a syntax error in an executed SQL statement, a connection failure, or calling the fetch method for an already cancelled or finished statement handle.
The DB API defines a number of errors that must exist in each database module. The following table lists these exceptions.
Sr.No.
Exception & Description
1
WarningUsed for non-fatal issues. Must subclass StandardError.
2
ErrorBase class for errors. Must subclass StandardError.
3
InterfaceErrorUsed for errors in the database module, not the database itself. Must subclass Error.
4
DatabaseErrorUsed for errors in the database. Must subclass Error.
5
DataErrorSubclass of DatabaseError that refers to errors in the data.
6
OperationalErrorSubclass of DatabaseError that refers to errors such as the loss of a connection to the database. These errors are generally outside of the control of the Python scripter.
7
IntegrityErrorSubclass of DatabaseError for situations that would damage the relational integrity, such as uniqueness constraints or foreign keys.
8
InternalErrorSubclass of DatabaseError that refers to errors internal to the database module, such as a cursor no longer being active.
9
ProgrammingErrorSubclass of DatabaseError that refers to errors such as a bad table name and other things that can safely be blamed on you.
10
NotSupportedErrorSubclass of DatabaseError that refers to trying to call unsupported functionality.
A regular expression is a special sequence of characters that helps you match or find other strings or sets of strings, using a specialized syntax held in a pattern. Regular expression are popularly known as regex or regexp.
Usually, such patterns are used by string-searching algorithms for “find” or “find and replace” operations on strings, or for input validation.
Large scale text processing in data science projects requires manipulation of textual data. The regular expressions processing is supported by many programming languages including Python. Python’s standard library has remodule for this purpose.
Since most of the functions defined in re module work with raw strings, let us first understand what the raw strings are.
Raw Strings
Regular expressions use the backslash character (‘\’) to indicate special forms or to allow special characters to be used without invoking their special meaning. Python on the other hand uses the same character as escape character. Hence Python uses the raw string notation.
A string become a raw string if it is prefixed with r or R before the quotation symbols. Hence ‘Hello’ is a normal string were are r’Hello’ is a raw string.
In normal circumstances, there is no difference between the two. However, when the escape character is embedded in the string, the normal string actually interprets the escape sequence, where as the raw string doesn’t process the escape character.
>>> normal="Hello\nWorld" >>>print(normal) Hello World >>> raw=r"Hello\nWorld" >>>print(raw) Hello\nWorld
In the above example, when a normal string is printed the escape character ‘\n’ is processed to introduce a newline. However because of the raw string operator ‘r’ the effect of escape character is not translated as per its meaning.
Metacharacters
Most letters and characters will simply match themselves. However, some characters are special metacharacters, and don’t match themselves. Meta characters are characters having a special meaning, similar to * in wild card.
Here’s a complete list of the metacharacters −
.^ $ *+ ? {}[] \ |()
The square bracket symbols[ and ] indicate a set of characters that you wish to match. Characters can be listed individually, or as a range of characters separating them by a ‘-‘.
Sr.No.
Metacharacters & Description
1
[abc]match any of the characters a, b, or c
2
[a-c]which uses a range to express the same set of characters.
3
[a-z]match only lowercase letters.
4
[0-9]match only digits.
5
‘^’complements the character set in [].[^5] will match any character except’5’.
‘\’is an escaping metacharacter. When followed by various characters it forms various special sequences. If you need to match a [ or \, you can precede them with a backslash to remove their special meaning: \[ or \\.
Predefined sets of characters represented by such special sequences beginning with ‘\’ are listed below −
Sr.No.
Metacharacters & Description
1
\dMatches any decimal digit; this is equivalent to the class [0-9].
2
\DMatches any non-digit character; this is equivalent to the class [^0-9].
3
\sMatches any whitespace character; this is equivalent to the class [\t\n\r\f\v].
4
\SMatches any non-whitespace character; this is equivalent to the class [^\t\n\r\f\v].
5
\wMatches any alphanumeric character; this is equivalent to the class [a-zAZ0-9_].
6
\WMatches any non-alphanumeric character. equivalent to the class [^a-zAZ0-9_].
7
.Matches with any single character except newline ‘\n’.
8
?match 0 or 1 occurrence of the pattern to its left
9
+1 or more occurrences of the pattern to its left
10
*0 or more occurrences of the pattern to its left
11
\bboundary between word and non-word and /B is opposite of /b
12
[..]Matches any single character in a square bracket and [^..] matches any single character not in square bracket.
13
\It is used for special meaning characters like \. to match a period or \+ for plus sign.
14
{n,m}Matches at least n and at most m occurrences of preceding
15
a| bMatches either a or b
Python’s re module provides useful functions for finding a match, searching for a pattern, and substitute a matched string with other string etc.
The re.match() Function
This function attempts to match RE pattern at the start of string with optional flags. Following is the syntax for this function −
re.match(pattern, string, flags=0)
Here is the description of the parameters −
Sr.No.
Parameter & Description
1
patternThis is the regular expression to be matched.
2
StringThis is the string, which would be searched to match the pattern at the beginning of string.
3
FlagsYou can specify different flags using bitwise OR (|). These are modifiers, which are listed in the table below.
The re.match() function returns a match object on success, None on failure. A match object instance contains information about the match: where it starts and ends, the substring it matched, etc.
The match object’s start() method returns the starting position of pattern in the string, and end() returns the endpoint.
If the pattern is not found, the match object is None.
We use group(num) or groups() function of match object to get matched expression.
Sr.No.
Match Object Methods & Description
1
group(num=0)This method returns entire match (or specific subgroup num)
2
groups()This method returns all matching subgroups in a tuple (empty if there weren’t any)
Example
import re
line ="Cats are smarter than dogs"
matchObj = re.match(r'Cats', line)print(matchObj.start(), matchObj.end())print("matchObj.group() : ", matchObj.group())
It will produce the following output −
0 4
matchObj.group() : Cats
The re.search() Function
This function searches for first occurrence of RE pattern within the string, with optional flags. Following is the syntax for this function −
re.search(pattern, string, flags=0)
Here is the description of the parameters −
Sr.No.
Parameter & Description
1
PatternThis is the regular expression to be matched.
2
StringThis is the string, which would be searched to match the pattern anywhere in the string.
3
FlagsYou can specify different flags using bitwise OR (|). These are modifiers, which are listed in the table below.
The re.search function returns a match object on success, none on failure. We use group(num) or groups() function of match object to get the matched expression.
Sr.No.
Match Object Methods & Description
1
group(num=0)This method returns entire match (or specific subgroup num)
2
groups()This method returns all matching subgroups in a tuple (empty if there weren’t any)
Example
import re
line ="Cats are smarter than dogs"
matchObj = re.search(r'than', line)print(matchObj.start(), matchObj.end())print("matchObj.group() : ", matchObj.group())
It will produce the following output −
17 21
matchObj.group() : than
Matching Vs Searching
Python offers two different primitive operations based on regular expressions, match checks for a match only at the beginning of the string, while searchchecks for a match anywhere in the string (this is what Perl does by default).
Example
import re
line ="Cats are smarter than dogs";
matchObj = re.match(r'dogs', line, re.M|re.I)if matchObj:print("match --> matchObj.group() : ", matchObj.group())else:print("No match!!")
searchObj = re.search(r'dogs', line, re.M|re.I)if searchObj:print("search --> searchObj.group() : ", searchObj.group())else:print("Nothing found!!")
When the above code is executed, it produces the following output −
No match!!
search --> matchObj.group() : dogs
The re.findall() Function
The findall() function returns all non-overlapping matches of pattern in string, as a list of strings or tuples. The string is scanned left-to-right, and matches are returned in the order found. Empty matches are included in the result.
Syntax
re.findall(pattern, string, flags=0)
Parameters
Sr.No.
Parameter & Description
1
PatternThis is the regular expression to be matched.
2
StringThis is the string, which would be searched to match the pattern anywhere in the string.
3
FlagsYou can specify different flags using bitwise OR (|). These are modifiers, which are listed in the table below.
Example
import re
string="Simple is better than complex."
obj=re.findall(r"ple", string)print(obj)
It will produce the following output −
['ple', 'ple']
Following code obtains the list of words in a sentence with the help of findall() function.
import re
string="Simple is better than complex."
obj=re.findall(r"\w*", string)print(obj)
One of the most important re methods that use regular expressions is sub.
Syntax
re.sub(pattern, repl, string,max=0)
This method replaces all occurrences of the RE pattern in string with repl, substituting all occurrences unless max is provided. This method returns modified string.
Example
import re
phone ="2004-959-559 # This is Phone Number"# Delete Python-style comments
num = re.sub(r'#.*$',"", phone)print("Phone Num : ", num)# Remove anything other than digits
num = re.sub(r'\D',"", phone)print("Phone Num : ", num)
It will produce the following output −
Phone Num : 2004-959-559
Phone Num : 2004959559
Example
The following example uses sub() function to substitute all occurrences of is with was word −
import re
string="Simple is better than complex. Complex is better than complicated."
obj=re.sub(r'is',r'was',string)print(obj)
It will produce the following output −
Simple was better than complex. Complex was better than complicated.
The re.compile() Function
The compile() function compiles a regular expression pattern into a regular expression object, which can be used for matching using its match(), search() and other methods.
Syntax
re.compile(pattern, flags=0)
Flags
Sr.No.
Modifier & Description
1
re.IPerforms case-insensitive matching.
2
re.LInterprets words according to the current locale. This interpretation affects the alphabetic group (\w and \W), as well as word boundary behavior (\b and \B).
3
re.M Makes $ match the end of a line (not just the end of the string) and makes ^ match the start of any line (not just the start of the string).
4
re.SMakes a period (dot) match any character, including a newline.
5
re.UInterprets letters according to the Unicode character set. This flag affects the behavior of \w, \W, \b, \B.
6
re.XPermits “cuter” regular expression syntax. It ignores whitespace (except inside a set [] or when escaped by a backslash) and treats unescaped # as a comment marker.
The sequence −
prog = re.compile(pattern)
result = prog.match(string)
is equivalent to −
result = re.match(pattern, string)
But using re.compile() and saving the resulting regular expression object for reuse is more efficient when the expression will be used several times in a single program.
Example
import re
string="Simple is better than complex. Complex is better than complicated."
pattern=re.compile(r'is')
obj=pattern.match(string)
obj=pattern.search(string)print(obj.start(), obj.end())
obj=pattern.findall(string)print(obj)
obj=pattern.sub(r'was', string)print(obj)
It will produce the following output −
7 9
['is', 'is']
Simple was better than complex. Complex was better than complicated.
The re.finditer() Function
This function returns an iterator yielding match objects over all non-overlapping matches for the RE pattern in string.
Syntax
re.finditer(pattern, string, flags=0)
Example
import re
string="Simple is better than complex. Complex is better than complicated."
pattern=re.compile(r'is')
iterator = pattern.finditer(string)print(iterator )for match in iterator:print(match.span())
It will produce the following output −
(7, 9)
(39, 41)
Use Cases of Python Regex
Finding all Adverbs
findall() matches all occurrences of a pattern, not just the first one as search() does. For example, if a writer wanted to find all of the adverbs in some text, they might use findall() in the following manner −
import re
text ="He was carefully disguised but captured quickly by police."
obj = re.findall(r"\w+ly\b", text)print(obj)
It will produce the following output −
['carefully', 'quickly']
Finding words starting with vowels
import re
text ='Errors should never pass silently. Unless explicitly silenced.'
obj=re.findall(r'\b[aeiouAEIOU]\w+', text)print(obj)
It will produce the following output −
['Errors', 'Unless', 'explicitly']
Regular Expression Modifiers: Option Flags
Regular expression literals may include an optional modifier to control various aspects of matching. The modifiers are specified as an optional flag. You can provide multiple modifiers using exclusive OR (|), as shown previously and may be represented by one of these −
Sr.No.
Modifier & Description
1
re.IPerforms case-insensitive matching.
2
re.LInterprets words according to the current locale. This interpretation affects the alphabetic group (\w and \W), as well as word boundary behavior(\b and \B).
3
re.MMakes $ match the end of a line (not just the end of the string) and makes ^ match the start of any line (not just the start of the string).
4
re.SMakes a period (dot) match any character, including a newline.
5
re.UInterprets letters according to the Unicode character set. This flag affects the behavior of \w, \W, \b, \B.
6
re.XPermits “cuter” regular expression syntax. It ignores whitespace (except inside a set [] or when escaped by a backslash) and treats unescaped # as a comment marker.
Regular Expression Patterns
Except for control characters, (+ ? . * ^ $ ( ) [ ] { } | \), all characters match themselves. You can escape a control character by preceding it with a backslash.
Following table lists the regular expression syntax that is available in Python −
Sr.No.
Pattern & Description
1
^Matches beginning of line.
2
$Matches end of line.
3
.Matches any single character except newline. Using m option allows it to match newline as well.
4
[…]Matches any single character in brackets.
5
[^…]Matches any single character not in brackets
6
re*Matches 0 or more occurrences of preceding expression.
7
re+Matches 1 or more occurrence of preceding expression.
8
re?Matches 0 or 1 occurrence of preceding expression.
9
re{ n}Matches exactly n number of occurrences of preceding expression.
10
re{ n,}Matches n or more occurrences of preceding expression.
11
re{ n, m}Matches at least n and at most m occurrences of preceding expression.
12
a| bMatches either a or b.
13
(re)Groups regular expressions and remembers matched text.
14
(?imx)Temporarily toggles on i, m, or x options within a regular expression. If in parentheses, only that area is affected.
15
(?-imx)Temporarily toggles off i, m, or x options within a regular expression. If in parentheses, only that area is affected.
16
(?: re)Groups regular expressions without remembering matched text.
17
(?imx: re)Temporarily toggles on i, m, or x options within parentheses.
18
(?-imx: re)Temporarily toggles off i, m, or x options within parentheses.
19
(?#…)Comment.
20
(?= re)Specifies position using a pattern. Doesn’t have a range.
21
(?! re)Specifies position using pattern negation. Doesn’t have a range.
22
(?> re)Matches independent pattern without backtracking.
23
\wMatches word characters.
24
\WMatches nonword characters.
25
\sMatches whitespace. Equivalent to [\t\n\r\f].
26
\SMatches nonwhitespace.
27
\dMatches digits. Equivalent to [0-9].
28
\DMatches nondigits.
29
\AMatches beginning of string.
30
\ZMatches end of string. If a newline exists, it matches just before newline.
31
\zMatches end of string.
32
\GMatches point where last match finished.
33
\bMatches word boundaries when outside brackets. Matches backspace (0x08) when inside brackets.
34
\BMatches nonword boundaries.
35
\n, \t, etc.Matches newlines, carriage returns, tabs, etc.
36
\1…\9Matches nth grouped subexpression.
37
\10Matches nth grouped subexpression if it matched already. Otherwise refers to the octal representation of a character code.
Regular Expression Examples
Literal characters
Sr.No.
Example & Description
1
pythonMatch “python”.
Character classes
Sr.No.
Example & Description
1
[Pp]ythonMatch “Python” or “python”
2
rub[ye]Match “ruby” or “rube”
3
[aeiou]Match any one lowercase vowel
4
[0-9]Match any digit; same as [0123456789]
5
[a-z]Match any lowercase ASCII letter
6
[A-Z]Match any uppercase ASCII letter
7
[a-zA-Z0-9]Match any of the above
8
[^aeiou]Match anything other than a lowercase vowel
9
[^0-9]Match anything other than a digit
Special Character Classes
Sr.No.
Example & Description
1
.Match any character except newline
2
\dMatch a digit: [0-9]
3
\DMatch a nondigit: [^0-9]
4
\sMatch a whitespace character: [ \t\r\n\f]
5
\SMatch nonwhitespace: [^ \t\r\n\f]
6
\wMatch a single word character: [A-Za-z0-9_]
7
\WMatch a nonword character: [^A-Za-z0-9_]
Repetition Cases
Sr.No.
Example & Description
1
ruby?Match “rub” or “ruby”: the y is optional
2
ruby*Match “rub” plus 0 or more ys
3
ruby+Match “rub” plus 1 or more ys
4
\d{3}Match exactly 3 digits
5
\d{3,}Match 3 or more digits
6
\d{3,5}Match 3, 4, or 5 digits
Nongreedy repetition
This matches the smallest number of repetitions −
Sr.No.
Example & Description
1
<.*>Greedy repetition: matches “<python>perl>”
2
<.*?>Nongreedy: matches “<python>” in “<python>perl>”
Grouping with Parentheses
Sr.No.
Example & Description
1
\D\d+No group: + repeats \d
2
(\D\d)+Grouped: + repeats \D\d pair
3
([Pp]ython(, )?)+Match “Python”, “Python, python, python”, etc.
Backreferences
This matches a previously matched group again −
Sr.No.
Example & Description
1
([Pp])ython&\1ailsMatch python&pails or Python&Pails
2
([‘”])[^\1]*\1Single or double-quoted string. \1 matches whatever the 1st group matched. \2 matches whatever the 2nd group matched, etc.
Alternatives
Sr.No.
Example & Description
1
python|perlMatch “python” or “perl”
2
rub(y|le))Match “ruby” or “ruble”
3
Python(!+|\?)“Python” followed by one or more ! or one ?
Anchors
This needs to specify match position.
Sr.No.
Example & Description
1
^PythonMatch “Python” at the start of a string or internal line
2
Python$Match “Python” at the end of a string or line
3
\APythonMatch “Python” at the start of a string
4
Python\ZMatch “Python” at the end of a string
5
\bPython\bMatch “Python” at a word boundary
6
\brub\B\B is nonword boundary: match “rub” in “rube” and “ruby” but not alone
7
Python(?=!)Match “Python”, if followed by an exclamation point.
8
Python(?!!)Match “Python”, if not followed by an exclamation point.
Special Syntax with Parentheses
Sr.No.
Example & Description
1
R(?#comment)Matches “R”. All the rest is a comment
2
R(?i)ubyCase-insensitive while matching “uby”
3
R(?i:uby)Same as above
4
rub(?:y|le))Group only without creating \1 backreference
Recursion is a fundamental programming concept where a function calls itself in order to solve a problem. This technique breaks down a complex problem into smaller and more manageable sub-problems of the same type. In Python, recursion is implemented by defining a function that makes one or more calls to itself within its own body.
Components of Recursion
As we discussed before Recursion is a technique where a function calls itself. Here for understanding recursion, it’s required to know its key components. Following are the primary components of the recursion −
Base Case
Recursive Case
Base Case
The Base case is a fundamental concept in recursion, if serving as the condition under which a recursive function stops calling itself. It is essential for preventing infinite recursion and subsequent stack overflow errors.
The base case provides a direct solution to the simplest instance of the problem ensuring that each recursive call gets closer to this terminating condition.
The most popular example of recursion is calculation of factorial. Mathematically factorial is defined as −
n! = n × (n-1)!
It can be seen that we use factorial itself to define factorial. Hence this is a fit case to write a recursive function. Let us expand above definition for calculation of factorial value of 5.
While we can perform this calculation using a loop, its recursive function involves successively calling it by decrementing the number till it reaches 1.
Example
The following example shows hows you can use a recursive function to calculate factorial −
deffactorial(n):if n ==1:print(n)return1#base caseelse:print(n,'*', end=' ')return n * factorial(n-1)#Recursive caseprint('factorial of 5=', factorial(5))
The above programs generates the following output −
5 * 4 * 3 * 2 * 1
factorial of 5= 120
Recursive Case
The recursive case is the part of a recursive function where the function calls itself to solve a smaller or simpler instance of the same problem. This mechanism allows a complex problem to be broken down into more manageable sub-problems where each them is a smaller version of the original problem.
The recursive case is essential for progressing towards the base case, ensuring that the recursion will eventually terminate.
Example
Following is the example of the Recursive case. In this example we are generating the Fibonacci sequence in which the recursive case sums the results of the two preceding Fibonacci numbers −
deffibonacci(n):if n <=0:return0# Base case for n = 0elif n ==1:return1# Base case for n = 1else:return fibonacci(n -1)+ fibonacci(n -2)# Recursive case
fib_series =[fibonacci(i)for i inrange(6)]print(fib_series)
The above programs generates the following output −
[0, 1, 1, 2, 3, 5] Binary Search using Recursion
Binary search is a powerful algorithm for quickly finding elements in sorted lists, with logarithmic time complexity making it highly efficient.
Let us have a look at another example to understand how recursion works. The problem at hand is to check whether a given number is present in a list.
While we can perform a sequential search for a certain number in the list using a for loop and comparing each number, the sequential search is not efficient especially if the list is too large. The binary search algorithm that checks if the index 'high' is greater than index 'low. Based on value present at 'mid' variable, the function is called again to search for the element.
We have a list of numbers, arranged in ascending order. The we find the midpoint of the list and restrict the checking to either left or right of midpoint depending on whether the desired number is less than or greater than the number at midpoint.
The following diagram shows how binary search works −
Python Recursion Example
The following code implements the recursive binary searching technique −
A Decorator in Python is a function that receives another function as argument. The argument function is the one to be decorated by decorator. The behaviour of argument function is extended by the decorator without actually modifying it.
In this chapter, we whall learn how to use Python decorator.
Defining Function Decorator
Function in Python is a first order object. It means that it can be passed as argument to another function just as other data types such as number, string or list etc. It is also possible to define a function inside another function. Such a function is called nested function. Moreover, a function can return other function as well.
The typical definition of a decorator function is as under −
defdecorator(arg_function):#arg_function to be decorateddefnested_function():#this wraps the arg_function and extends its behaviour#call arg_function
arg_function()return nested_function
Here a normal Python function −
deffunction():print("hello")
You can now decorate this function to extend its behaviour by passing it to decorator −
function=decorator(function)
If this function is now executed, it will show output extended by decorator.
Examples of Python Decorators
Practice the following examples to understand the concept of Python decorators −
Example 1
Following code is a simple example of decorator −
defmy_function(x):print("The number is=",x)defmy_decorator(some_function,num):defwrapper(num):print("Inside wrapper to check odd/even")if num%2==0:
ret="Even"else:
ret="Odd!"
some_function(num)return ret
print("wrapper function is called")return wrapper
no=10
my_function = my_decorator(my_function, no)print("It is ",my_function(no))
The my_function() just prints out the received number. However, its behaviour is modified by passing it to a my_decorator. The inner function receives the number and returns whether it is odd/even. Output of above code is −
wrapper function is called
Inside wrapper to check odd/even
The number is= 10
It is Even
Example 2
An elegant way to decorate a function is to mention just before its definition, the name of decorator prepended by @ symbol. The above example is re-written using this notation −
defmy_decorator(some_function):defwrapper(num):print("Inside wrapper to check odd/even")if num%2==0:
ret="Even"else:
ret="Odd!"
some_function(num)return ret
print("wrapper function is called")return wrapper
@my_decoratordefmy_function(x):print("The number is=",x)
no=10print("It is ",my_function(no))
Python’s standard library defines following built-in decorators −
@classmethod Decorator
The classmethod is a built-in function. It transforms a method into a class method. A class method is different from an instance method. Instance method defined in a class is called by its object. The method received an implicit object referred to by self. A class method on the other hand implicitly receives the class itself as first argument.
Syntax
In order to declare a class method, the following notation of decorator is used −
classMyclass:@classmethoddefmymethod(cls):#....
The @classmethod form is that of function decorator as described earlier. The mymethod receives reference to the class. It can be called by the class as well as its object. That means Myclass.mymethod as well as Myclass().mymethod both are valid calls.
Example of @classmethod Decorator
Let us understand the behaviour of class method with the help of following example −
classcounter:
count=0def__init__(self):print("init called by ", self)
counter.count=counter.count+1print("count=",counter.count)@classmethoddefshowcount(cls):print("called by ",cls)print("count=",cls.count)
c1=counter()
c2=counter()print("class method called by object")
c1.showcount()print("class method called by class")
counter.showcount()
In the class definition count is a class attribute. The __init__() method is the constructor and is obviously an instance method as it received self as object reference. Every object declared calls this method and increments count by 1.
The @classmethod decorator transforms showcount() method into a class method which receives reference to the class as argument even if it is called by its object. It can be seen even when c1 object calls showcount, it displays reference of counter class.
It will display the following output −
init called by <__main__.counter object at 0x000001D32DB4F0F0>
count= 1
init called by <__main__.counter object at 0x000001D32DAC8710>
count= 2
class method called by object
called by <class '__main__.counter'>
count= 2
class method called by class
called by <class '__main__.counter'>
@staticmethod Decorator
The staticmethod is also a built-in function in Python standard library. It transforms a method into a static method. Static method doesn’t receive any reference argument whether it is called by instance of class or class itself. Following notation used to declare a static method in a class −
Syntax
classMyclass:@staticmethoddefmymethod():#....
Even though Myclass.mymethod as well as Myclass().mymethod both are valid calls, the static method receives reference of neither.
Example of @staticmethod Decorator
The counter class is modified as under −
classcounter:
count=0def__init__(self):print("init called by ", self)
c1=counter()
c2=counter()print("class method called by object")
c1.showcount()print("class method called by class")
counter.showcount()
As before, the class attribute count is increment on declaration of each object inside the __init__() method. However, since mymethod(), being a static method doesn’t receive either self or cls parameter. Hence value of class attribute count is displayed with explicit reference to counter.
The output of the above code is as below −
init called by <__main__.counter object at 0x000002512EDCF0B8>
count= 1
init called by <__main__.counter object at 0x000002512ED48668>
count= 2
class method called by object
count= 2
class method called by class
count= 2
@property Decorator
Python’s property() built-in function is an interface for accessing instance variables of a class. The @property decorator turns an instance method into a “getter” for a read-only attribute with the same name, and it sets the docstring for the property to “Get the current value of the instance variable.”
You can use the following three decorators to define a property −
@property − Declares the method as a property.
@<property-name>.setter: − Specifies the setter method for a property that sets the value to a property.
@<property-name>.deleter − Specifies the delete method as a property that deletes a property.
A property object returned by property() function has getter, setter, and delete methods.
The fget argument is the getter method, fset is setter method. It optionally can have fdel as method to delete the object and doc is the documentation string.
Syntax
The property() object’s setter and getter may also be assigned with the following syntax also.
A Python closure is a nested function which has access to a variable from an enclosing function that has finished its execution. Such a variable is not bound in the local scope. To use immutable variables (number or string), we have to use the non-local keyword.
The main advantage of Python closures is that we can help avoid the using global values and provide some form of data hiding. They are used in Python decorators.
Closures are closely related to nested functions and allow inner functions to capture and retain the enclosing function’s local state, even after the outer function has finished execution. Understanding closures requires familiarity with nested functions, variable scope and how Python handles function objects.
Nested Functions: In Python functions can be defined inside other functions. These are called nested functions or inner functions.
Accessing Enclosing Scope: Inner functions can access variables from the enclosing i.e. outer scope. This is where closures come into play.
Retention of State: When an inner function i.e. closure captures and retains variables from its enclosing scope, even if the outer function has completed execution or the scope is no longer available.
Nested Functions
Nested functions in Python refer to the practice of defining one function inside another function. This concept allows us to organize code more effectively, encapsulate functionality and manage variable scope.
Following is the example of nested functions where functionB is defined inside functionA. Inner function is then called from inside the outer function’s scope.
When a closure is created i.e. an inner function that captures variables from its enclosing scope, it retains access to those variables even after the outer function has finished executing. This behavior allows closures to “remember” and manipulate the values of variables from the enclosing scope.
Example
Following is the example of the closure with the variable scope −
defouter_function(x):
y =10definner_function(z):return x + y + z # x and y are captured from the enclosing scopereturn inner_function
closure = outer_function(5)
result = closure(3)print(result)
Output
18
Creating a closure
Creating a closure in Python involves defining a nested function within an outer function and returning the inner function. Closures are useful for capturing and retaining the state of variables from the enclosing scope.
Example
In the below example, we have a functionA function which creates and returns another function functionB. The nested functionB function is the closure.
The outer functionA function returns a functionB function and assigns it to the myfunction variable. Even if it has finished its execution. However, the printer closure still has access to the name variable.
Following is the example of creating the closure in python −
deffunctionA(name):
name ="New name"deffunctionB():print(name)return functionB
myfunction = functionA("My name")
myfunction()
Output
New name
nonlocal Keyword
In Python, nonlocal keyword allows a variable outside the local scope to be accessed. This is used in a closure to modify an immutable variable present in the scope of outer variable. Here is the example of the closure with the nonlocal keyword.
Generators in Python are a convenient way to create iterators. They allow us to iterate through a sequence of values which means, values are generated on the fly and not stored in memory, which is especially useful for large datasets or infinite sequences.
The generator in Python is a special type of function that returns an iterator object. It appears similar to a normal Python function in that its definition also starts with def keyword. However, instead of return statement at the end, generator uses the yield keyword.
Syntax
The following is the syntax of the generator() function −
defgenerator():......yield obj it = generator()next(it)... Creating Generators
There are two primary ways to create generators in python −
Using Generator Functions Using Generator Expressions Using Generator Functions
The generator function uses 'yield' statement for returning the values all at a time. Each time the generators __next__() method is called the generator resumes where it left off i.e. from right after the last yield statement. Here's the example of creating the generator function.
def count_up_to(max_value): current = 1 while current <= max_value: yield current current += 1
# Using the generator counter = count_up_to(5) for number in counter: print(number) Output
1 2 3 4 5 Using Generator Expressions
Generator expressions provide a compact way to create generators. They use a syntax similar to list comprehensions but used parentheses i.e. "{}" instead of square brackets i.e. "[]"
gen_expr = (x * x for x in range(1, 6))
for value in gen_expr: print(value) Output
1 4 9 16 25 Exception Handling in Generators
We can create a generator and iterate it using a 'while' loop with exception handling for 'StopIteration' exception. The function in the below code is a generator that successively yield integers from 1 to 5.
When this function is called, it returns an iterator. Every call to next() method transfers the control back to the generator and fetches next integer.
def generator(num): for x in range(1, num+1): yield x return
it = generator(5) while True: try: print (next(it)) except StopIteration: break Output
1 2 3 4 5 Normal function vs Generator function
Normal functions and generator functions in Python serve different purposes and exhibit distinct behaviors. Understanding their differences is essential for leveraging them effectively in our code.
A normal function computes and returns a single value or a set of values whether in a list or tuple, when called. Once it returns, the function's execution is complete and all local variables are discarded where as a generator function yields values one at a time by suspending and resuming its state between each yield. It uses the yield statement instead of return.
Example
In this example we are creating a normal function and build a list of Fibonacci numbers and then iterate the list using a loop −
def fibonacci(n): fibo = [] a, b = 0, 1 while True: c=a+b if c>=n: break fibo.append(c) a, b = b, c return fibo f = fibonacci(10) for i in f: print (i) Output
1 2 3 5 8 Example
In the above example we created a fibonacci series using the normal function and When we want to collect all Fibonacci series numbers in a list and then the list is traversed using a loop. Imagine that we want Fibonacci series going upto a large number.
In such cases, all the numbers must be collected in a list requiring huge memory. This is where generator is useful as it generates a single number in the list and gives it for consumption. Following code is the generator-based solution for list of Fibonacci numbers −
def fibonacci(n): a, b = 0, 1 while True: c=a+b if c>=n: break yield c a, b = b, c return
f = fibonacci(10) while True: try: print (next(f)) except StopIteration: break Output
1 2 3 5 8 Asynchronous Generator
An asynchronous generator is a co-routine that returns an asynchronous iterator. A co-routine is a Python function defined with async keyword, and it can schedule and await other co-routines and tasks.
Just like a normal generator, the asynchronous generator yields incremental item in the iterator for every call to anext() function, instead of next() function.
Syntax
The following is the syntax of the Asynchronous Generator −
async def generator(): . . . . . . yield obj it = generator() anext(it) . . . Example
Following code demonstrates a coroutine generator that yields incrementing integers on every iteration of an async for loop.
import asyncio
async def async_generator(x): for i in range(1, x+1): await asyncio.sleep(1) yield i
async def main(): async for item in async_generator(5): print(item)
asyncio.run(main()) Output
1 2 3 4 5 Example
Let us now write an asynchronous generator for Fibonacci numbers. To simulate some asynchronous task inside the co-routine, the program calls sleep() method for a duration of 1 second before yielding the next number. As a result, we will get the numbers printed on the screen after a delay of one second.
import asyncio
async def fibonacci(n): a, b = 0, 1 while True: c=a+b if c>=n: break await asyncio.sleep(1) yield c a, b = b, c return
async def main(): f = fibonacci(10) async for num in f: print (num)
An iterator in Python is an object that enables traversal through a collection such as a list or a tuple, one element at a time. It follows the iterator protocol by using the implementation of two methods __iter__() and __next__().
The __iter__() method returns the iterator object itself and the __next__()method returns the next element in the sequence by raising a StopIterationexception when no more elements are available.
Iterators provide a memory-efficient way to iterate over data, especially useful for large datasets. They can be created from iterable objects using the iter()function or implemented using custom classes and generators.
Iterables vs Iterators
Before going deep into the iterator working, we should know the difference between the Iterables and Iterators.
Iterable: An object capable of returning its members one at a time (e.g., lists, tuples).
Iterator: An object representing a stream of data, returned one element at a time.
We normally use for loop to iterate through an iterable as follows −
for element in sequence:print(element)
Python’s built-in method iter() implements __iter__() method. It receives an iterable and returns iterator object.
Example of Python Iterator
Following code obtains iterator object from sequence types such as list, string and tuple. The iter() function also returns keyiterator from dictionary.
<str_iterator object at 0x7fd0416b42e0>
<list_iterator object at 0x7fd0416b42e0>
<tuple_iterator object at 0x7fd0416b42e0>
<dict_keyiterator object at 0x7fd041707560>
However, int id not iterable, hence it produces TypeError.
iterator =iter(100)print(iterator)
It will produce the following output −
Traceback (most recent call last):
File "C:\Users\user\example.py", line 5, in <module>
print (iter(100))
^^^^^^^^^
TypeError: 'int' object is not iterable
Error Handling in Iterators
Iterator object has a method named __next__(). Every time it is called, it returns next element in iterator stream. Call to next() function is equivalent to calling __next__() method of iterator object.
This method which raises a StopIteration exception when there are no more items to return.
Example
In the following is an example the iterator object we have created have only 3 elements and we are iterating through it more than thrice −
it =iter([1,2,3])print(next(it))print(it.__next__())print(it.__next__())print(next(it))
It will produce the following output −
1
2
3
Traceback (most recent call last):
File "C:\Users\user\example.py", line 5, in <module>
print (next(it))
^^^^^^^^
StopIteration
This exception can be caught in the code that consumes the iterator using try and except blocks, though it’s more common to handle it implicitly by using constructs like for loops which manage the StopIteration exception internally.
it =iter([1,2,3,4,5])print(next(it))whileTrue:try:
no =next(it)print(no)except StopIteration:break</pre>
It will produce the following output −
1
2
3
4
5
Custom Iterator
A custom iterator in Python is a user-defined class that implements the iterator protocol which consists of two methods __iter__() and __next__(). This allows the class to behave like an iterator, enabling traversal through its elements one at a time.
To define a custom iterator class in Python, the class must define these methods.
Example
In the following example, the Oddnumbers is a class implementing __iter__() and __next__() methods. On every call to __next__(), the number increments by 2 thereby streaming odd numbers in the range 1 to 10.
# Using the Fibonacci iterator
fib_iterator = Fibonacci(10)for number in fib_iterator:print(number)
It will produce the following output −
0
1
1
2
3
5
8
13
21
34
Asynchronous Iterator
Asynchronous iterators in Python allow us to iterate over asynchronous sequences, enabling the handling of async operations within a loop.
They follow the asynchronous iterator protocol which consists of the methods __aiter__() and __anext__() (added in Python 3.10 version onwards.). These methods are used in conjunction with the async for loop to iterate over asynchronous data sources.
The aiter() function returns an asynchronous iterator object. It is an asynchronous counter part of the classical iterator. Any asynchronous iterator must support ___aiter()__ and __anext__() methods. These methods are internally called by the two built-in functions.
Asynchronous functions are called co-routines and are executed with asyncio.run() method. The main() co-routine contains a while loop that successively obtains odd numbers and raises StopAsyncIteration if the number exceeds 9.
Like the classical iterator the asynchronous iterator gives a stream of objects. When the stream is exhausted, the StopAsyncIteration exception is raised.
Example
In the example give below, an asynchronous iterator class Oddnumbers is declared. It implements __aiter__() and __anext__() method. On each iteration, a next odd number is returned and the program waits for one second, so that it can perform any other process asynchronously.
