Python comprehensions, like decorators, are syntactic sugar constructs that help build altered and filtered lists, dictionaries, or sets from a given list, dictionary, or set. Using comprehensions saves a lot of time and code that might be considerably more verbose.

• Performing mathematical operations on the entire list

my_list = [2, 3, 5, 7, 11]
squared_list = [x**2 for x in my_list]    # list comprehension
# output => [4 , 9 , 25 , 49 , 121]
squared_dict = {x:x**2 for x in my_list}    # dict comprehension
# output => {11:121, 2:4, 3:9, 5:25, 7:49}

• Performing conditional filtering operations on the entire list

squared_list = [x**2 for x in my_list if x%2 != 0]    # list comprehension
# output => [9 , 25 , 49 , 121]
squared_dict = {x:x**2 for x in my_list if x%2 != 0}    # dict comprehension
# output => {11:121, 3:9, 5:25, 7:49}

• Combining multiple lists into one

a = [1, 2, 3]
b = [7, 8, 9]
[(x + y) for (x,y) in zip(a,b)]  # parallel iterators
# output => [8, 10, 12]
[(x,y) for x in a for y in b]    # nested iterators
# output => [(1,7),(1,8),(1,9),(2,7),(2,8),(2,9),(3,7),(3,8),(3,9)]

• Flattening a multi-dimensional list

my_list = [[10,20,30],[40,50,60],[70,80,90]]
flattened = [x for temp in my_list for x in temp]
# output => [10,20,30,40,50,60,70,80,90]

Note: List comprehensions have the same effect as the map method in other languages. They follow the mathematical set-builder notation rather than map and filter functions in Python.

Decorators in Python are functions that add functionality to an existing function without changing its structure. They are represented using @decorator_name and are called in a bottom-up fashion.

# decorator function to convert to lowercase
def lowercase_decorator(function):
    def wrapper():
        func = function()
        string_lowercase = func.lower()
        return string_lowercase
    return wrapper

# decorator function to split words
def splitter_decorator(function):
    def wrapper():
        func = function()
        string_split = func.split()
        return string_split
    return wrapper

@splitter_decorator   # executed next
@lowercase_decorator  # executed first
def hello():
    return 'Hello World'

hello()  # output => ['hello', 'world']

Decorators can also accept arguments and modify them before passing to the function. The inner nested function (wrapper) enforces encapsulation and remains hidden from the global scope.

# decorator function to capitalize names
def names_decorator(function):
    def wrapper(arg1, arg2):
        arg1 = arg1.capitalize()
        arg2 = arg2.capitalize()
        string_hello = function(arg1, arg2)
        return string_hello
    return wrapper

@names_decorator
def say_hello(name1, name2):
    return 'Hello ' + name1 + '! Hello ' + name2 + '!'

say_hello('sara', 'ansh')  # output => 'Hello Sara! Hello Ansh!'

Sometimes objects within the same scope have the same name but function differently. In such cases, scope resolution comes into play automatically in Python. A few examples:

• Python modules like math and cmath share many functions such as log10(), acos(), exp(). To avoid ambiguity, prefix them with the module: math.exp() and cmath.exp().
• Consider a global variable temp set to 10 and a local variable temp set to 20 inside a function. Python keeps their namespaces separate, so modifying the local variable does not affect the global one.

temp = 10   # global-scope variable
def func():
     temp = 20   # local-scope variable
     print(temp)

print(temp)   # output => 10
func()       # output => 20
print(temp)  # output => 10

This behavior can be overridden using the global keyword inside the function:

temp = 10   # global-scope variable
def func():
     global temp
     temp = 20   # modifies global variable
     print(temp)

print(temp)   # output => 10
func()       # output => 20
print(temp)  # output => 20

A namespace in Python ensures that object names in a program are unique and can be used without conflict. Python implements these namespaces as dictionaries with the 'name' as key mapped to a corresponding 'object' as value. This allows multiple namespaces to use the same name and map it to a separate object. A few examples of namespaces are:

• Local Namespace: Includes local names inside a function. The namespace is temporarily created for a function call and gets cleared when the function returns.
• Global Namespace: Includes names from various imported packages/modules used in the current project. Created when the package is imported and lasts until the execution of the script.
• Built-in Namespace: Includes built-in functions of core Python and built-in names for various exceptions.

The lifecycle of a namespace depends upon the scope of objects they are mapped to. If the scope of an object ends, the lifecycle of that namespace ends as well. Hence, it isn't possible to access inner namespace objects from an outer namespace.

• Memory management in Python is handled by the Python Memory Manager. The memory allocated by the manager is in the form of a private heap space dedicated to Python. All Python objects are stored in this heap, and being private, it is inaccessible to the programmer. However, Python provides some core API functions to work upon the private heap space.
• Additionally, Python has an in-built garbage collection to recycle the unused memory for the private heap space.

Lambda is an anonymous function in Python that can accept any number of arguments but can only have a single expression. It is generally used in situations requiring an anonymous function for a short time period. Lambda functions can be used in either of the two ways:

• Assigning lambda functions to a variable:

mul = lambda a, b : a * b
print(mul(2, 5))    # output => 10

• Wrapping lambda functions inside another function:

def myWrapper(n):
    return lambda a : a * n

mulFive = myWrapper(5)
print(mulFive(2))    # output => 10

• Negative indexes are the indexes from the end of the list, tuple, or string.
• Arr[-1] means the last element of array Arr[].

arr = [1, 2, 3, 4, 5, 6]
# get the last element
print(arr[-1])  # output 6
# get the second last element
print(arr[-2])  # output 5

• Python as a language is not strictly interpreted or compiled. Interpreted or compiled is a property of the implementation. Python is generally bytecode-interpreted (a set of interpreter-readable instructions).
• Source code is a file with .py extension.
• Python compiles the source code into a set of instructions for a virtual machine. The Python interpreter is an implementation of that virtual machine. This intermediate format is called bytecode.
• .py source code is first compiled to give .pyc, which is bytecode. This bytecode can then be interpreted by CPython or JIT-compiled by PyPy.

• .py files contain the source code of a program, whereas .pyc files contain the bytecode of your program. Bytecode is generated after compiling the .py file. Note that .pyc files are only created for imported files, not all files you run.
• Before executing a Python program, the interpreter checks for compiled files. If found, the virtual machine executes the .pyc. If not, it looks for the .py file, compiles it to .pyc, and then executes it.
• Having .pyc files saves compilation time on subsequent runs.

help() function in Python is used to display the documentation of modules, classes, functions, keywords, etc. If no parameter is passed to the help() function, an interactive help utility is launched on the console.
dir() function tries to return a valid list of attributes and methods of the object it is called upon. It behaves differently with different objects, as it aims to produce the most relevant data rather than the complete information.

• For Modules/Library objects, it returns a list of all attributes contained in that module.
• For Class objects, it returns a list of all valid attributes and base attributes.
• With no arguments passed, it returns a list of attributes in the current scope.

PYTHONPATH is an environment variable which you can set to add additional directories where Python will look for modules and packages. This is especially useful in maintaining Python libraries that you do not wish to install in the global default location.

Generators are functions that return an iterable collection of items, one at a time, in a set manner. Generators, in general, are used to create iterators with a different approach. They employ the use of yield keyword rather than return to return a generator object.
Let's try and build a generator for Fibonacci numbers:

# generate fibonacci numbers upto n
def fib(n):
    p, q = 0, 1
    while p < n:
        yield p
        p, q = q, p + q

x = fib(10)  # create generator object

# iterating using __next__()
x.__next__()  # output => 0
x.__next__()  # output => 1
x.__next__()  # output => 1
x.__next__()  # output => 2
x.__next__()  # output => 3
x.__next__()  # output => 5
x.__next__()  # output => 8
# error

# iterating using loop
for i in fib(10):
    print(i)  # output => 0 1 1 2 3 5 8

Python library offers a feature - serialization out of the box. Serializing an object refers to transforming it into a format that can be stored, so as to be able to deserialize it later to obtain the original object. Here, the pickle module comes into play.

Pickling:

• Pickling is the name of the serialization process in Python. Any object can be serialized into a byte stream and dumped as a file in memory. Pickle objects can be compressed further, and the serialization is portable across Python versions.
• The function used for this process is pickle.dump().

Unpickling:

• Unpickling is the inverse of pickling. It deserializes the byte stream to recreate the objects stored in the file and loads them into memory.
• The function used for this process is pickle.load().

Note: Python has another, more primitive, serialization module called marshall, which exists primarily to support .pyc files and differs significantly from pickle.

xrange() and range() are similar in functionality. Both generate a sequence of integers, but range() returns a Python list, whereas xrange() returns an xrange object.

So how does that make a difference?

Unlike range(), xrange() doesn’t generate a static list. It creates values on the go, using a technique commonly associated with generators, called "yielding". Yielding is crucial when memory is limited. Creating a static list as in range() can lead to MemoryError, while xrange() handles it optimally with minimal memory usage.

for i in xrange(10):    # numbers from 0 to 9
    print i               # output => 0 1 2 3 4 5 6 7 8 9

for i in xrange(1, 10):  # numbers from 1 to 9
    print i               # output => 1 2 3 4 5 6 7 8 9

for i in xrange(1, 10, 2):  # skip by two for next
    print i               # output => 1 3 5 7 9

Note: xrange has been deprecated as of Python 3.x. Now range behaves exactly like xrange did in Python 2.x, which made xrange() preferable over the original range() in Python 2.x.

In Python, the assignment statement (= operator) does not copy objects. It creates a binding between the existing object and the target variable name. To create copies of an object, we use the copy module. There are two ways of creating copies using this module:

Shallow Copy

A shallow copy is a bit-wise copy of an object. The copied object has an exact copy of the values in the original object. If any value is a reference to another object, only the reference addresses are copied.

Deep Copy

A deep copy copies all values recursively from source to target object. It duplicates even the objects referenced by the source object.

from copy import copy, deepcopy

list_1 = [1, 2, [3, 5], 4]

## shallow copy
list_2 = copy(list_1)
list_2[3] = 7
list_2[2].append(6)
list_2    # output => [1, 2, [3, 5, 6], 7]
list_1    # output => [1, 2, [3, 5, 6], 4]

## deep copy
list_3 = deepcopy(list_1)
list_3[3] = 8
list_3[2].append(7)
list_3    # output => [1, 2, [3, 5, 6, 7], 8]
list_1    # output => [1, 2, [3, 5, 6], 4]

This is done by using a method called issubclass() provided by Python. The method tells us if any class is a child of another class by returning true or false accordingly.

Example:

class Parent(object):
    pass   

class Child(Parent):
    pass   

# Driver Code
print(issubclass(Child, Parent))    # True
print(issubclass(Parent, Child))    # False

• We can check if an object is an instance of a class by making use of the isinstance() method:

obj1 = Child()
obj2 = Parent()

print(isinstance(obj2, Child))    # False 
print(isinstance(obj2, Parent))   # True

The __init__ method works similarly to constructors in Java. This method is run as soon as an object is instantiated. It is useful for initializing any attributes or default behaviour of the object at the time of instantiation.

Example:

class InterviewbuzzEmployee:

    # __init__ method / constructor
    def __init__(self, emp_name):
        self.emp_name = emp_name

    # introduce method
    def introduce(self):
        print('Hello, I am', self.emp_name)

emp = InterviewbuzzEmployee('Mr Employee')  # __init__ method is called here
emp.introduce()

The finalize() method is used for freeing up unmanaged resources and performing cleanup tasks before the garbage collection mechanism is invoked. This ensures better memory management and helps release system resources properly.

The new modifier is used to explicitly instruct the compiler to hide a base class method and use the new implementation defined in the derived class.

The override modifier, on the other hand, is used to extend or modify the abstract or virtual method defined in a base class. It allows a child class to provide a specific implementation of a method that was declared in the parent class.

An empty class does not have any members defined in it. It is created by using the pass keyword (the pass command does nothing in Python). We can still create objects for this class outside the class.

class EmptyClassDemo:
    pass

obj = EmptyClassDemo()
obj.name = "Interviewbuzz"
print("Name created= ", obj.name)

Output:
Name created = Interviewbuzz

Yes, it is possible if the base class is instantiated by other child classes or if the base class defines a static method.

Python does not make use of access specifiers like private, public, or protected explicitly. Instead, it imitates their behavior by using underscores:

• No underscore: Public variable (default).
• Single underscore (_): Indicates protected members.
• Double underscore (__): Indicates private members.

# to demonstrate access specifiers
class InterviewbuzzEmployee:
    # protected members
    _emp_name = None
    _age = None

    # private members
    __branch = None

    # constructor
    def __init__(self, emp_name, age, branch): 
        self._emp_name = emp_name
        self._age = age
        self.__branch = branch

    # public member
    def display(self):
        print(self._emp_name + " " + str(self._age) + " " + self.__branch)

Following are the ways to access parent class members within a child class:

class Parent(object):
    # Constructor
    def __init__(self, name):
        self.name = name    

class Child(Parent): 
    # Constructor
    def __init__(self, name, age):
        Parent.__init__(self, name)
        self.age = age

    def display(self):
        print(self.name, self.age)

# Driver Code
obj = Child("Interviewbuzz", 6)
obj.display()   # Output: Interviewbuzz 6

class Parent(object):
    # Constructor
    def __init__(self, name):
        self.name = name    

class Child(Parent):
    # Constructor
    def __init__(self, name, age):         
        # In Python 3.x, we can also use super().__init__(name)
        super(Child, self).__init__(name)
        self.age = age

    def display(self):
        print(self.name, self.age)

# Driver Code
obj = Child("Interviewbuzz", 6)
obj.display()   # Output: Interviewbuzz 6

Inheritance gives a class the power to access all attributes and methods of another class. It promotes code reusability and avoids redundancy. - The class inheriting from another is called a child class (derived class). - The class from which members are inherited is the parent class (superclass).

Types of Inheritance in Python

1. Single Inheritance

Child class derives members of one parent class.

# Parent class
class ParentClass:
    def par_func(self):
        print("I am parent class function")

# Child class
class ChildClass(ParentClass):
    def child_func(self):
        print("I am child class function")

# Driver code
obj1 = ChildClass()
obj1.par_func()
obj1.child_func()

2. Multi-level Inheritance

A child class inherits from a parent, which is further inherited by another class (grandchild).

# Parent class
class A:
   def __init__(self, a_name):
       self.a_name = a_name

# Intermediate class
class B(A):
   def __init__(self, b_name, a_name):
       self.b_name = b_name
       A.__init__(self, a_name)

# Child class
class C(B):
   def __init__(self, c_name, b_name, a_name):
       self.c_name = c_name
       B.__init__(self, b_name, a_name)
       
   def display_names(self):
       print("A name :", self.a_name)
       print("B name :", self.b_name)
       print("C name :", self.c_name)

# Driver code
obj1 = C('child', 'intermediate', 'parent')
print(obj1.a_name)
obj1.display_names()

3. Multiple Inheritance

A child class inherits from more than one parent class.

# Parent class1
class Parent1:
   def parent1_func(self):
       print("Hi I am first Parent")

# Parent class2
class Parent2:
   def parent2_func(self):
       print("Hi I am second Parent")

# Child class
class Child(Parent1, Parent2):
   def child_func(self):
       self.parent1_func()
       self.parent2_func()

# Driver code
obj1 = Child()
obj1.child_func()

4. Hierarchical Inheritance

A parent class is inherited by more than one child class.

# Base class
class A:
     def a_func(self):
         print("I am from the parent class.")

# 1st Derived class
class B(A):
     def b_func(self):
         print("I am from the first child.")

# 2nd Derived class
class C(A):
     def c_func(self):
         print("I am from the second child.")

# Driver code
obj1 = B()
obj2 = C()
obj1.a_func()
obj1.b_func()    # child 1 method
obj2.a_func()
obj2.c_func()    # child 2 method

To create a class in Python, we use the class keyword as shown below:

class InterviewbuzzEmployee:
   def __init__(self, emp_name):
       self.emp_name = emp_name

To instantiate or create an object from the class:

emp_1 = InterviewbuzzEmployee("Mr. Employee")

To access the attribute emp_name, we use the dot operator:

print(emp_1.emp_name)
# Prints Mr. Employee

To create methods inside the class, we include them under the class scope:

class InterviewbuzzEmployee:
   def __init__(self, emp_name):
       self.emp_name = emp_name
       
   def introduce(self):
       print("Hello I am " + self.emp_name)

The self parameter refers to the current instance of the class and is required for accessing attributes and methods. It must be the first parameter in any method defined inside the class.

The method introduce() can be accessed as:

emp_1.introduce()

Complete Example

class InterviewbuzzEmployee:
   def __init__(self, emp_name):
       self.emp_name = emp_name
       
   def introduce(self):
       print("Hello I am " + self.emp_name)
       
# create an object of InterviewbuzzEmployee class
emp_1 = InterviewbuzzEmployee("Mr Employee")
print(emp_1.emp_name)    # print employee name
emp_1.introduce()        # introduce the employee

This can be achieved by using the ~ (not/negation symbol) and isin() method as shown below.

import pandas as pd
df1 = pd.Series([2, 4, 8, 10, 12])
df2 = pd.Series([8, 12, 10, 15, 16])
df1 = df1[~df1.isin(df2)]
print(df1)

"""
Output:
0    2
1    4
dtype: int64
"""

Yes, they can, but with some bit of help. We need to add the parse_dates argument while we are reading data from the sources. Consider an example where we read data from a CSV file, we may encounter different date-time formats that are not readable by the pandas library. In this case, pandas provide flexibility to build our custom date parser with the help of lambda functions as shown below:

import pandas as pd
from datetime import datetime

dateparser = lambda date_val: datetime.strptime(date_val, '%Y-%m-%d %H:%M:%S')

df = pd.read_csv(
    "some_file.csv",
    parse_dates=['datetime_column'],
    date_parser=dateparser
)

We can achieve this by first performing the union of both series, then taking the intersection of both series. Then we follow the approach of getting items of union that are not there in the list of the intersection.

The following code demonstrates this:

import pandas as pd
import numpy as np

df1 = pd.Series([2, 4, 5, 8, 10])
df2 = pd.Series([8, 10, 13, 15, 17])

p_union = pd.Series(np.union1d(df1, df2))   # union of series
p_intersect = pd.Series(np.intersect1d(df1, df2))   # intersection of series

unique_elements = p_union[~p_union.isin(p_intersect)]
print(unique_elements)

"""
Output:
0     2
1     4
2     5
5    13
6    15
7    17
dtype: int64
"""

• Execute del df.index.name for removing the index by name.
• Alternatively, the df.index.name can be assigned to None.
• For example, if you have the below dataframe:

                Column 1
   Names             
   John               1
   Jack               2
   Judy               3
   Jim                4

• To drop the index name “Names”:

df.index.name = None
# Or run the below:
# del df.index.name
print(df)

        Column 1
John          1
Jack          2
Judy          3
Jim           4

To Delete Row/Column from DataFrame

• drop() method is used to delete row/column from dataframe.
• The axis argument is passed to the drop method where if the value is 0, it indicates to drop a row and if 1 it has to drop the column.
• Additionally, we can try to delete the rows/columns in place by setting the value of inplace=True. This makes sure that the job is done without the need for reassignment.
• The duplicate values from the row/column can be deleted by using the drop_duplicates() method.

A new column can be added to a pandas dataframe as follows:

import pandas as pd      

data_info = {
    'first' : pd.Series([1, 2, 3], index=['a', 'b', 'c']),    
    'second' : pd.Series([1, 2, 3, 4], index=['a', 'b', 'c', 'd'])
}    
  
df = pd.DataFrame(data_info)    

# To add new column 'third'
df['third'] = pd.Series([10, 20, 30], index=['a', 'b', 'c'])    
print(df)    

# To add new column 'fourth'
df['fourth'] = df['first'] + df['third']    
print(df)

Reindexing is the process of conforming a dataframe to a new index with optional filling logic. If the values are missing in the previous index, then NaN /NA is placed in the location. A new object is returned unless a new index is produced that is equivalent to the current one. The copy value is set to False. This is also used for changing the index of rows and columns in the dataframe.

We can identify if a dataframe has missing values by using the isnull() and isna() methods.

missing_data_count = df.isnull().sum()

We can handle missing values by either replacing the values in the column with 0 as follows:

df['column_name'].fillna(0)

Or by replacing it with the mean value of the column:

df['column_name'] = df['column_name'].fillna(df['column_name'].mean())

One dimensional array capable of storing different data types is called a Series. We can create pandas series from a dictionary object as shown below:

import pandas as pd    

dict_info = {'key1': 2.0, 'key2': 3.1, 'key3': 2.2}  
series_obj = pd.Series(dict_info)    
print(series_obj)    

Output:
key1    2.0
key2    3.1
key3    2.2
dtype: float64

If an index is not specified in the input method, then the keys of the dictionary are sorted in ascending order for constructing the index. In case the index is passed, then values of the index label will be extracted from the dictionary.

The dataframes can be combined using the following approaches:

df1.append(df2)

pd.concat([df1, df2])

df1.join(df2)

A dataframe is a 2D mutable and tabular structure for representing data labelled with axes - rows and columns.

import pandas as pd
dataframe = pd.DataFrame(data, index, columns, dtype)

• Pandas is an open-source, Python-based library used in data manipulation applications requiring high performance. The name is derived from “Panel Data” having multidimensional data. It was developed in 2008 by Wes McKinney for data analysis.
• Pandas are useful in performing 5 major steps of data analysis: load the data, clean/manipulate it, prepare it, model it, and analyze the data.

NumPy is one of the most popular, easy-to-use, versatile, open-source, Python-based, general-purpose packages used for processing arrays. NumPy stands for NUMerical PYthon.

It is highly famous for its optimized tools that result in high performance and powerful N-Dimensional array processing features, explicitly designed to work on complex arrays.

Due to its popularity, powerful performance, and flexibility to perform various operations like trigonometric, algebraic, and statistical computations, it is widely used in scientific computations and broadcasting functions.

Applications of NumPy