Data Manipulation Tool + pandas

Data manipulation refers to the process of transforming, cleaning, and preparing raw data into a more usable and insightful format. This often involves operations such as selecting specific data, filtering out irrelevant information, adding new features, handling missing values, aggregating data, and merging different datasets. Effective data manipulation is a crucial step in any data analysis, machine learning, or business intelligence workflow, as it directly impacts the quality and reliability of subsequent analyses.

Pandas is an open-source Python library widely recognized as the de facto standard for data manipulation and analysis. It provides powerful and flexible data structures, most notably the `DataFrame`, which is a two-dimensional, tabular data structure with labeled axes (rows and columns). Pandas excels at handling structured data, making it an indispensable tool for tasks like:

- Reading and Writing Data: Easily import data from various file formats (CSV, Excel, SQL databases, JSON, etc.) into DataFrames and export DataFrames back to these formats.
- Selection and Indexing: Efficiently select rows, columns, or specific data points using labels, integer positions, or boolean conditions.
- Filtering: Extract subsets of data based on criteria, such as rows where a certain column meets a specific condition.
- Adding and Modifying Columns: Create new columns based on existing ones, update column values, or assign constant values.
- Handling Missing Data: Identify, remove, or impute (fill) missing values (NaN) using various strategies (e.g., mean, median, forward-fill, backward-fill).
- Grouping and Aggregation: Group data by one or more columns and then apply aggregate functions (sum, mean, count, min, max, etc.) to each group. This is fundamental for summarizing data.
- Merging, Joining, and Concatenating: Combine multiple DataFrames based on common columns (like SQL joins) or stack them vertically/horizontally.
- Sorting: Arrange data by one or more columns in ascending or descending order.
- Applying Functions: Apply custom functions or lambda expressions to rows, columns, or individual elements for complex transformations.

Pandas' intuitive API, coupled with its performance (optimized C/Cython backend), makes it an extremely efficient and versatile tool for almost any data manipulation task, enabling data scientists and analysts to quickly explore, clean, and transform datasets for deeper insights.

import pandas as pd
import numpy as np

 1. Create a sample DataFrame
print("Original DataFrame:")
data = {
    'Product': ['Laptop', 'Mouse', 'Keyboard', 'Monitor', 'Webcam', 'Mouse', 'Laptop'],
    'Category': ['Electronics', 'Accessories', 'Accessories', 'Electronics', 'Accessories', 'Accessories', 'Electronics'],
    'Price': [1200, 25, 75, 300, 40, 30, 1300],
    'Quantity': [10, 50, 20, 5, 15, 40, np.nan],  np.nan for a missing value
    'CustomerID': [101, 102, 103, 101, 104, 102, 105]
}
df = pd.DataFrame(data)
print(df)
print("-" - 30)

 2. Select specific columns
print("Selected Columns ('Product', 'Price'):")
selected_columns_df = df[['Product', 'Price']]
print(selected_columns_df)
print("-" - 30)

 3. Filter rows based on a condition (Price > 100)
print("Filtered Rows (Price > 100):")
filtered_df = df[df['Price'] > 100]
print(filtered_df)
print("-" - 30)

 4. Add a new column: 'Total_Value' (Price - Quantity)
 First, handle missing 'Quantity' by filling with 0 for calculation
df['Quantity'].fillna(0, inplace=True)  Fill NaN with 0 before calculation
df['Total_Value'] = df['Price'] - df['Quantity']
print("DataFrame with 'Total_Value' column and NaNs handled:")
print(df)
print("-" - 30)

 5. Group by 'Category' and calculate the average 'Price' and total 'Quantity'
print("Grouped by 'Category' - Average Price and Total Quantity:")
grouped_df = df.groupby('Category').agg(
    Average_Price=('Price', 'mean'),
    Total_Quantity=('Quantity', 'sum'),
    Num_Products=('Product', 'count')
).reset_index()
print(grouped_df)
print("-" - 30)

 6. Sort the DataFrame by 'Price' in descending order
print("DataFrame sorted by 'Price' (descending):")
sorted_df = df.sort_values(by='Price', ascending=False)
print(sorted_df)
print("-" - 30)

 7. Apply a function to a column (e.g., double the price for products in 'Accessories')
def adjust_price(row):
    if row['Category'] == 'Accessories':
        return row['Price'] - 2
    return row['Price']

df['Adjusted_Price'] = df.apply(adjust_price, axis=1)
print("DataFrame with 'Adjusted_Price' (Accessories prices doubled):")
print(df)
print("-" - 30)