Exploratory Data Analysis Projects with Python Code: From Beginner to Advanced

by

1. Introduction

“If you torture the data long enough, it will confess to anything.” – Ronald Coase

I’ve learned over the years that raw data never tells the full story upfront. That’s where Exploratory Data Analysis (EDA) comes in—it’s like detective work for data scientists.

Before jumping into complex machine learning models, you need to understand your data inside out. I can’t count how many times I’ve seen bad models just because someone skipped a proper EDA.

Why EDA Matters in Real-World Data Science?

EDA isn’t just about pretty graphs and summary statistics—it’s about finding the truth hidden in the data.

Whether you’re working with financial data, healthcare records, or customer transactions, a solid EDA helps you spot trends, detect anomalies, and clean messy datasets before running any machine-learning model.

I’ve personally worked on datasets where simple outlier detection saved an entire project.

Imagine trying to predict housing prices but missing the fact that one house is listed for $100 million just because someone entered the wrong data. A simple box plot or IQR check could have flagged that instantly.

Why Python for EDA?

You might be wondering: Why Python? Why not R, SQL, or some fancy BI tool?

From my experience, Python strikes the perfect balance between flexibility and power. Libraries like pandas, seaborn, and matplotlib make it effortless to manipulate and visualize data, while plotly and sweetviz take it a step further with interactive and automated insights. The ecosystem is insanely rich, and the best part? You don’t have to keep switching between tools—everything you need is right in Python.

What You’ll Learn in This Guide

I’ll walk you through real-world EDA projects—not just toy examples. Whether you’re a beginner looking to clean your first dataset or an experienced data scientist exploring time-series anomalies, you’ll get hands-on projects that simulate real industry problems.

Here’s what’s coming up:
✅ Beginner: Titanic dataset—basic visualizations, missing data handling.
✅ Intermediate: Customer segmentation—clustering & feature engineering.
✅ Advanced: Financial market analysis—time-series EDA & predictive analytics.
✅ Bonus: Automating EDA with pandas-profiling & Sweetviz.

By the time you’re done, you won’t just “know” EDA—you’ll own it. Let’s dive in. 🚀

2. Prerequisites & Tools Needed

“A craftsman is only as good as his tools.” – And in data science, the right tools can make or break your analysis.

Over the years, I’ve tried almost every Python library for EDA, from the basics to the fancy automated tools. Some are game-changers, while others?

Let’s just say they collect dust in my environment. Here’s what I actually use day in, day out when working on EDA.

Essential Python Libraries for EDA

If you’re serious about digging deep into data, these libraries are non-negotiable:

✅ pandas – The backbone of every EDA workflow. If you’re not using pandas for data manipulation, you’re making your life harder than it needs to be. I use it for everything—loading datasets, handling missing values, merging tables, and quick statistics.

✅ numpy – The foundation for numerical operations. Even if you don’t directly use numpy often, many other libraries (like pandas and scikit-learn) run on top of it. I rely on it for efficient array operations and quick calculations.

✅ matplotlib & seaborn – If I need to quickly spot patterns in data, matplotlib and seaborn are my go-to tools. With seaborn, I can visualize distributions, correlations, and trends in just one line of code.

✅ plotly – Want to interact with your data instead of staring at static plots? plotly is a game-changer. I personally use it when presenting insights to stakeholders—it makes my life easier when I need to zoom in, filter, and customize charts on the fly.

✅ missingno – Missing data is always a headache. Before I used missingno, I’d spend way too much time manually checking for missing values. This library gives me a quick visual overview so I can decide whether to impute, drop, or flag missing data.

✅ sweetviz & pandas-profiling – Sometimes, I just want a full report in one click instead of writing 10+ lines of code. These libraries automatically generate detailed reports with everything from missing values to correlations, helping me save hours of manual work.

Development Environment: Where Should You Write Your Code?

This might surprise you: The right environment can speed up your EDA workflow significantly.

🔹 Jupyter Notebook – My personal favorite for EDA. I love the ability to run code cell by cell, visualize outputs instantly, and keep notes alongside my code. If you’re exploring a dataset for the first time, Jupyter is unbeatable.

🔹 VS Code – When my project gets bigger, or I need better version control, I move to VS Code. It’s lightweight, integrates well with Git, and works great when you’re handling larger datasets or multiple scripts.

Pro Tip: If your dataset is too big for Jupyter and keeps crashing, consider using Dask or moving to a cloud-based notebook like Google Colab or Kaggle Kernels.

Where to Get Real-World Datasets?

I’ve worked with all sorts of datasets, from clean, structured ones to messy, real-world data full of missing values and inconsistencies. If you’re looking for high-quality datasets to practice EDA, here’s where I go:

Kaggle – My first stop for any dataset. Whether it’s Titanic, customer churn, or financial data, Kaggle has it all. Plus, you get to see how others approach the same problem.

UCI Machine Learning Repository – If I need something a bit more research-focused, UCI is my go-to. Some of the datasets here have been used in academic papers and competitions.

Your Own Data – If you’re working in an industry setting, trust me—nothing beats analyzing your own company’s data. Real-world data is messy, complex, and requires far more problem-solving than any cleaned Kaggle dataset ever will.

3. Beginner-Level EDA Project: Analyzing Titanic Dataset

“Numbers have an important story to tell. They rely on you to give them a voice.” – Stephen Few

If there’s one dataset that almost every data scientist has worked with at some point, it’s the Titanic dataset. I remember the first time I used it—at first glance, it looked simple. But once I started digging in, I realized there’s a lot more than meets the eye.

The goal here is to explore the dataset, understand its structure, handle missing values, and create meaningful visualizations. Whether you’re new to EDA or just looking for a refresher, this is the perfect place to start.

Step 1: Load the Dataset

Before we do anything fancy, let’s load the dataset and take a quick look:

import pandas as pd  

# Load Titanic dataset  
df = pd.read_csv("https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv")  

# Quick overview  
df.head()

This might surprise you: Even though this dataset has been around forever, many people overlook hidden insights in it.

Step 2: Checking Missing Values

One of the first things I always do in EDA is check for missing data. You’d be surprised how many times missing values completely change the analysis.

# Checking for missing values
df.isnull().sum()

You’ll notice that Age, Cabin, and Embarked have missing values. Now, the real question is: What do we do about them?

  • Age: Instead of just filling it with the mean (which I see too often), a smarter approach is using median grouped by class. Higher-class passengers tend to be older.
  • Cabin: More than 75% missing—this tells me it’s best to drop this column (unless you want to engineer a feature like “Has_Cabin”).
  • Embarked: Only two missing values, so replacing them with the most common port makes sense.

Here’s how I’d handle it:

# Fill missing Age with median based on Pclass
df["Age"] = df.groupby("Pclass")["Age"].transform(lambda x: x.fillna(x.median()))

# Fill missing Embarked values with the most common category
df["Embarked"].fillna(df["Embarked"].mode()[0], inplace=True)

# Drop Cabin column
df.drop(columns=["Cabin"], inplace=True)

Step 3: Basic Statistics

This is where we get a feel for the data. I always run these two commands first:

df.info()
df.describe()
  • df.info() tells us data types and missing values (critical for preprocessing).
  • df.describe() helps spot outliers and skewed distributions (key for feature engineering).

For example, did you notice that Fare has a huge range? That’s a red flag—it means we might have extreme outliers.

Step 4: Univariate Analysis (Understanding Single Features)

This is where I start visualizing. Let’s check out the distribution of Age and Fare using seaborn:

import seaborn as sns
import matplotlib.pyplot as plt

# Age distribution
sns.histplot(df["Age"], bins=30, kde=True)
plt.title("Age Distribution of Titanic Passengers")
plt.show()

# Fare distribution
sns.histplot(df["Fare"], bins=40, kde=True)
plt.title("Fare Distribution")
plt.show()

Key Insights:

  • The Age distribution is right-skewed, meaning there are more younger passengers.
  • The Fare distribution is extremely skewed—a few people paid way more than others. This could be an outlier issue.

Step 5: Bivariate Analysis (Relationships Between Features)

Now, let’s see how survival rates differ across different categories.

Who had a better chance of survival? Men or women?

sns.countplot(x="Sex", hue="Survived", data=df)
plt.title("Survival Count by Gender")
plt.show()

No surprises here—women had a much higher survival rate.

What about ticket class?

sns.countplot(x="Pclass", hue="Survived", data=df)
plt.title("Survival Count by Ticket Class")
plt.show()

Again, 1st class passengers had a much better survival rate.

Step 6: Feature Correlations

One of my favorite parts of EDA is checking how features relate to each other. This is where a heatmap helps:

plt.figure(figsize=(10,6))
sns.heatmap(df.corr(), annot=True, cmap="coolwarm", linewidths=0.5)
plt.title("Feature Correlation Matrix")
plt.show()

A few things jump out immediately:

  • Fare and Pclass are negatively correlated (makes sense—higher-class tickets are expensive).
  • Survival is positively correlated with Fare and Pclass (richer passengers had better chances).

Final Thoughts

I’ve analyzed the Titanic dataset multiple times, and every time, I find something new. If you’re just starting with EDA, this dataset is perfect because it teaches you:

  • How to handle missing data properly.
  • The importance of visualizing distributions and relationships.
  • How to spot potential outliers and feature correlations.

Now that we’ve mastered the basics, let’s move on to something more complex—Customer Segmentation using real-world data.

4. Intermediate-Level EDA Project: Customer Segmentation Using Mall Customers Dataset

“If you don’t understand your customers, you don’t have a business—you have a hobby.”

I remember working on customer segmentation for the first time and realizing just how powerful EDA can be. It’s not just about pretty graphs—it’s about understanding behavior, uncovering patterns, and driving real business impact.

One dataset I love for this is the Mall Customers dataset. It’s simple enough to grasp but packed with insights that show how people spend money differently. The goal here?

Segment customers based on spending behavior so businesses can target them effectively.

Step 1: Load the Dataset & First Look

The first thing I always do? Load the dataset and explore its structure.

import pandas as pd

# Load dataset
df = pd.read_csv("https://raw.githubusercontent.com/mwaskom/seaborn-data/master/mall_customers.csv")

# Quick overview
df.head()

Here’s what we’re dealing with:

  • Gender: Categorical feature (important for segmentation)
  • Age: Crucial for identifying spending trends
  • Annual Income: A strong predictor of spending behavior
  • Spending Score: A rating from 1-100 based on shopping habits

At first glance, you might think this dataset is too small to be useful—but trust me, you’d be surprised what it can reveal.

Step 2: Handling Missing Values (If Any)

Before diving into clustering, I always check for missing values:

df.isnull().sum()

In this dataset, we’re lucky—no missing values. But in the real world, that’s rarely the case. If I had missing values here, my approach would depend on why they’re missing:

  • If data is missing completely at random, I’d use imputation (mean, median, mode).
  • If there’s a pattern, I’d explore more advanced techniques like regression imputation.

Step 3: Feature Engineering – Creating Meaningful Variables

This is where EDA gets fun. I like to create new variables that improve clustering results.

Example: Categorizing Age Groups

Instead of using raw age values, I often bin them into categories to capture behavior patterns:

df["Age Group"] = pd.cut(df["Age"], bins=[15, 25, 35, 50, 100], labels=["Young", "Mid-Age", "Adult", "Senior"])

Example: Income-to-Spending Ratio

Not everyone who earns more spends more. A high-income, low-spender is different from a low-income, high-spender.

df["Income_Spending_Ratio"] = df["Annual Income (k$)"] / df["Spending Score (1-100)"]

I’ve seen cases where this ratio predicts customer retention better than raw income or spending alone.

Step 4: Data Transformation (Standardizing for Clustering)

Clustering algorithms like KMeans are sensitive to scale, so I always standardize numerical features.

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
df_scaled = scaler.fit_transform(df[["Annual Income (k$)", "Spending Score (1-100)"]])

Why does this matter?

  • Without scaling, features like Annual Income (which has large values) would dominate the clustering process.
  • Standardization ensures all features contribute equally.

Step 5: Dimensionality Reduction (PCA & t-SNE Visualizations)

Sometimes, visualizing high-dimensional data is tricky, so I use PCA or t-SNE to reduce dimensions.

from sklearn.decomposition import PCA

pca = PCA(n_components=2)
df_pca = pca.fit_transform(df_scaled)

This helps visualize clusters in a 2D space, making segmentation patterns clearer.

Step 6: Clustering with KMeans & DBSCAN

Now, the exciting part—customer segmentation.

KMeans Clustering

from sklearn.cluster import KMeans

# Finding optimal clusters using the Elbow Method
wcss = []
for i in range(1, 11):
    kmeans = KMeans(n_clusters=i, init="k-means++", random_state=42)
    kmeans.fit(df_scaled)
    wcss.append(kmeans.inertia_)

# Plot WCSS to find the elbow point
import matplotlib.pyplot as plt

plt.plot(range(1, 11), wcss, marker="o")
plt.xlabel("Number of Clusters")
plt.ylabel("WCSS")
plt.title("Elbow Method for Optimal K")
plt.show()

The elbow point helps determine the optimal number of clusters. In most cases, it’s around K=3 or K=5.

Now, let’s apply KMeans:

kmeans = KMeans(n_clusters=5, random_state=42)
df["Cluster"] = kmeans.fit_predict(df_scaled)

Alternative: DBSCAN for Density-Based Clustering

KMeans works well for globular clusters, but what if customers have non-linear spending patterns? That’s where DBSCAN shines.

from sklearn.cluster import DBSCAN

dbscan = DBSCAN(eps=0.5, min_samples=5)
df["DBSCAN_Cluster"] = dbscan.fit_predict(df_scaled)

DBSCAN is great for detecting anomalies and finding complex groupings.

Step 7: Visualizing the Segments

One thing I always emphasize—clustering results are useless if you can’t explain them.

Pairplot for Cluster Interpretation

import seaborn as sns

sns.pairplot(df, hue="Cluster", diag_kind="kde")
plt.show()

This lets us see how clusters differ based on income, spending, and age.

Interactive 3D Plot with Plotly

For a more intuitive visualization:

import plotly.express as px

fig = px.scatter_3d(df, x="Annual Income (k$)", y="Spending Score (1-100)", z="Age",
                     color="Cluster", title="Customer Segmentation in 3D")
fig.show()

Key Takeaways from Customer Segmentation

From working on real-world segmentation projects, I’ve learned a few things:

  • Not all high-income customers are high spenders—many prefer saving.
  • Young customers tend to have extreme spending behaviors (either very high or very low).
  • Understanding spending clusters can help businesses personalize marketing strategies.

This might surprise you: Some of the most valuable customers aren’t high spenders—but consistent spenders. Companies often focus on big spenders, but identifying loyal, steady customers can be even more profitable.

5. Advanced-Level EDA Project: Analyzing Financial Market Data

“The stock market is filled with individuals who know the price of everything, but the value of nothing.” — Philip Fisher

If there’s one dataset that never behaves the way you expect, it’s financial data. I’ve spent hours analyzing stock prices, trying to spot trends, only to realize markets are chaotic by nature.

But here’s the thing—EDA in financial markets isn’t about predicting the future. It’s about understanding the past so you can make better-informed decisions.

For this project, I’ll walk you through an EDA on stock market or cryptocurrency data, using advanced techniques that go beyond basic line charts.

Step 1: Load the Data & Initial Exploration

First, let’s grab some stock market data. I personally prefer using the Yahoo Finance API because it gives you historical prices, volume, and other key indicators.

import pandas as pd
import yfinance as yf

# Load stock data for Tesla (TSLA)
df = yf.download("TSLA", start="2020-01-01", end="2024-01-01")

# Quick preview
df.head()

At first glance, you’ll see columns like Open, High, Low, Close, Adj Close, and Volume. But trust me—raw prices alone tell you very little. The real insights come from derived features and trend analysis.

Step 2: Time-Series Feature Engineering

In my experience, rolling averages and volatility measures are game-changers for market analysis.

1. Moving Averages (SMA & EMA)

Traders often rely on moving averages to smooth out price fluctuations.

df["SMA_50"] = df["Close"].rolling(window=50).mean()  # 50-day simple moving average
df["EMA_20"] = df["Close"].ewm(span=20, adjust=False).mean()  # 20-day exponential moving average

Why does this matter?

  • Short-term traders use the 20-day EMA to react to price swings.
  • Long-term investors prefer the 50 or 200-day SMA for broader trends.

I’ve seen cases where a simple crossover strategy (when a short-term EMA crosses above a long-term SMA) indicates strong bullish momentum.

Step 3: Autocorrelation Analysis – Does Past Data Predict Future Trends?

Financial markets aren’t purely random—they have short-term patterns. Autocorrelation helps detect how past prices influence future ones.

import matplotlib.pyplot as plt
from pandas.plotting import autocorrelation_plot

# Plot autocorrelation of daily returns
df["Returns"] = df["Close"].pct_change()
autocorrelation_plot(df["Returns"].dropna())
plt.show()

If you see significant autocorrelation at short lags, it suggests momentum or mean-reverting behavior, depending on the asset.

Step 4: Outlier Detection – Spotting Anomalies in Market Data

I remember analyzing a stock where a single day’s price movement distorted the entire trend. That’s why I always check for outliers using IQR and Z-score.

1. IQR Method

Q1 = df["Close"].quantile(0.25)
Q3 = df["Close"].quantile(0.75)
IQR = Q3 - Q1

df["Outlier"] = (df["Close"] < (Q1 - 1.5 * IQR)) | (df["Close"] > (Q3 + 1.5 * IQR))
df[df["Outlier"]]

2. Z-score Method

from scipy.stats import zscore

df["Z-Score"] = zscore(df["Close"])
df[df["Z-Score"].abs() > 3]  # Outliers beyond 3 standard deviations

These methods help flag unexpected price spikes or crashes—often due to earnings reports, macroeconomic news, or sudden liquidity issues.

Step 5: Seasonality Analysis – Do Markets Have Predictable Cycles?

It might surprise you, but financial markets have hidden seasonal patterns. Certain months, days, or even hours tend to perform differently.

1. Monthly Seasonality Trends

df["Month"] = df.index.month
df.groupby("Month")["Close"].mean().plot(kind="bar", title="Average Monthly Close Price")

Stocks like Apple (AAPL) tend to rally in Q4 due to holiday sales. Cryptos like Bitcoin often surge before halving events.

2. Decomposing Market Trends with Statsmodels

For deeper trend analysis, I use time-series decomposition:

from statsmodels.tsa.seasonal import seasonal_decompose

result = seasonal_decompose(df["Close"], model="multiplicative", period=252)  # Approx. 1 year
result.plot()
plt.show()

This breaks down price movements into trend, seasonality, and noise, helping identify long-term direction vs. short-term fluctuations.

Step 6: Predictive Analytics – Can EDA Give Early Insights?

While EDA isn’t about forecasting, I’ve seen it provide early warning signs for price movements.

One simple trick? Rolling volatility analysis.

df["Volatility"] = df["Returns"].rolling(window=30).std()
df["Volatility"].plot(title="30-Day Rolling Volatility")

Spikes in volatility often precede breakouts or reversals—helpful for both traders and risk managers.

Step 7: Interactive Visualizations with Plotly & mplfinance

Basic line charts are fine, but for financial data, interactive visualizations are far more insightful.

1. Candlestick Chart with Plotly

import plotly.graph_objects as go

fig = go.Figure(data=[go.Candlestick(x=df.index,
                open=df["Open"], high=df["High"],
                low=df["Low"], close=df["Close"])])

fig.update_layout(title="Tesla Stock Candlestick Chart")
fig.show()

2. Technical Indicators with mplfinance

import mplfinance as mpf

mpf.plot(df, type="candle", volume=True, style="yahoo", mav=(20, 50), title="TSLA Stock Analysis")

These charts make it easier to spot trends, support/resistance levels, and potential reversals.

Key Takeaways from Financial Market EDA

From my experience analyzing stocks and crypto, here are some golden rules:

  • Don’t rely on raw prices—derived features tell the real story.
  • Market data has structure, but also randomness—never overfit patterns.
  • Seasonality and volatility analysis provide valuable trading signals.
  • Interactive visualizations make trend analysis much clearer.

This might surprise you: The most useful EDA insights in finance often come from observing what others overlook—subtle trends, volatility shifts, or anomalies in trading volume.

6. Automating EDA: Pandas Profiling & Sweetviz

“If you automate a mess, you get an automated mess.” – Rod Michael

I’ll be honest—there was a time when I manually wrote dozens of lines of code just to explore a dataset. I’d check for missing values, generate distributions, analyze correlations… until I discovered automated EDA tools. That changed everything.

If you’ve ever found yourself repeating the same exploratory steps for every new dataset, you’ll love tools like Pandas Profiling and Sweetviz. With a single command, you get detailed, interactive reports that save hours of work.

But here’s the catch—automation is powerful, but it doesn’t replace human judgment. Let’s dive into when and how to use these tools effectively.

Why Automate EDA?

There’s a reason why experienced data scientists don’t waste time reinventing the wheel—we automate whenever possible. Pandas Profiling and Sweetviz offer:

  • Instant Insights: Get distributions, missing values, correlations, and warnings in minutes.
  • Visual Summaries: No need to manually plot histograms, heatmaps, or pairwise relationships.
  • Faster Iteration: Quickly spot issues before investing time in modeling.

But don’t get me wrong—automation isn’t magic. It won’t tell you the story behind the data—that’s still on you.

How to Use Pandas Profiling?

I’ve found Pandas Profiling to be a lifesaver for structured datasets. It generates a full EDA report in HTML format, complete with warnings and recommendations.

Installation:

If you don’t have it yet, install it first:

pip install pandas-profiling

Generate a Full EDA Report in One Line

import pandas as pd
from pandas_profiling import ProfileReport

# Load your dataset
df = pd.read_csv("your_dataset.csv")

# Create the profile report
profile = ProfileReport(df, explorative=True)

# Generate and display the report
profile.to_file("eda_report.html")

Key Features in the Report:

🔹 Missing Values: It flags columns with too many NaNs and suggests possible fixes.
🔹 Feature Correlations: It highlights multicollinearity issues (so you know which features are redundant).
🔹 Outlier Detection: Identifies extreme values that might distort your analysis.

When I Use Pandas Profiling:

  • Large tabular datasets (e.g., customer data, sales reports).
  • Quick sanity checks before deep diving into the data.

But for more visually engaging summaries, I prefer Sweetviz.

How to Use Sweetviz?

Sweetviz offers side-by-side dataset comparisons, which is great for feature selection and model evaluation.

Installation:

pip install sweetviz

Generate an Interactive Report

import sweetviz as sv

# Analyze dataset
report = sv.analyze(df)

# Show report
report.show_html("sweetviz_report.html")

Why I Like Sweetviz:

  • It’s more interactive—lets you compare train/test datasets easily.
  • Great for feature selection—shows which features contribute most to the target variable.
  • Visual & Intuitive—perfect for presenting insights to stakeholders.

When I Use Sweetviz:

  • Comparing datasets (e.g., train vs. test, different time periods).
  • Feature selection—spotting which variables impact the target.

Best Practices: When to Automate vs. When to Manually Analyze?

Here’s where experience comes into play. Not all EDA should be automated.

✅ Use automation for:

  • Large, structured datasets where you need a quick overview.
  • Comparing datasets train vs. test (Sweetviz shines here).
  • Early-stage EDA before feature engineering.

❌ Manually analyze when:

  • You need domain-specific insights—automation can’t understand business context.
  • Your dataset is unstructured (e.g., text, images, time-series)—most auto-EDA tools struggle here.
  • You need custom visualizations tailored to a specific problem.

I’ve learned that the best approach is a mix of both—use automation for the grunt work, but don’t blindly trust the results.

Conclusion & Next Steps

At this point, you’ve seen EDA at all levels—basic statistics, feature engineering, clustering, financial data analysis, and automation.

What Should You Do Next?

  • Start applying EDA to personal projects—pick a dataset and automate where it makes sense.
  • Learn Feature Engineering—EDA is just the beginning; the real power comes from transforming raw data into valuable features.
  • Explore Model Interpretability—once you’re comfortable with EDA, start looking at SHAP values, LIME, and feature importance techniques.
  • Try Automated ML Pipelines—tools like PyCaret and AutoML take automation a step further, handling both EDA and model selection.

Additional Resources:

  • Kaggle Notebooks: Explore real-world EDA case studies.
  • GitHub Repos: Check out open-source implementations of automated EDA tools.
  • Recommended Books: Python for Data Analysis by Wes McKinney (Pandas creator).

Final Thoughts

Automating EDA is a huge time-saver, but it’s not a replacement for your expertise. The best data scientists know when to automate and when to dive deep manually.

Remember: Tools don’t make great data scientists—curiosity does.

Now, it’s your turn—how will you apply these techniques in your projects?

Leave a Comment