Logistic Regression in Python – Splitting Data
One of the most important steps in any machine learning workflow is splitting the dataset into training and testing sets. Before building a Logistic Regression model in Python, we must divide the data properly to ensure fair evaluation and reliable performance measurement.
If we train and test a model on the same dataset, it will produce misleadingly high accuracy because it has already seen the data. This is why train-test split is essential for real-world machine learning systems.
In this guide, you will learn how to perform train-test split in Python using Scikit-Learn, understand why it matters, and follow best practices used in real machine learning projects.
Why Data Splitting is Important in Machine Learning
Data splitting is a critical step because it helps us evaluate how well a model performs on unseen data.
Key benefits:
- Ensures fair model evaluation
- Prevents overfitting
- Measures real-world performance
- Helps detect model weaknesses
- Improves generalization ability
Without splitting the dataset, the model may simply memorize patterns instead of learning meaningful relationships.
What is Train-Test Split?
Train-test split divides a dataset into two parts:
1. Training Set
The training set is used to train the Logistic Regression model.
- Typically 70%–80% of the dataset
- Used to learn patterns and relationships
- Model parameters are adjusted here
2. Testing Set
The testing set is used to evaluate the trained model.
- Typically 20%–30% of the dataset
- Used for final performance evaluation
- Helps simulate real-world predictions
Visual Representation of Data Splitting
Dataset
↓
-------------------------
| Training Set (80%) |
| Testing Set (20%) |
-------------------------
This separation ensures unbiased evaluation.
Import Required Library
We use Scikit-Learn for splitting data efficiently.
from sklearn.model_selection import train_test_split
Example Dataset for Train-Test Split
Let’s create a simple dataset to understand the concept clearly.
import pandas as pd
data = pd.DataFrame({
'Age': [22, 25, 30, 35, 40, 45, 50],
'Salary': [25000, 30000, 45000, 60000, 70000, 85000, 95000],
'Purchased': [0, 0, 0, 1, 1, 1, 1]
})
print(data)
This dataset represents customer behavior prediction.
Define Features and Target Variable
Before splitting, we separate input features and output labels.
X = data[['Age', 'Salary']]
y = data['Purchased']
Explanation:
- X → Input features (Age, Salary)
- y → Target variable (Purchased)
Perform Train-Test Split in Python
Now we split the dataset using Scikit-Learn.
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.25,
random_state=42
)
Explanation of Parameters
X
Input features used for training and testing.
y
Target variable to be predicted.
test_size
Defines the proportion of data used for testing.
Example:
test_size = 0.25 → 25% test data, 75% training data
random_state
Ensures reproducibility of results.
random_state = 42 → same split every time the code runs
This is important for consistent model evaluation.
Check Dataset Split Results
We can verify the size of each dataset.
print("Training set shape:", X_train.shape)
print("Testing set shape:", X_test.shape)
Example Output:
Training set shape: (5, 2)
Testing set shape: (2, 2)
This confirms the dataset has been split correctly.
View Training and Testing Data
Training Data:
print(X_train)
Testing Data:
print(X_test)
This helps us understand how data is distributed.
Why We Should NOT Use Full Dataset for Training
Using the entire dataset for training causes serious problems:
- The model memorizes data instead of learning patterns
- No proper evaluation is possible
- Leads to overfitting
- Real-world performance becomes unreliable
Train-test split solves this problem by separating learning and evaluation.
Understanding Data Leakage Problem
Data leakage happens when information from the test set influences training.
Common mistakes:
- Scaling before splitting
- Using test data in training
- Feature engineering on full dataset
Correct approach:
Split → Train → Test
This ensures fair and unbiased evaluation.
Best Practice Workflow for Machine Learning
A proper Logistic Regression workflow includes:
1. Load Dataset
2. Clean Data
3. Feature Engineering
4. Train-Test Split
5. Train Model
6. Evaluate Model
This is the standard pipeline used in real-world projects.
Train-Test Split with Feature Scaling (Important Rule)
Feature scaling should always be done after splitting the dataset.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
Important Rule:
- Fit scaler only on training data
- Transform both training and testing data
This prevents data leakage.
Complete Train-Test Split Example Code
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
# Dataset
data = pd.DataFrame({
'Age': [22, 25, 30, 35, 40, 45, 50],
'Salary': [25000, 30000, 45000, 60000, 70000, 85000, 95000],
'Purchased': [0, 0, 0, 1, 1, 1, 1]
})
# Features and target
X = data[['Age', 'Salary']]
y = data['Purchased']
# Train-test split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42
)
# Feature scaling
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
print("Data successfully split and prepared for model training")
Advantages of Train-Test Split
- Provides realistic model evaluation
- Prevents overfitting
- Improves model generalization
- Simulates real-world scenarios
- Ensures unbiased performance measurement
Common Mistakes to Avoid
Many beginners make these errors:
- Training on full dataset
- Applying scaling before splitting
- Ignoring random_state
- Choosing incorrect test size
- Causing data leakage
Avoiding these mistakes improves model reliability.
Real-World Importance of Train-Test Split
Train-test split is used in almost every machine learning application:
- Fraud detection systems
- Healthcare prediction models
- Customer behavior analysis
- Recommendation systems
- Financial risk prediction
It ensures models perform well in real-world environments, not just in training.
Conclusion
Train-test split is a fundamental step in building Logistic Regression models in Python. It ensures that your model is trained on one part of the data and evaluated on unseen data, providing a realistic measure of performance.
By using Scikit-Learn’s train_test_split() correctly, you can build more accurate, reliable, and production-ready machine learning models.
Understanding this step is essential for anyone learning machine learning and data science.
0 Comments