Model Evaluation Benchmark

Owner Name: Theerut Dokkatin

Student ID: 6604022630250

Project Goal

Implement and compare four model evaluation methods on a real medical classification dataset.

Course Requirement Coverage

• 4 methods implemented: Holdout, Random Subsampling, Repeated CV, Bootstrap
• At least 2 classifiers used
• Reported mean, std, 95% CI for each method-classifier pair
• Includes bar chart and boxplot visualizations
• Notebook contains code + output cells

Dataset

• Name: Breast Cancer Wisconsin
• Source: sklearn.datasets.load_breast_cancer
• Samples: 569
• Features: 30
• Task: Binary classification

Methods Implemented

1. Holdout (stratified, 10 iterations with different seeds)
2. Random Subsampling (R=60)
3. Repeated Stratified K-Fold (10x5)
4. Bootstrap OOB (B=300)

Classifiers

1. Logistic Regression
2. Linear SVM

Reported Metrics

• Accuracy
• Precision (macro)
• Recall (macro)
• F1-score (macro)
• Mean, Std, 95% CI for each method-classifier combination

Outputs

• Notebook: 6604022630250_Project.ipynb
• Summary CSV: outputs/metrics_summary.csv
• Long-format scores: outputs/all_scores_long.csv
• Figures:
  • figures/holdout_confusion_matrices.png
  • figures/benchmark_bar_boxplot.png

How To Run

1. Open notebook: 6604022630250_Project.ipynb
2. Run all cells from top to bottom
3. Confirm these files are generated:
   • outputs/metrics_summary.csv
   • outputs/all_scores_long.csv
   • figures/holdout_confusion_matrices.png
   • figures/benchmark_bar_boxplot.png

Submission Checklist

• Jupyter Notebook (.ipynb) with all code and output cells
• PDF export of the notebook
• All figures saved as PNG (dpi >= 150)
• README.md
• ZIP package named with student ID (example: 6604022630250_Project.zip)

Notes

• Test data is not used for training in Holdout.
• Repeated stratified splits are used to reduce variance.
• Pipelines are used to prevent data leakage.

Writing Style

• Core language: English
• Thai insight: ใช้ภาษาไทยเสริมเพื่ออธิบายเหตุผลเชิงตีความและข้อสังเกตสำคัญ