Pipeline
From raw survey data to distributed analysis
CSVSurvey dataset
→
HDFSDistributed storage
→
HiveSQL exploration
→
SparkProcessing + SQL
→
MLlibClassification
→
ZeppelinVisual analysis
Dataset snapshot
Key metrics
Exploratory analysis
Patterns in treatment-seeking responses
Treatment balance
The target variable is nearly balanced, which made it suitable for a simple binary-classification exercise.
Work interference
Among respondents who sought treatment, “Sometimes” was the most common reported level of work interference.
Wellness programs
Most treatment seekers reported that their employer did not provide a wellness program.
Top countries
The United States contributed the largest number of treatment-seeking responses in this survey sample.
Predictive modeling
MLlib model comparison
The original Zeppelin workflow indexed categorical variables, assembled features, used an 80/20 train-test split, and evaluated two classifiers using area under the ROC curve (AUC).
The original report called this value “accuracy,” but the evaluator shown in the implementation used areaUnderROC. The portfolio presentation uses the metric name supported by the code.
Data quality
Messy data was part of the analysis
Gender normalization
The raw survey includes 49 distinct gender labels. The Spark SQL analysis grouped common male/female variants and retained the remaining responses as a separate category.
Employee-range cleanup
Values such as Jun-25 and 01-May appeared in the company-size field, consistent with spreadsheet auto-formatting. The project normalized them in Spark SQL.
Age outliers
A small number of invalid ages are present in the raw data. The original visualization applied a range filter; this portfolio summary uses valid adult ages when discussing age trends.
Source & implementation
See the reconstructed Hadoop, Hive, Spark SQL and MLlib workflow
This page is a static portfolio companion. Hadoop and Spark are not running in the browser; the GitHub repository preserves the original workflow as reconstructed source files and implementation screenshots.
Open repository