2024-10-01 · 1 min read
Fraud Detection on EMSCAD: Why PR-AUC Beats Accuracy
TF-IDF + VADER features on 17,880 job postings: 0.80 fraud-class precision and 0.66 PR-AUC against a 0.048 base rate. Why accuracy is the wrong headline for rare-event detection.
The problem
Fraudulent job postings exploit text patterns and metadata inconsistencies. The challenge is catching them on a heavily imbalanced corpus with interpretable, real-time scoring.
The hybrid feature stack
Instead of deep learning alone, I engineered:
- TF-IDF vectorization for lexical fraud patterns
- VADER sentiment scores for emotional manipulation signals
- Metadata features from posting structure
Results
On the 17,880-posting EMSCAD dataset the model reaches 96.9% accuracy, but with only 4.8% of postings fraudulent, accuracy is easy. The number that matters is 0.80 precision on the fraud class and 0.66 PR-AUC against a 0.048 base rate (a fixed leakage bug had previously inflated results). Well-engineered classical features stay competitive on structured text.
Deployment
A real-time Dash dashboard visualizes fraud probabilities per posting, making the model actionable for screening workflows.
Takeaway
Feature engineering discipline beats model complexity on domain-specific text classification. Code at github.com/priyanshshahh/fraud-job-detection-pipeline.
Related project
Fraudulent Job Posting Detection
NLP fraud classifier on 17,880 EMSCAD job postings. 0.80 precision on the fraud class and 0.66 PR-AUC against a 0.048 base rate. 19 tests.