CART OPS (Clasificacion Automatica Real Time in Operations)

Pytesseract: For Optical Character Recognition (OCR) to extract text from images or scanned documents. Example: pytesseract.image_to_string() to convert images to text.

Pandas: For data manipulation, preprocessing, and handling dataframes. Example: Managing and cleaning data input from various sources.

NumPy: For numerical operations and handling arrays efficiently. Example: Performing matrix operations and calculations.

NLTK (Natural Language Toolkit):For natural language processing tasks such as tokenization, stemming, and stop word removal. Example: Preprocessing text data extracted from documents.

spaCy: For advanced NLP tasks, including named entity recognition and dependency parsing. Example: Extracting meaningful features from processed text

Scikit-learn: For machine learning algorithms including SVM, Random Forest, and PCA. Example: RandomForestClassifier() for classification tasks and PCA() for dimensionality reduction.

Imbalanced-learn: For handling imbalanced datasets with techniques like over-sampling (SMOTE) and under-sampling. Example: SMOTE() to create synthetic samples for minority classes.

TensorFlow/Keras or PyTorch: For building and training neural network models. Example: Implementing deep learning architectures for more complex classification tasks.

OpenCV: For image processing tasks that may be necessary to enhance images before OCR. Example: Preprocessing images (e.g., resizing, denoising) to improve OCR accuracy.

Matplotlib/Seaborn: For data visualization and model evaluation. Example: Plotting results to analyze classification performance.

Explanation:

OCR - object character recognition (Google Cloud Vision, Tesseract - Open Source, Rossum AI, Amazon Textract, Abbysoft Reader)

PDF splitter to images: ghostscript, imageMagick.

Data Preparation:

- Computer Vision - OCR, NLP (tokenization, stopword removal, stemming, equivalence classes, lowercase, data cleaning - dictionary matching, removal of chars, data decryption, handling different languages, TFIDF - convert a collection of raw documents to a matric of tfidf features), Feature extraction (if specific keyword appears in text), Principal Component Analysis - using randomized SVD, linear dimensionality reduction keeping only the most significiant vectors to project data to a lower dimensional space.

Model training:

- Linear classifiers,

- SVM

- Random Forest

- Neural Network

Model Selection:

- Precision, Recall, F1

Imballance:

- Oversampling: SMOTE, ADASYN, Random Undersampling

- Undersampling: Random, Near Miss, Cluster Centroids.

Challenges: OCR quality, standardized data input, NLP data cleaning, document misclasification, time and memory complexity, dealing with imbalance, dealing with similar classes and overalapping classes.

Contributions:

- Single model for multi-language context,

- Used class/ subclass hierarchy in prediction

- Used synthetic groups of subclasses for prediction

- Combined the emails multiple fields in classification

- Handled the imbalance by over/ under sampling