Ibrahim Abedrabbo

BRIEF
Implemented a solution directly from a published paper to predict 3D objects from fisheye cameras without training images, using our vanilla model trained on rectilinear images. Tasks included:
• Python implementation
• Porting to C++
• Integration with the main AI stack

The implementation has significantly reduced R&D time, as I did not train on any additional images. I utilized the existing 3D object detector.

METHODOLOGY
• Problem: Perspective images are shift-invariant, which means that an object moving along the X and Y axes with constant Z will result in the same appearance model and size. This is not true for fisheye images, where an object with the same depth will have a different appearance model and size when translated along the X and Y axes.
• Solution: The solution is to transform fisheye images into cylindrical projections, which are shift-invariant. Then, a pre-trained 3D object detector model is utilized, and the transformed images are passed to predict 3D objects.

IMPLEMENTATION STEPS
1. Transform fisheye images into cylindrical projections.
2. Inference on cylindrical Images using a pre-trained 3d Object detector.
3. Convert the detected 3D bounding box from 3D cylindrical space to real 3D space.

This project employs the Llama-2-7B model, fine-tuned for question-answering using two datasets: physics books and a scientific paper. The models adapted to the domains and provided representative answers.

Two phases for fine-tuning Llama-2:
1. Self-supervised learning (pre-training) on a text corpus in a causal learning fashion (CLM).
2. Supervised learning (fine-tuning) for question-answering.

DATASET
For self-supervised learning, a collection of physics books was used for one model, and a scientific paper for the other. Text was extracted and split into chunks using LangChain. For supervised learning, question-and-answer datasets were generated for each domain using Llama2-7 b-chat LLM via an automated script. The script augments text chunks into prompts, generating Q&As split into input (question) and target (answer).

TRAINING
For each domain:
1. Train the self-supervised model with text chunks.
2. Fine-tune the supervised model with the Q&A dataset.

Parameter-efficient techniques (PEFT) and QLoRA reduced trainable parameters from 7B to approximately 34M by freezing most of the network and training only adapters.

• For evaluation, Perplexity and Rouge metrics were used.

(Accepted Conference Paper; MedPRAI24)

(Accepted Conference Paper; CCIDSA-2024)

PAPER TITLE
Optimizing Neural Networks for Edge Devices via Structural Pruning: An Empirical Study with DepGraph

BRIEF
• The main goal in this project was to provide an experimental proof of structural model pruning for optimizing neural networks, resulting in higher inference FPS.
• The structural pruning technique from an open-source project was employed to prune the SwinTransformer-tiny model for classification on a subset of the ImageNet-1k dataset.
• The inference time was decreased by up to 45% compared to the original model while maintaining the accuracy of the original, unpruned model.

PRUNING AND TRAINING
The solution involves the following steps:
• Training the base vanilla model on the dataset.
• Apply structural pruning.
• Fine-tuning the pruned model on the same dataset.

MODEL PRUNING RESULTS
Two different pruning configurations were applied to the vanilla model with different pruning %. The number of trained parameters dropped significantly for both the pruned models. Also, inference speed was increased by roughly 23% for the first model and by nearly 44% for the second model.

ACCURACY COMPARISON
Pruning retains competitive performance across various evaluation metrics.

A real-world problem solution for autonomous driving.

BRIEF
• Introduced the TensorRT framework into the company's workflow, optimizing all the company's neural network graphs and achieving a 20%-40% increase in frame rate.
• Developed a C++ API for loading and initializing the engines and orchestrating the input, inference, and output during runtime.
• Automated the model deployment pipeline, converting models from PyTorch to ONNX and generating TensorRT engines for all the networks.
• Significantly reduced neural network inference time, enhancing overall system performance.
• Did the quantization of models with both FP16 and INT8 precisions while maintaining accuracy.
• Used both PTQ and QAT techniques (calibration data and quantize-aware training) for INT8 quantization.

WORK FLOW
1. The trained PyTorch models were converted to ONNX graphs.
2. ONNX graphs are then processed with the Polygraphy to have a dynamic batch dimension.
3. Finally, I converted ONNX graphs to TensorRT engines.

When starting the AI system, the engines are loaded via the C++ API and are ready to perform inference on new input.

MASTER'S THESIS
• Developed a machine learning framework for lattice fault detection and correction in a synchrotron storage ring.
• Designed and trained FCNN and LSTM models on simulated beam position monitor (BPM) signals to predict vertical quadrupole misalignments.
• Created a custom Python-based simulator to generate beam dynamics data for training.
• Benchmarked model robustness through BPM and tilt noise tolerance evaluation, cross-validation, and accumulated training.
• Achieved effective restoration of the beam’s vertical orbits to near-baseline conditions post-correction.
• Observed models reached 0.57 µm and 0.04 µm MAE, respectively, under realistic noise and tilt.
• Demonstrated the potential of ML-driven fault diagnosis and correction as a fast and accurate approach for modern accelerators, enabling stable, high-quality beams.

The project involved developing a BERT-based model to classify job proposal texts into categories such as 'very poor', 'poor', 'fair', 'good', and 'excellent'.

CONTRIBUTIONS
• Fine-tuned a family of BERT models, including BERT and DistilBERT, achieving 0.96 accuracy on training data and 0.89 on evaluation data.
• Led the end-to-end development, from data preprocessing to model deployment on Hugging Face.
• Utilized a small hand-crafted dataset and synthesized extra samples in alignment with the client's needs.
• Created insightful visualizations to monitor training progress and model performance.