Andrija Gajic

I created an application that analyzes full tennis match videos in collaboration with professional tennis players. The application extracts valuable information on players' and opponents' patterns to prepare for matches. The application combines several computer vision tasks, including object detection, object tracking, video classification, person re-identification, homography matrix extraction, etc.

It uses TrackNet for ball and court detection, as well as YOLOv5 and ReID for player detection. A homography matrix is created using court detection results, and the players and the ball are converted to the 2D frame. Bounces and hits are detected using the R3D video recognition network, while OCR is used for scoreboard detection.

Augmented virtuality systems merge real-world objects into virtual surroundings. The typical objects merged are a user's arms and body, which is done by performing semantic segmentation on the pixel level, determining whether each pixel corresponds to the body or background. Before this work, methods used color image processing for segmentation.

In our work, we proposed the usage of deep neural networks for semantic segmentation for mixed reality. Because the segmentation has to be done in real time, we used ThunderNet architecture as a starting point, further optimizing some of its bottlenecks for our cause, such as adding long skip connections between the encoder and decoder and changing the pyramid pooling module.

Since we moved to using deep neural networks for semantic segmentation, we also needed a large dataset, and there was no such dataset available online for mixed reality. Therefore, I was involved in the extraction of a semi-synthetic dataset. This involved recording egocentric videos of a user performing actions in front of green chroma, extracting from it the foreground mask, and then blending it with recorded egocentric videos of the background using an alpha matting algorithm.

The performance of convolutional neural networks for image classification has vastly increased in recent years. This success goes hand in hand with the need to explain and understand their decisions.

The problem of attribution deals specifically with the characterization of the response of convolutional neural networks by identifying the input features responsible for the model's decision. Perturbation-based attribution methods measure the effect of perturbations applied to the input image in the model's output. In this paper, we discussed the limitations of existing approaches and proposed a novel perturbation-based attribution method guided by semantic segmentation.

Our method inhibits specific image areas according to their assigned semantic label to link perturbations with a semantic meaning. The proposed semantic-guided attribution method enables us to delve deeper into scene recognition interpretability by obtaining the sets of relevant, irrelevant, and distracting semantic labels for each scene class.

Experimental results suggest that the method can boost research by increasing the understanding of convolutional neural networks while uncovering dataset biases that may have been included inadvertently.

An OpenCV project consisting of two parts: detection and tracking. Detection is used to initialize the tracker and help it reinitialize once the tracker loses the target. Template matching is used as an algorithm for detection with three different templates of a ball (in different scales) used. Once the detection is set, tracking takes place. Tracking is done using the channel and spatial reliability tracking algorithm. The precision of tracking is monitored, and if the difference between template and tracking prediction gets too high, the algorithm is reinitialized. The algorithm performs at approximately an 11FPS frame rate.

The goal of the project was to perform a classification of patients based on their primary tumor type, using the information about their genetic code. To prepare for the project, I completed a series of bioinformatics courses and learned about the main algorithms used in bioinformatics, such as alignment and assembly, and the most used data types. Then I became familiar with the Cancer Cell Line Encyclopedia (CCLE), which contains 947 BAM files of different cancer cells. Using these BAM files, I generated VCF files using Samtools.

The idea was to use the generated files and, based on the variations present in them, perform classification of the primary tumor type to either cervical or lung cancer. The classification was performed based on the mutations that happened in receptor tyrosine kinases (RTKs). The features selected were locations in the DNA code where the mutations occurred, the number of mutations in each of the RTKs, and the frequency of each alternative base compared to the other bases in one sample. The classification was performed using neural networks in the scikit-learn library.