This week I sucessfully rewrote the bee tracking/segmentation software in python and integrated the code into the existing system. I was able to reduce the label switching issue when bees were clustered; however, I had a few more ideas on how to solve this issue. After speaking with Dr. Peleg, she and I agreed that I could continue working on the project, even though the DREU program was technically over. I just learned that I was accepted to present my summer research at the 2023 NDiSTEM Conference, so Dr. Peleg and I will continue working on the project until then so we can have good results.

I had been having issues getting the BIO software to run on my MAC so I spent this week debugging the software and allowing it to run on MacOS. Along with this, I realized I knew how to recreate a majority of the system using OpenCV and python, so I spoke with Dr. Peleg about rewriting the entire system so it would work better with our existing tools. She was in support of the idea, as long as I could do it within a reasonable amount of time, so I worked on rewriting the system through week 10.

This week I did not work on the project because I was presenting at the 2023 National McNair Scholars conference.

This weeks was fairly simple, I mainly worked on drawing the bee trajectories on the screen so we could analyze how their position data changes over time. I also passively worked on the clustering issue.

This week I worked on a few different tasks. I continued to try resolving the clustering issues with the BIO system; however, along with this I created a script to translate the position/state data exported from the BIO system, into a format for the existing CollectiveScentingCV system. After creating this script, both systems could communicate and we had a complete pipleline.

This week I was running tests with the BIO software to track the positions of the honeybees. I was able to figure out the most common points of failure for the system. The most prominent of which was when the bees were clustered together, the system would often swap the labels of the bees. I also ran a few demo tests with the existing CollectiveScentingCV code to understand how to classify the behavior of the honeybees as “scenting” or “non-scenting.” Together Dr. Peleg and I decided that we should spend a bit more time attempting to reduce the clustering issues; however, if we couldn’t find a solution within two weeks then we would move on.

During week 4, my goal was to test a few other trackers and see what would work best for our research. This was a tricky task since my tests with XMem displayed I might not need an object segmentation tool, so instead I started to look for object tracking tools. During this week Dr. Peleg sent me this twitter post of an alternative object tracker called BIO. This tracker was made for low resource environments; however, it was trained to track many types of animals in potentially occluded environments. In order to run the tool, I had to learn how to setup a Virtual Machine on my laptop to emulate Windows, but after that I was able to test the tool. I was shocked because it worked very well on my preliminary tests, and was able to keep track of individual bees even within occluded environments. My only doubt was the fact that the tool relied on thresholding, where it turns the environment into a 0 (black pixel) or 1 (white pixel) based on if the object is of interest. This means that it could suffer from the same issue as the tracker previously used for the project, since that tracker could not keep track of individual bees within groups very well since it would lose track. This worry is what I will attempt to clarify with my tests during week 5.

Week 3 was filled with a variety of tasks. One such task was that I was able to raise the XMem pull request and get in contact with the creator to review my changes. This was exciting because it is one of the largest open source projects I have contributed to. Since it has over 1.2k stars, 100+ forks, and the fact that is heavily utilized in projects such as Segment Anything by Meta, I was able to make a meaningful contribuition to further the research community. For archival purposes, I have provided a list of my contributions to the project: With increased MPS support from PyTorch, this feature will be necessary for users to run XMem on-device.

• Updated code to use PyQt6
• Added checks for MPS instead of just CUDA
• Added code to correctly start the timeout sequence for the video player
• Tracking memory usage is not supported by PyTorch, so I patched a temporary fix where I just return their currently used memory, until PyTorch updates their MPS support
• Changed the shortcuts to be Cmd/Cntrl + (1 - N) and I also added a new widget to the GUI where users could manually select the object they wanted to select/edit. The keys 1 through N did not work if users were on Mac because of way Qt handles Mac shortcuts. Along with this, if users wanted to select more than 9 objects they previously could not.
  • Actually fixed this issue by adding self.setFocusPolicy(Qt.FocusPolicy.StrongFocus)
  • Forces the computer to give focus to XMem
• Added widget to export video of overlayed frames.

Week 2 I worked on many different things all at once. I primarily had a meeting with Golnar in which we discussed the setup for her experiments and the impact that might have on my tests. Essentially, the backlit system that was previously used made object detection and tracking very difficult because of the lack of distinguishing features among the bees.

The first week I primarily read the relevant literature surrounding my project. There was a significant body of work surrounding the pathfinding capabilities of honeybees, so I read 4 papers and 1 book to learn about these topics. Overall the readings made sense and were thoroughly interesting, although I had one major question regarding the lighting setup for the experiments. The papers I read utilized a backlit system, which would be less than ideal when utilizing computer vision to distinguish the bees. Lucklily, a quick meeting with Dr. Peleg informed me that recent work has modified this lighting system to better identify features of individual bees. I had a few questions about the specific experiments that were conducted; however, I will be meeting with Golnar Fard during week 2 to discuss these questions in more detail.