Week 2 I worked on many different things all at once. I primarily had a meeting with Golnar in which we discussed the setup for her experiments and the impact that might have on my tests. Essentially, the backlit system that was previously used made object detection and tracking very difficult because of the lack of distinguishing features among the bees.

Gonlar’s project utilized a front lit system so we decided that I should run tests with videos from both the backlit and front lit system to see where/if XMem (my proposed object detection/segmentation framework) breaks down. Golnar will come back July 10 so I intend to run tests until then, and afterwards we can decide if new data needs to be collected.

Along with this, I was able to run the demo provided for XMem; however, upon setting up the GUI and running my own tests, I noticed several issues with the code that I was able to fix.

• They were using PyQt5 which is now outdated by PyQt6, so I updated code to utilize the new system
• MPS not supported in their implementation of the PyTorch logic so Mac users could not utilize their gpus, so I modified all of the PyTorch code to check if users require MPS support.
  • This was quite difficult because MPS is still new to PyTorch at this time, so there were many broken features that I had to work around.
• The video player was not working and upon closer inspection, I realized the code for the timer was not written correctly because it did not account for the timeout sequence. I edited the code and the video started working properly.
• Clear memory was not working because PyTorch does not provied MPS support for accessing total system memory. I patched a temporary fix where I just return their currently used memory, until PyTorch updates their MPS support.
• The keys 1 through N were supposed to be used as shortcuts to select N objects; however, these shortcuts did not work if users were on Mac because of way Qt handles Mac shortcuts. Along with this, if users wanted to select more than 9 objects they could not. To solve these issues, I changed the shortcuts to be Cmd/Cntrl + (1 - N) and I also added a new widget to the GUI where users could manually select the object they wanted to select/edit.
• XMem did not support exporting frames to video so I added my own widget to the UI to do so. As long as users have rendered and saved all of the overlay frames, they do not have to wait any longer for the export because it uses those saved frames.

Updating XMem took a bit of work but I believe it is worth it because it is used for other world class systems such as Segment Anything and Track Anything

I aim to clean up my code then raise a pull request for XMem during week 3.

Finally, I had my meeting with Dr. Peleg on Friday and we discussed the end goals for the project. Essentially the goall will be to track the temporal data associated with an individual bee, and use this data to draw conclusions about the balance between searching and scenting (i.e. exploration vs exploitation) for the swarm. Along with this, I am planning on submitting our project to 2023 SACNAS, a conference for undergraduates engaged in summer research, and so we also discussed plans for writing and editing my abstract.