TL;DR

Uncountable is a project my team made in 24 hours during the 2023 HackOHI/O Hackathon, winning 1st place out of 200 teams. Using computer vision, Uncountable tracks surgical instruments during a procedure, alerting the surgeon if any instruments are left inside the patient.

Problem

RSI incidents occur when a surgical tool is accidentally left inside a patient, potentially leading to severe complications like infection and organ damage ¹. With about 1,500 incidents each year in the U.S. alone ², this problem is typically managed by manual counting or costly RFID tagging. We wanted to create a solution that would be both cost-effective and accurate, automating the process with computer vision, all in 24 hours.

Process

To create a computer vision software, we first needed some hardware: A “budget-conscious” cardboard box to simulate a patient’s body and a top-down web camera to capture the movement of the instruments. We found both in a matter of 30 minutes (running all around campus) and started the works.

We decided to base our model off YOLOv8, a state-of-the-art real-time object detection model, and enhanced its performance by collecting a custom dataset of instrument images. We took pictures of every related tool we could find at the hackathon: plastic knives, scissors, tissues, and hands in the view of the camera to fine-tune the model and backround subtraction.

The detection model was augmented with a tracking algorithm, which could detect if something was put inside of the box (aka the patient) or taken out.

I designed an Electron.js user interface that gives surgeons a clear, real-time overview of all instruments in use, with a live camera feed and a log of instrument entries and removals.

As you may know, Electron.js uses JavaScript, but our PyTorch model was in Python. To bridge the gap, a Python program was spawned as a separate child process with a Flask server, which communicated with the Electron.js UI through WebSocket API.

To make the entire system work seamlessly in real time, I built a multithreaded architecture where the CV model, tracking algorithm, and Python server ran concurrently, providing crucial speed for a surgical application.

Obviously, a JavaScript tool like Electron.js is not the best choice for a high-performance application, but it was best to tackle the problem in 24 hours due to its rapid development capabilities.

After a long night of coding, testing, debugging, and drinking Red Bull, we glued parts of the system together and … it worked! Our fine-tuned model detected instruments, tracking algorithm registered insertion/removal events, and the UI displayed everything in real-time. Our only limitations being the 24 hour limit and compute of a laptop, we built something that could potentially save lives.

Aftermath

Winning 1st place at the 2023 HackOHI/O Hackathon out of 200 teams and 800 participants at HackOHI/O 2023 was a huge honor and an affirmation of our team’s effort and dedication to solving a real-world problem. My collaborators, David Novikov, Zeyad Mansour, and Michael Lin, demonstrated outstanding skill and collaboration throughout the challenge.

See more