Leveraging ML and computer vision for real-time surgical scene recognition and enhancing operating room efficiency

Binariks approached this project with a data-driven and research-intensive methodology, ensuring that the solution would be both scalable and highly accurate in recognizing key surgical events. The project kicked off in June 2024 as a proof of concept (PoC), where we collaborated closely with the client to validate the feasibility of an AI-powered video recognition system for operating room management.

Our team – comprising a Project Manager, Senior Data Scientist, and Lead Data Scientist – began by exploring existing open-source computer vision models to determine the most effective approach. We evaluated MoviNets, TSM, 3DCNN, and TimeSformer, ultimately selecting TimeSformer due to its superior accuracy and efficiency in training and inference.

Initially, we planned to leverage MONAI.io for data labeling and model training, but after thorough investigation, we pivoted to a custom approach using AWS EC2 for model experimentation.

With a focus on real-time tracking, the solution was designed to provide medical staff and hospital management with actionable insights to optimize Operating Room Effectiveness (ORE) – not only streamlining hospital operations but also enhancing patient care by reducing delays and increasing surgical capacity.

To achieve the project objectives, Binariks followed a structured, research-driven approach, balancing accuracy, efficiency, and scalability. The implementation phase involved the following key steps:

1. Model selection

We conducted research on open-source computer vision models for video action recognition, evaluating MoviNets, TSM, 3DCNN, and TimeSformer, each trained on about 500 hours of annotated surgical videos from the client.

• MoviNets: Strong accuracy but excessive training time led to its rejection.
• 3D CNNs: Computationally expensive with poor accuracy on surgical videos.
• TimeSformer: Selected due to its transformer-based architecture, superior accuracy, and ability to handle temporal dependencies in video data.

2. Data preparation and preprocessing

• Utilized the client's labeled dataset of surgical videos, containing key events such as "wheels in," "operation start," "operation end," and "wheels out."
• Preprocessed the videos to align with model dataset requirements by segmenting events from full-length recordings.
• Standardized frame rates and resolutions to ensure consistency across the dataset.

3. Model training and implementation

• Used AWS EC2 GPU instances to train models efficiently, reducing computation time.
• Fine-tuned the neural networks using PyTorch,which was chosen due to its flexibility, ease of use, and dynamic computation graph essential for working with complex video models like TimeSformer.
• Achieved an initial accuracy of 82%, targeting 95% through continuous model refinement.
• Integrated MLflow for experiment tracking and hyperparameter optimization.

4. Deployment and real-time inference

• Deployed the trained TimeSformer model on AWS SageMaker, ensuring scalable and managed inference.
• Implemented GStreamer for video stream processing, enabling real-time detection of surgical events.
• Designed a CI/CD pipeline using Docker and Git, ensuring smooth model updates and version control.