Securing Medical IoT Devices with an MLOps Automated Pipeline
Medical devices like smart infusion pumps and connected heart monitors keep patients safe every day. However, more connected medical devices could also mean a wider attack surface across hospital networks.
When these connected medical devices become targets, patient lives could be put in danger. Anyone can train a machine learning model to catch cyberattacks. Keeping that model alive and well-behaved inside a hectic hospital network? Totally different story.
Here is what we did at MosChip. We put together a production-ready MLOps pipeline, testing it out using the CIC-IoMT-2024 benchmark data from the Canadian Institute for Cybersecurity.
To give you an idea: this dataset simulates the digital pulse of 40 different Internet of Medical Things (IoMT) gadgets. It maps out network traffic so we can flag standard behavior alongside 18 brutal cyber threats like port scans, sudden DDoS hits, and ARP spoofing.
Here is how we overcome the hardest parts of production machine learning solutions using tools like Apache Airflow, Great Expectations, DVC, MLflow, GitLab, Scikit learn, KServe, Grafana and Evidently AI.
To bring this architecture to life, our end-to-end MLOps solution served as an automated security engine built to protect modern healthcare environments. The solution organises the entire machine learning lifecycle through a reliable multi-stage pipeline: raw network data flows smoothly into Apache Airflow, where it undergoes structured data ingestion, strict validation, transformation, version-controlled repository tracking via DVC, and automated model training.
The trained artifacts and tracking metrics are then registered in MLflow to pick the top-performing model. From there, a GitLab CI/CD pipeline takes over to execute rigorous model evaluation, handle deployment, power real-time prediction services for medical staff, and continuously monitor infrastructure alongside live traffic using advanced drift detection. When data or concept drift sneaks in, the pipeline flags those cases and routes them straight to manual annotation. Once labelled, they flow right back into the training cycle.
For hospitals, this means having a literal 24/7 watchdog. It watches network traffic, tracks data coming off critical IoMT medical devices, and figures out if traffic is safe or tied to one of 18 specific cyber threats.
The model spits out these attack predictions in real time. That lets hospital security teams act fast like blocking malicious IPs or isolating a compromised device immediately. Meanwhile, our MLOps pipeline handles the heavy lifting in the background, constantly evaluating and tuning the model, so patient care never skips a beat.
Building Fault-Tolerant ML Pipelines That Survive Infrastructure Failures
Turning raw network logs into live threat predictions is great in theory but keeping that pipeline running smoothly in a high-stakes medical environment is definitely easier said than done.
The Challenge
In a live hospital environment monitoring critical IoMT devices, infrastructure is unpredictable. Network connections drop, cloud storage encounters connection errors, and incoming raw network data logs arrive corrupted or completely missing. When an automated pipeline breaks halfway through data ingestion or training, it leaves security teams blind to potential cyber threats and forcing engineers to scramble to figure out where things went wrong.
Why This Happens and Why It Hurts
Data engineering and machine learning workflows are messy. Without proper orchestration, a single random network blip means you’re stuck rerunning a multi-hour pipeline completely from scratch. Not only does that burn through expensive cloud compute and stall model updates, but it’s also a great way to burn our engineers who have to babysit failing jobs late at night.
How We Fixed It
We fixed this by plugging in Apache Airflow to handle the heavy lifting. Instead of manual oversight, Airflow maps out task dependencies, automatically handles retries when things flake out, and pings us when something actually needs attention.
When raw network logs or device profiling data roll in from our IoMT setup and are stored in a safe folder, Airflow funnels them from collection to validation and transformation. If a task fails because storage is empty, Airflow catches the error, waits a moment, and retries safely without trashing our previous progress.
We also paired Airflow with DVC (Data Version Control) to track our datasets like source code. Instead of bloating our Git repository with gigabytes of network traffic files, DVC saves tiny pointer files. After successful data registration, Airflow will start model training. If a pipeline crashes or a dataset gets corrupted, our team can roll back the clock instantly and pull up the exact data slice used for any past model run.
Curious about MLOps?
Read more to find out: The Enterprise Blueprint for Scalable MLOps Orchestration
Catching Model Failures with Shift-Left Validation for MLOps
The Challenge
Waiting until a model is fully deployed to find out it performs poorly is a recipe for disaster. By the time bad predictions hit your users, the damage is already done.
Why This Happens and Why It Hurts
Traditional software uses unit tests, but machine learning teams often skip testing data and model artifacts before pushing them live. If you feed garbage data into your training loop, your model learns garbage patterns. Catching these issues late in the game means costly rollbacks, downtime, and broken trust from the business.
How We Fixed It
We used a “shift-left” approach, meaning we tested everything as early as possible before it ever touched a production server.
The second raw data lands in our pipeline; we hook Great Expectations right into Airflow. It runs strict checks against our dataset columns like Header_Length, Rate, and TCP flags to make sure values actually make sense. If sensor data reports impossible readings or missing values spike past a threshold, Airflow kills the pipeline immediately and pings us.
We apply the same strict rules to model training. Using Optuna for Bayesian hyperparameter tuning, we test diverse classification algorithms automatically. Once a model finishes training, our GitLab CI/CD pipeline takes over on a self-hosted runner. It spins up a clean environment, caches dependencies, and runs automated Scikit-Learn benchmark tests. If a model fails to hit our strict quality score on a holdout test set, the pipeline rejects it on the spot. No bad models make it past this gate.
Engineering Comprehensive Model Quality Evaluation
The Challenge
Standard metrics like raw accuracy do not tell the whole story, especially when dealing with complex multi-class problems like cybersecurity threats across 40 different medical devices.
Why This Happens and Why It Hurts
Looking at a single accuracy percentage can hide major blind spots. Your model might look like it is performing well overall but completely fails to catch critical minority classes like a dangerous Malformed Data attack or an ARP Spoofing attempt hidden inside normal hospital network traffic.
How We Fixed It
We built a thorough evaluation stage that goes way beyond basic numbers. During this step, a custom subprocess application runs our testing inputs and uses Scikit-Learn to compute precision, recall, accuracy, and detailed confusion matrices across all 18 attack categories.
To make the results easy for stakeholders to digest without digging through raw logs, we integrated Llama via Groq. The evaluation script feeds the performance metrics straight into Llama, which generates an automated, human-readable performance summary. It compiles everything into a clean, interactive HTML report.
If the model clears all quality checks, our system wraps the trained .pkl model file and a lightweight FastAPI app into a Docker container, pushes it to Docker Hub, and rolls it out using KServe on a local KIND Kubernetes cluster. KServe even gives us scale-to-zero capabilities, meaning cloud costs drop to zero when traffic is quiet.
Detecting Data Drift Before Model Performance Degrades
The Challenge
A model that works great today can slowly fail tomorrow. Real-world data changes over time, and when user behavior or network patterns shift, model accuracy silently drops.
Why This Happens and Why It Hurts
In cybersecurity and medical IoT, hackers constantly change their tactics, and medical devices get firmware updates that alter network traffic patterns. This creates covariate drift and concept drift. Without active monitoring, your model will just keep serving up confident hallucinations or dead-wrong predictions.
How We Fixed It
We tackled this by setting up Prometheus and Grafana for live observability, giving us instant visibility into infrastructure metrics, latency, and confidence scores.
For this specific IoMT demo, we added a clever safeguard: an “unknown” attack class designed to catch novel, untrained cyberthreats based on prediction confidence thresholds. If the volume of predictions falling into this “unknown” bucket crosses 10%, Grafana fires an automated alert.
That alert triggers Evidently AI, which immediately analyzes our live production traffic against our original training baseline. Evidently checks for both covariate and concept drift, packages the findings into a detailed diagnostic report, and emails it directly to our engineering team.
When a model starts losing its grip due to data drift, we route those examples into a review queue for labelling. Once they’re tagged, they drop right back into our DVC dataset store and automatically kick off a retraining run in Airflow.
Conclusion
Moving machine learning past the local notebook stage is notoriously tough. Still, trading manual guesswork for actual automation, robust tests, and proactive tracking changes the game, models start behaving like reliable software instead of walking on eggshells.
At MosChip, we help engineering teams bridge that exact gap. Whether you are trying to automate your data pipelines, secure your cloud infrastructure, or set up practical MLOps workflows from scratch, our engineers work side-by-side with your team to turn machine learning ideas into dependable tools you can rely on every single day. To know more about our expertise, drop us a line and we will arrange an introductory call with our MLOps team.
FAQs
Think of MLOps as an automated security guard for a hospital’s network. In healthcare, connected medical devices generate a massive stream of data every second, and hackers are constantly changing their tactics. MLOps takes the machine learning models trained to spot these cyberattacks and keeps them running reliably behind the scenes. It automates everything from cleaning incoming network data and testing models for accuracy to updating them when new security threats appear. This ensures that the AI protecting critical medical devices never misses a beat and stays accurate, reliable, and capable of adapting to emerging cyber threats in real time.
In our production architecture, KServe serves as the model deployment and serving layer running on a local KIND Kubernetes cluster. It takes the packaged model artifacts (the trained .pkl file and a lightweight FastAPI wrapper) from Docker Hub and scales them efficiently to power real-time attack predictions for medical and security staff. Crucially, KServe provides scale-to-zero capabilities, ensuring that cloud compute costs drop to zero during periods of quiet hospital network traffic without losing readiness for instant scaling when activity picks up.
Post-deployment testing shifts away from traditional static unit tests and relies heavily on real-time observability and continuous monitoring. We achieve this by tracking live infrastructure metrics, latency, and prediction confidence scores using Prometheus and Grafana. To catch novel or unexpected security threats, we utilize confidence threshold triggers, such as an automated alert if “unknown” attack classifications exceed 10%. Furthermore, tools like Evidently AI compare live production traffic against training baselines to detect covariate and concept drift, automatically feeding drifted data back into an automated retraining loop.
