Building a Real-Time Flood Rescue AI with Isaac Sim and TensorRT

🏗️ 1. Synthetic Data Generation with Isaac Sim

One of the biggest challenges in disaster response AI is the lack of training data. You can't safely fly a drone or drive a boat into a real flood just to collect images.

To overcome this, I used NVIDIA Isaac Sim and its Replicator extension. By creating a photorealistic 3D environment simulating a flooded area, I was able to generate diverse, annotated synthetic datasets perfectly tailored for our models. This allowed us to control lighting, water levels, and object placement to ensure the models were robust.

🧠 2. Dual-Model Architecture: YOLO + UNet

This project requires two distinct types of visual understanding:

1. YOLO (You Only Look Once): Trained to perform high-speed object detection. It identifies stranded persons, obstacles (like debris), and safe goal areas in the image frame.
2. UNet: Trained to generate spatial Grid Maps. By performing semantic segmentation on the input images, UNet produces a top-down view of navigable water vs. non-navigable obstacles, allowing a drone or boat to chart a safe path.

- Result Image of Yolo

- Result Image of UNet

⚡ 3. Real-Time Inference on Edge Devices (C++ & TensorRT)

Running two dense neural networks simultaneously on an embedded system like the Jetson Nano requires extreme optimization. Python overhead is often too high for strict real-time requirements.

To achieve maximum performance, I ported the inference pipeline to C++ using the TensorRT (NvInfer) API.

TensorRT Implementation Highlights

I created custom C++ wrapper classes for both models to handle memory allocation, format conversions, and GPU execution seamlessly.

The UNet TensorRT Wrapper:

// A snippet of the UNet TensorRT wrapper handles RGB conversion and dual grid map outputs
void infer(const cv::Mat& bgr, cv::Mat& gridA, cv::Mat& gridB) {
    cv::Mat resized, rgb;
    cv::resize(bgr, resized, {inW_, inH_});
    cv::cvtColor(resized, rgb, cv::COLOR_BGR2RGB);
    
    // Memory copy to device
    cudaMemcpyAsync(buffers_[inIdx_], hostIn.data(), hostIn.size()*sizeof(float),
                    cudaMemcpyHostToDevice, stream_);
                    
    // Execute inference
    ctx_->enqueueV2(buffers_.data(), stream_, nullptr);

    // Fetch dual outputs (Grid A and Grid B)
    gridA = fetch(outIdxs_[0]);
    gridB = fetch(outIdxs_[1]);
}

The YOLO TensorRT Wrapper:

The YOLO implementation handles bounding box detection, applying Non-Maximum Suppression (NMS) to filter redundant boxes, and directly overlaying them onto the output video frame.

The Unified Execution Script

The resulting application is invoked from a single command-line script, taking the compiled TensorRT engines (.engine files) and the input video feed, outputting a highly optimized result video along with the grid map frames.

./trt_yolo \
  --engine ../yolo_model.fp16.416.engine \
  --engine-obst ../obst_model.fp16.416.engine \
  --engine-goal ../goal_model.fp16.416.engine \
  --input ../output_toslow.mp4 \
  --grid-out ../output_grid/ \
  --save ../result.mp4 \
  --display

🚀 Conclusion

By combining the synthetic data generation capabilities of Isaac Sim with the high-performance edge inference of TensorRT, this system demonstrates how AI can be effectively deployed in critical, time-sensitive disaster scenarios.

The dual-model approach ensures that a rescue vehicle not only knows what is in its path, but how to navigate around it safely.