Babysitting assistant for the OriginBot Home Assistant Program

This blog mainly talks about how to take care of your baby through OriginBot, and when the baby's face is not within the range of the camera, send a message to the DingTalk group to notify the family to check it in time.

Preface

I had a baby last month, in order to facilitate the care of the baby, I bought a camera with a baby care function, but the product is not very good, the most important face occlusion function can not be used, and then it was returned. After returning the product, I came up with the idea of using OriginBot to do a similar function, so I had this blog~

Functional Flow Diagram (Architecture Diagram)

The specific process or structure is as follows:

In fact, the whole is not complicated, there is a MIPI camera on OriginBot, and then use the human body detection and tracking of Horizon TogetheROS.Bot to detect whether there is a face in the camera in real time, if there is no face, send a piece of data to the backend, and then the backend will send a message to the DingTalk group to tell the family. In the DingTalk group, you need to create a webhook in advance.

The following will be divided into three parts: human detection, judging whether there is a face, and back-end operation to record.

Human detection

This part uses the out-of-the-box functionality of Horizon TogetheROS.Bot, after starting OriginBot, run the following command from the command line:

# 配置tros.b环境
source /opt/tros/setup.bash

# 从tros.b的安装路径中拷贝出运行示例需要的配置文件。
cp -r /opt/tros/lib/mono2d_body_detection/config/ .

# 配置MIPI摄像头
export CAM_TYPE=mipi

# 启动launch文件
ros2 launch mono2d_body_detection mono2d_body_detection.launch.py
           

At this time, you can view the detection effect through the http://IP:8000, this module detects the human body, human head, face, human hand detection frame, detection box type and target tracking ID and human body key points, etc., I only take the face part, of course, you can also add such as human body detection in the future.

After the above command is run, execute the ros2 topic list on OriginBot, there should be a topic called hobot_mono2d_body_detection, this is what we need, we will subscribe to this topic, and then analyze the data sent in it to determine whether there are faces

Determine if there is a face in the camera

按照TogetheROS.Bot的文档说明，hobot_mono2d_body_detection的消息类型是ai_msgs.msg.PerceptionTargets, 具体如下：

# 感知结果

# 消息头
std_msgs/Header header

# 感知结果的处理帧率
# fps val is invalid if fps is less than 0
int16 fps

# 性能统计信息，比如记录每个模型推理的耗时
Perf[] perfs

# 感知目标集合
Target[] targets

# 消失目标集合
Target[] disappeared_targets
           

The disappeared_targets is the focus of our attention, if the face appears in the disappeared_targets, it means that there was a face before, but now there is none, and at this time it is necessary to send data to the backend for further processing.

In order to determine whether there is a face or not, I wrote a Node, and the code is as follows:

import rclpy
from rclpy.node import Node
from ai_msgs.msg import PerceptionTargets
from cv_bridge import CvBridge
import time

from api_connection import APIConnection

BabyMonitorMapping = {
    # 这里的k-v要根据后端Django中的值来确定
    "face": "看不到脸",
    "body": "不在摄像头范围内",
}


class FaceDetectionListener(Node):
    """ 检测宝宝的脸是不是在摄像头中 """
    def __init__(self):
        super().__init__("face_detection")
        self.bridge = CvBridge()
        self.subscription = self.create_subscription(
            PerceptionTargets, "hobot_mono2d_body_detection", self.listener_callback, 10
        )
        self.conn = APIConnection()
        self.timer = time.time()
        self.counter = 0

    def listener_callback(self, msg):
        targets = msg.targets
        disappeared_targets = msg.disappeared_targets
        targets_list = []
        disappeared_targets_list = []
        if disappeared_targets:
            for item in disappeared_targets:
                disappeared_targets_list.append(item.rois[0].type)
        if targets:
            for item in targets:
                targets_list.append(item.rois[0].type)
        print(f"检测到的对象如下：{targets_list}")
        print(f"消失的对象如下：{disappeared_targets_list}")
        if disappeared_targets_list:
            self.sending_notification(disappeared_targets_list)

    def sending_notification(self, disappeared_targets_list):
        for item in disappeared_targets_list:
            if BabyMonitorMapping.get(item):
                event = BabyMonitorMapping.get(item)
                if self.counter == 0:
                    # 这里baby的ID是模拟的，应该去数据库中查
                    data = {"event": event, "baby": "6b56979a-b2b9-11ee-920d-f12e14f97477"} 
                    self.conn.post_data(item=data, api="api/monitor/face-detection/")
                    self.counter += 1
                    self.timer = time.time()
                else:
                    if time.time() - self.timer >= 60.0:
                        # 60秒不重复发消钉钉消息
                        data = {"event": event, "baby": "6b56979a-b2b9-11ee-920d-f12e14f97477"}
                        self.conn.post_data(item=data, api="api/monitor/face-detection/")
                        self.timer = time.time()
                        self.counter += 1


def main(args=None):
    rclpy.init(args=args)
    try:
        face_detection_listener = FaceDetectionListener()

        rclpy.spin(face_detection_listener)
    except KeyboardInterrupt:
        print("终止运行")
    finally:
        face_detection_listener.destroy_node()
        rclpy.shutdown()


if __name__ == "__main__":
    main()           

The code as a whole is not difficult, but there are a few necessary explanations:

1. BabyMonitorMapping

The purpose of this dictionary is to map the fields in TogetheROS.Bot to the fields in the backend for easy later use

Click on OriginBot Home Assistant Program - Babysitting Assistant - Guyueju to view the full article