Is there a way to sense unknown objects?

[gongxinhengParticipant]

I am using Rain AI to simulator our real robot. So, I can’t depend on the AI sensor to detect an object with name. Instead, I use openCV library to analyze the images the camera captures (eg. by analyzing the objects’ shape and color to determine what they are) and make decision. How do I implement this with Rain AI? Any ideas?

[SigilKeymaster]

Well a “simple” solution would be to do your image analysis and then create an Entity and Visual Aspect that properly describe the object in space (in some 3D space that you have anchored to your real world through your robot I imagine). At that point the normal RAIN sensors would be able to work.

An alternative would be to create a custom sensor that actually calls out to do the image analysis, and then creates the Entity/Visual Aspects on the fly and keeps track of them. That might a more interesting approach.

I can help you start the initial code for either of those depending on your needs.

• This reply was modified 11 months, 2 weeks ago by  Sigil. Reason: reworded

[gongxinhengParticipant]

Thank, Sigil. It sounds great. I would like to go with the custom sensor since it seems more flexible.

[SigilKeymaster]

Saw your response, just haven’t gotten back around to this just yet. Will post some code a bit later tonight.

[SigilKeymaster]

Ok, so here’s what I threw together. It’s more of a template, but it does work as it is.

First is the Image Sensor. I assumed, since you are relying on hardware, that you wouldn’t need things like a visual distance on the sensor in Unity, however you could inherit from VisualSensor (instead of RAINSensor) if you want some of those things.

using RAIN.Entities;
using RAIN.Entities.Aspects;
using RAIN.Perception.Sensors;
using RAIN.Serialization;
using System.Collections.Generic;
using UnityEngine;
[RAINSerializableClass]
public class ImageSensor : RAINSensor
{
    private List<RAINAspect> _matches = new List<RAINAspect>();
    private EntityRig _imageEntity = null;
    public override IList<RAINAspect> Matches
    {
        get { return _matches.AsReadOnly(); }
    }
    public override void Sense(string aAspectName, MatchType aMatchType)
    {
        // First thing we do when we sense is check to see if there are any new objects
        // to add to our image aspects, this would have to go out to your image code
        // which hopefully can add some unique factor to what is detected so we don't
        // get the same thing over and over
        AddNewAspects();
        // Because we added our aspects to the sensor manager (through the AddAspect call)
        // any sensor can access it, and we can also access any other sensors
        // data as well (if we had multiple robots in the same world they could feed
        // each other their sensor information this way)
        IList<RAINAspect> tDetectedAspects = SensorManager.Instance.GetAspects(ImageAspect.ImageAspectType);
        // It's getting late, so I'm going to assume all of the aspects with the 
        // right aspect name I can see (which is our unique description in my case) 
        _matches.Clear();
        for (int i = 0; i < tDetectedAspects.Count; i++)
        {
            if (tDetectedAspects[i].AspectName == aAspectName)
                _matches.Add(tDetectedAspects[i]);
        }
    }
    public override RAINAspect IsDetected(object aGameObject, string aAspectName)
    {
        // This is called when we are checking to see if a gameobject is within range
        // In our case we accept any aspect that matches the name we're looking for
        // and not checking for distance or any of the like.
        // Normally this handles GameObject, Entity, and RAINAspect
        if (aGameObject is GameObject)
        {
            EntityRig tEntityRig = ((GameObject)aGameObject).GetComponentInChildren<EntityRig>();
            if (tEntityRig != null)
            {
                foreach (RAINAspect tAspect in tEntityRig.Entity.Aspects)
                {
                    if (tAspect is ImageAspect && (aAspectName == "" || tAspect.AspectName == aAspectName))
                        return tAspect;
                }
            }
        }
        else if (aGameObject is Entity)
        {
            foreach (RAINAspect tAspect in ((Entity)aGameObject).Aspects)
            {
                if (tAspect is ImageAspect && (aAspectName == "" || tAspect.AspectName == aAspectName))
                    return tAspect;
            }
        }
        else if (aGameObject is RAINAspect)
        {
            RAINAspect tAspect = (RAINAspect)aGameObject;
            if (tAspect is ImageAspect && (aAspectName == "" || tAspect.AspectName == aAspectName))
                return tAspect;
        }
        return null;
    }
    public override float Score(RAIN.Entities.Aspects.RAINAspect aAspect)
    {
        // This is only used internally on the PhysicalSensor, likely we will remove it from the base class
        // but I'm just going to default it to 0 for now.
        return 0;
    }
    private void AddNewAspects()
    {
        // I'm just going to fake it here since I don't have an actual camera/image analysis application
        List<ImageData> tImages = new List<ImageData>();
        if (Time.time > 2)
            tImages.Add(new ImageData() { ID = "Dog", PositionInSpace = new Vector3(2, 0.5f, 2) });
        if (Time.time > 4)
            tImages.Add(new ImageData() { ID = "Cat", PositionInSpace = new Vector3(2, 0.5f, -2) });
        if (Time.time > 6)
            tImages.Add(new ImageData() { ID = "Human", PositionInSpace = new Vector3(-2, 0.5f, 2) });
        // We only need one Entity for these guys, and it can hold all of our aspects
        if (_imageEntity == null)
        {
            _imageEntity = EntityRig.AddRig(new GameObject() { name = "ImageEntity" });
            _imageEntity.VisualMode = RAIN.Core.VisualModeEnum.AlwaysOn;
            _imageEntity.Serialize();
        }
        // Make sure our aspects aren't already on the entity
        IList<RAINAspect> tAspects = _imageEntity.Entity.Aspects;
        for (int i = 0; i < tImages.Count; i++)
        {
            // Add the aspect if we don't have it already
            for (int j = 0; j <= tAspects.Count; j++)
            {
                // Reached the end, so add it as an aspect
                if (j == tAspects.Count)
                {
                    _imageEntity.Entity.AddAspect(new ImageAspect()
                    {
                        AspectName = "ImageAspect",
                        PositionOffset = tImages[i].PositionInSpace,
                        CameraImage = tImages[i],
                    });
                    break;
                }
                // Just in case we have other aspects
                ImageAspect tImageAspect = tAspects[j] as ImageAspect;
                if (tImageAspect == null)
                    continue;
                // Already have it, so break
                if (tImages[i].ID == tImageAspect.CameraImage.ID)
                    break;
            }
        }
    }
}

Next up is my custom aspect that the above sensor looks for, it’s pretty empty right now, I put in some fake data for giggles.

using RAIN.Entities.Aspects;
using RAIN.Serialization;
[RAINSerializableClass]
public class ImageAspect : RAINAspect
{
    public const string ImageAspectType = "imageaspect";
    // We could add any data we like (for the most part), I just added this for example
    [RAINSerializableField]
    private ImageData _cameraImage = null;
    public override string AspectType
    {
        get { return ImageAspectType; }
    }
    public ImageData CameraImage
    {
        get { return _cameraImage; }
        set { _cameraImage = value; }
    }
}

Last is just a helper class representing the image data that would have come from your external code:

using RAIN.Serialization;
using UnityEngine;
// Just a fake class for the example
[RAINSerializableClass]
public class ImageData
{
    public string ID
    {
        get { return _id; }
        set { _id = value; }
    }
    public Texture2D CameraImage
    {
        get { return _cameraImage; }
        set { _cameraImage = value; }
    }
    public Vector3 PositionInSpace
    {
        get { return _positionInSpace; }
        set { _positionInSpace = value; }
    }
    [RAINSerializableField]
    private string _id = "";
    [RAINSerializableField]
    private Texture2D _cameraImage = null;
    [RAINSerializableField]
    private Vector3 _positionInSpace = Vector3.zero;
}

So all this worked for me, I was able to drop it into a scene and create a simple behavior tree to sense with it and I would see the aspects show up over time (due to my little time hack in the AddNewAspects call).

My thoughts are that you would replace the ImageData class with something useful that you get from your hardware/library. You would also replace the code at the top of AddNewAspects in ImageSensor to actually go get that data from your hardware/library.

Just a note, the Sense and IsDetected calls in the sensor need to be fast, so you can’t wait too long for any hardware to come back, maybe just poll or thread something off and check back later.

Let me know if you have questions, if I’m way off base, or if you need more information.

[Author]

Posts