Can Unity be used to simulate & develop robotic navigation systems

I'm aware this was posted and answered some time ago but I'm curious for further discussion because this is relevant to me now.

I think that if your objectives are simulating navigation algorithms and creating engaging demonstrations of your work then Unity is a viable tool, but if your objectives are simulating the more physical aspects (the kinematic model of your robot, detailed sensor model), or integrating with real-world sensors etc then Unity would likely fall short.

The other day I found this 2011 paper from a conference in Mexico. They built a mobile robot simulation in Unity and offer up some informative points. http://www.uv.mx/rossum/CD/papers/005.pdf

My masters project is very similar - a mobile robot simulation which intends to enable the user to create their own mazes, customize the sensor configuration on robots, and provides an interface for writing your own navigation algorithms etc. It's in development and you can try it out here https://github.com/explosivose/botnavsim/releases/tag/v0.3.0-alpha

Thanks for your post. I will look into the document and app which you supplied. It will be interesting...

Hi,

I've worked on AAA game, and robotics. I have used Unity for both.

Unity has a lot of functionality, and the trick is not to get carried away and start designing an Agent/NPC that attempts to navigate the world using the Nav Mesh system. If you do that, when you migrate the code into your real world robot, the robot would need the game engine to move around ! I suppose you could actually take the maths used in your bespoke c# scripts and grab a 3d library instead - but you get the jist - you are not writing the robot code to use a game engine. See answer above from Fatte - I think they thought you meant what I've just described.

What you will be doing is using Unity to simulate a real world, so you can train and improve your robot navigation code.

The real benefit of using unity, and I think this is what you are really asking, is if you can train and develop algorithms for navigation using Unity as a world simulator. So in this case, yes you can use Unity. Let's say you want to move a robot around your house. There are 2 obstacle types (like most models) - static and dynamic. The static are the doorways, stairs, kitchen units etc. The dynamics are objects left around that dont move on their own ie the hoover (unless you have a robot hoover !!), and the objects that can move on their own ie your robots girlfriend, the dog etc. (note giant yoga balls can be considered this type with 0 velocity !)

Measure the rooms of the house in terms of floor space and put them into Unity. Connect the rooms etc via doorways etc. Create some assets for the objects that are dynamic but dont move too much ie sofa, chairs, tables etc - these are unlikely to be moved from the rooms they are mapped in, but can still be moved. Then make assets for objects that can be anywhere ie are moved around a lot but dont move on their own ie single chairs, and then finally make assets that can move themselves ie the dog, house guest.

Now create the level asset of your simulation in unity - randomize the dynamic objects., and you have a world sim. Add an update for the movement to the dynamic objects that can move themselves simply rotations, translations (no scaling !!) - you can use the navmesh to help with the movement simulation of these objects - remember none of this simulation code goes on the robot. And now you have a dynamic world simulation of your house

Now the good bit - hopefully it's an easy robot - eg a tracked robot not a humanoid - measure your robot, be careful with this - it needs a front direction (call it North), and dont measure just the foot print - does anything lean out/bend over etc. Model this as your character, and make sure you model the movement using your north direction (unless it can move in all directions equally well etc).

If you have moving parts on the robot that affect the size/footprint - this needs to be mapped or you can constrain the robot to worse case scenario - this can be very limiting (take volume of largest possible size)

Once you have the robot mapped, make sure you can randomize its location only on suitable surfaces ie not on any dynamic object, on floors where it can operate (ie it it can move on fitted rugs etc).

Now this is the really fun bit, the robot needs to have a set of atomics, things it can do in the real world ie move 1 cm forward, rotate etc You need to spend some time mapping these movements, but start simple ie forward, backward, left track move only, right track move only. Program these movements one in each function for example. See genetic programming for algorithms using atomics -this is very interesting - making new "genes" with fitness, random etc.

Use an algorithm so that each update on the scene allows the algorithm to do something to the robot s position.

If its an alg that requires training - then set a destination point for the robot, and then let the robot try each movement until it reaches its destination - woooaa - going to take a while ! Use heuristics - so choose a rule that progresses the robot to its goal, -this requires a way to measure progress etc.

For example add the permissible movement functions to a list, chose an item from the list, measure the distance to the destination, if the robot is closer then it's a winner move, keep that move (or try them all - chose the one that brings closest), but with this approach you will hit valleys and troughs, false peaks etc.

Once its trained well enough and you are happy with the number of moves/time (depends how you measure success), the code is ready to move to the robot. The only thing that changes is each update function (scene frame), needs a main loop to call it - this will be easy as all firmware runs like this ie no end. Then change the atomics functions to instead of moving your robot in the sim, actual control the HW that does those moves.

So all this does mean you can use Unity as a simulator, if you have more questions feel free to ask.

I used Unity for my robotic navigation system training. My robot has 6 raspberry Pis - a Pi4 vision module that uses opencv for face/object recog, a Pi4 Speech module Pi that uses pocket sphinx, speech simulator is HW, a Pi3 brain module Pi that controls the other modules via 0 message Q, creates and tracks jobs etc, a Pi3 mobility module - uses various navigation and tracking systems, a Pi4 Mind module - contains the personality and knowledge systems, and decision making - uses semantic networks, ANNs, and a personality engine, and finally a Pi4 Fun Module - plays games, movies, music, surfs web etc. all via a mini projector.