How do you set an order for 2D Colliders that overlap?

After a bit of testing, I've come to a discovery, which might cast some light on this "problem".

TL;DR: Go to where it says "For 2D in Unity" in bold text, if you don't want to learn, but just get the facts.

None of the physics layers, sprite sorting layers or sorting orders seem to have ANY impact on which object a ray hits first. The sprite-stuff only makes a difference to which order the sprites are rendered in, and the physics layers just let you define which objects can collide with each other. It kind of has to be this way, because otherwise these things would be tied together! An objects' colliders could be on a parent or child object, and there could even be one collider acting on one physics layer only, and another one acting on another layer only. It makes sense that they don't factor into which object is in front of the other.

Example: Let's say we have two objects; "Front" and "Back". They both have a BoxCollider2D and a SpriteRenderer.

I set "Front" to Sprite Sorting Layer "Foreground", and sprite "Back" to Sprite Sorting Layer "Background". Now "Front" is always rendered on top of "Back", but they both still have the same Z-position, so the ray hits either of them seemingly at random.

Now, if I set the Z-axis of "Front" to be 10 (deeper into the screen), which will make it move BEHIND sprite "Back" in the Scene-view, it will still be rendered AFTER i.e. ON TOP of sprite "Back", because of the order of the Sprite Sorting Layers, but a screen-ray (see GetRayIntersection() below) will always hit sprite "Back" first, because its Z-position is closer to the camera.

It's sort of the same with the physics layers. If you used them to control which object is in front of another, then you'd be locked into some irritating situations.

For 2D in Unity:

Physics sorting layers = help you control which types of colliders can hit which types of colliders, and also which of them should ignore raycasts i.e. which of them should be ignored when we do our raycast-clicks (more on them further down).

Sprite sorting layers and sorting orders = control the sprite rendering queue, so the sprites are rendered in the correct order, and makes it possible for you to help Unity batch properly.

Z-position = lets you control which gameobjects are "physically" in front of the others, without affecting sprite rendering order or physics interactions, but affects which 2D collider is considered closest to the screen when casting a ray into the screen as opposed to across the screen.

2D Raycasting = casts a ray through the (x,y) world space coordinate system i.e. across the screen, using given start- and end-coordinates, and returns information about the first collider it hits, if any. If two or more objects have a collider at the exact same coordinates, it will be a luck of the draw. Makes total sense.

GetRayIntersection() = just like raycasting, but can cast a ray "into" the screen of a 2D game, returning information about the first collider it hits.

Solution: Don't put several objects, that are all supposed to be clickable, on top of each other with the same Z-position. And to find the frontmost collider, DO NOT use Physics2D.Raycast. That is for casting rays in (x,y) space, you know, across the screen.

For hitting the frontmost collider at a screen point, use something like this:

 Vector3 worldPoint = Camera.main.ScreenToWorldPoint(Input.mousePosition);
 worldPoint.z = Camera.main.transform.position.z;
 Ray ray = new Ray(worldPoint, new Vector3(0, 0, 1));
 RaycastHit2D hitInfo = Physics2D.GetRayIntersection(ray);

Code stolen from this stackoverflow post.

It can definitely be shortened to alleviate some instantiation. The "new Vector3(0, 0, 1)" makes the ray shoot directly into the screen, from the clicked position (worldPoint), after worldPoint has had its Z-position changed to that of the camera, instead of the default 0. So it sort of "shoots from the screen" and "into the game", hitting the first collider it meets. You can go ahead and filter the layers it can hit, by using some of the overloaded versions of Physics2D.GetRayIntersection.

2021 Update:

For getting the frontmost collider where you've clicked, you can use Physics2D.OverlapPoint() instead:

 Collider2D colliderHit = Physics2D.OverlapPoint(Camera.main.ScreenToWorldPoint(Input.mousePosition));

You can check if colliderHit is null, and if it isn't, then it is the frontmost thing it hit. You can adjust the Z-range it should search in and which layers to search in, by using the other parameters for the Physics2D.OverlapPoint() function.

For finding all the colliders under the point where you clicked, you can use Physics2D.OverlapPointAll() or Physics2D.OverlapPointNonAlloc() the latter of which is preferred for memory reasons. It requires you to supply an array of sufficient size to put the results in, which you then clear after you've used it. Since people click a lot, you don't want to garbage collect results for every click, thus we use the NonAlloc version and eat the cost of keeping our little array around; it should not need to be very large, anyway. You can see an example in kboudai's answer below, where results[0] is the frontmost thing hit and any subsequent hits in the array are in Z-order, positive to negative. The actual declaration of the "results" array is missing from kboudai's answer, but it'd just be something like this in your class:

 private Collider2D[] results = new Collider2D[5];

5 is the size of the array, but also the "depth" i.e. the maximum number of collisions we want to detect. OverlapPointNonAlloc simply writes the collisions into the array in order, and if the array is not large enough, it just skips the rest, so we only get the first 5. Then you MUST clear that array (set all its places to null) after you've used it. Otherwise, if you detected 5 collisions last time, but only 3 this time, then you'll have an array with the 3 new ones and the last 2 of the 5 from the last click. It's annoying, but it saves A LOT of garbage collection!

I made it work by putting them on the same layer then making their z-position closer to the camera.

No need for 3D.

For example:

 GameObject A;
 GameObject B;
 
 int Layer = 2;
 
 void Start(){
     A.layer = Layer;
     A.transform.position.z = -1;
     B.layer = Layer;
     B.transform.position.z = 0;
 }
 
 void onClick(){
     Vector2 p = Camera.main.ScreenToWorldPoint(pos);
     int hitNum = Physics2D.OverlapPointNonAlloc(p, results, 1<<Layer);
     if(hitNum > 0){
         // Should be A, assuming the camera's z is -10
         Collider2D hit = results[0];
     }
 }

I've just found out that in such situations you can use 3D colliders, which will support correct z-ordering. You may have problems using 2D and 3D colliders at once, in such case you'll need to move entirely to 3D colliders.

There is no straight-forward answer to solve this without a complicated work-around.

Personally i would just switch my 2DColliders to 3D ones. The change might be a bit of a hassle but IMO much easier than adding these complicated raycasts to every click event in your game.

You can also my obviously see how a collider interacts within it's own space. You can still make everything else look 2D with no complication at all.

Try the example here

http://docs.unity3d.com/Documentation/Components/Layers.html Read where it says "Casting Rays Selectively".