How to zoom in correctly to a certain part of an object with mouse scroll?

Hi, currently facing a challenge trying to come up with a good way to do the following: I have a character inside a Customization screen, the character should rotate, pan and zoom, first two are done but my issue come with the zoom, Zoom in to be precise Zoom In should go to whatever part of the body the mouse is located at (if the mouse is on top of a hand, it should zoom and change the camera position to match the hand).

Restrictions i have is that the camera should be perspective and the zoom is based on movement (z axis) not Fov, which complicate the task a litte bit.

Retrieving mouse position and raycasting to world position sounded like a good approach but to my surprise it does not work as intended.

When Zoom in occurs i create a ray and retrieve the world location of mouse position

 private Vector3 GetScrollDirection()
 {
        Ray ray = cam.ScreenPointToRay(Input.mousePosition);
        RaycastHit hit;
        Physics.Raycast(ray,out hit);
        return hit.point;
 }

Then with result i lerp positions of camera with hit point maintainig the z axis as zoom permits

What i expected: x and y values are constant whenever the hit point is received from mouse position Result: is a weird snappy behaviour that constanly changes the x and y axis whenever i zoom in

Thanks and any help wll be welcomed