Problem with profiler spikes caused by Semaphore.WaitForSignal

I am working on a 2D pc game and just recently began networking it using photon PUN. By the time I got everything working, I was realizing there were huge lag spikes caused by Gfx.WaitForPresentOnGfxThread and Semaphore.WaitForSignal that made the game unplayable.

In a two player game, these spikes only happen when the player on the other instance of the game passes tree sprites or a castle area. I found that deactivating the spriterenderer component on the leaves of the trees fixes the problem there, and deactivating the 2D light components in the castle area fixes the problem there.

However, I need both the leaf sprites and 2D lighting in my game, I don't know how to workaround it. Additionally, there are plenty of other spriterenderer components in the scene with the exact same materials and none of them seem to be causing issues. Same with 2D lighting in other places. It could be due to the number of sprites/2D lights in such a small area but I'm not sure.

Any help would be great!