Comment by mschuster91

What OP says

> Turned out when slowed down SPI bus, it started working.

reeks of either RF interference issues or power rail issues. Power is not the strongest domain of anything RPi so that's where I'd look first.