It can be numerous things, like misconfigured port forwarding on your router and/or torrent client on your side or on there side. This doesn't mean that it blocks all handshake but it does limit how well your/there client will exchange data. Other possibility is that Client is set to a limit of a active connection (total or by torrent). It can keep some connection so that it will switch to another one when it will be done with other connections (because tracker is sending list of peers according to some timer, it doesn't ask for new peers every time it has finished with one, probably to not overload tracker server with repeated queries).
Those are the first reasons that come to my mind but probably are others possibilities.

Edit: I don't personally think these are people/agencies logging data, agencies that try to enforce infringment will download data from you so they have proof that you are sharing.

something watching, logging connections to everyone connected to that torrent

Might be, FWIW there are quite a few ways to torrent in a rather private way, namely require encrypted connection, have a blocklist, require to be behind a VPN, etc ... but in the end you still share data with strangers, that's the core premise. The whole point is to facilitate the sharing of data reliably but then who joins the pool is outside of the protocol itself.