The longer periods between finding new blocks can be caused by the pool having temporary bad luck. For example, if the pool's hash rate and the current network difficulty predict that the pool will find 1 block every 4 hours, this is only the average. In the short-term, the pool may find a block just 1 hour after the previous one, and then not find the next block for 7 hours after that. This would result in the pool luck being 100% luck during that 2-block period, despite having one round go much longer than expected.
Pool found block faster than average → good luck (>100%).
Needed more time → tough luck (<100%).
The important thing to understand is that past luck doesn't impact future mining whatsoever. In the long run, the luck is always 100% — smoothing out any rough edges. Therefore, it doesn't make sense to try to game the luck by connecting after a period of high luck or disconnecting after a period of low luck.