The use of drone swarms for search and rescue is being increasingly explored. Because of the time-sensitive and life-critical nature of these missions they are very mentally demanding of the search and rescue operators. It is therefore apparent that incorporation of artificial intelligence at the right levels will be a deciding factor in the effectiveness of these systems. We set out to investigate how explanations about object detections made by the drone swarm can be used to improve the performance of operators engaged in search and rescue missions. We conducted an online study with 8 participants involved with the Danish Emergency Services, in which they were tasked with responding to AI-generated alerts under varying workload while being provided distinct levels of explanation detail. A combination of performance measures and subjective measures showed that under high cognitive load, participants become significantly faster at responding to alerts, and that this increase in speed is not necessarily at the cost of accuracy. Our findings also provide insight into the challenge of mitigating a drop-off in user expertise due to over-trust in the AI. We discuss the findings and provide implications for the design of alerts for search and rescue.