In this session, we will explore the transformative potential of Large Language Models (LLMs) in medicine and healthcare, emphasizing their diverse applications and benefits. We will delve into the risks posed by adversarial attacks on LLMs, which can lead to unsafe and unreliable outputs. Highlighting a research case study, which will showcase effective guardrailing strategies designed to prevent such unsafe responses, particularly in patient-facing applications, ensuring the safety and reliability of AI-driven medical chatbots.