Development and Reliability Assessment of an Artificial Intelligence-Driven Urticaria Support (AIDUS) Chatbot

Background: Chronic urticaria (CU) severely impairs patients’ quality of life. Correctly diagnosing and treating CU can take years, so patients seek answers from the Internet to manage the condition. Objective: We aimed to build a chatbot, Artificial Intelligence-Driven Urticaria Support (AIDUS) for patients with CU and treating physicians and to evaluate its reliability in providing high-quality CU-specific information compared with Chat Generative Pre-Trained Transformer (ChatGPT)-3.5 and ChatGPT-4o. Methods: AIDUS was developed by an expert committee of urticaria and artificial intelligence specialists using JavaScript and OpenAI's (https://www.clay.com/dossier/openai-headquarters-office-locations) ChatGPT large language model. PubMed was systematically reviewed to ensure AIDUS contained high-quality information. The chatbot was populated exclusively with selected peer-reviewed CU publications authored by the Charité University, Berlin research group, published after 2014. A total of 254 publications were integrated using ChatGPT-3.5 as the underlying algorithm. We developed A set of 100 validated questions based on current CU knowledge to evaluate the performance of AIDUS. The program was run on the same questions several times and compared for consistency. We tested performance with different chunk- and overlap-size settings to optimize AIDUS's efficiency and accuracy. Results: AIDUS outperformed general ChatGPT models in terms of accuracy, consistency, and stability in answering CU-specific questions. AIDUS demonstrated higher average accuracy (94.6%) across multiple test runs compared with ChatGPT-3.5 (42.6%) and ChatGPT-4o (85.7%). Conclusions: AIDUS provides reliable, high-quality information about CU, addressing patients’ and physicians’ needs for accurate, relevant answers based on peer-reviewed medical literature. AIDUS remains a means of assistance and does not replace consultation with a physician.