Evaluation of AI Model Scientific Reasoning Skills is a hands-on tutorial designed to equip researchers with practical skills and conceptual grounding in the application of LLMs to scientific challenges.

Large Language Models (LLMs) are becoming capable of solving complex problems while presenting the opportunity to leverage them for scientific applications. However, even the most sophisticated models can struggle with simple reasoning tasks and make mistakes.

This tutorial focuses on best practices for evaluating LLMs for science applications. It guides participants through methods and techniques for testing LLMs at basic and intermediate levels. It starts with the fundamentals of LLM design, development, application, and evaluation while focusing on scientific application. Participants will also learn various complementary methods to rigorously evaluate LLM responses in benchmarks and end-to-end scenario settings. The tutorial features a hands-on session where participants use LLMs to solve provided problems.

Participants will learn the principles and approaches for the use of LLMs as scientific assistants and how these can be evaluated with respect to scientific knowledge and reasoning skills, such as:

• Use cases of LLMs for scientific applications
• Importance of prompting and performance
• Basic of LLM evaluation,
• Evaluation of LLMs for science and engineering
• Advanced evaluation techniques of LLMs for Science and Engineering
• Hands-on