LUMI-GreenNLP webinar on large language model (LLM) training

This technical webinar is an online-only event organized jointly with CSC and GreenNLP, in connection with a HPLT-project hackathon. This is a lightweight event discussing possible collaboration on LUMI LLM training.

GreenNLP project is building resources for sustainable NLP. In this spirit we at CSC and GreenNLP would be inviting LUMI customers to discuss their experiences on training LLMs to avoid duplication of (optimization) work. We have noticed that many of LUMI users work on very similar problems and face similar challenges. We would like to provide this webinar as a platform for our users to share what they are working on, and share tips and tricks related to getting your LLM training running efficiently on LUMI. The objective would be to discuss technical matters with other like-minded people.

The event would begin with short (5-10 minutes) presentations of various LUMI LLM projects and descriptions of technical implementations and challenges related to training and scale. We are still open to presentations and would gladly hear more about your project. We could for example be interested in:
– Short description about what you are doing (model, data, scale)
– Software stack (For example PyTorch, Hugging Face transformers, Megatron-LM, DeepSpeed, FSDP, DDP, Singularity containers)
– Scaling properties when increasing node count
– In terms of optimization work, what did you need to tune to achieve good throughput (Data, Communication, Model architecture, Software)
– Possible throughput or FLOPS efficiency (For example with stack x and model y and n nodes we achieved X% of theoretical max per GCD, amounting to Y FLOPS)
– Were there some problems you could not solve or are still working on

After an opportunity to present the projects, there would be room to have discussion related to what kind of collaboration and coordination there could be in this space to avoid wasting human and computational resources on duplicate work.

Zoom: https://cscfi.zoom.us/j/68034158484?pwd=alhhU2tBRmU1Z0JXU1RoOTE0anp6QT09

Preliminary schedule (CEST):
Introductions 17-17:15
Project presentations 17:15-18:15
Discussion 18:15-19:00