I met the same problem and solved it by reinitialization. U can deepcopy an original model before u wrap it with deepspeed.initialize.Load your checkpoint to the original model, and run ...