upgrade liger to 0.4.0

[winglian]

Description

Motivation and Context

How has this been tested?

Screenshots (if appropriate)

Types of changes

Social Handles (Optional)

[NanoCode012]

Do we need to wait for their latest release or point to this commit for GA fix? linkedin/Liger-Kernel#333

[bursteratom]

@NanoCode012 @winglian tried to run this particular branch just now but ran into this error

File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/triton/compiler/code_generator.py", line 1066, in visit_Attribute                                  [114/1839]
             return getattr(lhs, node.attr)                                                                                                                                   [113/1839]
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                             ]
[rank0]: AttributeError: 'tensor' object has no attribute 'cast'

which happens during

File "/workspace/axolotl/src/axolotl/core/trainer_builder.py", line 678, in compute_loss                                                                                     
[rank0]:     return super().compute_loss(model, inputs, return_outputs=return_outputs)                                                                                                  
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I'm using the default axolotl template on runpod and made sure to install the dependencies associated with this branch

And my yaml is as follows:

base_model: NousResearch/Meta-Llama-3.1-8B

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true

strict: false

datasets:
    - path: tatsu-lab/alpaca
      type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.02
output_dir: ./outputs/out

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 1
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
evals_per_epoch: 2
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
  - full_shard
  - auto_wrap
fsdp_config:
  fsdp_limit_all_gathers: true
  fsdp_sync_module_states: true
  fsdp_offload_params: true
  fsdp_use_orig_params: false
  fsdp_cpu_ram_efficient_loading: true
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_backward_prefetch: BACKWARD_PRE
special_tokens:
  pad_token: <|finetune_right_pad_id|>
  eos_token: <|eot_id|>