Make tokenizer_path to be non-mandatory

[A9isha]

Description

Simplify the parameters for running on MaxText by removing tokenizer_path as a required argument.

FIXES: b/490520651

Notice 1: Once all tests pass, the "pull ready" label will automatically be assigned.
This label is used for administrative purposes. Please do not add it manually.

Notice 2: For external contributions, our settings currently require an approval from a MaxText maintainer to trigger CI tests.

Tests

Ran locally on a v5p-8 the following commands:

# With tokenizer path
python3 -m src.maxtext.trainers.post_train.rl.train_rl src/maxtext/configs/post_train/rl.yml   model_name=llama3.1-8b-Instruct   tokenizer_path=meta-llama/Llama-3.1-8B-Instruct   load_parameters_path=/path/to/checkpoint   run_name=maz-8b-$RANDOM   base_output_directory=/path/to/gcs/bucket   hf_access_token=<HF_TOKEN> dataset_name=gsm8k steps=4 rollout_tensor_parallelism=-1 rollout_data_parallelism=1 rollout_expert_parallelism=1

# Without tokenizer path
python3 -m src.maxtext.trainers.post_train.rl.train_rl src/maxtext/configs/post_train/rl.yml   model_name=llama3.1-8b-Instruct   tokenizer_path=meta-llama/Llama-3.1-8B-Instruct   load_parameters_path=/path/to/checkpoint   run_name=maz-8b-$RANDOM   base_output_directory=/path/to/gcs/bucket   hf_access_token=<HF_TOKEN> dataset_name=gsm8k steps=4 rollout_tensor_parallelism=-1 rollout_data_parallelism=1 rollout_expert_parallelism=1

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[SurbhiJainUSC]

Can we do the similar change for SFT?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[A9isha]

Can we do the similar change for SFT?

Yes done now

[SurbhiJainUSC]

What if MODEL_NAME is not present in HF_IDS?

[A9isha]

Please see my response here

[SurbhiJainUSC]

Instead of explicitly adding Instruct model name to this list, can we somehow derive it from the base model? In future, we might need Instruct model for gemma and then we would have to update this list.

[SurbhiJainUSC]

You have added qwen3-omni-30b-a3b-Instruct to HF_IDS, but it is not present in ModelName list.

[A9isha]

Yes, I think the idea for this PR is to allow on the fly tokenizer_path calculation for the popular models used in the tutorials to make it easier for first time users. We still have the option of passing tokenizer_path. So, for the models which are not supported for this on the fly calculations we will default to using the previous maxtext/assets/tokenizers/tokenizer.llama2 default and if the user intended to use something else they would pass tokenizer_path

Great point about qwen3-omni-30b-a3b-Instruct, I will remove that change.