[chore] log quant config to the user_agent

[sayakpaul]

What does this PR do?

To get finegrained understanding of which quant configs are most used, allow quant configs to also go in user_agent.

Example value:

user_agent['quant_config']='{"modules_to_not_convert": null, "quant_method": "torchao", "quant_type": {"default": {"_data": {"granularity": {"_data": {"dim": -1}, "_type": "PerRow", "_version": 1}, "group_size": null, "set_inductor_config": true}, "_type": "Int8WeightOnlyConfig", "_version": 2}}}'

For:

from diffusers import AutoModel, TorchAoConfig
import torch 
from torchao.quantization import Int8WeightOnlyConfig

model = AutoModel.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    subfolder="transformer",
    torch_dtype=torch.bfloat16,
    quantization_config=TorchAoConfig(Int8WeightOnlyConfig(group_size=None, version=2))
)

Additionally, also pass the model class to user_agent to better understand which quantization classes are usually used for a particular class of models.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[SunMarc]

Thanks a lot !

[Disty0]

This change broke loading of pre-quants.

Changing this line:

user_agent["quant_config"] = json.dumps(quantization_config.to_dict(), sort_keys=True)

To

user_agent["quant_config"] = json.dumps(hf_quantizer.quantization_config.to_dict(), sort_keys=True)

The line above works fine.
Aka using hf_quantizer.quantization_config instead of quantization_config works fine.

Simple reproduction code:

import torch
import diffusers

dit = diffusers.Ideogram4Transformer2DModel.from_pretrained("ideogram-ai/ideogram-4-nf4-diffusers", subfolder="transformer", device_map="cpu", torch_dtype=torch.bfloat16)
print(dit)

Error trace:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 4
      1 import torch
      2 import diffusers
      3 
----> 4 dit = diffusers.Ideogram4Transformer2DModel.from_pretrained("ideogram-ai/ideogram-4-nf4-diffusers", subfolder="transformer", device_map="cpu", torch_dtype=torch.bfloat16)
      5 print(dit)

File ~/AI/Apps/diffusion-trainer/venv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py:88, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
     84         validate_repo_id(arg_value)
     86 kwargs = smoothly_deprecate_legacy_arguments(fn_name=fn.__name__, kwargs=kwargs)
---> 88 return fn(*args, **kwargs)

File ~/AI/Apps/diffusion-trainer/venv/lib/python3.12/site-packages/diffusers/models/modeling_utils.py:1152, in ModelMixin.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
   1150 # In order to ensure popular quantization methods are supported. Can be disabled with `disable_telemetry`
   1151 user_agent["quant"] = hf_quantizer.quantization_config.quant_method.value
-> 1152 user_agent["quant_config"] = json.dumps(quantization_config.to_dict(), sort_keys=True)
   1154 # Force-set to `True` for more mem efficiency
   1155 if low_cpu_mem_usage is None:

AttributeError: 'NoneType' object has no attribute 'to_dict'

[sayakpaul]

#13906