Revisiting Diffusion Fine-Tuning for Unsupervised Domain Adaptation

Overview

MUSE reformulates diffusion-based UDA for the multi-target setting: instead of repeating source–target fine-tuning for every target domain, a single source-guided diffusion adaptation process is shared across multiple unlabeled targets.

Problem

Existing diffusion-based UDA pipelines are typically target-specific, requiring a separate source–target fine-tuning process for each target domain.

Shared–Private Adaptation

MUSE separates source-supervised semantics from target-specific appearance using a shared semantic branch and lightweight target-private style cores.

Target-specific Generation

After multi-target fine-tuning, MUSE activates the corresponding target branch to generate labeled target-style bridge samples for downstream UDA.

Goal. Improve the accuracy–efficiency trade-off by sharing reusable semantic adaptation while preserving target-specific visual variation.

MUSE Pipeline

Shared

Semantic branch

Labeled source images use class-conditional prompts and update a shared low-rank semantic branch.

Private

Target style cores

Each target activates only its lightweight private core on top of shared style projection factors.

Two-pass

Decoupled optimization

Pass A updates semantics from source data; Pass B freezes semantics and updates target-conditioned style parameters.

Adaptive

Target sampling

Targets with larger recent denoising losses receive more target-side updates during multi-target adaptation.

Overview of the MUSE pipeline with frozen SDXL components, shared-private adapters, source-conditioned and target-conditioned passes, and adaptive target sampling. — MUSE performs branch-decoupled diffusion adaptation with a source-supervised shared semantic branch and target-private style branches, then activates the corresponding target branch for target-specific bridge generation.

Target-specific Bridge Generation

1. DDIM inversion translation

Instance-preserving bridge: invert a labeled source image and denoise it with a target-specific branch. The generated sample keeps the source label while shifting visual appearance toward the target domain.

2. Class-conditional generation

Diversity bridge: start from Gaussian noise and use class prompts with the target-specific branch to synthesize additional labeled target-style samples.

Bridge generation examples on miniDomainNet using DDIM inversion and Gaussian-noise class-conditional synthesis. — Bridge generation on miniDomainNet: DDIM inversion translates a Real-domain lion to Clipart, Painting, and Sketch; Gaussian-noise class-conditional generation synthesizes broccoli in target styles.

miniDomainNet Results

miniDomainNet uses Clipart (C), Painting (P), Real (R), and Sketch (S), producing twelve ordered transfer tasks. Values below are target-domain top-1 accuracy (%). Pairwise methods adapt/generate separately for source–target pairs, while MUSE uses multi-target generation.

84.37%SSRT + MUSE average

64.02%MCC + MUSE average

65.04%ELS + MUSE average

2.44×SDXL fine-tuning speedup vs. Terra

Method	Diffusion Gen.	C→P	C→R	C→S	P→C	P→R	P→S	R→C	R→P	R→S	S→C	S→P	S→R	Avg.
ERM	—	39.48	53.27	42.93	49.55	68.18	41.99	49.30	55.52	37.35	54.60	45.33	53.08	49.22
DANN	—	45.94	56.12	49.40	50.72	65.61	50.07	55.15	60.55	49.95	58.54	54.64	58.99	54.64
AFN	—	49.23	60.11	51.11	55.60	70.59	51.78	55.84	60.41	47.46	60.69	56.36	62.28	56.79
CDAN	—	47.99	58.50	51.17	56.36	68.71	53.01	61.15	62.85	53.44	60.89	55.90	60.88	57.57
MDD	—	48.53	61.75	52.32	59.74	70.62	55.43	62.18	62.22	54.04	63.07	58.55	64.50	59.41
SDAT	—	50.97	62.42	53.91	60.57	69.97	55.85	64.39	64.83	55.86	64.07	59.43	64.28	60.55
MCC	—	51.95	67.73	52.66	60.96	76.61	54.67	64.15	64.02	50.34	63.64	59.68	69.78	61.35
ELS	—	50.11	61.45	53.02	60.77	70.61	56.04	62.43	64.16	54.89	63.93	59.19	64.47	60.09
SSRT	—	77.30	86.35	75.24	80.79	88.10	74.92	84.29	80.16	78.10	81.75	75.71	86.67	80.78
Diffusion-based baselines: pairwise generation
MCC + Terra	Pairwise	52.64	68.26	51.88	60.45	77.24	55.41	63.85	64.52	50.86	63.21	59.88	70.68	61.57
ELS + Terra	Pairwise	52.68	68.57	54.37	60.23	70.78	56.87	63.57	63.25	54.51	64.22	60.36	64.54	61.16
SSRT + Terra	Pairwise	79.84	89.05	76.35	80.48	88.41	75.56	82.84	78.25	78.40	80.95	77.14	88.41	81.31
MCC + DCDM	Pairwise	55.28	70.21	54.42	63.25	77.05	55.74	65.16	65.01	51.96	65.76	60.83	71.09	62.98
ELS + DCDM	Pairwise	56.14	68.52	55.51	64.96	73.44	58.23	64.67	64.36	57.66	67.23	62.56	69.90	63.60
SSRT + DCDM	Pairwise	78.26	87.42	77.56	81.64	87.15	76.82	81.92	80.35	76.51	82.34	76.92	88.21	81.26
Our method: multi-target generation
MCC + MUSE	Multi-target	58.02	72.82	56.54	66.23	74.56	57.74	63.42	65.87	54.67	66.74	60.98	70.64	64.02
ELS + MUSE	Multi-target	59.18	73.61	57.13	65.81	75.81	58.62	64.31	65.32	58.94	67.59	61.93	72.24	65.04
SSRT + MUSE	Multi-target	80.68	91.16	80.05	85.60	91.63	79.25	85.44	81.79	80.05	84.49	80.21	92.11	84.37

Avg. is over all twelve tasks. SSRT + MUSE obtains the highest value on every reported miniDomainNet transfer task in this table.

Generator fine-tuning efficiency

Dataset	# Domains	Terra (h) ↓	MUSE (h) ↓	Speedup ↑
Office-31	3	96.78	51.74	1.87×
Office-Home	4	196.36	80.79	2.43×
miniDomainNet	4	197.68	81.04	2.44×

Feature-space Visualization

The t-SNE plots compare source-domain samples, target-domain samples, DDIM-inversion adapted source samples, and generated target-domain samples across Office-31, Office-Home, and miniDomainNet.

t-SNE visualizations for Office-31, Office-Home, and miniDomainNet comparing source, target, DDIM-inversion adapted source, and generated target samples. — Representative feature-space visualizations on Office-31, Office-Home, and miniDomainNet.

Citation

@inproceedings{qi2026muse,
  title={Revisiting Diffusion Fine-Tuning for Unsupervised Domain Adaptation},
  author={Qi, Xuan and Wei, Yi and Berardini, Daniele and Pastore, Vito Paolo and Murino, Vittorio},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026}
}