Cosine merge

Cosine Similarity merge

https://github.com/wkpark/sd-webui-model-mixer/pull/90

Cosine merge by recoilme https://github.com/recoilme/losslessmix/blob/master/weightedsim.py#L41-L56

original code.

            simab = torch.nn.functional.cosine_similarity(theta_a, theta_b, dim=0)
            ...
            k = (simab - sims.min())/(sims.max() - sims.min())
            k = k - alpha # max k is 1. so it is about (1 - alpha)
            k = k.clip(min=0.0, max=1.0) # k could be minus, clip needed
            theta_0[key] = weighted_sum(theta_0[key], theta_1[key], k) # k is a tensor

Cosine merge and Simple Cosine merge (used by Model-Mixer)

           ...
           simab = torch.nn.functional.cosine_similarity(theta_a, theta_b, dim=0).abs() # use abs()
           ...
            k = (simab - sims.min())/(sims.max() - sims.min()) # k always k > 0 and k < 1
            k = k.mean() * alpha if "Simple" in calcmode else k * alpha # k.mean() * alpha for Simple Cosine.
            theta_0[key] = weighted_sum(theta_0[key], theta_1[key], k) # k is a float if Simple Cosne

Inv. Cosine merge (Cosine distance) (used by Model-Mixer)

           ...
           simab = torch.nn.functional.cosine_similarity(theta_a, theta_b, dim=0).abs() # use abs()
           ...
            k = (simab - sims.min())/(sims.max() - sims.min()) # k always k > 0 and k < 1
            k = 1.0 - k if "Inv" in calcmode else k # use cosine similarity or cosine distance
            k = k.mean() * alpha if "Simple" in calcmode else k * alpha # k.mean() * alpha for Simple Cosine.
            theta_0[key] = weighted_sum(theta_0[key], theta_1[key], k) # k is a float if Simple Cosne