Finegrain/refiners: A microframework on top of PyTorch with first-class citizen APIs for foundation model adaptation

mirror of https://github.com/finegrain-ai/refiners.git synced 2024-11-21 13:48:46 +00:00

A microframework on top of PyTorch with first-class citizen APIs for foundation model adaptation https://refine.rs/

background-generation background-removal controlnet diffusion-models dinov2 image-generation ip-adapter lcm lcm-lora lora sam sdxl segment-anything segment-anything-model shadow-generation stable-diffusion t2i-adapter text-to-image textual-inversion

Find a file

Laurent e423ba4291 Some checks failed CI / lint_and_typecheck (push) Has been cancelled Details Spell checker / Spell check (push) Has been cancelled Details remove a couple `from torch import ...` from the code		2024-08-21 08:46:46 +00:00
.github/workflows	bump setup-rye action to v3	2024-06-27 14:11:07 +02:00
assets	README: upgrade hello world	2023-10-20 18:28:31 +02:00
docs	add refiners.foundationals.latent_diffusion.multi_diffusion to mkdocstrings	2024-07-11 15:23:02 +02:00
notebooks	deprecate outdated notebooks/basics.ipynb	2024-02-02 14:12:59 +01:00
scripts	implement foreground conditioned ic light	2024-08-12 12:09:23 +02:00
src/refiners	remove a couple `from torch import ...` from the code	2024-08-21 08:46:46 +00:00
tests	implement foreground conditioned ic light	2024-08-12 12:09:23 +02:00
.gitignore	add DINOv2-FD metric	2024-04-03 16:45:00 +02:00
CONTRIBUTING.md	refactor dinov2 tests, check against official implementation	2024-04-02 10:02:43 +02:00
LICENSE	Update LICENSE	2024-02-02 14:08:09 +01:00
mkdocs.yml	write Training 101 guide	2024-02-26 14:44:02 +01:00
pyproject.toml	bump pillow and pyright	2024-07-17 19:22:16 +02:00
README.md	add IC-Light bullet point in Latest News	2024-08-12 12:09:23 +02:00
requirements.docs.txt	(pyproject.toml) move doc deps inside their own `project.optional-dependencies`	2024-02-02 11:08:21 +01:00
requirements.lock	fix typing issues coming from torch 2.4 version ; typing is not guaranteed for torch < 2.4	2024-08-02 12:02:00 +02:00

README.md

The simplest way to train and run adapters on top of foundation models

Manifesto | Docs | Guides | Discussions | Discord

Latest News 🔥

Added IC-Light to manipulate the illumination of images
Added Multi Upscaler for high-resolution image generation, inspired from Clarity Upscaler (HF Space)
Added HQ-SAM for high quality mask prediction with Segment Anything
Added SDXL-Lightning
Added Latent Consistency Models and LCM-LoRA for Stable Diffusion XL
Added Style Aligned adapter to Stable Diffusion models
Added ControlLoRA (v2) adapter to Stable Diffusion XL
Added Euler's method to solvers (contributed by @israfelsr)
Added DINOv2 for high-performance visual features (contributed by @Laurent2916)
Added FreeU for improved quality at no cost (contributed by @isamu-isozaki)
Added Restart Sampling for improved image generation (example)
Added Self-Attention Guidance to avoid e.g. too smooth images (example)
Added T2I-Adapter for extra guidance (example)
Added MultiDiffusion for e.g. panorama images
Added IP-Adapter, aka image prompt (example)
Added Segment Anything to foundation models
Added SDXL 1.0 to foundation models
Made possible to add new concepts to the CLIP text encoder, e.g. via Textual Inversion

Installation

The current recommended way to install Refiners is from source using Rye:

git clone "git@github.com:finegrain-ai/refiners.git"
cd refiners
rye sync --all-features

Documentation

Refiners comes with a MkDocs-based documentation website available at https://refine.rs. You will find there a quick start guide, a description of the key concepts, as well as in-depth foundation model adaptation guides.

Awesome Adaptation Papers

If you're interested in understanding the diversity of use cases for foundation model adaptation (potentially beyond the specific adapters supported by Refiners), we suggest you take a look at these outstanding papers:

Projects using Refiners

https://github.com/brycedrennan/imaginAIry

Credits

We took inspiration from these great projects:

tinygrad - For something between PyTorch and karpathy/micrograd
Composer - A PyTorch Library for Efficient Neural Network Training
Keras - Deep Learning for humans

Citation

@misc{the-finegrain-team-2023-refiners,
  author = {Benjamin Trom and Pierre Chapuis and Cédric Deltheil},
  title = {Refiners: The simplest way to train and run adapters on top of foundation models},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/finegrain-ai/refiners}}
}