![ControlNet + More Tutorial [ StreamDiffusionTD 0.2.2 ]](https://img5.saketami.com/storage/10/ge/pg/9a2cd0-019e8036-a4d7-7d44-a1f2-6ba50cc444af.jpg)
ControlNet + More Tutorial [ StreamDiffusionTD 0.2.2 ]
Added 2024-09-05 16:03:01 +0000 UTCHappy to bring an update of StreamDiffusionTD with some fixes + a video covering many things in regards to ControlNet + StreamDiffusion. Hopefully it is both helpful + inspiring. I have a few more tutorials / overviews of new operators in the works that will be posted soon.
Thank you for the support as always !
Get the 0.2.2 NEW TOX here >>> https://www.patreon.com/posts/111459780
0.2.2 Changelog here >>> https://www.patreon.com/posts/111459875
NEW ControlNet Tutorial >>> https://www.youtube.com/watch?v=gS2Am5DFZO4
NEW DotUpdater - See this for updating to new version of my operators in your project [also covered in ControlNet tutorial] >>> https://youtu.be/MpSnq8-a1Go
Like always, please let me know if you run into issues. The troubleshoot forum on the discord and @ me on there is the best / preferred way.
Comments
Is there any tutorial or post somewhere on how to install other checkpoints or LoRAS to use with Stream diffusion?
Jonatan Ewald
2025-06-22 15:24:57 +0000 UTC...Loading sd15 ControlNet model: control_v11p_sd21_depth.safetensors Failed to load ControlNet model https://huggingface.co/thibaud/controlnet-sd21/blob/main/control_v11p_sd21_depth.safetensors due to: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect. Traceback (most recent call last): File "D:\AI TOUCH\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 637, in _load_model controlnet = ControlNetModel.from_single_file(controlnet_file, config_file=config_file, use_safetensors=True, local_files_only=clip_exists, torch_dtype=torch.float16).to(device=self.device, dtype=self.dtype) File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\loaders\single_file.py", line 617, in from_single_file controlnet = download_controlnet_from_original_ckpt( File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1846, in download_controlnet_from_original_ckpt controlnet = convert_controlnet_checkpoint( File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1125, in convert_controlnet_checkpoint set_module_tensor_to_device(controlnet, param_name, "cpu", value=param) File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\accelerate\utils\modeling.py", line 292, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect.
Shio Visconti
2025-05-19 10:27:21 +0000 UTCissue, when i start streaming with controlnet ON, show me this error: "ValueError: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect. " help pleaseee
Ignacia .Hurtado
2025-01-22 03:56:08 +0000 UTCHello, when start streaming it says in the console : " V2V + ControlNet are not supported and will be disabled. A matching Triton is not available, some optimizations will not be enabled. Error caught was: No module named 'triton'. " Do you have an idea where this issue comes from and how I can resolve it ?
Dji
2024-12-22 17:31:13 +0000 UTChey, can you tell me how to do the pose controller? from your video I saw depth and edge but couldn't find any pose related object in TD
KasΔ±m AΓ§Δ±kbaΕ
2024-12-11 10:41:08 +0000 UTCI can't do it on streaming. cmd doesn't load data. can you please help me
Raja Singa
2024-11-01 09:54:21 +0000 UTCyeah Im having problems with this too - earlier versions could
Gus
2024-10-24 14:04:31 +0000 UTCDid it ever finish loading? maybe restart whole process. Depending on the amount of vram on your gpu, it could be having trouble fitting on it, I would recommend 8gigs at least.
dotsimulate
2024-10-07 21:51:25 +0000 UTCYou can negative prompt certain things with negative weight on LoRA models, which is a bit more limiting but could work for certain things.
dotsimulate
2024-10-07 21:50:14 +0000 UTCnegative prompts don't work with lcm / 1 step processes such as streamdiffusion unfortunately.
dotsimulate
2024-10-07 21:49:40 +0000 UTC[issue] ...Loading sd-turbo (2.1 distilled) model pipeline... ...Loading sd-turbo (2.1 distilled) model: sd-turbo Loading pipeline components...: 0%| | 0/5 [00:00
Andrea Bucci
2024-10-07 21:18:21 +0000 UTChow do I enable the additional settings for negative prompts?
Edmond Yang
2024-10-07 17:41:09 +0000 UTCThank you so much for the controlnet tutorial. Would you also share how to use openpose in stream diffusion ? How to get the colorful skeleton form video source? Thanks in advance.
Dan Fong
2024-10-05 18:09:28 +0000 UTCHi all, I'm enjoying this tool, thanks DotSimulate. But I'd like to know if you have found models better than SD-TURBO; do you have any advice?
Ismael Abrego
2024-10-02 18:20:34 +0000 UTCWhat would be the equivalent of img2img in comfy? I get good results when I send a video thru stream diffusion but I would like to increase the quality and fps, and I'm looking for the equivalent with stable diffusion / comfyui. Thanks !
Frédéric Bussière
2024-10-01 07:37:18 +0000 UTCsame issue. fixed with pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 thanks dot!
jeevanpillay
2024-09-30 10:56:39 +0000 UTCsame
TITO FERRARI
2024-09-29 16:05:14 +0000 UTCI have the same error when I run the stream, Did you manage to solve it?
Diskorsia
2024-09-27 09:36:57 +0000 UTClo voy a ver, Gracias Dot!!
Julii Haid
2024-09-24 02:21:33 +0000 UTCTraceback (most recent call last): File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 683, in _load_model pipe = pipeline_class.from_pretrained( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1090, in from_pretrained cached_folder = cls.download( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1856, in download cached_folder = snapshot_download( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\huggingface_hub\_snapshot_download.py", line 216, in snapshot_download raise LocalEntryNotFoundError( huggingface_hub.errors.LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, pass 'local_files_only=False' as input. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 687, in _load_model except huggingface_hub.utils._errors.LocalEntryNotFoundError: AttributeError: module 'huggingface_hub.utils' has no attribute '_errors' Failed to load model due to: module 'huggingface_hub.utils' has no attribute '_errors'
Julii Haid
2024-09-23 21:13:22 +0000 UTCthis issue is caused by hugging face having a newer version. Click on .tox file on Manual Activation VENV then type this to downgrade huggingface to the right version : pip install huggingface-hub==0.24.6 Then launch it and it should download dependencies (I had to relaunch twotimes but now its working)
Johao MikaΓ«l Marin GagnΓ©
2024-09-23 17:15:59 +0000 UTCWhy prompt No internet connection detectedγ
ε±
2024-09-23 10:18:03 +0000 UTCis there only 512x512 ? or is there a way to have a better resolution?
Mariana Kamke
2024-09-23 01:19:30 +0000 UTC# StreamDiffusionTD huggingface_hub.utils._errors troubleshooting guide 1. Hit "Manual Activate" (install page) pulse to open command window. 2. Check package versions: ``` pip list ``` Look for: - torch: 2.1.0 - huggingface-hub: 0.24.6 - numpy: 1.26.4 3. Check CUDA: ``` python -c "import torch; print(torch.cuda.is_available())" ``` Should print "True". 4. If versions wrong or CUDA unavailable, check CUDA version: ``` nvidia-smi ``` 5. Install correct packages ( AND replace cu121 with cu118 if you have cuda11.8): ``` pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 ``` 6. Recheck CUDA (step 3). 7. If you continue to have an issue, please join the discord as it is way easier to troubleshoot on there. And post in the troubleshooting forum the details from above.
dotsimulate
2024-09-22 02:09:08 +0000 UTCsee above. Also follow up on discord. It is impossible to get notified or keep up with this via patreon comments.
dotsimulate
2024-09-22 02:01:45 +0000 UTC# StreamDiffusionTD huggingface_hub.utils._errors troubleshooting guide 1. Hit "Manual Activate" (install page) pulse to open command window. 2. Check package versions: ``` pip list ``` Look for: - torch: 2.1.0 - huggingface-hub: 0.24.6 - numpy: 1.26.4 3. Check CUDA: ``` python -c "import torch; print(torch.cuda.is_available())" ``` Should print "True". 4. If versions wrong or CUDA unavailable, check CUDA version: ``` nvidia-smi ``` 5. Install correct packages (replace cu121 if needed): ``` pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 ``` 6. Recheck CUDA (step 3). 7. If you continue to have an issue, please join the discord as it is way easier to troubleshoot on there. And post in the troubleshooting forum the details from above.
dotsimulate
2024-09-22 02:00:15 +0000 UTCYes. I am looking at this. It seems to not be installing all of the pip packages it should. The issue started about 5 days ago or so. still tracking down what is causing it but there are a few fixes that have worked for people in the discord. I will summarize below.
dotsimulate
2024-09-22 01:31:19 +0000 UTCgot the same thing trying to use any model other than sdturbo
Sasha BΓ©dard
2024-09-21 18:28:01 +0000 UTCI'm having the same issue today...
Jack Kalish
2024-09-21 18:01:57 +0000 UTChi! i have this issue: ...Loading sd-turbo (2.1 distilled) model pipeline... ...Loading sd-turbo (2.1 distilled) model: sd-turbo Traceback (most recent call last): File "D:\touchAI\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 683, in _load_model pipe = pipeline_class.from_pretrained( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1090, in from_pretrained cached_folder = cls.download( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1856, in download cached_folder = snapshot_download( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\huggingface_hub\_snapshot_download.py", line 216, in snapshot_download raise LocalEntryNotFoundError( huggingface_hub.errors.LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, pass 'local_files_only=False' as input. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\touchAI\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 687, in _load_model except huggingface_hub.utils._errors.LocalEntryNotFoundError: AttributeError: module 'huggingface_hub.utils' has no attribute '_errors' Failed to load model due to: module 'huggingface_hub.utils' has no attribute '_errors'
Luca Corsi
2024-09-21 16:44:30 +0000 UTCHello. Thank you for the amazing work! does 0.2.2 work on mac? Oh I just saw in changelog that you will update Mac-support. I wanna test that :)
Victor E Morales
2024-09-18 05:00:51 +0000 UTCMac install guide?
Stuart Smith
2024-09-16 17:45:22 +0000 UTCWhy can't I download streamDiffusion? The message "no internet connection detected "is always displayed+1
Q ε°
2024-09-15 15:40:20 +0000 UTCWhy can't I download streamDiffusion? The message "no internet connection detected "is always displayed
WDH2021
2024-09-15 13:33:42 +0000 UTCCan you record a tutorial for a plug-in to load the civitai Lora model library? I downloaded several model libraries and loaded them in the models in the plug-in, but the effect I got was similar to the effect of not binding the plug-in. I'm not sure whether my model library is not running now. Is there any way to judge that my model library is being used normally?
fan ze
2024-09-13 14:16:41 +0000 UTCAmaaazing work man! Very cool stuff. Iβm curious, as I have a couple of GPUβsβ¦. would there be a way for 2 frames to be processed at once on say a 4090 and 4080 setup? Iβve seen some chatter on this being the way to leverage them, in lieu of actual multi-gpu crunching on each frame. Appreciate it!
Kevin Friel
2024-09-13 02:05:15 +0000 UTChello, im newbie, i already install the streamdiffusion 0.2.2, after install the step 1 and step 2, i try to run start stream, the command prompt stuck in last log here Loading pipeline components...: 0%| | 0/5 [00:00
Yanandra Ardi Putra P
2024-09-11 03:28:43 +0000 UTCThanks for the fast fix!! Always appreciated.
Garry
2024-09-10 16:45:43 +0000 UTCWhen I click Download StreamDiffusion in Install, the message "No internet connection detected" is displayed.
WDH2021
2024-09-09 04:51:25 +0000 UTCwww.patreon.com/posts/dotupdater-1-0-3-111520883
dotsimulate
2024-09-06 04:31:01 +0000 UTCah! wow. I didn't realize that wasn't uploaded.
dotsimulate
2024-09-06 04:18:15 +0000 UTCThank you for the great work! Where do you link to the DotUpdater tox?
Al Order
2024-09-06 02:45:18 +0000 UTCHey Garry. I have fixed this and reuploaded 0.2.2 Thanks for letting me know.
dotsimulate
2024-09-05 21:21:13 +0000 UTC[Issues] : I recieve a "Script Error" when I pulse the second Install step (Venv~) By the way, thanks for the great work!! [Edit] : 0.2.0 works well
Garry
2024-09-05 19:46:43 +0000 UTC
