ComfyUI With Mimic Motion Change Video Background
Added 2024-09-02 15:23:46 +0000 UTC
Video Tutorial : https://youtu.be/kBUdLRWFmI8
Things you need to know:
MimicMotion Blog : https://tencent.github.io/MimicMotion/
Github : https://github.com/Tencent/MimicMotion
MimicMotion ComfyUI Custom Node: https://github.com/kijai/ComfyUI-MimicMotionWrapper
Workflow for Patreon Supporters : https://www.patreon.com/posts/111263092/
Comments
You have to notice if you have a VAE in your vae loader or link up with your checkpoint model.
Benjamin Law
2024-09-25 18:13:39 +0000 UTCHello) In the AnimateDiff Refiner workflow section an error pops up... 'VAE' object has no attribute 'vae_dtype'..is there anything you can do?
Eugene Feder
2024-09-25 17:52:27 +0000 UTCIf you paste that name on Google. You can find this one. https://huggingface.co/crishhh/animatediff_controlnet/blob/main/controlnet_checkpoint.ckpt Or i have posted a list of AnimateDiff related links in Patreon before. If you are new here, you can check it out. Thanks
Benjamin Law
2024-09-25 06:32:48 +0000 UTCIn the AnimateDiff Refiner workflow section, in Load Advanced ControlNet Models. I could not find where to download "Animate_controlnet_checkpoint.ckpt". My guess this is a rename? But I'm not sure which model or ckpt to download to get this part to work.
John C
2024-09-25 06:24:44 +0000 UTCYes, one more thing, do not use a stick man or 3D dummy for motion reference. If you do, put a face on it before run it. Cause the DWPose cannot find the face pose, and the output face part will be morphing, remember that this framework also so motion for face.
Benjamin Law
2024-09-09 09:40:16 +0000 UTCThe problem was with outdated Diffusers. Everything is working, but my character is pretty destroyed and unrecognizable after Mimicmotion. Here is the clip: https://drive.google.com/file/d/101VUN9canxPNbFG5yVxeISrZW5Qeh_WV/view?usp=sharing Can you tell me what i am doing wrong? I didn't change anything, except raising steps from 20 to 30, cause is was even worse with 20 steps. I have one more question: how can I replace the background with a video that has a different aspect ratio? For example, the background video is 16:9 and the character Mimicmotion is 1:2. I tried to do it myself with the compositing nodes but it didn't work. Thanks in advance, sorry if I'm boring with questions.
Nenad Kuzmanovic
2024-09-09 05:49:20 +0000 UTCfor get pose problem. it happened to me with one of my reference video with lots of people on the background. And I use the SEG group above Mimic Motion group, and segment only the character for capture pose. Then pass the output, to Mimic Motion. So in Bypass group node, enable Segmentation group. In MinicMotion Output group, set the incoming GET Node "ref_video" to "Mask_Img_SEG_OBJ". That's what I did to solve get pose issue when there's too much people in the background, which distracts the Get Pose. I am not sure about your reference video, but you can try this method.
Benjamin Law
2024-09-05 17:43:32 +0000 UTCInstalling Mimic motion did not help. I've captured a video clip. https://drive.google.com/file/d/1KgUX062DgrRwAalsCAnjokr14As-ffAI/view?usp=sharing
Nenad Kuzmanovic
2024-09-05 15:19:19 +0000 UTCYou need to update the custome node. It happen sometime when I saw this kind of message, update Comfyui and nodes, then error gone.
Benjamin Law
2024-09-05 08:47:08 +0000 UTC2024-09-05 03:24:40.354 - execution.py:386 - ERROR - !!! Exception during processing !!! need at least one array to stack 2024-09-05 03:24:40.356 - execution.py:387 - ERROR - Traceback (most recent call last): File "C:\Comfy_UI_V2\ComfyUI\execution.py", line 317, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) File "C:\Comfy_UI_V2\ComfyUI\execution.py", line 192, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) File "C:\Comfy_UI_V2\ComfyUI\execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "C:\Comfy_UI_V2\ComfyUI\execution.py", line 158, in process_inputs results.append(getattr(obj, func)(**inputs)) File "C:\Comfy_UI_V2\ComfyUI\custom_nodes\ComfyUI-MimicMotionWrapper\nodes.py", line 451, in process detected_bodies = np.stack( File "C:\Comfy_UI_V2\ComfyUI\venv\lib\site-packages\numpy\core\shape_base.py", line 445, in stack raise ValueError('need at least one array to stack') ValueError: need at least one array to stack 2024-09-05 03:24:40.358 - main.py:138 - INFO - Prompt executed in 32.83 seconds I've tried everything, update, create nodes from scratch etc..
Nenad Kuzmanovic
2024-09-05 01:26:22 +0000 UTC
