Countermeasures

After the immediate urgency of the raid disruption has been dealt with, it was time to keep extending the ways to detect and deal with malicious actors on the horde as I do not wish to keep it into invite-only mode indefinitely, as that would add an extra step to the most critically important aspect of the horde. Having enough workers.

However as I started building on this, I kept coming with more an more ideas I wanted to add, so in the the end, it became one of the most concentrated feature additions to the Horde, so I wanted to write this post to enumerate and explain all the new things possible currently.

User Modifications

With the addition of the moderator flag for user, it quickly became apparent that they could use more power to help out. So first came the extension of the /api/v2/users/{ID} endpoint.

On the PUT request, we now have the "worker_invite" variable which is the amount of workers an account can have. During the invite-only mode of the horde, this is the absolute max amount of workers that user can have, but this also serves a purpose outside invite-only mode. Normally outside invite-only mode an account is still restricted to a max of 8 workers. This more than most everyone will need, but it could allow someone to onboard like 8 GPUs without asking for permission. The max limit will also prevent someone from creating a ton of new malicious workers in a very short amount of time which do not use the stable diffusion at all but rather just send pre-existing pictures. If someone has a legitimate reason to need more than this amount of workers on the same account, the moderator team can easily extend it with this setting

The "public_workers" setting on a user is an opt-in value that the user can set themselves, which would not only show the list of worker IDs on the user's info, but also show the owner's username on each worker page. I initially kept this information hidden, but in the interest of flexibility, I now give the option for anyone to reveal this info if they wish.

The new "username" option allows anyone to change their own username to something else via REST API. Even if they have a pseudonymous account.

At this point it will be a good idea to mention that the recent countermeasures include multiple profanity filters for usernames, server names and server info. This is to prevent malicious actors from putting hateful messages in them. It's not perfect, but it does the job well enough for now.

I now also have the capacity to give scheduled Kudos to specific accounts. This will allow me to not just handle my patreon reward tier automatically, but also for other purposes. The first use of this for example, is giving the new moderators 100000 kudos per month for helping out.

Last setting, but definitely not least is the capability to set a user as trusted. 

Before I explain what this does, I need to mention that the horde will from now on include IP checking. Specifically I will be limiting the amount of workers on the same IP, to avoid someone just spawning an infinite amount of workers by simply creating new accounts. On top of that I am also restricting VPN access, as that can be used to bypass this restriction. However there are legitimate reasons why someone might want to use a VPN, so the trusted role helps bypass these limits

But the purpose of the trusted role is not merely that. No,  this is part of my most important countermeasure, which is two tiers of workers. Trusted and untrusted. Trusted workers are those who's owners have been set as trusted. These workers are always used by all request to the horde. However all workers instead start as untrusted. 

Untrusted workers work the same way, with one important difference that the horde will not give them requests which explicitly requested only trusted workers to fulfill them. This will allow integrations such as chat bots in public locations to ensure they someone malicious cannot easily jump in to disrupt their operation.

However anyone can opt-in to use untrusted workers, and since those workers will tend to not be as busy as trusted workers, one could get significant speed boost by allowing those workers to serve them as well. There is just always a risk that someone will anyway find a way jump over the other countermeasures I've devised and try to wreak havoc before a human moderator can handle it. Having a different entrusted pool, will allow everyone in the horde to quickly press a button on their end and continue as normal with barely any disruption.

This feature also adds an automatic way for an untrusted worker to become trusted. You see, the untrusted worker's owner does not accumulate all the kudos they get for generations and uptime. Instead half their generated kudos and all their uptime kudos is put aside in the "evaluating_kudos" pool. When that pool amasses a predefined amount of kudos, the owner is automatically set as "trusted" and all their evaluating kudos is given to them. 

The aim of this approach is to make it a significant investment for a malicious worker to join the trusted pool. Meaning days or weeks of generating for the horde, just to troll it and get removed within minutes by a mod. The hope is that this investment will discourage all but the most dedicated actors.

Unfortunately this approach is not as optimal for the speed of the horde, but I hope that once people are used to the use and meaning of untrusted workers, they will be more willing to use them.

User Info

But not just the PUT has been modified. The user's GET response has also been extended. Now both mods and users themselves can pass their own API key in the GET request's Headers. If the key belongs to the user in question or a mod, more information will be revealed, which would otherwise be hidden.

The new unprivileged information that can be seen on a user's info includes

* worker_invited: How many workers this user can join to the horde

* worker_count: How many workers this user currently has in the horde

* monthly kudos: How much kudos this user receives per month, and when was the last datetime they did so

* trusted: If they're a trusted user

* moderator: if they're a moderator of the horde

The following information is only shown to privileged access

* evaluating_kudos: How much kudos this untrusted account has accumulated. 

* worker_ids: The list of worker IDs owned by this user. This is public if public_workers is set to true.

Worker Modifications

Worker endpoints also received some love. In the PUT type, the API key of the owner or a mod of a worker can now change their name. Previously there was no way to do it, as changing the name via the bridgeData would just create a new worker. This feature will now allow the name change, but it needs some caution. In order to do this, the worker should be first stopped and its name changed in its bridgeData. Then while it's stopped, the name change operation should happen, and only then should the worker be started again.

Worker Info

On the GET side, we also allow privileged and unprivileged information to be shown to the worker's owner and mods.

The new "owner" field will show the user's ID only if this is a privileged access, or the public_workers of the owner is set to true.

the "trusted" field will be true, only if the worker's owner is trusted and it means that this user is in the trusted pool.

Forgot your username?

A lot of people are opting for pseudonymous accounts instead of logging in with oauth. And sometimes the user's ID is not shown. Othertimes, integrators have asked for a way to verify that a provided API key is a valid user. Well, say not more because now the "/v2/find_user" endpoint exists, which when sent an API key in the header, will return the use info of the user it belongs to!

Other

This post is already way to long, but suffice to say I have not even mentioned all the countermeasures I've added to Horde, such as suspicion rating, ip checking etc. But I hope they will prove sufficient for when we reopen the worker invites next!