Defederate, no question.

Are you gonna tell us which instance is doing this?

I’ve toyed around with LLM-based moderation tools but it never really panned out. It was too hit or miss to be relied upon even with the temperature parameters turned way down in an attempt to get consistent results. Granted, I was using a small local model and not feeding it to one of the big players.

To give an example, I tried to keep it focused by creating one custom model per rule to enforce. An example prompt to mod calls for violence was basically:

ROLE: You are a forum moderator who does not want users calling for violence.  Examine the input and analyze whether it violates any constraints. 

KNOWLEDGE:
- {list of dog-whistle slang for calling for murder}

CONSTRAINTS:
- Content should not advocate violence
- Content should not normalize violence
- Content should not escalate tensions or fan flames
- Content should avoid promoting harmful stereotypes
- Content should not utilize broad, sweeping generalizations
- Content should not use dehumanizing language
- Content should not undermine human rights, due process, or the rule of law

FORMAT YOUR RESPONSES AS JSON:
{
  reason: [A one to two sentence summary],
  score: [On a scale of 0 to 10, how severe is the content advocating violence]
}

The score part of the response was my band-aid to get around the high number of both false positives and false negatives. Any score 7 or higher caused the item to be passed to the mod queue along with the reason, and I would review its actions later.

Ultimately it was slow and still somewhat unreliable, so I abandoned the idea after running it for a little less than a day since I can 't run bigger models to get better results fast enough to keep up. Using a cloud based service was out of the question for many, many reasons, both financial and ethical.

To answer your question, as long as the models were locally hosted and properly tuned/tested, I’m fine with it in theory, except for the ideology part; that’s pretty messed up. While I don’t want my submissions used to train anyone’s model and take measures to prevent my own instance from being used as a data source, I remain aware that once I post something, I have no control over its fate the moment it federates out.

[1] Yes, I know that’s like half the comments that get posted around here. My goal was to try to have it mod things so posts were bases for actual discussions instead of being a knee-jerk rage factory.

How did you discover this?

Aside from the ethical implications of profiling users or of using a corporatly owned server and model to execute this, I see nothing uniquely concerning about this practice that isnt already a risk of federated social media generally.

Every mod on every instance is free to use whatever tools or standards for moderation they want - that’s an intentional byproduct of federation. Similarly, the collection of this data for use with llms is a bygone conclusion at this point - there was never any way of preventing that from happening with a federated network.

I think the only thing here to talk about is the way these questions are being framed as a question of intra-instance policy. We already have communities where moderation abuse can be called out and adjudicated- why pose this as a question of instance administration when there doesnt seem to be any evidence for it?

Wthout going into the issue itself, it is such a ridiculous waste to use an llm for something that a far simpler model could do like 100x faster and locally for essentially free…

Just search for “machine learning text moderation” and you will find all kinds of options. Not to talk about the fact that a simple 4B LLM could do this as well.

One thing I really hate is how LLMs have completely overshadowed the entire ML/AI field and people just use them for everything.

Using a trillion parameter LLM model for basic text moderation is like using a gaming rig to play candy crush.

You stay far, FAR away from that shit, is what you do.

Scanning people’s entire history for political leanings, etc? That’s some deeply dystopian stuff right there.

It’s easy to forget that these sorts of communities are dictatorships with only as much transparency as the owner wants to share. Usually they’re benevolent dictators, so we don’t think about it too much. But they can change in a heartbeat - and we don’t ever really know what they’re really thinking, or doing behind the scenes.

When the mask slips and they reveal this sort of thing, thinking we’ll just accept it and keep living under their rule, it’s time to read the red flags and GET OUT.

Hopefully someone compiles a list of places that do this stuff, so we can avoid them like the plague <3

You talk about instances utilizing this tooling, but in your comments you admit it’s just some mods. This is misleading, as talking about instances doing it assumes admin access and relevant instance policy, something which invites calls for defederation (as can be clearly be seen from the comments in your post).

A random mod doing something is not the same as an instance doing it. Literally anyone can be a mod and they don’t get any more access than an anonymous account by doing so.

This is the second time in one week I see you throwing careless statements like chum in the water. I can’t help but notice a pattern emerging.

I think LLMs could be useful tools for moderation, you might even can get away with smaller models for it, but I don’t think people should be outsourcing them to big corpos, due to ability to manipulate the models.

I’m really not fond of the profiling by automated means, but it seems like an inevitable consequence of the design of the threadiverse. Everything is public and easily accessible by anyone that would like to profile you.

I certainly disapprove of moderation based on ideology. Moderation should be based on quality of the content and if it fits in the publicly readable rules. Definitely not some hidden analytics or if the user completely fits in the in-group of the moderator.

I will admit that this might be a good way to find and filter out LLM based bots that are only there to promote or manipulate the conversation. But it should still be done according to public rules.

Rimu farming more drama?

I don’t like this happening, and there should be transparency in all moderation decisions, but some of these points make no sense.

There is essentially no expectation of privacy on threadiverse platforms. Everything is public and probably already being used to train models.

There is no private messaging system. Direct messages are unencrypted and potentially visible to any instance admins. They and should not be used to share anything sensitive.

I think this will exemplify the beauty of federation. If I find out my instance mods are running all of my comments through a company’s ai model, I’ll switch instances. This is in great disparity to something like Instagram or Snapchat where every photo I post is immediately fed to ai and my only options are: be okay with it, never post, or delete Instagram.

I don’t think the privacy issues here are too salient. Pretty much everything on the fediverse is public already and have likely federated outside any particular region like the EU, so GDPR doesn’t really have any teeth. The exception to that would be if instance admins are using database access to also feed private messages to an LLM (especially a corporate LLM). I know that the “private” in private messages on the fediverse can be conditional…but it should at least be considered private from LLMs as an expectation since those messages are inaccessible to things like scraper bots or listening instances designed just to harvest data.

My biggest concerns here would be twofold:

1. False positives - LLM sycophancy is a thing. So, I worry that if you ask an LLM to dig through a big pile of text looking for a thing, that it will tell you that it found that thing…even if it is completely removed from context or completely made up. The false positive rate might be low (I have no idea), but I guess I just don’t trust the LLM enough to let it take the wheel with stuff like this.
2. Outsourcing moderation - LLMs are not going to be up to the task of moderating everything, just ask digg. However, tools to help moderators effectively do their jobs are helpful as well. There is a balance to be struck here. I think, for me, something like asking an AI essentially, should I ban this person, just feels like you are outsourcing your decision making too much. It is too far on the automation side of the scale for my tastes.

All that said, people can run their instances how they want. I don’t really have strong opinions on LLMs/AI in general, I just kinda hate big tech companies. That is my foundational belief in the work that I do for the fediverse - fuck big tech and the oligarchy they have built/funded in my country. That is really the only axe I have to grind in all this.

I would like to see some ROBOT9000 esque oddball meme communities overtly based on heavy algorithmic moderation, can be LLM but wouldn’t have to be. Weird rules strictly enforced by robots, could be fun.