I've posted about this before as it relates to mod tools.^1^
The search part isn't all that difficult, there are open source search engines that are easy enough for admins to configure a decent search feature. The more difficult issue is aggregating the data from all our instances to a single source where we can make queries with those existing search engine tools.
I am going to spend some time this weekend working on a proof of concept for a search engine for mod tools. Big picture solution is:
- Instance admins regularly dump anonymized (i.e. no PII) post and comment data to a public source (possibly torrent, possibly sftp)
- Other instance admins download each others data and feed it into their search db (e.g. Elasticsearch)
- Mods & users create tools using this data
BTW: this isn't a novel idea:
- This is what pushshift is for reddit (check out their FAQ/wiki). We're missing mod tools big time and searching/aggregating is huge part of mod tools.
- Up until recently, like last week, Stack Exchange provided a regular dump of their data to the Internet Archive for posterity's sake
EDIT: Linked my OG post on the subject ^1^