Best way to index and search leaked databases?

I am in the tb range with breaches collected, obviously it would be highly impractical to search this through grep or any tools, I would just combine everything by the first line alphabetically, but this would only work for searching by email or whatever happens to be listed first.

Commercial breach searching sites have collections larger than mine and search them almost instantly, does anyone have an idea of how they are set up?

What about creating a database which you can use say jupyter to programmatically search through it on the fly? You probably will want to break it up to multiple db's and have a rather robust system running it, you can even write say a script to sort usernames and passwds and output those results to a txt file