Announcing SteemShovel - Steem Search Engine (pre-alpha)
SteemShovel.com
Disclaimer : This is pre-alpha, in-development software so expect it to crash and/or be unavailable from time to time while I continue to work on it. Thanks for understanding :)
I'm excited to announce the very first rough draft of my Steem Search engine - Steem Shovel. I was frustrated with not being able to find older posts on Steemit so I set out to build my own search engine. I have some experience with ElasticSearch and it seemed like a great fit for indexing Steem posts and comments.
What's it do?
SteemShovel runs continuously indexing new content from new blocks. It will index both comments and posts and make them full-text searchable. Moving forward I plan to allow you to limit your search to specific authors and also restrict searches to certain tags. Right now it's just a full on mega search against all of the fields/text.
What powers SteemShovel?
The underlying application code is written in Node.JS while the search engine is powered by ElasticSearch. I use the SteemJS module (https://github.com/adcpm/steem) to talk to the Steem process.
The application is load balanced using AWS EC2 servers behind an Elastic LoadBalancer. This gives me the ability to add more servers / horsepower at will if anyone decides to use this app.
Why can't I find the post I'm looking for?
The indexing process is currently making its way through all of the Steem blocks. A second process is indexing new posts as they appear but there is a big gap between the two. Because of this it might be a while before you are able to find every single post you're looking for. Hang tight, they're coming!
What are some improvements coming to SteemShovel?
Well, I'm glad you asked! I have a pretty ambitious set of goals for this project so we'll see how it goes. The main focus now is to work on the indexing and build some more filters that you can use to search. Beyond that I want to improve the look of the site and make sure it's working well on mobile.
Once the basics are covered I have tons of features I want to work on - log in, voting, UI improvements to show threads of comments like Steemit does.
But wait, there's more!
I have also built this to be an API that anyone can talk to. All you have to do is direct an http client to https://steemshovel.com/search/?search={your search term} and you'll get some JSON to do with as you please.
If you have any questions or suggestions as to what you'd like to see this thing do, please comment and let me know.
@kaptainkrayola thanks for the heads-up! This is one of many steem tools.
@roelandp is making a collection of them at http://steemtools.com/
Keep it up!
I'll get the form filled out there to get listed directly. Thanks!
Great tool dude!
I've listed it on SteemTools!
It's fantastic that you have added a JSON api, this is really helpful!
Supercool that you are indexing...
Feature request, undoubtly already on the list:
Hats off! Applause!
This is much needed! Thank you!
Alright - you totally deserve the upvote and this comment.
Great job providing for the community. I have no doubt you will make:
You rock man!
Great!
Thanks for the support everyone. I think I'll use the money from this post to get a proper web designer to work on the site as well as some contracting hours from another node.js dev to help build out some more of the features.
It already looks pretty great IMO
Just W O W
Thank you so much for your hard work. Hope you're appropriately rewarded. Come on people, get behind @kaptainkrayola!
I do love your enthusiasm. Thanks for the support!
The built-in Steemit search is diabolical! You are providing a much needed service. Huge respect and gratitude.
Very nice idea, thank you for sharing
I love how fast it is, and the results seem to be pretty good. Kudos on the implementation so far.
I only have 1 minor feature request, which is to have the ability to show only results from posts in certain time period (ie. last 24hr, past week, past month, past year...).
I'll add some of those filters to the to-do list for sure. Thanks for the input
way to go! good job! just what i need!