Today Yahoo released an intriguing service called "Pipes". Based on the concept of unix pipes, it allows you to filter and process RSS feed data in a chain - using a very cool web graphical user interface. The result can be spit out in a variety of formats that can then be used in your RSS reader, on your blog, in a Greasemonkey script or even as the building blocks of more complex pipes.
There is still a lot of room for improvement on the tool (the wish list of functionality grows by the second!) but it's an awesome and inspiring start. The only major complaint from people is the lack of easy-to-understand documentation. Don't worry though - I'll present a step-by-step guide to get you going!
The simplest - listing an RSS feed
Ok, we can't get much more basic than this, so it's a good place to start. Taking an RSS feed, and doin' nothin' to it...
- Open the "Sources" dropdown list and drag the Fetch module on to the workspace.
- In the "url" textbox enter your feed address. For this example we will use http://www.digg.com/rss/index.xml
- Now, join the output of the Fetch module to the input of the Pipe Output module. When you click and drag a pipe from a module's output node you will see that all input nodes that accept that output will light up. This makes it easy to see where you are allowed to join stuff!
Now we have joined our modules, the RSS feed is piped to the output. Click on the Pipe Output module and the final output is displayed in the debugger. This is how you can test your pipe sets without having to run them from your pipe lists.
If you join something to the wrong place, you can delete the join by clicking on the output node, and dragging to a blank space. - Click on the "Save" button in the top right of your workspace. Enter a name in the text box in the top left and hit Save.
- To test the pipe, go back to your pipe homepage and click on your pipe's name. Click the "Run this Pipe" link.
Notice that when you select the Fetch box, the Debugger at the bottom of the screen will load the RSS feed and show you the returned results. The debugger shows you the output after every module in your pipe chain.
The Pipe Preview will load the Digg RSS feed. Phew - there's ya first pipe done!
A more complex example
Now that you've got the basics down, it's just a matter of playing around with the other modules to see what you can do. Here's what we'll do here - we'll make a feed that combines the latest stories from Slashdot with the stories from Digg that have over a certain number of diggs. Then we'll filter out all Digg stories that have "amazing" in the title, and get rid of any Slashdot stories from Zonk.
- Add two Fetch modules from the available Sources. To one add the URL http://www.digg.com/rss/index.xml and to the other http://rss.slashdot.org/Slashdot/slashdot. Next add a Union module from the Operators section. Join the two Fetch outputs to the Union inputs. (The same thing can be done adding extra urls to one Fetch module, but I just did it this way for now so we can join some more stuff!)
Clicking on the Union module you will see that both RSS feeds are output in the debugger window.
- Now we will customise the results to suit our tastes... Add a Filter module from the Operators list. We want to "Block" items that match "Any" of our criteria.
Click on the Rules dropdown list - notice it only contains the values "Title", "Body", "Publication Date". Join the output of the Union module to the input of the Filter.
The dropdown box changes to a label that says "Updating...". Once it is loaded, it will contain a list of all the fields you can filter on from the data feeds! NOTE: This can be a little buggy. If it doesn't appear to properly refresh the lists, try removing the link and re-adding it. This should do the trick.
Now, let's filter away crap we don't want
- First of all, let's get rid of low-dugg stories:
From the dropdown list select "digg:diggCount" "is less than" and in the textbox type "500".
Our output will only return Digg stories that have over 499 diggs. It will return all Slashdot stories. - Next click on the plus to add a new Rule. Let's make this one "Title" "Does not contain" "amazing".
- For the final rule, let's not get Zonk's posts: "dc:creator" "Does not contain" "Zonk"
Ahhh that's good. Now just join the Filter output to the Pipe Output and we've got our aggregation.
User Input
Finally, what if we want to change the minimum number of Diggs needed to be included? Well one way is to add a User Input field. Let's do that.
- Select Number Input from the User Inputs. Drop it next to our Filter. Fill in the following values (without the parenthesis!):
Name: numDiggs (internal variable name)
Prompt: Miniumn Diggs (this appears on the input screen)
Position: 0 (Order of input boxes)
Default: 500 (A default value for the input screen)
Debug: 500 (A value just for you when you're testing) - Drag the output from the Number Input control to the little input on the diggCount field in the Filter.
Change the Debug value, and refresh the Pipe Output. You will see the number of Digg stories changes.
If save your pipe and go back to your pipes home page, when you load this pipe you will be prompted to enter the "Minimum Diggs" in a textbox. Awesome!
That's it really. Now you just need to start adding in other modules and figuring out what they do. The only tricky ones are the For-Each modules. But don't worry about them too much now.
The best thing about Yahoo Pipes is you can use one of it's many available methods to "Subscribe" to your new mashed up feed. Now all you gotta do is put on your thinking helmets and come up with some killer mash-ups! Magic!
27 Comments
Excellent tutorial, got me past the guessing stage, now Ive got a feed output I’ve been trying to get for a while.
Now how do i publish it on my site? Havent got that far yet…
Thanks so much for the writeup. It got my wheels spinning. It’s the graphical environment that folks like KnowNow should have used to build a developer community around their feed server. I’ve been a sucker for GUIs since 1984.
Mike: This is a great tutorial, excellent details. Cheers!
– Daniel Raffel
Pipes dev team member
Very nice and elegantly to the point – the Union Module instruction made me less afraid to try more Modules! -Joe, check Dinarius’s Homepage ‘source code’ for code that will let you show RSS on your site. MrSpeak3r’s Union tip is going to help us change everything on our feeder. THANKS BIG TIME!
excellent. you got me up and running. built a chowhound filter for cambridge restaurants.
Thanks for the tutorial – visual patching always confises me – but I realized that it’s more that I can’t “visualize” what each patch will do, even though I can conceive of it programatically (sometimes anyway, lol) – my main problem has been getting figured out what each piece is generating, and then what some of the pieces are for.
Again, great tutorial. Can you make one for Quartz Composer now? =P
“Now how do i publish it on my site? Havent got that far yet…”
If you go to your main pipes page, you can “run” your pipe (don’t know why you can’t run from the pipe page, maybe in the future). If you run your pipe (this is covered in the tutorial) the resulting page will have an RSS and JSON link at the bottom.
There’s a couple places you can go from here (hopefully there will be a way to generate a widget/webbadge of a pipe in the future), but you might look at the JSON bit (http://www.json.org/js.html) which would allow you to parse the JSON feed in javascript on your page, and write it out however you want.
I’m still trying to figure that out myself, but I’ve seen tons of pages on it on the various javascript resource sites.
In regards to publishing the results of pipes – the Pipes devs have hinted that they are working on some simple ways to embed them in your pages. I’ve never tried working with rss feeds, so I’m not 100% sure the best way to do it – I reckon it’d be pretty easy though – I’m sure there are tutorial out there already.
The only way I’ve used the results so far is in this greasemonkey script – I use the JSON output from the Pipe to do groovy stuff client side.
JSON output is easy to get from Pipes; you want to add _render=json and (optionally) _callback=yourCallback. I’ve just put up a working Web badge to return Technorati search results via Pipes; source is online and should be fairly easy to implement and understand.
How much damage might do to whatever else I now have on mp personal PC ????????????
How much damage might I be able to do, & to what ???
very good tutorial. From the yahoo page, I didn’t even know what pipes were. Now I can start trying them out. Thanks.
This was a great tutorial. It helped me get started. I also created a nice series of video tutorials. With your permission, here is the link:
http://usefulvideo.blogspot.com/2007/02/yahoo-pipes-tutorials.html
One way of getting an interface to a Pipe is to use a Grazr widget (http://grazr.com) fed by a small Grazrscript file to generates the search box (http://docs.grazr.com/script/tutorial/).
I have posted a small example at http://blogs.open.ac.uk/Maths/ajh59/009318.html, which offers a Grazr interface to an Amazon book search pipe.
tony
Here’s a really quick’n’dirty GRazr interface generator for a yahoo pipe with a single text entry box (such as a search pipe). Simply run a pipe with a magic search term, take a copy of the rss feed url, paste it into a form and get a grazr interface to your search pipe
http://ouseful.open.ac.uk/grazr/gPipeTest2.php
tony
Please have a look at http://soarack.blogspot.com/ for an alternative to yahoo pipes
The site looks great ! Thanks for all your help ( past, present and future !)
Couldn’t get your second tutorial to work. The filter would only recognize one of the feeds’ items (the one that was connected furthest to the left of the union operator). I had to have two filters going into one union… Must be an issue with the pipes. Thanks for the tut though!
Hmm, sounds a bit odd Jeff – do you have a link to the pipe, I could take a look if it’s giving you grief!
I’ve had success adding multiple feeds into one Fetch Feeds module without using a Union module. Is there a reason why I should use one way versus the other?
You are correct mattechi – It’s a lot nicer to just add multiple feeds without the union. I only included the union here to demonstrate the ideas.
The only reason you’d do it this way is if you want to process or filter the feed BEFORE you merge it with the others.
For example if you have one feed that has heaps of results, you might want to only get the top 10 of those before you merge it with others, to prevent the final feed being over-run with the first feed’s items. Or something.
Wow, love this tool! But how do I publish the results onto my web page to use the content to enhance my SEO?
This is a great step by step tutorial, thanks for the effort, it really helps a noob like myself :)
Anyone interested in Yahoo! Pipes might be interested in my two books “Working with Yahoo! Pipes, No Programming Required” and “Mashup Case Studies with Yahoo! Pipes”.
http://www.lotontech.com/it_books.htm
Hi
Do you have any Tutorial on How to make these Yahoo Pipes useful in generating Content for hosted WordPress blogs?
Tutorial on (a) a WordPress page, named News, that generates News content related to specific keywords
Tutorial on (b) a WordPress side bar widget, that also generates News content via RSS
I would appreciate these Tutorials, please.
Thanks. Your post is extremely helpful.
May Ong
I would like to suggest a great new site that organizes your RSS feeds.
It employs a bayesian filter for RSS feeds where you can train the filter what you like and
what you don’t like. It’s free, try it at http://www.filteredrss.com.
How can i use it to translate a hole site ?
Hello,
Great tutorial, but I had a problem getting started.
I could not find “Fetch”. I see 6 selections of “Fetch (CSV, Feed, Data, Page)”, but not just Fetch.
Please explain…
Thank you,
4 Trackbacks/Pingbacks
Crazy 72 hours
It’s been almost 72 hours since our beta launch. Needless to say, the response to Pipes has been overwhelming for everyone on the team (and our servers!). I have finally had a chance to relax/sleep after addressing some of the immediate issues….
[…] http://www.mrspeaker.net/2007/02/10/yahoo-pipes/ […]
[…] http://www.mrspeaker.net/2007/02/10/yahoo-pipes/ […]
[…] http://www.mrspeaker.net/2007/02/10/yahoo-pipes/ […]