Introduction to Yahoo Pipes

The Yahoo! Pipes logoToday Yahoo released an intriguing service called "Pipes". Based on the concept of unix pipes, it allows you to filter and process RSS feed data in a chain - using a very cool web graphical user interface. The result can be spit out in a variety of formats that can then be used in your RSS reader, on your blog, in a Greasemonkey script or even as the building blocks of more complex pipes.

There is still a lot of room for improvement on the tool (the wish list of functionality grows by the second!) but it's an awesome and inspiring start. The only major complaint from people is the lack of easy-to-understand documentation. Don't worry though - I'll present a step-by-step guide to get you going!


The simplest - listing an RSS feed

Ok, we can't get much more basic than this, so it's a good place to start. Taking an RSS feed, and doin' nothin' to it...

  1. Open the "Sources" dropdown list and drag the Fetch module on to the workspace.
    first step diagram
  2. In the "url" textbox enter your feed address. For this example we will use http://www.digg.com/rss/index.xml
  3. second step diagram
    Notice that when you select the Fetch box, the Debugger at the bottom of the screen will load the RSS feed and show you the returned results. The debugger shows you the output after every module in your pipe chain.

  4. Now, join the output of the Fetch module to the input of the Pipe Output module. When you click and drag a pipe from a module's output node you will see that all input nodes that accept that output will light up. This makes it easy to see where you are allowed to join stuff!

    Now we have joined our modules, the RSS feed is piped to the output. Click on the Pipe Output module and the final output is displayed in the debugger. This is how you can test your pipe sets without having to run them from your pipe lists.
    second step diagram
    If you join something to the wrong place, you can delete the join by clicking on the output node, and dragging to a blank space.

  5. Click on the "Save" button in the top right of your workspace. Enter a name in the text box in the top left and hit Save.
  6. To test the pipe, go back to your pipe homepage and click on your pipe's name. Click the "Run this Pipe" link.
    second step diagram

The Pipe Preview will load the Digg RSS feed. Phew - there's ya first pipe done!

A more complex example

Now that you've got the basics down, it's just a matter of playing around with the other modules to see what you can do. Here's what we'll do here - we'll make a feed that combines the latest stories from Slashdot with the stories from Digg that have over a certain number of diggs. Then we'll filter out all Digg stories that have "amazing" in the title, and get rid of any Slashdot stories from Zonk.

  1. Add two Fetch modules from the available Sources. To one add the URL http://www.digg.com/rss/index.xml and to the other http://rss.slashdot.org/Slashdot/slashdot. Next add a Union module from the Operators section. Join the two Fetch outputs to the Union inputs. (The same thing can be done adding extra urls to one Fetch module, but I just did it this way for now so we can join some more stuff!)

    second pipe, first step diagram

    Clicking on the Union module you will see that both RSS feeds are output in the debugger window.

  2. Now we will customise the results to suit our tastes... Add a Filter module from the Operators list. We want to "Block" items that match "Any" of our criteria.

    Click on the Rules dropdown list - notice it only contains the values "Title", "Body", "Publication Date". Join the output of the Union module to the input of the Filter.

    The dropdown box changes to a label that says "Updating...". Once it is loaded, it will contain a list of all the fields you can filter on from the data feeds! NOTE: This can be a little buggy. If it doesn't appear to properly refresh the lists, try removing the link and re-adding it. This should do the trick.

    Now, let's filter away crap we don't want

  3. First of all, let's get rid of low-dugg stories:

    From the dropdown list select "digg:diggCount" "is less than" and in the textbox type "500".
    Our output will only return Digg stories that have over 499 diggs. It will return all Slashdot stories.

  4. Next click on the plus to add a new Rule. Let's make this one "Title" "Does not contain" "amazing".
  5. For the final rule, let's not get Zonk's posts: "dc:creator" "Does not contain" "Zonk"

second pipe, second step diagram

Ahhh that's good. Now just join the Filter output to the Pipe Output and we've got our aggregation.

User Input

Finally, what if we want to change the minimum number of Diggs needed to be included? Well one way is to add a User Input field. Let's do that.

  1. Select Number Input from the User Inputs. Drop it next to our Filter. Fill in the following values (without the parenthesis!):
    Name: numDiggs (internal variable name)
    Prompt: Miniumn Diggs (this appears on the input screen)
    Position: 0 (Order of input boxes)
    Default: 500 (A default value for the input screen)
    Debug: 500 (A value just for you when you're testing)

  2. Drag the output from the Number Input control to the little input on the diggCount field in the Filter.
    Change the Debug value, and refresh the Pipe Output. You will see the number of Digg stories changes.

second pipe, third step diagram

If save your pipe and go back to your pipes home page, when you load this pipe you will be prompted to enter the "Minimum Diggs" in a textbox. Awesome!

second pipe, final step diagram

That's it really. Now you just need to start adding in other modules and figuring out what they do. The only tricky ones are the For-Each modules. But don't worry about them too much now.

The best thing about Yahoo Pipes is you can use one of it's many available methods to "Subscribe" to your new mashed up feed. Now all you gotta do is put on your thinking helmets and come up with some killer mash-ups! Magic!

27 Comments

  1. Joe wrote:

    Excellent tutorial, got me past the guessing stage, now Ive got a feed output I’ve been trying to get for a while.

    Now how do i publish it on my site? Havent got that far yet…

    Saturday, February 10, 2007 at 5:44 am | Permalink
  2. Mike McGrath wrote:

    Thanks so much for the writeup. It got my wheels spinning. It’s the graphical environment that folks like KnowNow should have used to build a developer community around their feed server. I’ve been a sucker for GUIs since 1984.

    Saturday, February 10, 2007 at 11:36 am | Permalink
  3. Mike: This is a great tutorial, excellent details. Cheers!

    - Daniel Raffel
    Pipes dev team member

    Saturday, February 10, 2007 at 1:09 pm | Permalink
  4. BryanA wrote:

    Very nice and elegantly to the point – the Union Module instruction made me less afraid to try more Modules! -Joe, check Dinarius’s Homepage ‘source code’ for code that will let you show RSS on your site. MrSpeak3r’s Union tip is going to help us change everything on our feeder. THANKS BIG TIME!

    Saturday, February 10, 2007 at 3:53 pm | Permalink
  5. B. Collins wrote:

    excellent. you got me up and running. built a chowhound filter for cambridge restaurants.

    Saturday, February 10, 2007 at 11:14 pm | Permalink
  6. illovich wrote:

    Thanks for the tutorial – visual patching always confises me – but I realized that it’s more that I can’t “visualize” what each patch will do, even though I can conceive of it programatically (sometimes anyway, lol) – my main problem has been getting figured out what each piece is generating, and then what some of the pieces are for.

    Again, great tutorial. Can you make one for Quartz Composer now? =P

    “Now how do i publish it on my site? Havent got that far yet…”

    If you go to your main pipes page, you can “run” your pipe (don’t know why you can’t run from the pipe page, maybe in the future). If you run your pipe (this is covered in the tutorial) the resulting page will have an RSS and JSON link at the bottom.

    There’s a couple places you can go from here (hopefully there will be a way to generate a widget/webbadge of a pipe in the future), but you might look at the JSON bit (http://www.json.org/js.html) which would allow you to parse the JSON feed in javascript on your page, and write it out however you want.

    I’m still trying to figure that out myself, but I’ve seen tons of pages on it on the various javascript resource sites.

    Sunday, February 11, 2007 at 1:00 am | Permalink
  7. Mr. Speaker wrote:

    In regards to publishing the results of pipes – the Pipes devs have hinted that they are working on some simple ways to embed them in your pages. I’ve never tried working with rss feeds, so I’m not 100% sure the best way to do it – I reckon it’d be pretty easy though – I’m sure there are tutorial out there already.

    The only way I’ve used the results so far is in this greasemonkey script – I use the JSON output from the Pipe to do groovy stuff client side.

    Monday, February 12, 2007 at 1:22 pm | Permalink
  8. JSON output is easy to get from Pipes; you want to add _render=json and (optionally) _callback=yourCallback. I’ve just put up a working Web badge to return Technorati search results via Pipes; source is online and should be fairly easy to implement and understand.

    Wednesday, February 14, 2007 at 9:50 am | Permalink
  9. Don Meader wrote:

    How much damage might do to whatever else I now have on mp personal PC ????????????

    Thursday, February 15, 2007 at 10:22 am | Permalink
  10. Don Meader wrote:

    How much damage might I be able to do, & to what ???

    Thursday, February 15, 2007 at 10:24 am | Permalink
  11. notAspammor wrote:

    very good tutorial. From the yahoo page, I didn’t even know what pipes were. Now I can start trying them out. Thanks.

    Saturday, February 17, 2007 at 12:30 am | Permalink
  12. jc wrote:

    This was a great tutorial. It helped me get started. I also created a nice series of video tutorials. With your permission, here is the link:

    http://usefulvideo.blogspot.com/2007/02/yahoo-pipes-tutorials.html

    Monday, February 19, 2007 at 6:40 am | Permalink
  13. Tony Hirst wrote:

    One way of getting an interface to a Pipe is to use a Grazr widget (http://grazr.com) fed by a small Grazrscript file to generates the search box (http://docs.grazr.com/script/tutorial/).

    I have posted a small example at http://blogs.open.ac.uk/Maths/ajh59/009318.html, which offers a Grazr interface to an Amazon book search pipe.

    tony

    Monday, February 19, 2007 at 7:55 pm | Permalink
  14. Tony Hirst wrote:

    Here’s a really quick’n'dirty GRazr interface generator for a yahoo pipe with a single text entry box (such as a search pipe). Simply run a pipe with a magic search term, take a copy of the rss feed url, paste it into a form and get a grazr interface to your search pipe

    http://ouseful.open.ac.uk/grazr/gPipeTest2.php

    tony

    Tuesday, February 20, 2007 at 2:16 am | Permalink
  15. ovi wrote:

    Please have a look at http://soarack.blogspot.com/ for an alternative to yahoo pipes

    Wednesday, March 14, 2007 at 2:45 pm | Permalink
  16. morganusvitus wrote:

    The site looks great ! Thanks for all your help ( past, present and future !)

    Thursday, April 5, 2007 at 11:16 pm | Permalink
  17. Jeff wrote:

    Couldn’t get your second tutorial to work. The filter would only recognize one of the feeds’ items (the one that was connected furthest to the left of the union operator). I had to have two filters going into one union… Must be an issue with the pipes. Thanks for the tut though!

    Wednesday, June 13, 2007 at 4:10 am | Permalink
  18. Mr. Speaker wrote:

    Hmm, sounds a bit odd Jeff – do you have a link to the pipe, I could take a look if it’s giving you grief!

    Wednesday, June 13, 2007 at 5:03 pm | Permalink
  19. mattechi wrote:

    I’ve had success adding multiple feeds into one Fetch Feeds module without using a Union module. Is there a reason why I should use one way versus the other?

    Friday, June 15, 2007 at 7:10 am | Permalink
  20. Mr. Speaker wrote:

    You are correct mattechi – It’s a lot nicer to just add multiple feeds without the union. I only included the union here to demonstrate the ideas.

    The only reason you’d do it this way is if you want to process or filter the feed BEFORE you merge it with the others.

    For example if you have one feed that has heaps of results, you might want to only get the top 10 of those before you merge it with others, to prevent the final feed being over-run with the first feed’s items. Or something.

    Friday, June 15, 2007 at 9:19 am | Permalink
  21. Bob wrote:

    Wow, love this tool! But how do I publish the results onto my web page to use the content to enhance my SEO?

    Tuesday, October 30, 2007 at 5:27 pm | Permalink
  22. Rob wrote:

    This is a great step by step tutorial, thanks for the effort, it really helps a noob like myself :)

    Friday, January 11, 2008 at 9:21 am | Permalink
  23. Tony Loton wrote:

    Anyone interested in Yahoo! Pipes might be interested in my two books “Working with Yahoo! Pipes, No Programming Required” and “Mashup Case Studies with Yahoo! Pipes”.

    http://www.lotontech.com/it_books.htm

    Friday, June 6, 2008 at 8:40 pm | Permalink
  24. May Ong wrote:

    Hi

    Do you have any Tutorial on How to make these Yahoo Pipes useful in generating Content for hosted WordPress blogs?

    Tutorial on (a) a WordPress page, named News, that generates News content related to specific keywords

    Tutorial on (b) a WordPress side bar widget, that also generates News content via RSS

    I would appreciate these Tutorials, please.

    Thanks. Your post is extremely helpful.

    May Ong

    Wednesday, July 2, 2008 at 5:44 pm | Permalink
  25. web man wrote:

    I would like to suggest a great new site that organizes your RSS feeds.
    It employs a bayesian filter for RSS feeds where you can train the filter what you like and
    what you don’t like. It’s free, try it at http://www.filteredrss.com.

    Wednesday, July 16, 2008 at 1:16 am | Permalink
  26. How can i use it to translate a hole site ?

    Monday, July 28, 2008 at 5:11 pm | Permalink
  27. Inquire_98 wrote:

    Hello,
    Great tutorial, but I had a problem getting started.

    I could not find “Fetch”. I see 6 selections of “Fetch (CSV, Feed, Data, Page)”, but not just Fetch.

    Please explain…

    Thank you,

    Wednesday, December 2, 2009 at 12:43 pm | Permalink

4 Trackbacks/Pingbacks

  1. Pasha Sadri's Blog on Saturday, March 31, 2007 at 2:01 pm

    Crazy 72 hours

    It’s been almost 72 hours since our beta launch. Needless to say, the response to Pipes has been overwhelming for everyone on the team (and our servers!). I have finally had a chance to relax/sleep after addressing some of the immediate issues….

  2. [...] http://www.mrspeaker.net/2007/02/10/yahoo-pipes/ [...]

  3. Mashups with Yahoo Pipes | Dave Wallace on Monday, March 7, 2011 at 4:05 pm

    [...] http://www.mrspeaker.net/2007/02/10/yahoo-pipes/ [...]

  4. Week 3 | Exploring interfaces on Wednesday, March 14, 2012 at 8:07 am

    [...] http://www.mrspeaker.net/2007/02/10/yahoo-pipes/ [...]

Post a Comment

Captcha! Please type 'radical' here: *
How did you find this thingo? *