We've been getting some questions about what's going on with Pipes these day from various blogs and our message boards, so here's a small update about its progress and what we're working on.

Since its v1 launch two years ago, Pipes has had over 24 feature releases. The Pipes system of today is quite different from its first version, with a sizable number of significant new features and fixes. Some of the things we've changed and added include 25 new modules, favorites, new output formats, search, private fields, a variety of badges, a new web site and so on.

Along the way, we've learned a lot and collected feedback from all of you. We're aware that there are currently some bugs and performance issues, and we're working hard to correct them. We understand your frustration and updating the underlying Pipes' data engine and infrastructure to address these issues is a priority for us.

Today, Pipes serves hundreds of millions requests monthly and its usage continues to grow. As part of the Yahoo! Open Strategy, we're excited about the ongoing potential for Pipes and can't wait to see what developers like you use it for next.

Thanks to everyone who uses Pipes. We value developer feedback greatly, so keep it constructive and keep it coming - ­ it helps us make Pipes better.

YQL has become a great way to extend Pipes. You can harvest the power of YQL by using YQL statements in the YQL module. Another way of using even more of YQL's power is to use the Pipes Webservice module. This enables you to use YQL's Execute functionality to create javascript functions that will work on your data in any Pipe.

In this blog post we'll go through the steps on how to use YQL Execute to power the Pipes Webservice Module - for the impatient, here's the example Pipe that we'll be creating.

In order to use YQL Execute, you'll need a website to host your XML file. The XML file is where YQL Execute allows you to manipulate data by using server side javascript. We can then post data to YQL via the Pipes Webservice module to allow further processing of our Pipes Data.

Some users on the Pipes message boards asked if we could add a "yindex" element that numbers each item in the output. This becomes trivial when using YQL Execute!

The first thing we will create is our XML file for YQL Execute. You can copy and paste the structure of the XML below, editing only whats inside the execute node.
The execute content is wrapped within a CDATA statement. Since we're dealing with json, we'll import Crockfords json parser and stringifier by using the y.include() method. First, we parse the json data using JSON.parse(), then loop through the json structure and add new elements by using pdata.items[i].new_element_here within the loop. "yindex" is the new item we want added to our Pipes data structure. We then assign a number to yindex within the loop and then use JSON.stringify() to bring it back together. Finally, we assign our "output" variable as our response.object. Then we want to host this XML file on a publicly accessible web host.

Now we'll set up our Pipe. We need to use the URL builder to setup the URL that the Webservice module will access.

In the URL Builder module, we'll use

• http://query.yahooapis.com as our base
• v1/public/yql as our Path element
• These name/values as our query params
  • q: use "http://paul.donnelly.org/yql/yindex.xml" as yindex; select * from yindex where data = @data
  • format: json
  • diagnostics: false

And simply wire the URL builder module into the url terminal in the Web Service module. Be sure to use query.results.result.items as the Path to item list.

Here is the example Pipe that takes the Pipes blog feed, uses YQL execute as a Webservice module to create an additional element in the Pipes data output: "yindex" which adds a number to each item in the Pipes results. Another example YQL Execute statement is: use "http://paul.donnelly.org/yql/random.xml" as ran; select * from ran where data = @data that randomizes a given feed.

It's been an awesome 2 years!

Pipes continues to be one of the most popular mashup and feed processing tools around the web. We would like to thank everyone for using our services, reporting bugs and giving great suggestions!

Here is a short list of what happened since our last birthday.

• fixed international encoding bug
• released YQL module
• released badges
• released php output
• improved caching

This year we spent quite some time working on the YQL engine and we hope to migrate Pipes onto much of that infrastructure down the road.

We would also like to publicly acknowledge and thank community member hapdaniel for providing Pipes guidance and expertise on our message boards and around the internet, as well as other members such as Tony Loton and Dawn Foster (geekygirldawn) for being such great advocates.

Unfortunately as some of you have experienced, Pipes RSS feeds have been returning a "999" error when running them through Feedburner.

As we posted on our message boards last week, we've tracked the issue down to Feedburner's recent migration onto Google's platforms (more details can be found on Feedburners status blog http://feedburnerstatus.blogspot.com/). During this migration, the IPs used by Feedburners to access feeds have changed and we were not made aware of this change before it happened. As a result, Pipes has been blocking some of their new IP's that came online. We have since added those IP's to our whitelist and we are in the process of unblocking them.

For now, If you do run your Pipes RSS through Feedburner please hold off on doing so until the IP whitelist propagates.

We just added a new source module: YQL.

YQL complements Pipes by allowing you to fetch, aggregate and mash up data from many sources using a SQL-like SELECT syntax. To find out more about YQL click here.

To use YQL, simply enter a YQL statement (select * from feed where url='http://digg.com/rss/index.xml') into the textarea. To drill down further into the result set you can use either the Pipes Sub-element module or by using projection in a YQL statement. For example: select title from feed where url='http://digg.com/rss/index.xml' returns only the titles from the Digg RSS feed. Here is an example Pipe using YQL as a source.

The YQL module has 2 output modes: Results only or Diagnostics and results. Diagnostics provides additional data such as: count, language type and more.

You can test your query in the YQL console by clicking on the "Try in the console" link. The YQL console provides sample queries and shows what data tables are available to query against.

Here are some interesting queries to get you started:

• Fetch two rss feeds, Digg and Mixx and sort them by pubDate
  select * from rss where url in ('http://digg.com/rss/index.xml','http://feeds.mixx.com/MixxPopular') | sort(field="pubDate")
• Find Flickr photos that are tagged "fog" and are in San Francisco
  select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text="san francisco, ca") and tags = "fog")

We also listened to your suggestions and made it easier to access your Pipes RSS/JSON urls by putting it on the main Use this Pipe banner menu. It should speed up access to these widely used formats.

As some of you know, fetching Twitter feeds via Pipes often produces a 406 error in our editor, and the Pipe does not run successfully.

We have been in contact with Twitter trying to resolve this problem and supplied them several test cases. Based on various tests we've performed it appears that Twitter's servers are blocking requests from our outbound caches. At the moment they are still looking into it, but until they uncover the reason any Pipes using Twitter feeds will only work occasionally.

If there is any change in status regarding this situation, we will let you know.

Thanks - The Pipes team

It's been a few months since we launched Pipes badges. We've seen them all over the web and just wanted to point out a few. Sites that use Pipes badges really have ranged the gamut from a map/photo mashup at Calistoga Inn's web site to powering Blondies home page.

• http://www.builderstudiovt.com - nice use of the Pipes photo badge.
• http://www.blondie.net/under_construction.shtml - powering Blondies blog aggregation and news.
• http://www.kingmanbicycleoutfitters.com - nice, simple cycling news aggregation feed.
• http://www.cakephp.org/planet - helping power CakePHP's super feed.
• http://www.sofobomo.org/blogs.php - this site dedicates a single page to a Pipes badge.
• http://www.calistogainn.com/directions.html - Calistoga Inn has a Pipe map badge that plots photos and places of interest on a map near their Inn.
• http://nassauwalks.org/ - really nice Pipe map badge of future places this group will meet and walk. The badge purposely loads last so that page can render so the user can read and use the site immediately.
• http://nd.water.usgs.gov/ - even the government uses a Pipe badge!

A special thanks to John Herren for an awesome post on our newly added support for serialized php output. In his post he shows how to use Pipes serialized php output and even gives benchmarks on decoding serialized php vs json.

Corvida over at ReadWriteWeb.com just recently posted an excellent compilation of useful Pipes. She broke it down into three categories - Social Submission and Aggregation Yahoo Pipes, Pricing Alert Yahoo Pipes - Catch That Deal! and Media Yahoo Pipes.

Xefer is a really cool mashup that uses Yahoo! Pipes to get twitter data and uses Googles chart to display it. Also check out Summize to see what people are saying about Yahoo! Pipes on Twitter.

Until now JSON output has been the only way to obtain all the data flowing through a Pipe. Starting today we've added a second way of getting all the data - serialized PHP Pipe output.

To get this, go to your Pipe -> More options -> Get as PHP. Right click the "Get as PHP" link to copy link location (or copy shortcut if using IE).

After obtaining the link location, you can do something as simple as this to get Pipe output data as an array into your PHP script.

// Pipes Request
$req = 'http://pipes.yahoo.com/pipes/pipe.run?_id=ZKJobpaj3BGZOew9G8evXg&_render=php';
// Make the request
$phpserialized = file_get_contents($req);
// Parse the serialized response
$phparray = unserialize($phpserialized);

To view the array in your web browser, use this piece of PHP code to help visualize the array.

print_r($phparray);

To view a code sample on how to do a request go here. To read more about serialized PHP at Yahoo!, go here.

It's then up to you on how you want to display or use the Pipe data. The possibilities are endless.

Did you ever want an easy way of putting Pipes output onto your website or blog?

Introducing - Pipes badges!

We currently have three types of badges - map, image and list. Map badges are available if there is geocoded data in the feed. Image badges are available if there are media images in the feed. List badges will always be present if there is valid data.

Read more about how to configure them here.

You'll notice that on the Pipe info pages there is a new link called "Get as a Badge."

Click on the "Get as a Badge" link and a wizard will guide you on adding it to certain destinations like your WordPress blog, iGoogle page or any web site (that supports Javascript).

For those of you that are more technically inclined, we are using a script node replacement technique that uses JSON embedded configuration (This is based on Kent Brewsters “Case Harded Javascript”).

It's Pipes first birthday and we've grown so much!

We'd like to thank our users and developers for making Pipes so successful. It really makes us enjoy getting up in the morning and coming to work.

Since our launch (way back in 07), we have been focused on listening to what you want and prioritizing those features in each release. This has meant adding over 20 new modules, redesigning our site to make Pipes easier to find and use, and numerous backend enhancements. As always, please give us your suggestions here.

It's amazing to see the variety of Pipes that people have been building using the simple visual editor we provide. These range from translating text from one language to another, to searching for the best apartments near a park, to monitoring hundreds of RSS feeds for certain topics.

In the past year we've adding some cool features such as, Flash support (tutorial link), iPhone support (wired buzz), Private Field support (external tutorial link) and Fetch Page module (external tutorial link) just to name a few.

We also have had some awesome tutorials created by our amazing users. Here are some recent articles:

• A brief reintroduction to Yahoo! Pipes - Part 1 of 5
• Yahoo Pipes - A case study of using the Regex Module

Pipes Books! Download these and print them out for your bleary eyes:

• Working with Yahoo! Pipes, No Programming Required
• O'Reilly Short Cuts - Yahoo! Pipes

We have some new cool features coming in the near future - find out about them on this blog!

In the mean time, check out these recent Pipes we've made. yahoo.search.image - by Jonathan (our fearless co-founder of Pipes) - shows how a Pipe can be used to wrap REST-style APIs and Feed Item Search that looks at the search results of any query and finds RSS items in those results. craigslist / zillow mashup - created by Paul (gui gluer) - cross references Zillow and craigslist to find out what Zillow thinks the estimated price should be in craigslist and yahoo buzz image search shows 5 images associated with each Yahoo! buzz item.