When using source modules other than Fetch Feed, the data structure is often not reflective of a RSS structure. When viewing the Pipe as RSS or on the Pipe info page, the results seem to be empty and many users ask on the message boards "why can't I see my data?" So we created 2 new modules, the Create RSS module and RSS Item Builder Module.

The Create RSS operator module makes it easy to convert an entire list of items into an RSS stream when the input data is not in RSS format, e.g., the fields are not named correctly for RSS display and output. Both the common required fields can be set, as well as the optional, but frequently used, media extensions. To rename non-RSS elements to a RSS structure, simply select an existing element name from the drop-down list provided.

This Create RSS example uses the YQL module to get the top music artists of the week. Because the data isn't in a RSS structure, we'll use the Create RSS Module to convert the data to RSS.

This is all done easily by mapping the item.element to the RSS Element you want. For example, I want item.name as my Title, item.ItemInfo.ChartPosition as my Description field, and item.website as my Link. Simply find the item in the dropdown next to the RSS Item you want it defined as.

If you want to further customize your fields, using the Regex Module is the easiest way. For example, in my description I want to prepend this text: "This weeks Chart Position: " in front the text currently being rendered as the description field. I first find the item.description field and I use "^" in the "replace" text area to signify that I want to start at the beginning of the string. In the "with" text area I enter "This weeks Chart Position: ". Check the debugger pane that it came out correctly and we're done!

With the RSS Item Builder module, you can create a single-item RSS data source by assigning values to one or more RSS attributes. RSS Item Builder can be used to create a single new RSS item from scratch, or reformat and restructure an existing item into an RSS structure. When the RSS Item Builder is used as a sub-module to the Loop module, the values from each original item can be converted to RSS. Unlike the CreateRSS module, each attribute within the RSS Item Builder can be "wired", so you can take values from other modules.

In this RSS Item Builder example, we'll use the RSS Item Builder to create a traffic RSS feed for San Francisco.

First we'll use the YQL module to get the traffic data for zip code 94123. We'll use the Loop module and embed a RSS Item Builder sub-module inside by drag and dropping it onto the target area.

We'll map item.Title to Title, item.Description to Description, we'll wire in an email address using the String Builder Module as my Author field, and map item.UpdateDate as my GUID.

We also want the media:content image URL to map to item.ImageUrl so we have that extra meta data that might be read by certain RSS readers.

Since I just want an RSS structure, I'll check off the "emit all results" radio button.

We also want to see the map of where the traffic is in our Description. Using the Regex module, select item.description in the drop down and in the "replace" textbox I use "$" which is regex for appending something to the end of the string and i'll use the String Builder again to create the image that will be appended.

Check the debugger pane and we're done!

YQL has become a great way to extend Pipes. You can harvest the power of YQL by using YQL statements in the YQL module. Another way of using even more of YQL's power is to use the Pipes Webservice module. This enables you to use YQL's Execute functionality to create javascript functions that will work on your data in any Pipe.

In this blog post we'll go through the steps on how to use YQL Execute to power the Pipes Webservice Module - for the impatient, here's the example Pipe that we'll be creating.

In order to use YQL Execute, you'll need a website to host your XML file. The XML file is where YQL Execute allows you to manipulate data by using server side javascript. We can then post data to YQL via the Pipes Webservice module to allow further processing of our Pipes Data.

Some users on the Pipes message boards asked if we could add a "yindex" element that numbers each item in the output. This becomes trivial when using YQL Execute!

The first thing we will create is our XML file for YQL Execute. You can copy and paste the structure of the XML below, editing only whats inside the execute node.
The execute content is wrapped within a CDATA statement. Since we're dealing with json, we'll import Crockfords json parser and stringifier by using the y.include() method. First, we parse the json data using JSON.parse(), then loop through the json structure and add new elements by using pdata.items[i].new_element_here within the loop. "yindex" is the new item we want added to our Pipes data structure. We then assign a number to yindex within the loop and then use JSON.stringify() to bring it back together. Finally, we assign our "output" variable as our response.object. Then we want to host this XML file on a publicly accessible web host.

Now we'll set up our Pipe. We need to use the URL builder to setup the URL that the Webservice module will access.

In the URL Builder module, we'll use

• http://query.yahooapis.com as our base
• v1/public/yql as our Path element
• These name/values as our query params
  • q: use "http://paul.donnelly.org/yql/yindex.xml" as yindex; select * from yindex where data = @data
  • format: json
  • diagnostics: false

And simply wire the URL builder module into the url terminal in the Web Service module. Be sure to use query.results.result.items as the Path to item list.

Here is the example Pipe that takes the Pipes blog feed, uses YQL execute as a Webservice module to create an additional element in the Pipes data output: "yindex" which adds a number to each item in the Pipes results. Another example YQL Execute statement is: use "http://paul.donnelly.org/yql/random.xml" as ran; select * from ran where data = @data that randomizes a given feed.

We just added a new source module: YQL.

YQL complements Pipes by allowing you to fetch, aggregate and mash up data from many sources using a SQL-like SELECT syntax. To find out more about YQL click here.

To use YQL, simply enter a YQL statement (select * from feed where url='http://digg.com/rss/index.xml') into the textarea. To drill down further into the result set you can use either the Pipes Sub-element module or by using projection in a YQL statement. For example: select title from feed where url='http://digg.com/rss/index.xml' returns only the titles from the Digg RSS feed. Here is an example Pipe using YQL as a source.

The YQL module has 2 output modes: Results only or Diagnostics and results. Diagnostics provides additional data such as: count, language type and more.

You can test your query in the YQL console by clicking on the "Try in the console" link. The YQL console provides sample queries and shows what data tables are available to query against.

Here are some interesting queries to get you started:

• Fetch two rss feeds, Digg and Mixx and sort them by pubDate
  select * from rss where url in ('http://digg.com/rss/index.xml','http://feeds.mixx.com/MixxPopular') | sort(field="pubDate")
• Find Flickr photos that are tagged "fog" and are in San Francisco
  select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text="san francisco, ca") and tags = "fog")

We also listened to your suggestions and made it easier to access your Pipes RSS/JSON urls by putting it on the main Use this Pipe banner menu. It should speed up access to these widely used formats.

Until now JSON output has been the only way to obtain all the data flowing through a Pipe. Starting today we've added a second way of getting all the data - serialized PHP Pipe output.

To get this, go to your Pipe -> More options -> Get as PHP. Right click the "Get as PHP" link to copy link location (or copy shortcut if using IE).

After obtaining the link location, you can do something as simple as this to get Pipe output data as an array into your PHP script.

// Pipes Request
$req = 'http://pipes.yahoo.com/pipes/pipe.run?_id=ZKJobpaj3BGZOew9G8evXg&_render=php';
// Make the request
$phpserialized = file_get_contents($req);
// Parse the serialized response
$phparray = unserialize($phpserialized);

To view the array in your web browser, use this piece of PHP code to help visualize the array.

print_r($phparray);

To view a code sample on how to do a request go here. To read more about serialized PHP at Yahoo!, go here.

It's then up to you on how you want to display or use the Pipe data. The possibilities are endless.

Did you ever want an easy way of putting Pipes output onto your website or blog?

Introducing - Pipes badges!

We currently have three types of badges - map, image and list. Map badges are available if there is geocoded data in the feed. Image badges are available if there are media images in the feed. List badges will always be present if there is valid data.

Read more about how to configure them here.

You'll notice that on the Pipe info pages there is a new link called "Get as a Badge."

Click on the "Get as a Badge" link and a wizard will guide you on adding it to certain destinations like your WordPress blog, iGoogle page or any web site (that supports Javascript).

For those of you that are more technically inclined, we are using a script node replacement technique that uses JSON embedded configuration (This is based on Kent Brewsters “Case Harded Javascript”).

The Pipes team spent a hectic 24 hours as part of the recent internal Y! hack day (video coming soon) creating an iPhone interface for Pipes. We thought it was so useful we wanted to get it out as soon as possible. Now everyone can access their mash-ups while mobile by going to iphone.pipes.yahoo.com. In addition to running any of your pipes and getting the output in a suitably styled interface, one of the coolest parts of our hack was integrating Pipes with the built-in iPhone mapping application. You can put any of your geo Pipe results on the iPhone just by pressing the "Map" button:

So you can display results from Y! local, as well as answering the question posed by one of our more "urgent" Pipes: the restroom locator. Some other featured pipes that work well while mobile include Live Traffic Results (what exactly is causing that snarl up), and Price watcher - for those of us standing in Frys to work out if that really is a good deal...

We suggest tagging your pipes with "iphoneapp" so we can find and feature the best mobile mashups in the future.

We've added far more advanced searching, querying, and ranking capabilities to Pipes. Now you can find the types of Pipes that interest you by refining your search on a number of criteria, including the types of output they produce, what tags they have, what data sources they are mashing up, and in any combination! Pipes are also now ranked and sorted in a new way, allowing more "interesting" Pipes to appear earlier in each search.

Simply start searching in the usual way (type some text into the search box or hit "Browse") and then you'll get a list of attribute values down the left hand side of the page that are common to one or more of the search results. These values show you how many of the items in your search contain a particular tag, or access a certain data source, or output a given type of data and so on. Clicking on a particular value (e.g. the tag) will re-search and show only those items in your original search that also have that value. You can keep refining your search by adding more and more attribute values until you find the right Pipes. Refined too much? No problem, just go to the "Current Search" box and click on the "x" next to that value to stop restricting the results that way.

When there are more than 10 values for an attribute they become hidden. Press "More +" to the right of the attribute name (e.g. Tag, or Source) to show them all.

While every release of Pipes fixes many bugs, we don't often talk about them. However the latest Pipes update addresses one of the biggest problems reported on the message boards and now Pipes supports a wider diversity language encodings and character sets found in many feeds. We still have a few remaining types to nail down but many more feeds should now work well.

Another frequently requested enhancement finally arrives on the Regex module, now our power users (you know who you are) can make use of the "m" "i" "g" and "s" flags.

We've also enabled run counting - yes, finally they're back! We've re-counted all those hanging chads, errr, runs, and updated the statistics on each Pipe. Take a look at your Pipes and see how popular they have been.

Fed up with that avatar? You can now use any picture or yourself so other developers can see how pretty you really are. You can change your picture without even leaving your Pipes page:

We setup an open crossdomain.xml file at pipes.yahooapis.com. This update allows Flash developers to get the output of a Pipe in JSON or RSS formats.

To access your Pipe from Flash you need to take the following steps:

1. Find the JSON or RSS URL for the Pipe you want to call. After you run a Pipe you can find these links in the "Subscribe" tab. Here's an example URL for the RSS feed of a Pipe:
http://pipes.yahoo.com/pipes/pipe.run?_id=5BiciiOz2xGc28CWpRr.og&query=rss&username=joshua&_render=rss

2. Replace the domain name in the above URL from "yahoo" to "yahooapis". Here's an example of the change:
http://pipes.yahooapis.com/pipes/pipe.run?_id=5BiciiOz2xGc28CWpRr.og&query=rss&username=joshua&_render=rss

It was only a matter of time before we included a module that utilized the power of regular expressions. If you're at all familiar with them, you've probably already been craving for a module like this in Pipes. Now you can use Perl-like regular expressions to help you filter and replace with more flexibility. Learn more about regular expressions or check out one of our example Pipes that uses the module.

As if that wasn't enough, we've also improved on the editor's load time so when you load up a Pipe, it should load up on the canvas faster!