We've been getting some questions about what's going on with Pipes these day from various blogs and our message boards, so here's a small update about its progress and what we're working on.

Since its v1 launch two years ago, Pipes has had over 24 feature releases. The Pipes system of today is quite different from its first version, with a sizable number of significant new features and fixes. Some of the things we've changed and added include 25 new modules, favorites, new output formats, search, private fields, a variety of badges, a new web site and so on.

Along the way, we've learned a lot and collected feedback from all of you. We're aware that there are currently some bugs and performance issues, and we're working hard to correct them. We understand your frustration and updating the underlying Pipes' data engine and infrastructure to address these issues is a priority for us.

Today, Pipes serves hundreds of millions requests monthly and its usage continues to grow. As part of the Yahoo! Open Strategy, we're excited about the ongoing potential for Pipes and can't wait to see what developers like you use it for next.

Thanks to everyone who uses Pipes. We value developer feedback greatly, so keep it constructive and keep it coming - ­ it helps us make Pipes better.

We just added a new source module: YQL.

YQL complements Pipes by allowing you to fetch, aggregate and mash up data from many sources using a SQL-like SELECT syntax. To find out more about YQL click here.

To use YQL, simply enter a YQL statement (select * from feed where url='http://digg.com/rss/index.xml') into the textarea. To drill down further into the result set you can use either the Pipes Sub-element module or by using projection in a YQL statement. For example: select title from feed where url='http://digg.com/rss/index.xml' returns only the titles from the Digg RSS feed. Here is an example Pipe using YQL as a source.

The YQL module has 2 output modes: Results only or Diagnostics and results. Diagnostics provides additional data such as: count, language type and more.

You can test your query in the YQL console by clicking on the "Try in the console" link. The YQL console provides sample queries and shows what data tables are available to query against.

Here are some interesting queries to get you started:

• Fetch two rss feeds, Digg and Mixx and sort them by pubDate
  select * from rss where url in ('http://digg.com/rss/index.xml','http://feeds.mixx.com/MixxPopular') | sort(field="pubDate")
• Find Flickr photos that are tagged "fog" and are in San Francisco
  select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text="san francisco, ca") and tags = "fog")

We also listened to your suggestions and made it easier to access your Pipes RSS/JSON urls by putting it on the main Use this Pipe banner menu. It should speed up access to these widely used formats.

This week we've got three more updates to our site: Give you and your Pipes easy to read URLs; RSS and JSON output from most pages; and quick links to various social bookmarking sites.

First, are you fed up with those impossible to remember Pipe URLs? We certainly were, so now we've implemented a way for you to create nice human readable (and memorable!) URLs for your Pipe page, and all of your Pipes. It's as simple as two clicks for yourself - find the web path URL at the top of your "My Pipes" page, click "edit", choose a nice name for yourself, and hit "Save". Done!

Once you've got a nice name, its time to move on to your Pipes. Again, it's as easy as two clicks. Go to your Pipe's page, press "edit" next to the web path shown just above the actions, type in the nice name for the Pipe, and press "Save".

Secondly, wherever you can find a list of Pipes on our site, such as our search pages or anyones "My Pipes" page, you can now get that same list as JSON or RSS. So now you can keep up to date with Pipes being created by other great developers, or mash them up in Pipes!

Finally, once you find a Pipe you find interesting, why not share it with others outside of your fellow developers? Just to left of each Pipe's run page you can find quick links to add the Pipe into various social bookmarking sites.

We've added far more advanced searching, querying, and ranking capabilities to Pipes. Now you can find the types of Pipes that interest you by refining your search on a number of criteria, including the types of output they produce, what tags they have, what data sources they are mashing up, and in any combination! Pipes are also now ranked and sorted in a new way, allowing more "interesting" Pipes to appear earlier in each search.

Simply start searching in the usual way (type some text into the search box or hit "Browse") and then you'll get a list of attribute values down the left hand side of the page that are common to one or more of the search results. These values show you how many of the items in your search contain a particular tag, or access a certain data source, or output a given type of data and so on. Clicking on a particular value (e.g. the tag) will re-search and show only those items in your original search that also have that value. You can keep refining your search by adding more and more attribute values until you find the right Pipes. Refined too much? No problem, just go to the "Current Search" box and click on the "x" next to that value to stop restricting the results that way.

When there are more than 10 values for an attribute they become hidden. Press "More +" to the right of the attribute name (e.g. Tag, or Source) to show them all.

There are some exciting improvements in Pipes to announce including additional modules and improved documentation.

We've added a new module under Sources:

• The Fetch Site Feed module uses a web site's auto-discovery links (link rel tags) to find an RSS, RDF or Atom feed. If multiple feeds are discovered, only the items in the first one discovered are fetched. This module is a simpler alternative to the Feed Auto-Discovery Module which returns feed URLs (not the content in them). Here's an example that uses the Fetch Site Feed module to retrieve the first five items from a feed. The feed is auto-discovered, with the site's URL provided by the user via a URL User Input module.

We've also added a new module under String:

• The String Tokenizer module splits a string into an array of strings. A delimiter string (often just a single character) tells the module where to split the input string. The delimiter string doesn't appear in the output list. Here's an example that connects a Text User Input module to the String Tokenizer to create an array of search terms. The term delimiter is a comma. These terms are passed into a Loop module containing a Yahoo! Search sub-module. The Search sub-module is site restricted to Wikipedia, and the loop is configured to only output the first result for each search. If you enter the string "frog,vanilla,pulp fiction" into the text user input, the resulting feed will have three items.

Finally, we've made some improvements to our Module Reference Documentation. Every Module in the Pipes Editor now has a enhanced description including details about what it does, the types of data it can input and output and links to examples.

We've made some major updates to Pipes. The big news is that we have 8 new modules to announce!

There are 2 new modules under Sources:

• The Feed Auto-Discovery module detects feed URLs that are embedded in a webpage using auto-discovery links (such as link rel tags). This example demonstrates how you could use the Feed Auto-Discovery module to find the RSS feed for a given webpage and then fetch its content.
  
• The Item Builder module allows you to create a new single item data source by defining a set of key-value attributes. This example demonstrates how you can use it to build up a new feed.
  

There are 4 new modules under Operators:

• The Loop module will run an enclosed sub-module or Pipe one time for each item in the input feed. Its output options can be set to 'emit' or 'assign' results. Emit results will replace the content of the original input item, here's an example. Assign results will insert the content into a specified attribute, here's an example. We introduced this module in order to support a wider range of modules, and data types, that can be looped over. You can use the Loop module with any module or Sub-Pipe except User Inputs, Operators, or Deprecated modules. Note: Old Skool Pipes users will notice that this module is similar to the For Each modules.
  
• The Reverse module reverses the order of the items contained in the input feed. This example demonstrates how it works.
  
• The Sub-element module will emit only the specified attribute or array for each item in the input feed. This example demonstrates how you could use it to take an RSS feed and emit only a specific item attribute for each item.
  
• The Tail module emits the last 'n' items from the input feed. This example demonstrates how you could use it to emit only the last 5 items in a feed.
  

There are 2 new modules under String:

• The Translate module automatically translates a string from one language into another. Select the language combination you desire from the pulldown menu. Note: Old Skool Pipes users will notice that this module is similar to the BabelFish module. Unlike the BabelFish module, the Translate module can be used within a Loop module and operate on any specified element attribute. This example demonstrates how you could use it to translate the Title and Description of a feed from one language into another.
  
• The String Regex module allows you to apply Perl-like regular expressions to attributes in a string. This example takes the delicious RSS feed and replaces the title for each item with its "dc:subject" element.
  

Some other updates worth noting:

• The String Concatenate module has been renamed the String Builder. It works exactly as before but if you're new to it here's an example that demonstrates how you can use it to build up strings!
• In a few cases several older modules have been Deprecated and replaced by what we consider to be improved modules. The For Each: Annotate and For Each: Replace modules have been replaced by the Loop module. And the BabelFish module has been replaced by the Translate module. The Deprecated modules will continue to work but we're discouraging their use.
• In the Editor, we've updated how you can modify the properties for your Pipe. In the past, there was a button labeled Publish in its place there's a new Properties button that brings up a dialog window that allows you to modify your Pipes name, description and tags. You now have the option to either make these changes so only you see them or to publish them along with your Pipe.
  
• And, as usual we also fixed miscellaneous bugs that have either been reported or that we stumbled upon. Thanks for continuing to share your suggestions!

On Wednesday, June 6th, Pipes will be offline for approximately 2 hours starting at 6:30 PM PST. We'll be performing a routine, scheduled update and expect regular service to return by around 8:30 PM PST. Click here to find out what time the site will be offline where you are located.

Thanks for being patient, the Pipes are flowing again! For those just tuning in, we had some unplanned downtime on Wednesday, May 17th. The service is back to normal.

Thanks to everyone who has continued to provide the Pipes team with valuable suggestions. Today, we pushed a number of improvements that we want to tell you more about.

1) You can now use subfields in the Regex and Rename Modules. It's also possible to refer to an item using dot syntax: ${published.timezone}.

If something in the preview pane is nested three levels deep, foo->bar->baz, you can refer to it as ${foo.bar.baz} in Regex replacements. Here's an example to get you started.

2) Titles in a Pipe that contain quotes and/or angle brackets no longer get mangled

3) You can now use URLs that have urlencoded characters (like %22) in the Fetch Module.

4) Modules can now contain arbitrary text. In the past, fields that contained values that looked like tags (for example: <something>) were getting unexpectedly escaped or stripped.

5) Certain BBC feeds that were broken now work.

We pushed several new updates to Pipes based on the comments we've been receiving at the Pipes Suggestion Board. They include:

* A new Split Module, for creating duplicate copies of feeds.

* Significant bugfixes to both the Filter and Regex Modules, so that pattern matching will now work correctly with Chinese and Russian text, and Unicode in general.

* A significant bugfix to the Translate Module, it should no longer mangle encoding.

* The introduction of a new URL syntax across the site (note: these changes should be backwards compatible for Pipes that users have already subscribed to.)

* The list of sources associated with a Pipe now appears on the run page.

As always, we look forward to hearing what you think!