Some background: The current Pipes engine is nearly 3 years old and was created quite quickly to see if the Pipes idea would be successful. It was. Wildly so. Its been a good stretch for that software and the system design, but the time has come to put something else in its place. During the last year the Pipes team has been developing a second generation engine, which is used in the YQL product (developer.yahoo.com/yql). Its much more stable, performs far better that the current engine, and we've been working on migrating the old Pipes-specific functionality out of the v1 engine to this new engine. We're at a "beta" stage now, where most of the big bugs are out, and we're shaking down the system to see what we've missed. Some things to note:

• strict equality between the output of Pipes 1 and Pipes 2 is not a requirement. However, the Pipe should run and give output that makes sense.
• there will be bugs, this isn't ready for primetime.
• the mechanism to run Pipes on the new engine is given to you in confidence.
• the Pipes 2 engine is a new "baseline" for new functionality, not the end goal. It gives us a scalable platform to build features and capabilities which we've been unable to add to the old one, including more sophisticated quota management, Pipe stat tracking, caching and enhanced modules (like scripting).

What we'd like to get from you:

• general feedback. It works the same, it doesn't, etc
• specific bugs. What doesn't work. Please give us the Pipe ID and if you can pinpoint where things are going awry the better. Once you find a bug (for example with a specific module) it will probably be common to many Pipes so please try and avoid duplication.

Please post your feedback here.

So how do you get going?

• Its easy - First login to Pipes, then use the upgrade links on the pipe.info pages or in the Pipes editor.
• You can only upgrade your own Pipes.
• If you see a Pipe you want running in V2 but do not own it, you can clone it and then upgrade it.

What's next:

• Auto-upgrade all Pipes that work the same in both engines.
• After a number of months we'll retire the old engine

Thanks for trying out the V2 engine!

The Pipes Team

If anyone has been using YQL execute to power the Pipes webservice module, in your execute statement JSON.stringify is no longer needed.

Our previous post about using YQL to power the Pipes webservice module has been updated: http://blog.pipes.yahoo.net/2009/06/01/using-yql-execute-to-power-the-pipes-webservice-module/

We've been working with Craigslist over the last week to see how both products can better work with each other. After providing more information about how to track and ban Pipes that violate their TOS (or are considered generally abusive), they have opened Craigslist RSS feeds back to us.

Pipes usage has grown exceptionally since its launch nearly three years ago. Unfortunately, the original architectural design and implementation of the engine has been showing it's age for a while now.

To address these we've been actively working on moving from the current engine to a new data flow engine based on the one YQL uses for some time now. We've talked a little about this before.

As Pipes has become more popular so has amount of both deliberate and accidental abuse on the system (abuse=running large numbers of complex pipes with frequently changing parameters). We've solved many of these, and are working on others. jQuery, for example, cache-busts on every request so those of you using that library to fetch a Pipe are getting blocked way more than you should (jQuery should really add an "if IE" statement before doing that).

We're making great progress with the new engine and we'll be sending out information to a number of you in the next few weeks to start testing it. In the meantime we're continuing to upgrade other parts of our infrastructure and many of these issues should start to go away, but won't be remedied until the new engines comes fully online.

If you use Pipes a lot and would like to be involved in testing the new engine, please send an email to pipes-questions [at] yahoo-inc.com

As many of you are aware, Craigslist recently started blocking Yahoo Pipes. We're currently working with them to establish why they did this and what can be done. In the meantime here are several alternative ways to use Pipes and Craigslist together:

1. As of this writing, you still can use the YQL module to fetch craigslist RSS feeds.

In the YQL module use this example statement:

select * from feed where url = "http://sfbay.craigslist.org/apa/index.rss"

2. You can also use Feedburner to act as a proxy.

Create a Feedburner account and burn the craigslist RSS feed.

Example feed:
http://feeds.feedburner.com/craigslist/OvXO

3. Create your own proxy using script hosting sites.

http://pipesproxy.jgate.de/?url=http://sfbay.craigslist.org/rea/index.rss

jgate.de is similar to what appjet was.

http://www.scriptlets.org also works, but as of this writing has issues.

4. You can also host your own proxy (or use appengine) and use that URL as your fetch feed link.

The Pipes and YQL team is looking to expand! Passionate about Pipes? Want to be part of a great team and be involved in the evolution of Pipes and YQL? Got Java skills?

For more information see http://careers.yahoo.com/jdescription.php?frm=jsres&oid=25394

Please see this posting: http://developer.yahoo.net/blog/archives/2009/08/term_extraction_stays.html

The Term Extractor Module will be moved out of the deprecated folder back into the String folder shortly.

Attention Pipe Developers and Users.

The Term Extractor module listed in the String folder in the Pipes editor will be discontinued and cease to work after August 31st. It will be moved to the Deprecated folder shortly. This is the only module that will be affected at this time.

The closing of the Term Extractor module is due to the closing of Yahoo's Term Extraction web service. (http://developer.yahoo.com/search/content/V2/termExtraction.html)

Please modify your Pipes that use this module and look to external services that may do the same, or create your own and tie it in via the Web Service module. For some Pipes, the Yahoo! Shortcuts module may be a suitable replacement.

As many of you might have seen PopFly (and previously Google Mashup Editor) are in the process (or have already) shut down. Its a great shame since as we know *a lot* of the applications created on these visual mashup editors used Pipes as their source of data. Indeed, everything starts with data...

The good news is Pipes continues to grow and is going strong and there are lots of cool ways you can use Pipes on your site - whether its using our built in map, image or list widgets, or via other visual widgets like widgetbox, or creating your own applications directly.

We've been getting some questions about what's going on with Pipes these day from various blogs and our message boards, so here's a small update about its progress and what we're working on.

Since its v1 launch two years ago, Pipes has had over 24 feature releases. The Pipes system of today is quite different from its first version, with a sizable number of significant new features and fixes. Some of the things we've changed and added include 25 new modules, favorites, new output formats, search, private fields, a variety of badges, a new web site and so on.

Along the way, we've learned a lot and collected feedback from all of you. We're aware that there are currently some bugs and performance issues, and we're working hard to correct them. We understand your frustration and updating the underlying Pipes' data engine and infrastructure to address these issues is a priority for us.

Today, Pipes serves hundreds of millions requests monthly and its usage continues to grow. As part of the Yahoo! Open Strategy, we're excited about the ongoing potential for Pipes and can't wait to see what developers like you use it for next.

Thanks to everyone who uses Pipes. We value developer feedback greatly, so keep it constructive and keep it coming - ­ it helps us make Pipes better.