We've been working with Craigslist over the last week to see how both products can better work with each other. After providing more information about how to track and ban Pipes that violate their TOS (or are considered generally abusive), they have opened Craigslist RSS feeds back to us.
Pipes usage has grown exceptionally since its launch nearly three years ago. Unfortunately, the original architectural design and implementation of the engine has been showing it's age for a while now.
To address these we've been actively working on moving from the current engine to a new data flow engine based on the one YQL uses for some time now. We've talked a little about this before.
As Pipes has become more popular so has amount of both deliberate and accidental abuse on the system (abuse=running large numbers of complex pipes with frequently changing parameters). We've solved many of these, and are working on others. jQuery, for example, cache-busts on every request so those of you using that library to fetch a Pipe are getting blocked way more than you should (jQuery should really add an "if IE" statement before doing that).
We're making great progress with the new engine and we'll be sending out information to a number of you in the next few weeks to start testing it. In the meantime we're continuing to upgrade other parts of our infrastructure and many of these issues should start to go away, but won't be remedied until the new engines comes fully online.
If you use Pipes a lot and would like to be involved in testing the new engine, please send an email to pipes-questions [at] yahoo-inc.com
As many of you are aware, Craigslist recently started blocking Yahoo Pipes. We're currently working with them to establish why they did this and what can be done. In the meantime here are several alternative ways to use Pipes and Craigslist together:
1. As of this writing, you still can use the YQL module to fetch craigslist RSS feeds.
In the YQL module use this example statement:
select * from feed where url = "http://sfbay.craigslist.org/apa/index.rss"
2. You can also use Feedburner to act as a proxy.
Create a Feedburner account and burn the craigslist RSS feed.
Example feed:
http://feeds.feedburner.com/craigslist/OvXO
3. Create your own proxy using script hosting sites.
http://pipesproxy.jgate.de/?url=http://sfbay.craigslist.org/rea/index.rss
jgate.de is similar to what appjet was.
http://www.scriptlets.org also works, but as of this writing has issues.
4. You can also host your own proxy (or use appengine) and use that URL as your fetch feed link.
The Pipes and YQL team is looking to expand! Passionate about Pipes? Want to be part of a great team and be involved in the evolution of Pipes and YQL? Got Java skills?
For more information see http://careers.yahoo.com/jdescription.php?frm=jsres&oid=25394
Please see this posting: http://developer.yahoo.net/blog/archives/2009/08/term_extraction_stays.html
The Term Extractor Module will be moved out of the deprecated folder back into the String folder shortly.
Attention Pipe Developers and Users.
The Term Extractor module listed in the String folder in the Pipes editor will be discontinued and cease to work after August 31st. It will be moved to the Deprecated folder shortly. This is the only module that will be affected at this time.
The closing of the Term Extractor module is due to the closing of Yahoo's Term Extraction web service. (http://developer.yahoo.com/search/content/V2/termExtraction.html)
Please modify your Pipes that use this module and look to external services that may do the same, or create your own and tie it in via the Web Service module. For some Pipes, the Yahoo! Shortcuts module may be a suitable replacement.
As many of you might have seen PopFly (and previously Google Mashup Editor) are in the process (or have already) shut down. Its a great shame since as we know *a lot* of the applications created on these visual mashup editors used Pipes as their source of data. Indeed, everything starts with data...
The good news is Pipes continues to grow and is going strong and there are lots of cool ways you can use Pipes on your site - whether its using our built in map, image or list widgets, or via other visual widgets like widgetbox, or creating your own applications directly.
We've been getting some questions about what's going on with Pipes these day from various blogs and our message boards, so here's a small update about its progress and what we're working on.
Since its v1 launch two years ago, Pipes has had over 24 feature releases. The Pipes system of today is quite different from its first version, with a sizable number of significant new features and fixes. Some of the things we've changed and added include 25 new modules, favorites, new output formats, search, private fields, a variety of badges, a new web site and so on.
Along the way, we've learned a lot and collected feedback from all of you. We're aware that there are currently some bugs and performance issues, and we're working hard to correct them. We understand your frustration and updating the underlying Pipes' data engine and infrastructure to address these issues is a priority for us.
Today, Pipes serves hundreds of millions requests monthly and its usage continues to grow. As part of the Yahoo! Open Strategy, we're excited about the ongoing potential for Pipes and can't wait to see what developers like you use it for next.
Thanks to everyone who uses Pipes. We value developer feedback greatly, so keep it constructive and keep it coming - it helps us make Pipes better.
When using source modules other than Fetch Feed, the data structure is often not reflective of a RSS structure. When viewing the Pipe as RSS or on the Pipe info page, the results seem to be empty and many users ask on the message boards "why can't I see my data?" So we created 2 new modules, the Create RSS module and RSS Item Builder Module.
The Create RSS operator module makes it easy to convert an entire list of items into an RSS stream when the input data is not in RSS format, e.g., the fields are not named correctly for RSS display and output. Both the common required fields can be set, as well as the optional, but frequently used, media extensions. To rename non-RSS elements to a RSS structure, simply select an existing element name from the drop-down list provided.
This Create RSS example uses the YQL module to get the top music artists of the week. Because the data isn't in a RSS structure, we'll use the Create RSS Module to convert the data to RSS.
This is all done easily by mapping the item.element to the RSS Element you want. For example, I want item.name as my Title, item.ItemInfo.ChartPosition as my Description field, and item.website as my Link. Simply find the item in the dropdown next to the RSS Item you want it defined as.
If you want to further customize your fields, using the Regex Module is the easiest way. For example, in my description I want to prepend this text: "This weeks Chart Position: " in front the text currently being rendered as the description field. I first find the item.description field and I use "^" in the "replace" text area to signify that I want to start at the beginning of the string. In the "with" text area I enter "This weeks Chart Position: ". Check the debugger pane that it came out correctly and we're done!
With the RSS Item Builder module, you can create a single-item RSS data source by assigning values to one or more RSS attributes. RSS Item Builder can be used to create a single new RSS item from scratch, or reformat and restructure an existing item into an RSS structure. When the RSS Item Builder is used as a sub-module to the Loop module, the values from each original item can be converted to RSS. Unlike the CreateRSS module, each attribute within the RSS Item Builder can be "wired", so you can take values from other modules.
In this RSS Item Builder example, we'll use the RSS Item Builder to create a traffic RSS feed for San Francisco.
First we'll use the YQL module to get the traffic data for zip code 94123. We'll use the Loop module and embed a RSS Item Builder sub-module inside by drag and dropping it onto the target area.
We'll map item.Title to Title, item.Description to Description, we'll wire in an email address using the String Builder Module as my Author field, and map item.UpdateDate as my GUID.
We also want the media:content image URL to map to item.ImageUrl so we have that extra meta data that might be read by certain RSS readers.
Since I just want an RSS structure, I'll check off the "emit all results" radio button.
We also want to see the map of where the traffic is in our Description. Using the Regex module, select item.description in the drop down and in the "replace" textbox I use "$" which is regex for appending something to the end of the string and i'll use the String Builder again to create the image that will be appended.
Check the debugger pane and we're done!
YQL has become a great way to extend Pipes. You can harvest the power of YQL by using YQL statements in the YQL module. Another way of using even more of YQL's power is to use the Pipes Webservice module. This enables you to use YQL's Execute functionality to create javascript functions that will work on your data in any Pipe.
In this blog post we'll go through the steps on how to use YQL Execute to power the Pipes Webservice Module - for the impatient, here's the example Pipe that we'll be creating.
In order to use YQL Execute, you'll need a website to host your XML file. The XML file is where YQL Execute allows you to manipulate data by using server side javascript. We can then post data to YQL via the Pipes Webservice module to allow further processing of our Pipes Data.
Some users on the Pipes message boards asked if we could add a "yindex" element that numbers each item in the output. This becomes trivial when using YQL Execute!
The first thing we will create is our XML file for YQL Execute. You can copy and paste the structure of the XML below, editing only whats inside the execute node. 
The execute content is wrapped within a CDATA statement. Since we're dealing with json, we'll import Crockfords json parser and stringifier by using the y.include() method. First, we parse the json data using JSON.parse(), then loop through the json structure and add new elements by using pdata.items[i].new_element_here within the loop. "yindex" is the new item we want added to our Pipes data structure. We then assign a number to yindex within the loop and then use JSON.stringify() to bring it back together. Finally, we assign our "output" variable as our response.object. Then we want to host this XML file on a publicly accessible web host.
Now we'll set up our Pipe. We need to use the URL builder to setup the URL that the Webservice module will access.
In the URL Builder module, we'll use
And simply wire the URL builder module into the url terminal in the Web Service module. Be sure to use query.results.result.items as the Path to item list.
Here is the example Pipe that takes the Pipes blog feed, uses YQL execute as a Webservice module to create an additional element in the Pipes data output: "yindex" which adds a number to each item in the Pipes results. Another example YQL Execute statement is: use "http://paul.donnelly.org/yql/random.xml" as ran; select * from ran where data = @data that randomizes a given feed.
Pipes Blog is powered by WordPress | Entries (RSS) and Comments (RSS).