Posts Tagged ‘Yahoo! Pipes’

Weekly Link Roundup – 2

Links published on May 04, 2009:

Math world, simple mental calculations or what’s going on with education?

Most interesting math blog post I have read for a long time. Have a read guys and tell me how you find it.

Programmer Competency Matrix:

Panoramic Image & Viewer related resources:

File conversion utility for panoramic photos:

Automatic image stitching:

Panoramic Image App (Commercial):

Panorama Gallery:

Panoramic Viewer:

TDD Links:
A Conversation with Martin Fowler
Mocks Aren’t Stubs
Learn about .NET Unit Testing with Typemock

Old Version Downloads:

10 Youtube URL Tricks You Should Know About

100 Best JavaScript Resources

Performance killer: Disk I/O

Top 20 Best Agile Development Books, Ever

Top 100 Best Software Engineering Books, Ever

Top 20 Programming Books

Top 10 Concepts That Every Software Engineer Should Know

Securing ASP.NET Applications

The ADO.NET Entity Framework must read:

Introducing the Entity Framework [MSDN]
The ADO.NET Entity Framework Overview
ADO.NET Tech Preview: Entity Data Model

Introduction to the Entity Framework [Code Project]

Windows Communication Foundation must read:

Fundamental Windows Communication Foundation Concepts
What Is Windows Communication Foundation?

A Guide to Designing and Building RESTful Web Services with WCF 3.5

Howto Dynamically Insert Javascript And CSS:

This is a short and sweet little tutorial to show you how to dynamically insert a new Javascript (or style sheet) into your web pages, and explores how you can exploit cross-domain includes in your applications.

As JSON and DHTML start to get pushed more and more to the forefront of web-based applications, the web designer is faced with a new problem: how to dynamically insert a script element into an existing web page. It won’t take long to figure out that ajax loads and innerHTML injections won’t work.

Yahoo! Pipes and JQuery : Goodbye Same Origin Policy

URL Shorteners:

Interview Tips:

LINQ to Entities:

LINQ Learning Guide: LINQ to Entities,295582,sid8_gci1299600,00.html

The ADO.NET Entity Framework Overview:

Parallelism Videos Galore

Writing LINQ Queries with the Entity Framework:

ASP.NET Dynamic Data Overview

Inheritance and Associations with Entity Framework Part 1

Model Domain Objects with the Entity Framework

Lambda the Ultimate : Programming Languages Weblog

Introduction to Domain Specific Languages

Good blog site on Patterns:

Pattern site on Hexagonal architecture

JSINQ – LINQ to Objects for JavaScript

JSINQ: LINQ style access for JavaScript objects

LINQ to SQL: SQL-CLR Type Mismatches (LINQ to SQL)

LINQ to SQL : Object Identity (LINQ to SQL)

Most interesting math blog post I have read for a long time. Have a read guys and tell me how you find it.

Scraping your way to RSS feeds!

I was looking for a way to get regular updates from a job site about a particular category even though the site doesn’t offer any sort of feed.

Then I stumbled upon a site called

What I basically did was to have an RSS feed ready for the site. It’s pretty elementary with the help of the above site really. You don’t even need to register in order to create an RSS feed for a certain site.

Only problem was that I didn’t have the RSS feed in XML format. I had to go to the web site to view so. Also the feed couldn’t really be customized in any ways.

There’s another site named which does pretty much the same. Mind you none of the above sites are perfect yet they do a reasonable job of it.

So I googled a bit more and stumbled upon which let me actually write expression for creating the feed.

Here‘s what I came up with as an RSS feed version of this page. It lets you use ‘search patterns’ using regular expression and ‘output templates’. It’s a handy site even with all its limitations for unpaid package like polling intervals, maximum feed limit etc. Do give it a try.

I know there are several good articles like Creating a generic Site-To-RSS tool, When RSS Fails: Web Scraping with HTTP and How To: Scrape a Web Page to RSS Feed for doing the kind of the same.

What’s more I don’t know if you know this but both Yahoo! and MSN provides search result in RSS format.

Here’s the result using Yahoo! web search service for ASP.NET MVC and here‘s MSN’s version for the same.

But of course, it would help if Google was to have an XML feed of their normal Search engine positioning (SERPS) like Yahoo! & MSN do.

What it does provide though is an RSS feed for searching blogs. Try this.

There’s another gem I figured which actually lets you run XPath query for scraping into a web page for RSS. It can be used to search in an HTML document in a pretty straightforward way.

Well this has been a very long ride for scraping your way to another site but what if you want to stop others doing the same :). Enough of RSS Scraping, Scavenging, Stealing, and Content Theft, no? Talk about having a dose of one’s own medicine, right?

Anyway, have a look at What Do You Do When Someone Steals Your Content or better still have a read about the antonym of Scraping in IT terminology Information Trapping.

To wrap things up, do remember there are words like Copyright and Intellectual property / Intellectual Property Protection in the dictionary :). So use it in a positive way and enjoy the Scrapventure!

Update on 9th April, 2009: It was unfair on my part to leave off tools like Yahoo! Pipes and While Yahoo! Pipes is a less than straightforward means to achieving our objective, it has powerful features like Visual query development which are missing from the rest. But I think what makes Yahoo! Pipes unique is that you can chain together arbitrary number of previous queries (pipes) and thus mash them up into one which would have all your filters/queries. It also provides input facilities. More on Yahoo! Pipes later on subsequent post perhaps when I would guide you through the process., on the other hand, is a very straightforward means to achieving what we want. It’s quite efficient and intelligent with parsing too. Give it a try.

Update on 16th April, 2009: Microsoft Popfly mashup creator is another candidate for honorable mention 🙂