Understanding URL scraping
URL scraping is the process where you automatically extract and filter URLs of WebPages that have specific features. The features that you are looking for vary depending on your goal. For example, if you are looking for a site where you can place your comment and get back link juice, you should go for WebPages that allow dofollow comments.
Techniques for URL scraping
There are many techniques that you can use to get the URL that you are looking for. Some of these techniques include:
Copy pasting: this is where you visit a given site and check whether it has the features that you are looking for. For example, if you are interested in dofollow links, you should visit a number of sites and find out if they have your target links. You should then identify the ones that have the features that you are looking for and compile a list.
Text grepping: this is a technique that allows you to search plain text on websites that match a regular expression. Although, the technique was designed for Unix, you can also use it on other operating systems.
HTTP programming: here you retrieve the WebPages that have the features that you are looking for. You should then note the URL of the pages. To retrieve the pages you have to post HTTP requests using a remote server that uses socket programming.
HTML Parser: a HTML parser allows you to mine data by detecting a common template, script or code on a specific website or Webpage. To be able to detect the script or code you have to use one of the many programming languages: HTQL, Java, PHP, XQuery and Python. Once the data is extracted, it's translated and packaged in a way that you are able to easily understand it.
DOM parsing: This is a technique where you retrieve dynamic content that has been generated by client side scripts that execute in a web browser such as Google Chrome, Mozilla Firefox or any other browsers.
URL scraping software: this is the easiest way of scraping URLs as all you need is high quality software that will do all the work for you. You should identify the features that you are interested in and then give command to the software. The software will go through all the sites on the internet and extract the URLs of the pages that have your target features.
Source: http://www.amazines.com/article_detail.cfm/6180373?articleid=6180373
URL scraping is the process where you automatically extract and filter URLs of WebPages that have specific features. The features that you are looking for vary depending on your goal. For example, if you are looking for a site where you can place your comment and get back link juice, you should go for WebPages that allow dofollow comments.
Techniques for URL scraping
There are many techniques that you can use to get the URL that you are looking for. Some of these techniques include:
Copy pasting: this is where you visit a given site and check whether it has the features that you are looking for. For example, if you are interested in dofollow links, you should visit a number of sites and find out if they have your target links. You should then identify the ones that have the features that you are looking for and compile a list.
Text grepping: this is a technique that allows you to search plain text on websites that match a regular expression. Although, the technique was designed for Unix, you can also use it on other operating systems.
HTTP programming: here you retrieve the WebPages that have the features that you are looking for. You should then note the URL of the pages. To retrieve the pages you have to post HTTP requests using a remote server that uses socket programming.
HTML Parser: a HTML parser allows you to mine data by detecting a common template, script or code on a specific website or Webpage. To be able to detect the script or code you have to use one of the many programming languages: HTQL, Java, PHP, XQuery and Python. Once the data is extracted, it's translated and packaged in a way that you are able to easily understand it.
DOM parsing: This is a technique where you retrieve dynamic content that has been generated by client side scripts that execute in a web browser such as Google Chrome, Mozilla Firefox or any other browsers.
URL scraping software: this is the easiest way of scraping URLs as all you need is high quality software that will do all the work for you. You should identify the features that you are interested in and then give command to the software. The software will go through all the sites on the internet and extract the URLs of the pages that have your target features.
Source: http://www.amazines.com/article_detail.cfm/6180373?articleid=6180373
No comments:
Post a Comment