GD Experience

Here is the very first version of HTTP Crawler.

The tool is based on C++ language and implements cURL library functionalities. It lets you setup scenarios that can be run on a set of URLs to reproduce the actions a human would perform when using browsing a web site. The tool runs currently as a win32 version but can be compiled to any platform thanks to cURL's portability.

The setup of the tool is very simple. You start by creating 1 scenario file per request you want to submit to the web server and link them together with the NextStep setting. As long as the bot finds a NextStep variable, it will move onto the next scenario file... until it finally gets to an empty setting. The 2 scenario files below are processed by the bot and simply login to GDE web site and then display a normal page (sessions & cookies are persistant throughout the bot scenario)

Each scenario file can handle GET or POST requests. When you specify a POST request via the ReqType setting, you define the different POST elements with the Params variable.  The example below shows how to get passed the login form of this website

HTTP Crawler features :

• Supports HTTP & HTTPS
• Supports Microsoft integrated auth system (activated on IIS servers)
• Generates log files in an output folder with HTTP response code, detailed response times and pages code which let you check the crawling is working correctly

Command line settings : HttpCrawler.exe <crawler_config> [repeat_frequency repeat_count]

• crawler_config : mandatory field to indicate which is the first scenario file the bot has to load. Links to the next scenario steps are done within the configuration files (via NextStep setting)
• repeat_frequency : optional field used to tell the bot to loop the scenario every x seconds
• repeat_count : optional field used in conjunction with repeat_frequency to indicate how many times to bot has to loop the scenario

A sample bat file is provided with HttpCrawler. Please post any questions related to HttpCrawler in the forums section.

Download HTTPCrawler v1.0 Now

HTTPGetGui is a tiny tool designed for single url monitoring. It is based on cURL and MFC libraries and it configured through a simple configuration file. In the current version, HTTPGetGui supports HTTP and HTTPS protocols (must be specified in the config file) in addition to the IIS Windows integrated auth feature.

The configuration file lets you specify an URL, the http or https switch, the IIS domain/user and password (if applicable). HTTPGetGui will then loop at the specified frequency and attempt to reach the url you wish to monitor. If the response time if below the timeout value, the tool will raise a popup window to inform the user of the abnormal response time.

Finally, all the tool activity is traced in a log file for further analysis.

Download HTTPGetGui Now