Ask the Expert

Automating Web site download process

I am a newbie programmer, primarily working with ASP for use in a mortgage company which I own. I am in search of advice for a project that I am working on. I have checked numerous sites including searchWin2000 without success. I am hoping you may be able to give some direction.

I want to automate the downloading of information from a Web site. Currently, I open a browser and type in the following:
http://drawbridge-red.gte-es.com/drawbridge/nabor/drawbridge.dll?GetListings &user=kMCfnNAB&pass=netKworK&vendor_id=Kmc870&product_id=DrKMCfn&property_ty pe=0&status=5&max_rows=20&TranslateOptions=TRUE&date_update=02/08/2001

Requires Free Membership to View

This opens the download manager. I then specify the location to download to. At that point, a flat text file is downloaded to my server's hard disk. From there, I process the text file using SQL Server 2000, which in turn links to numerous ASP applications.

What I am trying to accomplish is automating the actual downloading of this text file. Once I have that program written, I would schedule it using NT Scheduler or SQL Server Agent, thus giving me a completely automated function. Unfortunately, to this point, I have been unsuccessful in finding a solution to automating the download. Any direction you give would be enormously appreciated.


If you are a new programmer, I want you on my team! It sounds like you are doing some interesting development and you are making things happen. Instead of using the browser to download the data, you need to make a small application to do it for you. That way you can control exactly how the data is downloaded. I would recommend that you look at using Visual Basic for the task. It is relatively easy to learn and if you can handle ASP, you can handle Visual Basic. In order to make the downloading easy to program, I would also recommend getting a third party library that encapsulates HTTP and FTP for you. There are numerous companies developing them and any VB related site or magazine has plenty of ads you can use to evaluate. After you get your small download program written, you can also use the scheduler to automate it to run before the SQL 2000 processing.


This was first published in February 2001

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: