AutoHotkey Homepage AutoHotkey Community
Let's help each other out
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

getting HTML source of a webpage without openning it
Goto page Previous  1, 2, 3  Next
 
Post new topic   Reply to topic    AutoHotkey Community Forum Index -> Ask for Help
View previous topic :: View next topic  
Author Message
engunneer



Joined: 30 Aug 2005
Posts: 6847
Location: Pacific Northwest, US

PostPosted: Tue Jul 08, 2008 4:22 pm    Post subject: Reply with quote

it sounds to me like you downloaded the linux version of the program (they typically have no file extension, hence the "empty" icon)

It seems you are missing a few basic pieces of information that I and others were assuming you had. I can try to write up a how-to for installing libcurl, but I am not able to do so immediately.
_________________
Unless otherwise noted, all code is untested.
Common Answers: 1.(Loops, Viruses, etc.) 2. Search 3.RTFM
Back to top
View user's profile Send private message Visit poster's website
Benny-D



Joined: 29 Feb 2008
Posts: 137

PostPosted: Tue Jul 08, 2008 10:14 pm    Post subject: Reply with quote

engunneer wrote:
It seems you are missing a few basic pieces of information that I and others were assuming you had. I can try to write up a how-to for installing libcurl, but I am not able to do so immediately.


It's okay, take your time. Thank you for your willingness to help me, anyway.

I just discovered something that may shed some light on my situation.

In ''Win32 2000/XP'' file that I downloaded (and unzipped) from the package list on cURL Downloads page (the name of the package: Win32 - Generic; version: 7.18.2; type: libcurl; provider: Günter Knauf; size: 1.54 MB), if I add '.txt' extension to the names of some of those ''empty-icon'' files like ''README'', ''RELEASE-NOTES'', ''COPYING'', ''libcurl.pc'', they turn into normal notepad icons and I can open them and read their content! (The text looks perfectly well. For example, I can see the ''cURL logo'' made of underlines and slashes in the header of ''README'' file).

I tried to do the same thing with some ''empty-icon'' files in ''curl-7.18.2'' file (downloaded from Source Archives section on cURL Downloads page), and the result was a little bit different: the icons of files still changed into notepads, but the text inside of those files didn't look nice when I openned them. Some lines were shifted, there were some black squares in the middle of lines (the ones that appear when you miss either `n or `l when making a text file in AHK), and, of course, the the ''cURL logo'' in the header of ''README'' file was all twisted.

It's so strange! Why do I need to add extensions manually to each file's name? It seems that some important Windows system files are missing on my computer.
Back to top
View user's profile Send private message
HugoV



Joined: 27 May 2007
Posts: 650

PostPosted: Wed Jul 09, 2008 7:40 am    Post subject: Reply with quote

Benny-D wrote:
It's so strange! Why do I need to add extensions manually to each file's name? It seems that some important Windows system files are missing on my computer.
You don't need to. You downloaded the libcurl.dll so there is nothing to click on (nothing will happen if you double click or open the DLL). You don't need the DLL anyway because it is too
complicated.

You don't need the source either.

You need curl.exe which is here:
http://curl.haxx.se/download/curl-7.18.2-win32-nossl.zip

Unzip that file and you will have a curl-7.18.2 FOLDER, open
that folder and you will see curl.exe there. The other files
are plain text files which you can open in your word processor
or browser.

If you want to use curl.exe place it in the same directory as
your script and use the AHK run command and correct CURL
command line parameters (check the CURL manual for that
which can be found in the curl-7.18.2\DOCS folder)

May I ask: how old are you?
Back to top
View user's profile Send private message
taotesea



Joined: 08 Jul 2008
Posts: 1

PostPosted: Wed Jul 09, 2008 8:55 am    Post subject: Reply with quote

Very Happy
_________________
http://taotesea.spaces.live.com
Back to top
View user's profile Send private message Visit poster's website
Benny-D



Joined: 29 Feb 2008
Posts: 137

PostPosted: Wed Jul 09, 2008 2:08 pm    Post subject: Reply with quote

HugoV wrote:

Unzip that file and you will have a May I ask: how old are you?


Thank you for your instructions. I am about to follow on them.

I am 34. Why are you asking? Is it because my questions are so silly?
Back to top
View user's profile Send private message
HugoV



Joined: 27 May 2007
Posts: 650

PostPosted: Wed Jul 09, 2008 10:09 pm    Post subject: Reply with quote

No specific reason, just wondering Wink

Not to complicat things, but have a look at this
http://www.autohotkey.com/forum/viewtopic.php?t=33506
as it might help with an all AHK solution (e.g. no need to
install curl, wget etc)
Back to top
View user's profile Send private message
Benny-D



Joined: 29 Feb 2008
Posts: 137

PostPosted: Thu Jul 10, 2008 5:04 am    Post subject: Reply with quote

WOW!!! HugoV, thank you so much for all the links. To tell you the truth, now I am perplexed - should I stick to AHK only or try to continue exploring cURL way? Both ways look quite luring. Perhaps, I'll try both ways and see which one is better.

So far, the only thing that I need is to be able to collect HTML codes of those pages (of a website) that I can only enter by providing my password , and I want to do it automatically - without openning pages by any web browser.

Anyway, thank you very much again.
Back to top
View user's profile Send private message
tank



Joined: 21 Dec 2007
Posts: 1033

PostPosted: Thu Jul 10, 2008 3:28 pm    Post subject: Reply with quote

much as i like to try defer people to IE7 scripts
the curl aproach for your described need would be more versitile unless you never plan on using https sites

However if not https sites and you dont mind the learning and writing your own functions to extract what you need then yes by all means in the very end the link hugoV osted is the very best method of doing what you want
_________________
Read this
Com
Automate IE7 with Tabs
Back to top
View user's profile Send private message
Benny-D



Joined: 29 Feb 2008
Posts: 137

PostPosted: Thu Jul 10, 2008 5:17 pm    Post subject: Reply with quote

Thanks for this remark. Sooner or later I'll have to move on to using https sites anyway; so I think I better take the cURL way.
Back to top
View user's profile Send private message
Benny-D



Joined: 29 Feb 2008
Posts: 137

PostPosted: Thu Jul 10, 2008 6:35 pm    Post subject: Reply with quote

HugoV wrote:
If you want to use curl.exe place it in the same directory as your script and use the AHK run command and correct CURL
command line parameters (check the CURL manual for that
which can be found in the curl-7.18.2\DOCS folder)


How do I do it? I don't think the following AHK code would be correct:
Code:
Run, curl http://www.netscape.com/
(it didn't work for me)
Do you mean using DllCall?

And also, how would I run commands in cURL without AHK? For example, if I want to download the above-mentioned page (http://www.netscape.com/), where should I place ''curl http://www.netscape.com/'' line (command) to have it run?

And then, after it was downloaded, where should I be looking for the downloaded page (or its HTML code)? In my clipboard?
Back to top
View user's profile Send private message
Benny-D



Joined: 29 Feb 2008
Posts: 137

PostPosted: Fri Jul 11, 2008 12:31 am    Post subject: Reply with quote

I get the feeling that I don't know the very basics of cURL -- I simply don't know how to run commands there -- and these basics are so basic, they are not even explained there.

It was much easier with AHK -- such things like how to create your first AHK file and run it were explaind clearly and in detail in the Quick Tutorial of the AHK Help file.
Back to top
View user's profile Send private message
HugoV



Joined: 27 May 2007
Posts: 650

PostPosted: Fri Jul 11, 2008 6:47 am    Post subject: Reply with quote

In the curl MANUAL, you will see on the first page a section DOWNLOAD TO A FILE which will get you started. Also you might need to check your firewall and allow curl internet access. Also note the link I provides does not allow you to access https sites, you would need the SSL version of curl (see the curl download page). The curl documentation is quite good, so read it a few times.
Back to top
View user's profile Send private message
Slanter



Joined: 28 May 2008
Posts: 397
Location: Minnesota, USA

PostPosted: Fri Jul 11, 2008 7:11 am    Post subject: Reply with quote

This may seem a little off-topic at this point, but I promise, it may simplify things for the original aim of this script Laughing

Are you using firefox to log in to the website? UrlDownloadToFile uses IE, so in order for the cookies to be set to automatically log in, you would have to first log in using IE. Try doing this, if it doesn't work I apologize for wasting time, but if it does you won't even need to mess with cURL, you'll just have to re-login via IE every once in a while (like... once a year or something).
_________________
Unless otherwise stated, all code is untested

(\__/) This is Bunny.
(='.'=) Cut, copy, and paste bunny onto your sig.
(")_(") Help Bunny gain World Domination.
Back to top
View user's profile Send private message Visit poster's website
tank



Joined: 21 Dec 2007
Posts: 1033

PostPosted: Fri Jul 11, 2008 2:09 pm    Post subject: Reply with quote

in almost all cases of a web form you can automate a log in by doing somethng like
http://www.mysite.com/path/filename.html?username=myname&password=mypass

username is replaced with the name atribute of the feild collecting the username
and the same is true of the password
another way to get all the feild names in a query string is to do this

parste the source manually for the form name
consider
Code:
<form name=thisform method=post action=http://somesite></form>

you would do the following

Code:
javascript:void(document.forms('thisform').method='Get')

paste that into the address bar and then hit go

then fill out said form and submit it
in most cases the result will be a url and querystring with all the submited feild namesand values
use that url for your urldownloadtofile

there are some more advanced options available but it seems at your level it might be best to start this way
_________________
Read this
Com
Automate IE7 with Tabs
Back to top
View user's profile Send private message
Benny-D



Joined: 29 Feb 2008
Posts: 137

PostPosted: Sat Jul 12, 2008 10:26 am    Post subject: Reply with quote

Slanter wrote:
... you'll just have to re-login via IE every once in a while (like... once a year or something).


As you have rightly noted, this was the original reason why I started this thread -
I knew how to use UrlDownloadToFile command (yes, I am using IE, not Firefox)
and I did dowload in this way many free-access pages (their HTML code), but
when I stumbled upon a website where my password was needed and, hence,
the problem of having to enter my password each time, I decided to ask for help.

I tried doing this before (logging in using IE prior to running my script containing
UrlDownloadToFile command), and it didn't work at that time.

After I read this post of yours, I decided to try again, and... can you imagine, it
worked! But I am a little bit scared now - what if it doesn't work again? And also,
if I take this way I will then have to re-sign in everyday as they only have ''Keep
me signed in for today'' box (not for a week or month), but I don't want to do that,
I mean I want it to be done automatically. Plus, I am quite intrigued by cURL now.


Last edited by Benny-D on Sat Jul 12, 2008 1:57 pm; edited 1 time in total
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    AutoHotkey Community Forum Index -> Ask for Help All times are GMT
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You can post new topics in this forum
You can reply to topics in this forum


Powered by phpBB © 2001, 2005 phpBB Group