Jump to content


Photo

dropdownlist : too many items, continuation too long


  • Please log in to reply
8 replies to this topic

#1 jyloup

jyloup
  • Members
  • 74 posts

Posted 30 June 2012 - 05:40 AM

[attachment=0]<!-- ia0 -->All_cities_from_Lastminute.rar<!-- ia0 -->[/attachment]Hello,

i've been asking some questions about the Lastminute program that i've still kept developping.
Now it's working quite fine and i'm ready to edit it on Script section.
However, just before doing it, i need to add in the GUI's departure dropdownlist all the cities allowed by this website http://www.travelage...m?idpart=944547
So, to grab all cities stored in it, I checked first the source code of website but no city were displayed on it.
But, I noticed that by typing the 3 first letters of any city in "departure" box, the website autocompletes them by displaying on a small list all cities matching with those 3 letters.

And that's cool ! because now, when small list is displayed, the source code is also changing and contains all the cities from the small list matching to the 3 letters. In source code it seems to be jQuery code, Ajax or something like that...but that's not the point.

So, inspired by a sinkfaze's code on this page :http://www.autohotke...=87247&p=542160 , i did this wily code below to grab all cities by typing automatically all triplets from AAA to ZZZ
(it took 2 complete days to run all letters (26x26x26 combinations from AAA to ZZZ), without touching computer because it sends letters directly on visible webpage so i couldn't touch other windows !
And it worked :D !! I finally got a text file with about 9200 cities stored by Lastminute website,
that is, all the cities in the world having an airport i guess...

Pwb := ComObjCreate("InternetExplorer.Application")
Pwb.Navigate("http://www.travelagency.travel/vol/open-jaw.cfm?idpart=944547")
Pwb.Visible:=True
while pwb.readystate < 4 ; waits page is loaded
sleep, 500
while !inStr(pwb.document.documentElement.innertext, "Aller/retour")
Sleep, 200

Pwb.document.aspnetForm["ctl00$m_cphMain$m_ctrlFlightSearchMini_openjaw$rbtTrip"][0].fireEvent("onclick") ; clicks on Round-trip radio
pwb.document.aspnetForm["ctl00$m_cphMain$m_ctrlFlightSearchMini$m_txtDepart"].focus()
; clicks on departure box
Pwb.document.aspnetForm["ctl00$m_cphMain$m_ctrlFlightSearchMini$m_txtDepart"].click()

; runs all letters from aaa to zzz ("a" ascii code is 97)
Loop, 26 {
	i:=A_index + 96	
	Sendinput, % Chr(i)

	Loop, 26 {
		j:=A_index + 96		
		Sendinput, % Chr(j)

		Loop, 26 {
			k:=A_index + 96			
			Sendinput, % Chr(k)
			Sleep, 4000 ; waits some seconds for the small suggestion list to be displayed
                                         ; and the source main page code to be changed

; in the source code, i noticed cities displayed in the small list were between "<A ...." tag
loop % (cities_html:=Pwb.document.getElementsbytagname("A")).length
{
; and classname of those cities were either "ac_odd ui-corner-all" or "ac_even ui-corner-all"
if (cities_html[i].classname = "ac_odd ui-corner-all") OR (cities_html[i].classname = "ac_even ui-corner-all") {
cities := cities_html[i].innertext
FileAppend, %cities%|`n, All_cities_from_Lastmiute.txt
}
		}		
			Sendinput, {BACKSPACE}
			}
			
		Sendinput, {BACKSPACE}
		}
		
	Sendinput, {BACKSPACE}  ; erases typed letters to put next ones
}
Msgbox All cities grabbed from Lastminute ! Finished !

the "|" in Fileappend, %cities%| is to be ready to include directly the resust text file in a dropdownlist in separated all_cities.ahk file :

gui, add, dropdownlist, r20 w400 vCity,
 (
 city 1 |
 city 2 |
 city 3 |
 etc...
 )

Then i include this list in the main code :
#Include all_cities.ahk
Gui, add, button, gDisplay, Show selected city
Gui, show
return

Display:
msgbox %city%
return

The script works up to about 360 cities lines, but beyond, the script returns an error :
"Continuation section too long".
But i would like to include all the 9200 cities in the dropdownlist.

I tried to divide the list with several consecutive "( .... )" pairs.
(
...
)
but it still doesn't work.
Could anyone help me to deal with this VERY long dropdownlist ? It could be very great and will
end my program. To make test, i joined the text file with all cities.

I have also a intermediate question, just for challenge cause i need it no more... ;) :

in the above code to grab cities, is there a quicker way to grab all cities, without viewing and clicking physically on the website? to make it work in background mode? (i mean by using pwb:=False), thanks

#2 sinkfaze

sinkfaze
  • Moderators
  • 6089 posts

Posted 30 June 2012 - 06:01 AM

I would put the list of cities in a separate text file and parse the text file into a dropdownlist at load time.

#3 jyloup

jyloup
  • Members
  • 74 posts

Posted 30 June 2012 - 08:02 AM

Ok thanks again Sinkfaze !!
that's exactly that and it works fine with Loop, read, Filename
(known to be quicker than Fileread)

Loop, read, All_cities_from_Lastminute.txt
{
ListAllcities.=A_loopreadline
}
Gui, add, dropdownlist, vCity w500 r20, %ListAllcities%|
Gui, Add, button, gDisplay, Display selected city
Gui, show
return

Display:
Gui, submit, nohide
CityCode:=Substr(City, -3,3)
msgbox %CityCode%
return


It takes 2 seconds to load the script, but all cities are in the list !! :D
Now i can put the complete program on script section.

#4 Lexikos

Lexikos
  • Administrators
  • 8845 posts

Posted 01 July 2012 - 03:24 AM

You're sure that is quicker than FileRead? That would be contrary to my expectation.
; Read the entire file at once.
FileRead, ListAllCities, All_cities_from_Lastminute.txt
; Remove any kind of newline.
ListAllCities := RegExReplace(ListAllCities, "\R")


#5 jyloup

jyloup
  • Members
  • 74 posts

Posted 01 July 2012 - 05:49 AM

Yes indeed, using Fileread and Regexreplace is slightly quicker (but not so) than Loop, read one.
In the help file i read about Loop, read method:

Retrieves the lines in a text file, one at a time (performs better than FileReadLine).

So i thought it was quicker also than Fileread, but i didn't think to use regexreplace, which
seems very powerful.

I did this script comparison test between your method and Sinkfaze's one (on 10 trials) and i got this :

Number of following trials : 10
SINKFAZE METHOD
Average time to load list : 374 ms
Average time to load GUI : 5461 ms
LEXIKOS METHOD
Average time to load list : 291 ms
Average time to load GUI : 4902 ms


Here is the comparison script, it should be "fair" normally.
nb_trials:=10

total_time1_Sinkfaze =
total_time2_Sinkfaze =
total_time1_Lexikos =
total_time1_Lexikos =

Fileappend, Number of following trials : %nb_trials% `n , Comparisontimes.txt

; SINKFAZE METHOD : Calculus of average loading time
Loop, %nb_trials%
{
ListAllcities = ; empties the variable at the beginning of the loop	
Gui, Add, button,, Display the selected City
Starttime:=A_tickCount
Loop, read, Allcities.txt
{
ListAllcities.=A_loopreadline
}
time1_Sinkfaze_method:=A_tickCount - Starttime
;Msgbox %time1_Sinkfazemethod%
Gui, add, dropdownlist, vCity w500 r20, %ListAllcities%|
;Guicontrol,,ListAllcities, %Listallcities%
Gui, show
time2_Sinkfaze_method:=A_tickCount - Starttime
total_time1_Sinkfaze+=time1_Sinkfaze_method
total_time2_Sinkfaze+=time2_Sinkfaze_method
;msgbox %time2_Sinkfazemethod% ms
Gui, destroy
}

EnvDiv, total_time1_Sinkfaze, %nb_trials%
avr_time1_Sinkfaze:=total_time1_Sinkfaze
EnvDiv, total_time2_Sinkfaze, %nb_trials%
avr_time2_Sinkfaze:=total_time2_Sinkfaze
Fileappend,  SINKFAZE METHOD `n Average time to load list : %avr_time1_Sinkfaze% ms `n Average time to load GUI : %avr_time2_Sinkfaze% ms `n, Comparisontimes.txt
total_time1_Sinkfaze =
time1_Sinkfaze_method =
total_time2_Sinkfaze =
time2_Sinkfaze_method =
avr_time1_Sinkfaze = 
avr_time2_Sinkfaze = 

; empties all Sinkfaze variables to not "loose time" for next loop (fair comparison)


; LEXIKOS METHOD : Calculus of average loading time
Loop, %nb_trials%
{
ListAllcities = ; empties the variable at the beginning of the loop
Gui, Add, button,, Display the selected City
Starttime:=A_tickCount
FileRead, ListAllCities, Allcities.txt
ListAllCities := RegExReplace(ListAllCities, "\R")
time1_Lexikos_method:=A_tickCount - Starttime
;Msgbox %time1_Lexikosmethod%
Gui, add, dropdownlist, vCity w500 r20, %ListAllcities%|
;Guicontrol,,ListAllcities, %Listallcities%
Gui, show
time2_Lexikos_method:=A_tickCount - Starttime
total_time1_Lexikos+=time1_Lexikos_method
total_time2_Lexikos+=time2_Lexikos_method
;msgbox %time2_Lexikosmethod% ms
Gui, destroy
}

EnvDiv, total_time1_Lexikos, %nb_trials%
avr_time1_Lexikos:=total_time1_Lexikos
EnvDiv, total_time2_Lexikos, %nb_trials%
avr_time2_Lexikos:=total_time2_Lexikos

Fileappend,  Lexikos METHOD `n Average time to load list : %avr_time1_Lexikos% ms `n Average time to load GUI : %avr_time2_Lexikos% ms `n `n, Comparisontimes.txt
total_time1_Lexikos =
time1_Lexikos_method =
total_time2_Lexikos =
time2_Lexikos_method =
avr_time1_Lexikos = 
avr_time2_Lexikos = 

Fileread, Content, Comparisontimes.txt

msgbox, %Content%


#6 Lexikos

Lexikos
  • Administrators
  • 8845 posts

Posted 01 July 2012 - 12:44 PM

Retrieves the lines in a text file, one at a time (performs better than FileReadLine).

FileReadLine is not FileRead. If you were to use FileReadLine to read the entire file line by line, it would indeed be very slow.

[...] but i didn't think to use regexreplace, which seems very powerful.

I could have used StringReplace, which would be (at least marginally) faster, but didn't know which line ending format your files have.

#7 jyloup

jyloup
  • Members
  • 74 posts

Posted 02 July 2012 - 07:23 AM

Oh yeah even quicker with Strinreplace ! 10 times quicker about than parsing and regexreplace method ! i replaced the Regexreplace command by :

Stringreplace, ListAllCities, ListAllCities, `n, , All
; replaces the command line:
; ListAllCities := RegExReplace(ListAllCities, "\R")
; all lines ending format is : `n

i got this result :

Number of following trials : 5
SINKFAZE METHOD
Average time to load list : 265 ms
Average time to load GUI : 2845 ms
Lexikos METHOD
Average time to load list : 43 ms
Average time to load GUI : 2795 ms

Thanks anyway for you help.

#8 jyloup

jyloup
  • Members
  • 74 posts

Posted 02 July 2012 - 08:17 AM

Lol, i realize in fact none of this is necessary to load the text file.
I tried to load directly the list without removing new lines,
and it still works !!!!
(i remind that the text file is as follows :

city1|
city2|
etc...
cityN|

i just put :

FileRead, ListAllCities, Allcities.txt
;Stringreplace, ListAllCities, ListAllCities, `n, , All
;Fileappend, %ListallCities%, ListallcitiesBis.txt
Gui, Add, Dropdownlist, x90 y55 w450 h20 R10 Sort Limit vFrom, %ListAllcities%

So, finally Autohokey dropdownlist recognizes also new line in separated file, it's good to know ! :D

#9 jyloup

jyloup
  • Members
  • 74 posts

Posted 02 July 2012 - 10:27 AM

Lexikos could you please explain me a strange thing happening in dropdownlist,
about the incremential search of an item :
when i load a list text file, incremential search is working for some files, but not for other files.

;Listallcities:=Regexreplace(Listallcities,"\R")
;Fileappend, %Listallcities%, AllcitiesIncrementWorking.txt
FileRead, ListAllCities,  AllcitiesIncrementWorking.txt
Gui, Add, dropdownlist, x90 y55 w550 R20 vFrom Sort,%ListAllcities%

From the text file created above, i can search an item with incremential function,
when i type the 3 first letters (Sort option)

But from the original text file (the one with 1 city by line), which can be loaded although, the increment doesn't work.

It doesn't work neither from the text file created with Stringreplace command. However, the text files "AllcitiesBisIncrementNotworking.txt" below and " AllcitiesIncrementWorking.txt" above
look like exactly the same, without new line :

;FileRead, ListAllCities, Allcities.txt
;Stringreplace, Listallcities, Listallcities,`n, ,All
;Fileappend, %Listallcities%, AllcitiesBisIncrementNotworking.txt
Fileread, %Listallcities%, AllcitiesBisIncrementNotworking.txt
Gui, Add, dropdownlist, x90 y55 w550 R20 vFrom Sort,%ListAllcities%

Why the "sort" function doesn't work for this file ? and for the original list file with new lines?
thanks.