Jump to content


Photo

Sikuli Screenshot based cross platform automation


  • Please log in to reply
14 replies to this topic

#1 LiquidGravity

LiquidGravity
  • Members
  • 154 posts

Posted 29 January 2010 - 11:43 PM

I found this pretty interesting. Computer automation so easy a caveman (or your boss) could do it. :lol:

If it has a GUI, you can probably use it with Sikuli. Sikuli is an open source scripting app that uses a combination of very simple commands like click, type, and wait, and screenshots to tell Sikuli what to manipulate. There's no internal API support, it just searches the screen for the image in the screenshot—meaning you can use it with pretty much anything. Seriously, the world is your oyster.

Sikuli Automates Almost Anything with Screenshot Ease

So who is going to code the whole thing in AutoHotkey with ImageSearch?

#2 majkinetor

majkinetor
  • Fellows
  • 4511 posts

Posted 03 February 2010 - 09:40 AM

Thx for sharing this. Very nice app plus its portable.

Should be definitely made with AHK.

#3 Learning one

Learning one
  • Members
  • 1294 posts

Posted 05 February 2010 - 11:27 AM

caveman (or your boss)

:D :lol: hahaha

#4 gronne

gronne
  • Members
  • 6 posts

Posted 09 February 2010 - 01:07 AM

I've played around with Sikuli quite a bit in the last week. It's an interesting idea but it's not ready (it's Alpha, not Beta quality).

It's really slow compared to AHK and it's image recognition engine wasn't sophisticated enough to discern which file menu I was clicking on when trying to create macros.

#5 florinn

florinn
  • Members
  • 45 posts

Posted 10 June 2010 - 05:07 AM

I love the way you can take small screenshots directly from your screen and the script integrates them from the left menu. If a similar GUI could be done in ahk with some more options in the left panel, AND if it would be able to integrate the .bmp files in your compiled script it would be AWESOME.
Was not able to find the loop command yet in sikuli since I am not familiar with python :oops: and I dont think you can compile.

#6 majkinetor

majkinetor
  • Fellows
  • 4511 posts

Posted 15 June 2010 - 07:30 AM

This can be done with AHK.

- Using RichEdit control from Forms framework let you add images and code next to each other. Currently, adding bitmaps is not yet implemented but there are 2 ways: COM & RTF. Plenty of code exists around to do this. The other way would be to use Internet Explorer control. You could even use ordinary text control and use special sign to link images in the code (for instance, images could be shown when user hovers this symbol).

- Scripts that monitor what you do on the screen and take a screenshot of the area already exist on the forums.

- Injecting bitmap into text is also done on various places on forum (using MIME representation).

It doesn't look that hard to be honest. I would personaly go with RTF bitmap + RichEdit editor. For instance thisis Delphi code to convert bitmap to RTF code. RTF code can be then set via SetText to show the image. Another Delphi code is here. COM technique is described here. If anybody wants to translate some of these to AHK we would have all necessary prerequisites to make this app in AHK.

#7 Guests

  • Guests

Posted 10 November 2010 - 01:23 PM

It'ld be great if someone knowledgeable be interested in this project.
I just found this thread about Sikulia new experimental programing tool, and tird it.
With it I was able to automate opera without pain and quickly.

This can be done with AHK.

- Using RichEdit control from Forms framework let you add images and code next to each other. Currently, adding bitmaps is not yet implemented but there are 2 ways: COM & RTF. Plenty of code exists around to do this. The other way would be to use Internet Explorer control. You could even use ordinary text control and use special sign to link images in the code (for instance, images could be shown when user hovers this symbol).

- Scripts that monitor what you do on the screen and take a screenshot of the area already exist on the forums.

- Injecting bitmap into text is also done on various places on forum (using MIME representation).

It doesn't look that hard to be honest. I would personaly go with RTF bitmap + RichEdit editor. For instance thisis Delphi code to convert bitmap to RTF code. RTF code can be then set via SetText to show the image. Another Delphi code is here. COM technique is described here. If anybody wants to translate some of these to AHK we would have all necessary prerequisites to make this app in AHK.



#8 awannaknow

awannaknow
  • Members
  • 369 posts

Posted 10 November 2010 - 01:24 PM

previous post was mine, was not logged . . .
As I cannot correct, I meant :
I just found this thread about Sikulia new experimental programing tool, and tird it.
I just found this thread about Sikuli a new experimental programing tool, and tried it.
I'll propose this idea here :
Brainstorming-WishList: Post your script idea or wish here!

#9 TheGreatSwami Woo

TheGreatSwami Woo
  • Members
  • 237 posts

Posted 04 July 2011 - 09:58 AM

verson of sikuli is now at 1.0rc2
http://sikuli.org/download.shtml
what's new in the later version
http://sikuli.org/do...in-sikulix.html

#10 awannaknow

awannaknow
  • Members
  • 369 posts

Posted 19 August 2011 - 07:31 PM

Finally I made a few scripts with Sikuli and found it really ssssllllooowww ! ! !
And then recently, I got accustomed to ahk commands : PixelGetColor and PixelSearch and found I could do same things as with Sikuli.
It's as easy to do, but it's much much fasterrrrrr ! ! !
:p

Haven't, yet, had opportunity to try ahk command : ImageSearch

But ,i'm sure it's pretty useful . . .

:wink:

#11 TheGreatSwami Woo

TheGreatSwami Woo
  • Members
  • 237 posts

Posted 20 August 2011 - 04:45 PM

Finally I made a few scripts with Sikuli and found it really ssssllllooowww ! ! !
And then recently, I got accustomed to ahk commands : PixelGetColor and PixelSearch and found I could do same things as with Sikuli.
It's as easy to do, but it's much much fasterrrrrr ! ! !
:p

Haven't, yet, had opportunity to try ahk command : ImageSearch

But ,i'm sure it's pretty useful . . .

:wink:


Sample code?

#12 awannaknow

awannaknow
  • Members
  • 369 posts

Posted 20 August 2011 - 05:59 PM

Sample code?


From Sikuli ? or . . . ahk ?
Ok, look at how I automated this :
1 - without Automation. Start and stop manually : 1mn 03 seconds

http://www.youtube.c...kKrM?hl=fr&fs=1
2 - Automated.
Start : the first 2 screens are skipped at lightening speed, then the menu is automated lightening fast to reach the loading screen. Then immediate closing (not pixel, just code) : 23 seconds
http://www.youtube.c...T2Ts?hl=fr&fs=1


I searched to embed the videos directly but don't know if it's possible, so just links . . .


Here is the code, it works for my screen (1280x1024) under XP 32 bits and its AHK classic not AHK_L.
If you try it, plz, tell me if it work for your pc. :
/*
*********************************************************
*					RA2 Yuri's Revenge 					*
*			Launch and Stop in Skirmish mode            *
*********************************************************
*/
Ahk_File_Name = Yuri Skirmish.ahk
FrontEndWindow = ahk_class Yuri's Revenge
Path = put your path here
ExeName = Yuri.exe
Icon_Path = put your Icon path here
Icon_Name = put your Icon name here
;Menu, Tray, Icon, %Icon_Path%%Icon_Name%,
Menu, Tray, Icon, %Path%%ExeName%,
CoordMode,ToolTip,Screen

applicationname = RA2 Yuri's Revenge Launch and Stop in Skirmish mode

TRAYMENU:
Menu,Tray,Tip, %applicationname%

/*
*********************************************************
		Read the registry for 1rst run or else
*********************************************************
*/
RegRead, v_reg_yuri, HKEY_CURRENT_USER, Software\Microsoft\Windows\ShellNoRoam\MUICache, C:\EA Games\Command & Conquer Red Alert(tm) II\RA2\gamemd.exe

/*
*********************************************************
*		On / Off Hotkey or HotString Switch             *
*********************************************************
*/
switch = 0 ; Set Variable Switch to 0 Before starting 
;^y:: ;; Ctrl y Hotkey; Launching with Ctrl+y desactivated because launched from same Ctrl+y shortcut from Liberkey Menu
  IfWinNotExist, %FrontEndWindow% 
	  {
	  Run, %Path%%ExeName% ; Launching Yuri's revenge
	  WinWait, %FrontEndWindow%,
	  IfWinNotActive, %FrontEndWindow%, , WinActivate, %FrontEndWindow%, 
	  WinWaitActive, %FrontEndWindow%
	  switch = 1  ; 1 = On : Yuri's revenge is open
	  ;Sleep, %Sleep_1%
	  ; Loop when OutputVar is not-equal to pixel color 0xC67D29

	  While vBlue != 0xC67D29 ; means : until the variable "vBlue" doesn't contain this color (0xC67D29) continue to loop
	  ;not-equal (<> or !=). The operators != and <> are identical in function.
	  PixelGetColor, vBlue, 393, 262 
	  ;MsgBox, 256 OutputVar is %OutputVar% = 0xC67D29
	  IfEqual, vBlue, 0xC67D29 ; If vBlue variable is equal to 0xC67D29
	  ;MsgBox, 258 OutputVar is %OutputVar% = 0xC67D29
	  Send, {ESC} ;Skip Animation
	  WinWait, %FrontEndWindow%,
	  IfWinNotActive, %FrontEndWindow%, , WinActivate, %FrontEndWindow%, 
	  ;Sleep, %Sleep_2%
	  WinWaitActive, %FrontEndWindow%
	  ; Loop when OutputVar is not-equal to pixel color 0x3138FF
	  
	  While vRed != 0x3138FF
	  ;not-equal (<> or !=). The operators != and <> are identical in function.
	  PixelGetColor, vRed, 108, 253 
	  ;MsgBox, 277 OutputVar is %OutputVar% = 0x3138FF
	  IfEqual, vRed, 0x3138FF
	  ;MsgBox, 279 OutputVar is %OutputVar% = 0x3138FF
	  Control, Check,, Button2, %FrontEndWindow% ;click Single Player ClassNN:	Button2 Text:	GUI:SinglePlayer
	  WinWait, %FrontEndWindow%,
	  IfWinNotActive, %FrontEndWindow%, , WinActivate, %FrontEndWindow%, 
	  WinWaitActive, %FrontEndWindow%
	  ; Loop when OutputVar is not-equal to pixel color 0x00FFFF  x 702 y 305
	  ;Loop until OutputVar is equal to pixel color 0x00FFFF
	  While vYellowSkir != 0x00FFFF
	  PixelGetColor, vYellowSkir, 702, 305 
	  ;MsgBox, 314 OutputVar is %OutputVar% = 0x00FFFF
	  IfEqual, vYellowSkir, 0x00FFFF
	  ;MsgBox, 316 OutputVar is %OutputVar% = 0x00FFFF  

	  Control, Check,, Button3, %FrontEndWindow% ;Click Skirmish ClassNN:	Button2 Text:	GUI:LoadSavedGame ClassNN:	Button3 Text:	GUI:Skirmish
	  WinWait, %FrontEndWindow%,
	  IfWinNotActive, %FrontEndWindow%, , WinActivate, %FrontEndWindow%, 
	  WinWaitActive, %FrontEndWindow%

	  ; Loop when OutputVar is not-equal to pixel color 0x00FFFF  x 692 y 261
	  ;Loop until OutputVar is equal to pixel color 0x00FFFF
	  While vYellowStar != 0x00FFFF
	  PixelGetColor, vYellowStar, 692, 261 
	  ;MsgBox, 352 OutputVar is %OutputVar% = 0x00FFFF
	  IfEqual, vYellowStar, 0x00FFFF
	  ;MsgBox, 354 OutputVar is %OutputVar% = 0x00FFFF  
	  
	  Control, Check,, Button2, %FrontEndWindow% ;Click StartGame
	  }
	Else
	  {
	  ^y:: ;; Ctrl y Hotkey
	  IfWinExist, %FrontEndWindow%,
	  WinClose, %FrontEndWindow%
	  Process,close, gamemd.exe
	  IfWinNotExist, %FrontEndWindow%
	  switch = 0 ; 0 = Off : %FrontEndWindow% is closed
	  ;IfWinExist [, WinTitle, WinText, ExcludeTitle, ExcludeText]

	  ;PostMessage, 0x112, 0xF060,,, WinTitle, WinText  ; 0x112 = WM_SYSCOMMAND, 0xF060 = SC_CLOSE
	  
	  ExitApp
	  }
	  Return

Also as a reference, there is a very complete script and thread still going on, here :
OPIE fishing bot for WOW

#13 kife

kife
  • Guests

Posted 02 September 2011 - 11:21 PM

you can't compare sikuli's image search with pixel stuff in ahk. i've only been using it a few days but, the closest thing would be to make a 1 pixel region, and do region.exists() using a 1 pixel image.

ahk's imagesearch would be a better comparison, but i haven't tried it yet either, simply because it's not as easy to use. also, i don't think you can restrict it to regions like in sikuli.



Finally I made a few scripts with Sikuli and found it really ssssllllooowww ! ! !
And then recently, I got accustomed to ahk commands : PixelGetColor and PixelSearch and found I could do same things as with Sikuli.
It's as easy to do, but it's much much fasterrrrrr ! ! !
:p

Haven't, yet, had opportunity to try ahk command : ImageSearch

But ,i'm sure it's pretty useful . . .

:wink:



#14 nimda

nimda
  • Members
  • 4301 posts

Posted 03 September 2011 - 03:04 AM

also, i don't think you can restrict it to regions like in sikuli.

Wrong, you can.
<!-- m -->http://d.ahk4.me/~ImageSearch<!-- m -->

#15 DustTheWind

DustTheWind
  • Members
  • 1 posts

Posted 23 July 2012 - 09:31 PM

what I really like about Sikuli is that it compounds commands and shortens the coding necessary. Not only does it let you capture on the screen either a picture to click or even a region to click it also lets you compound the capture process, variable storage, search, and click process into one command.

example would be dragDrop(image1,image2). It saves me time to just click a button and select an image in that process. If I learn jython i can expand my codes quite easily. With the region restrict image recognition gets a lot faster.

What I don't like in sikuli is a lack of making many scripts accessable by short cut keys. his is also what I like about Autohotkey.

However I have to say programming equivilant commands in AHK is more complicated and time consuming than it was in Sikuli. I only wish I could combine them some how.

I think the best software is software that saves us time learning/programing so we can acomplish more. Humans have been able to accomplish more because we found ways to shirk grinding wheat, ect, using animals first, then wind, then gasoline/solar. Bassically letting other stuff automate what took our time and energy. Giving us more time to accomplish other things.

I would highly recomend a way to make the coding easier to learn/program for AHK and simplify/compound the processes now that I have seen there is a better way.