False Positive Battle

Discuss Autohotkey related topics here. Not a place to share code.
Forum rules
Discuss Autohotkey related topics here. Not a place to share code.
User avatar
DataLife
Posts: 382
Joined: 29 Sep 2013, 19:52

False Positive Battle

Post by DataLife » 24 Feb 2021, 14:48

Well after 12 years of fighting the antivirus companies over false positives it appears they have won.

14 out of 70 antivirus products on VirusTotal detects something and I can not get most of them to respond to false positive reports. Some AVG, Avast, SecureAPlus replied back that it is not a false positive.

Some Symantec, VBA32, verified it is a false positive.

Now ZoneAlarm is quarantining my exe every time I compile it. I added exceptions to fix it on my PC.

I can no longer distribute my autohotkey program to people I do not personally know. It can not be available to the general public compiled.

Any ideas?

I suppose I can distribute my source code with a copy of Autohotkey with instructions on how to run it in a zip file.

Would I distribute a batch file with it to run Autohotkey with my source code as a parameter? How would that look to people who are not into coding.
Check out my scripts. (MyIpChanger) (ClipBoard Manager) (SavePictureAs)
All my scripts are tested on Windows 10, AutoHotkey 32 bit Ansi unless otherwise stated.

gregster
Posts: 6230
Joined: 30 Sep 2013, 06:48

Re: False Positive Battle

Post by gregster » 24 Feb 2021, 16:40

I am really sorry to hear that. I mean it's a well-known issue - still your experience seems extreme.

To understand what's going on, let me ask some questions.
14 seems a lot, even for AHK. Did you get any info why it was deemed no false positive? And are you applying additional compressing/obfuscation/encryption techniques to your compiled script? Which AHK version do you use? And what is your script doing?

Edit: I just tested with a small compiled script (AHK v1.1.1.33.02 64 bit) and got 2 false positives, by Cynet and Zillya - whoever this is ); Edit: on a repeated run which included more engines it seems, a third engine - Jiangmin - came also up - I think that's a one of the usual suspects for false positives. Final score 3/70
These detections are probably purely based on the AHK source code which is available in every compiled AHK script.


SOTE
Posts: 1265
Joined: 15 Jun 2015, 06:21

Re: False Positive Battle

Post by SOTE » 24 Feb 2021, 21:25

I think us AHKers will have to concentrate a bit more on VirusTotal itself. That is, sign up on VirusTotal directly (https://www.virustotal.com/gui/join-us), and then comment about AntiVirus companies that are doing AutoHotkey dirty or are unresponsive to users. Another route is that you can send comments to VirusTotal without joining them (but you can't vote or have your comments show on the result pages), by using this link- https://www.virustotal.com/gui/contact-us

Arguably one of the poster children for what is going bad is the AntiVirus company, Jiangmin, that appears to not respond to users, customers, or complaints about their product. A Google search of them will reveal complaints about their company and product going back several years. Jiangmin is not the only one, as there are several problematic companies that VirusTotal lists for their own reasons. But, something to consider is how many of your users are using Jiangmin products? So despite seeing negative detections, how severely it will affect you can be relative to what country you are in and your situation.

As for community users, it appears that joesecurity was/is doing some damage to AutoHotkey's reputation by ranking it very low and as malware. It's unknown how he comes up with his results or what his credentials are. He also links to his website (joesandbox.com), from off of VirusTotal, so clearly has a business interest. Why VirusTotal is allowing such conflicts of interests and linking to competing websites with conflicting results and uncertified credentials is not understood. Something else that VirusTotal should be called out on.

User avatar
DataLife
Posts: 382
Joined: 29 Sep 2013, 19:52

Re: False Positive Battle

Post by DataLife » 24 Feb 2021, 23:42

gregster wrote:
24 Feb 2021, 16:40
I am really sorry to hear that. I mean it's a well-known issue - still your experience seems extreme.

To understand what's going on, let me ask some questions.
14 seems a lot, even for AHK. Did you get any info why it was deemed no false positive? And are you applying additional compressing/obfuscation/encryption techniques to your compiled script? Which AHK version do you use? And what is your script doing?

Edit: I just tested with a small compiled script (AHK v1.1.1.33.02 64 bit) and got 2 false positives, by Cynet and Zillya - whoever this is ); Edit: on a repeated run which included more engines it seems, a third engine - Jiangmin - came also up - I think that's a one of the usual suspects for false positives. Final score 3/70
These detections are probably purely based on the AHK source code which is available in every compiled AHK script.
They have not provided any extra info. Quote "Our virus specialists have been working on this request and they confirmed this detection is correct." and Quote "The following files were found to be suspicious*, and hence were not flagged as wrong detections:" Actually not "files" just one file.

After compiling with Ahk2Exe I use Inno Setup to turn my exe into an installation file.

I get 7 detections if I only compile with Ahk2Exe and not use Inno Setup.

Some of the antivirus companies use the same definitions, I believe ZoneAlarm and Kaspersky use the same definitions. I believe ALYac, cScan and BitDefender use the same definitions. So the real number of detections are lower, but I suspect people trust their antivirus products and will discard my program.
Check out my scripts. (MyIpChanger) (ClipBoard Manager) (SavePictureAs)
All my scripts are tested on Windows 10, AutoHotkey 32 bit Ansi unless otherwise stated.

User avatar
DataLife
Posts: 382
Joined: 29 Sep 2013, 19:52

Re: False Positive Battle

Post by DataLife » 24 Feb 2021, 23:46

I would be willing to try compiling with AutoHotkey_H. I suppose when I install AutoHotkey_H it will also install its own compiler?

I know where the bin files are located but I do not know what you mean by using exe instead of bin.
Check out my scripts. (MyIpChanger) (ClipBoard Manager) (SavePictureAs)
All my scripts are tested on Windows 10, AutoHotkey 32 bit Ansi unless otherwise stated.

User avatar
DataLife
Posts: 382
Joined: 29 Sep 2013, 19:52

Re: False Positive Battle

Post by DataLife » 25 Feb 2021, 00:01

SOTE wrote:
24 Feb 2021, 21:25
I think us AHKers will have to concentrate a bit more on VirusTotal itself. That is, sign up on VirusTotal directly (https://www.virustotal.com/gui/join-us), and then comment about AntiVirus companies that are doing AutoHotkey dirty or are unresponsive to users. Another route is that you can send comments to VirusTotal without joining them (but you can't vote or have your comments show on the result pages), by using this link- https://www.virustotal.com/gui/contact-us

Arguably one of the poster children for what is going bad is the AntiVirus company, Jiangmin, that appears to not respond to users, customers, or complaints about their product. A Google search of them will reveal complaints about their company and product going back several years. Jiangmin is not the only one, as there are several problematic companies that VirusTotal lists for their own reasons. But, something to consider is how many of your users are using Jiangmin products? So despite seeing negative detections, how severely it will affect you can be relative to what country you are in and your situation.

As for community users, it appears that joesecurity was/is doing some damage to AutoHotkey's reputation by ranking it very low and as malware. It's unknown how he comes up with his results or what his credentials are. He also links to his website (joesandbox.com), from off of VirusTotal, so clearly has a business interest. Why VirusTotal is allowing such conflicts of interests and linking to competing websites with conflicting results and uncertified credentials is not understood. Something else that VirusTotal should be called out on.
I think the problem would be solved if VirusTotal would require the Antivirus companies that they use to respond to False Positive reports or their Antivirus program would not be used by VirusTotal.
I have been submitting False Positive reports for more then 12 years. Rising and Cylance has never responded or fixed a False Positive.
Check out my scripts. (MyIpChanger) (ClipBoard Manager) (SavePictureAs)
All my scripts are tested on Windows 10, AutoHotkey 32 bit Ansi unless otherwise stated.

SOTE
Posts: 1265
Joined: 15 Jun 2015, 06:21

Re: False Positive Battle

Post by SOTE » 25 Feb 2021, 00:15

DataLife wrote:
24 Feb 2021, 23:42
They have not provided any extra info. Quote "Our virus specialists have been working on this request and they confirmed this detection is correct." and Quote "The following files were found to be suspicious*, and hence were not flagged as wrong detections:" Actually not "files" just one file.
After compiling with Ahk2Exe I use Inno Setup to turn my exe into an installation file.
The questions asked by Gregster are relevant, because it can possibly help narrow down the issue.
gregster wrote:
24 Feb 2021, 16:40
And are you applying additional compressing/obfuscation/encryption techniques to your compiled script?
Which AHK version do you use?
And what is your script doing?
For instance, some people use MPRESS or UPX, and these can increase your detection hits. AHK_H already has some built-in compression, this is the first box (Use compression to reduce size of resulting executable). So using AHK_H's compression and MPRESS or UPX all together, might not be necessary. AHK_H's compression is less likely to cause a false-positive. Plus you have to consider how small you really need the executable to be, especially in these days of terabyte drives.

The version of AHK_H can be a factor, in terms of people seeing the same results that you are seeing and troubleshooting.

What your script is doing is also relevant. For instance, is it connecting to the internet, messing with the registry, or doing something borderline? Other factors like your executable has no information and isn't digitally signed, can add to detection hits. It's not necessarily one factor, but it can be an executable that has a lot of suspicious things going on that causes heuristics to send out alerts. Like an executable that is using a packer (MPRESS), is not digitally signed, doing odd things to the registry, and is trying to connect to a suspicious web server.

SOTE
Posts: 1265
Joined: 15 Jun 2015, 06:21

Re: False Positive Battle

Post by SOTE » 25 Feb 2021, 00:21

DataLife wrote:
25 Feb 2021, 00:01
I think the problem would be solved if VirusTotal would require the Antivirus companies that they use to respond to False Positive reports or their Antivirus program would not be used by VirusTotal.
I have been submitting False Positive reports for more then 12 years. Rising and Cylance has never responded or fixed a False Positive.
I totally agree that various companies that VirusTotal allows are very suspect and they being on there is possibly more a business decision than what is good for users or quality. I'm quite amazed how an open-source interpreter, where every line of code can be viewed and tested, is any kind of problem for expert programmers at such AntiVirus companies. With some companies, I suspect they grab signatures from elsewhere (doing little to no testing themselves) and incorporate them into their product.

It appears we have to be more vigilant with VirusTotal, not just the AntiVirus companies, as to what some of these companies on their website are doing. Overall, VirusTotal does have an affect on the reputation and public image of AutoHotkey, including how much users, newbies, and casuals might trust it.

User avatar
DataLife
Posts: 382
Joined: 29 Sep 2013, 19:52

Re: False Positive Battle

Post by DataLife » 25 Feb 2021, 00:57

SOTE wrote:
25 Feb 2021, 00:15
DataLife wrote:
24 Feb 2021, 23:42
They have not provided any extra info. Quote "Our virus specialists have been working on this request and they confirmed this detection is correct." and Quote "The following files were found to be suspicious*, and hence were not flagged as wrong detections:" Actually not "files" just one file.
After compiling with Ahk2Exe I use Inno Setup to turn my exe into an installation file.
The questions asked by Gregster are relevant, because it can possibly help narrow down the issue.
gregster wrote:
24 Feb 2021, 16:40
And are you applying additional compressing/obfuscation/encryption techniques to your compiled script?
Which AHK version do you use?
And what is your script doing?
For instance, some people use MPRESS or UPX, and these can increase your detection hits. AHK_H already has some built-in compression, this is the first box (Use compression to reduce size of resulting executable). So using AHK_H's compression and MPRESS or UPX all together, might not be necessary. AHK_H's compression is less likely to cause a false-positive. Plus you have to consider how small you really need the executable to be, especially in these days of terabyte drives.

The version of AHK_H can be a factor, in terms of people seeing the same results that you are seeing and troubleshooting.

What your script is doing is also relevant. For instance, is it connecting to the internet, messing with the registry, or doing something borderline? Other factors like your executable has no information and isn't digitally signed, can add to detection hits. It's not necessarily one factor, but it can be an executable that has a lot of suspicious things going on that causes heuristics to send out alerts. Like an executable that is using a packer (MPRESS), is not digitally signed, doing odd things to the registry, and is trying to connect to a suspicious web server.
My program does not write to the registry.
It does download updates to my program from Sourceforge.
I do not use Mpress or UPX.
My program is not digitally signed.

Quote " AHK_H already has some built-in compression" Is this by default or only when choosing to reduce the size of the resulting exe?

I suppose I could alter my source code to not access a webpage to see if it reduces the False Positives. But it would not take care of 14 detections.
Check out my scripts. (MyIpChanger) (ClipBoard Manager) (SavePictureAs)
All my scripts are tested on Windows 10, AutoHotkey 32 bit Ansi unless otherwise stated.

User avatar
DataLife
Posts: 382
Joined: 29 Sep 2013, 19:52

Re: False Positive Battle

Post by DataLife » 25 Feb 2021, 01:05

SOTE wrote:
25 Feb 2021, 00:21
DataLife wrote:
25 Feb 2021, 00:01
I think the problem would be solved if VirusTotal would require the Antivirus companies that they use to respond to False Positive reports or their Antivirus program would not be used by VirusTotal.
I have been submitting False Positive reports for more then 12 years. Rising and Cylance has never responded or fixed a False Positive.
I totally agree that various companies that VirusTotal allows are very suspect and they being on there is possibly more a business decision than what is good for users or quality. I'm quite amazed how an open-source interpreter, where every line of code can be viewed and tested, is any kind of problem for expert programmers at such AntiVirus companies. With some companies, I suspect they grab signatures from elsewhere (doing little to no testing themselves) and incorporate them into their product.

It appears we have to be more vigilant with VirusTotal, not just the AntiVirus companies, as to what some of these companies on their website are doing. Overall, VirusTotal does have an affect on the reputation and public image of AutoHotkey, including how much users, newbies, and casuals might trust it.
Quote "where every line of code can be viewed and tested"

My SavePictureAs script has over 16000 lines of my code and then alot more code for the functions that I did not create.

Could a antivirus company be able to inspect line for line?

Seems to make their virus definitions better they would want to do that.
Check out my scripts. (MyIpChanger) (ClipBoard Manager) (SavePictureAs)
All my scripts are tested on Windows 10, AutoHotkey 32 bit Ansi unless otherwise stated.

HotKeyIt
Posts: 2274
Joined: 29 Sep 2013, 18:35
Contact:

Re: False Positive Battle

Post by HotKeyIt » 25 Feb 2021, 03:54

DataLife wrote:
24 Feb 2021, 23:46
I would be willing to try compiling with AutoHotkey_H. I suppose when I install AutoHotkey_H it will also install its own compiler?

I know where the bin files are located but I do not know what you mean by using exe instead of bin.
No need to install, simply extract and run compiler, from Bin File Drop down select AutoHotkey.exe insead AutoHotkeySC.bin.

SOTE
Posts: 1265
Joined: 15 Jun 2015, 06:21

Re: False Positive Battle

Post by SOTE » 25 Feb 2021, 05:10

DataLife wrote:
25 Feb 2021, 00:57
[My program does not write to the registry.
It does download updates to my program from Sourceforge.
I do not use Mpress or UPX.
My program is not digitally signed.

Quote " AHK_H already has some built-in compression" Is this by default or only when choosing to reduce the size of the resulting exe?

I suppose I could alter my source code to not access a webpage to see if it reduces the False Positives. But it would not take care of 14 detections.
1) Not sure how you have your update functionality set up, but maybe make checking for updates a more manual user initiated process versus something automatic and at install. Programs that automatically seek out webservers (especially those with bad reputations) tend to be more scrutinized and trip alerts. Perhaps have a GUI pop up after starting the program and the last update was more than 90 days (or whatever reasonable time frame). The user then selects "Yes" or "No" for the update.

2) The AHK_H compression I was referring to is with Ahk2Exe (AHK_H version). It's one of the options you can choose when compiling your script.

3) On digital signature certificates, you can do OV code signing certificates (regular ones and can be done for free) or EV code signing certificates (usually bought and for professional developers or companies). They can help, depending on which type used. OV code signing certificates help a little bit, by clarifying who the author of the program is, and this can be a factor with heuristic or behavioral detection. Malware tends to not have information about the program and not be signed. But, you might still see false-positive because you have to establish your reputation and this takes time. EV code signing certificates are relatively expensive, but you are much less likely to get hit with a false-positive and it will instantly give you a "good reputation" with Microsoft (Windows OS and Edge browser) and Google (thus Chrome and Firefox browsers). For independent developers, the cost might outweigh the benefit, depending on the situation. They are not necessary, especially for hobbyist or non-professionals, but is a "nice to do" type of thing.
DataLife wrote: Could a antivirus company be able to inspect line for line?
For an interpreted programming language like AutoHotkey, usually the AntiVirus companies are first looking at the interpreter (autohotkey.exe) or the DLL version. That can be a specific signature in their database that might be designated as malware or suspicious. Usually in the second case, you are dealing with heuristic or behavioral analysis. The AntiVirus software will have a list of things that might set off an alert. So it's the actions of the software (like say screwing around with settings in the registry), not necessarily specific code, that can set it off. However, it's not to say that specific code or machine code isn't one of the things also on the lists that the AntiVirus software is comparing or looking at. When a specific kind of malware becomes popular, experts will start digging into the code more deeply (if it's a scripting language) or using segments of machine code to help identify it.
DataLife wrote: I get 7 detections if I only compile with Ahk2Exe and not use Inno Setup.
For an alternative installer that is AHK only, you might want to check the thread below. Based on what you typed, an AHK only install solution, might cut down your false-positive detection rate from 14 to 7. Might be worth a try.
https://www.autohotkey.com/boards/viewtopic.php?f=6&t=6723
(Base Frame for AutoHotKey Scripts (Installer ...))

lexikos
Posts: 7510
Joined: 30 Sep 2013, 04:07
GitHub: Lexikos

Re: False Positive Battle

Post by lexikos » 05 Apr 2021, 23:06

Running a script distributed in .ahk form doesn't need to be complicated. You can include AutoHotkey.exe but give it the same base name as your script. Myscript.exe will read Myscript.ahk from the current directory, by default.

Whether that will solve the problem is a separate matter.

sarahmath
Posts: 1
Joined: 07 May 2021, 14:36

Re: False Positive Battle

Post by sarahmath » 07 May 2021, 14:41

HotKeyIt wrote:
24 Feb 2021, 18:03
Have you tried compiling AutoHotkey_H using exe instead of bin?
4/70 on 32-bit Unicode MD: https www.virustotal.com /gui/file/2f6e9e179fed4de8f6b0b6b5c60e7b7f9dca9f2695c57159cfd86d76116b4aba/detection Broken Link for safety
3/70 on 64-bit MD: https www.virustotal.com /gui/file/7bb8b3b89e8d01eedc48db01e4c6d9890d0eb2f3c4d99369f253a2ff71cabe7d/detection Broken Link for safety

thanks for your sharing.

Post Reply

Return to “General Discussion”