10 replies to this topic

[Lexikos]

Clipboard = %Clip0% does not perform any text conversions; it merely copies binary data back onto the clipboard. WinClip appears to work the same way. However, if I'm not mistaken, it discards the CF_TEXT and CF_OEMTEXT formats. ClipboardAll saves the first text format, which may be CF_TEXT, CF_OEMTEXT or CF_UNICODETEXT. It sounds like you're getting (non-Unicode) CF_TEXT, and in the process of saving and restoring it, the locale information is lost.

I'll look into this some more later.

[Lexikos]

I have verified that AutoHotkey_L correctly saves and restore the original binary format of CF_TEXT, and the CF_LOCALE object which the OS uses to translate from CF_TEXT to CF_UNICODETEXT. Storing CF_TEXT with character value 128 and CF_LOCALE 1049 produces a Russian character, and saving and restoring the clipboard reproduces it correctly.

What I find odd is that the input language is used by default:

When you close the clipboard, if it contains CF_TEXT data but no CF_LOCALE data, the system automatically sets the CF_LOCALE format to the current input language.
Source: Standard Clipboard Formats

Normally, non-Unicode applications use the system default ANSI code page for all strings. While the input language can change at any time, the system code page remains constant and is the same for all applications until the OS restarts (at which time it can change). So I don't know under what circumstances ANSI text copied to the clipboard would actually be in the format defined by the input language rather than the system code page.

That aside, as far as I can tell, AutoHotkey does not affect the interpretation or binary value of CF_TEXT. However, if an application copies both CF_TEXT (first) and CF_UNICODETEXT (second), only the first format is kept. It seems likely that the CF_TEXT data would be in the system code page, which (if set to US English) could not contain Russian characters, regardless of the input language. You would be able to paste the text into Unicode-aware applications up until AutoHotkey discards the CF_UNICODETEXT data.

Now, I don't have a clue why any application would store both formats explicitly, since the system does automatic conversion. I would like to know:

[*:3vg17qd9]Which clipboard formats you observed before and after performing the assignment.
[*:3vg17qd9]The actual binary data/encoding of each text format on the clipboard.
[*:3vg17qd9]Where you are copying from and pasting to.

[Lexikos]

Thanks, you've confirmed my suspicion. Word puts CF_TEXT before CF_UNICODETEXT, whereas Notepad doesn't. (Confirmed on my Windows 7 system with Word 2010 and a script calling EnumClipboardFormats.)

FYI, some detailed comments from the source code:

// EnumClipboardFormats() retrieves all formats, including synthesized formats that don't
// actually exist on the clipboard but are instead constructed on demand. Unfortunately,
// there doesn't appear to be any way to reliably determine which formats are real and
// which are synthesized (if there were such a way, a large memory savings could be
// realized by omitting the synthesized formats from the saved version). One thing that
// is certain is that the "real" format(s) come first and the synthesized ones afterward.
// However, that's not quite enough because although it is recommended that apps store
// the primary/preferred format first, the OS does not enforce this. For example, testing
// shows that the apps do not have to store CF_UNICODETEXT prior to storing CF_TEXT,
// in which case the clipboard might have inaccurate CF_TEXT as the first element and
// more accurate/complete (non-synthesized) CF_UNICODETEXT stored as the next.
// In spite of the above, the below seems likely to be accurate 99% or more of the time,
// which seems worth it given the large savings of memory that are achieved, especially
// for large quantities of text or large images. Confidence is further raised by the
// fact that MSDN says there's no advantage/reason for an app to place multiple formats
// onto the clipboard if those formats are available through synthesis.
// And since CF_TEXT always(?) yields synthetic CF_OEMTEXT and CF_UNICODETEXT, and
// probably (but less certainly) vice versa: if CF_TEXT is listed first, it might certainly
// mean that the other two do not need to be stored. There is some slight doubt about this
// in a situation where an app explicitly put CF_TEXT onto the clipboard and then followed
// it with CF_UNICODETEXT that isn't synthesized, nor does it match what would have been
// synthesized. However, that seems extremely unlikely (it would be much more likely for
// an app to store CF_UNICODETEXT *first* followed by custom/non-synthesized CF_TEXT, but
// even that might be unheard of in practice). So for now -- since there is no documentation
// to be found about this anywhere -- it seems best to omit some of the most common
// synthesized formats:
// CF_TEXT is the first of three text formats to appear: Omit CF_OEMTEXT and CF_UNICODETEXT.
// (but not vice versa since those are less certain to be synthesized)
// (above avoids using four times the amount of memory that would otherwise be required)
// UPDATE: Only the first text format is included now, since MSDN says there is no
// advantage/reason to having multiple non-synthesized text formats on the clipboard.

Typical of Microsoft to not following their own recommendations - using multiple text formats and putting the least preferred one first.

I see two possible solutions:
[*:6xogzppp]Always store only CF_UNICODETEXT, assuming any applications requesting CF_TEXT will get a correctly-synthesized value.
[*:6xogzppp]Always store all text formats, despite the increased memory usage.

[Lexikos]

Nothing is dropped when restoring the clipboard. As I said earlier, ClipboardAll saves only the first text format (i.e. it depends on the order in which data was placed onto the clipboard). The next release will unconditionally save CF_UNICODETEXT.