How many have Spinning HDs vs SSDs?

[kunkel321]

It's AHK-related because I'm determining how important it is for my script to write log data as it is collected, vs caching the data and doing periodic scheduled writes... Presumably, if lots of folks use old-style spinning hard discs, then minimizing how often disc-writes happen will be more important(?)

[TAC109]

Windows caches access to permanent disks anyway. Are you sure it will help performance to write your own caching scheme as well?

Cheers

[andymbody]

I personally cache and write to logs in batches. Usually every few seconds or so, depending on the app. I don't want to wear out my SSD either. I have read that SSDs have a limited life expectancy and reducing the number of writes can extend the life. I don't know how true that is.

Slightly off topic... I'm currently working on a log processor, in hopes of extracting log info in a much simpler way than viewing the raw data (using regex). If you have tips in this arena, I'd be glad to hear them.

[kunkel321]

Thanks for the feedback.

Windows caches access to permanent disks anyway. Are you sure it will help performance to write your own caching scheme as well?

I'm actually not sure... My main reason for doing this is because I just think it might be annoying to people if their HD has to wake up and spin every time there's an autocorrection. Andy's reply is also relevant here, though...

I personally cache and write to logs in batches. Usually every few seconds or so, depending on the app. I don't want to wear out my SSD either. I have read that SSDs have a limited life expectancy and reducing the number of writes can extend the life. I don't know how true that is.

That's interesting... I had just assumed that the SSD and RAM would have the same life span... I guess it makes sense that the technology has to be different though, since the SSD "remembers" its state.

I Googled, and here is the Google AI part of the search results:

RAM (Random Access Memory) usually lasts longer than SSDs (Solid-State Drives) because RAM is volatile memory, while SSDs use non-volatile RAM. RAM is "erased" when the system powers down, but SSDs use flash memory electronically, so the cells wear out over time. However, modern SSDs are designed to have a long lifespan and can last for many years under normal usage conditions. The average lifespan of RAM is between 3-5 years, while the average lifespan of an SSD can vary depending on the quality of the drive, usage patterns, and the specific technology used.

Solid state drives (SSDs) offer faster data access due to their rich internal parallelism. They have numerous advantages, like faster boost-up speed, energy efficiency, better resistance, and compact size. They are also more durable when compared to other storage options. However, SSDs also have their limitations. SSDs store data using flash memory electronically. The cells wear out with time and become less efficient, affecting the drive's usable life cycle.

Can RAM last longer than flash memory like an SSD in terms of lifespan? Yes, because RAM is basically “erased” when the system powers down. SSD's use non-volatile RAM which hold's it's data after the system is powered off. Those “cells” in the SSDs wear out.
SSDs wear out due to the degradation of their insulating layers, while RAM lasts longer thanks to the use of capacitors. To reach a petabyte of total written data, the average consumer would have to use their computer more or less nonstop for a decade, maybe even multiple decades. However, SSDs have a feature called wear leveling, which means the SSD's controller will spread out the reads and writes over all of the SSD's cells, instead of hitting the same few cells over and over again wearing them out faster.

Activities like copying music from another drive, downloading files, editing files, or backing up your phone, and so on require you to write to the drive, which affects the drive's usable life cycle. Watching movies, reading PDF files, or viewing photos doesn't count as writing, and has no effect on an SSD's life expectancy

This definitely makes me think that cashing to RAM, then doing periodic writes is the way to go.

[kunkel321]

Slightly off topic... I'm currently working on a log processor, in hopes of extracting log info in a much simpler way than viewing the raw data (using regex). If you have tips in this arena, I'd be glad to hear them.

My AHK experience, in this regard, has been limited to analyzing my logs of hotstrings. Your "Jim" regexes (or parts of them) often get used in them. But the most important part is usually to have a loop inside a loop. That way I can consecutively compare each string with each item in the log, thus determining the number of duplicates.

Depending on what your data is, though, and what insights you need to get from it, you might need something entirely different. Is your data a list of strings, or more like a table of numbers?

You should start an aptly-named thread in this "General Discussion" subforum...

[andymbody]

That way I can consecutively compare each string with each item in the log, thus determining the number of duplicates.

So your logs are primarily to discover hotstring dups?

Is your data a list of strings, or more like a table of numbers?

Thousands of time-stamped events. Which I need to extract related activity and cross-references of other related and unrelated activity, patterns, etc. Fun stuff. Something that AI will be good at one day very soon. I want to have an interface that will allow regex needles to be written and applied on the fly, with immediate updates of extracted results. Something that can be for general use but also customizable for particular needs. I envision the 'Everything' App, but for logged data. Maybe I should talk to David about creating the app for me. He could probably incorporate much of the code he uses with Everything. But he did mention that much of the code is very specific, and not designed for reuse. I've got most of the code written, just need to place it in a nice interface. It's always the trim work (and error handling) that takes the longest, right?

You should start an aptly-named thread in this "General Discussion" subforum...

I may take this advice because my search for ready-made tools and tips have been unfruitful so far.

Anyway, sorry if I hijacked your thread with the log discussion. Thanks for the details about SSDs.

[TAC109]

Thanks for the feedback.

Windows caches access to permanent disks anyway. Are you sure it will help performance to write your own caching scheme as well?

I'm actually not sure... My main reason for doing this is because I just think it might be annoying to people if their HD has to wake up and spin every time there's an autocorrection. Andy's reply is also relevant here, though...

This article gives an overview of how Windows file caching works. I think that you will see that writing a caching routine in your code will have little affect on how often disks are accessed.

Cheers

[andymbody]

how Windows file caching works

News to me... after all this time! Thank you!

[kunkel321]

... Anyway, sorry if I hijacked your thread with the log discussion.

All good. It allows me to bump my poll to the top with this reply... LOL.

Regarding my own use of logs, I suppose it's not "duplicates" per se, but rather "frequency of occurrence." For example these data:

Spoiler

12<< and 5-- for ::ther::there <removed>
12<< and 8-- for :*:wih::whi <removed>
7<< and 1-- for :?*:daty::day <removed>
6<< and 11-- for :?*:asr::ase
6<< and 4-- for ::fro::for
6<< and 6-- for :*:tyo::to
5<< and 1-- for ::wat::way <removed>
5<< and 1-- for ::whic::which <removed>
5<< and 10-- for :*:if is::it is
5<< and 2-- for :?:ualy::ually
4<< and 0-- for :*:with in::within <fixed>
4<< and 1-- for :*:tyhe::they
4<< and 8-- for :*:puch::push
3<< and 0-- for :*:helf::held
3<< and 1-- for :?*:ernt::erent
3<< and 1-- for :?*:tatn::tant
3<< and 10-- for :*C:ime::imme
3<< and 12-- for :?*:grama::gramma
3<< and 15-- for :*:recie::recei
3<< and 18-- for :*:senc::sens
3<< and 2-- for :?:realy::really
3<< and 3-- for ::fo::of

It sounds like you might need something more like a "filter-as-you type" though(?) If so, be sure to check out this
viewtopic.php?f=6&t=85394
It filters the results superfast and the later versions of it even highlight the search terms, which is pretty cool. Unfortunately it's v1 code. Also, that author abandoned the project when he converted it so a Sublime plug in, so there will never be a v2 version made.

[kunkel321]

... This article gives an overview of how Windows file caching works. ....

It's new information for me as well... Thanks for sharing this.

Interestingly, it mentions 256 kb "regions" in one place, and mentions analyzing 1/8th of the cache in another. I wonder if this means that the cache is about 2 mbs?

[andymbody]

be sure to check out this

This is excellent to get me started on the filtering mechanism! Thank you! I have never seen this on the forum. Such a tiny script that is very powerful.

I took the liberty of making a v2 version... then realized there were more advanced versions posted later in the thread... . So I have a v2 version of the original, and might convert the latest version another time.

Thank you very much!
Andy

[lexikos]

If the OS is installed on a HDD and it spins down, it will inevitably cause delays. No one wants that to happen while the computer is in use. And even if the computer is idle, I suppose that you shouldn't buffer your log data indefinitely. If the computer is so idle that the OS drive can be put to sleep, perhaps the system itself should be asleep; in that case, your script won't be logging anything.

I work on a variety of computers owned by clients during the course of my day job. There is generally a very wide performance gap between computers that have the OS on a HDD and those that have an SSD. I would assume that HDD owners are accustomed to delays, because there is usually a night and day difference in regard to OS boot time in particular, often app startup times as well, and task-switching in extreme cases (where there is insufficient RAM). Most computers with the OS on a HDD are configured that way because they are old, which exacerbates the problem in various ways. Even for old hardware, it is rare that upgrading to an SSD doesn't make a significant difference.

New computers in this region (for several years now) nearly always have an SSD. Putting the OS on a HDD in a new computer is sacrilege; it can make an otherwise fast computer feel like it is a decade old. Prices are close enough that cheaping out isn't worth the cost to productivity and sanity. If one must pinch pennies, combining a small SSD for the OS with a large HDD for storage is much better than putting the OS on a HDD.

Early SSDs were infamous for their limited lifespan, but newer technologies have essentially remedied this. Although they still technically have a finite lifetime, what device doesn't? SSDs are much more reliable than HDDs in my professional experience (which relates to consumers and small businesses, not data centres).

I do have a 2TB HDD for secondary storage, but I will likely replace it with a 4TB SSD when it fails or runs out of space.

All this to say, I would not go out of my way to cater to computers where the OS is on a HDD (nor would I be worried about frequent writes wearing out an SSD).

[william_ahk]

I used to have the system installed on an HDD and there was this one software that writes its config periodically to the disk, and I can hear the noise every 2 seconds. It was absolutely annoying and I uninstalled it.

But in the case of logs I suppose there is no other way around the constant writes, if you buffer it the data won't be reliably saved anymore.