pro_latest v1.2064 release [17th September 2020] (Patreon)

Hi all,

I've just released a pretty big update to pro_latest and pro_1.2064 which addresses quite a few bugs and adds some new features. All in all, there have been changes to 57 files with 4,896 additions and 4,346 deletions.

The biggest change is to how much RAM Tdarr uses. In my mock 100,000 library test, with each file having a reasonable 4 streams, Tdarr used 504MB RAM - just above the target. At 200,000 files it used 950MB while building the queues in just 12 seconds whereas previously it would have taken several minutes with that number of files. I have no comparison to give with the previous pro_latest version as it crashes if I try to use a library of that size but the RAM usage is anywhere from 4-20x less depending on what Tdarr is doing. 

Some more RAM stats (I should do a graph of this at some point):

Idle @ 1 file - 43.4MB

Old @

 10,000 files - 221.3MB

 20,000 files - 429.2MB

 30,000 files - 550MB

New @ 

 10,000 files - 91.8MB

 20,000 files - 139.5MB

 30,000 files - 180MB

The new file scanner is also 10-20 times faster from my tests. Previously, files were scanned with FFprobe, Exiftool, MediaInfo and optionally CCExtractor. Now, files are initially only scanned with FFprobe and the data is then stripped down to reduce RAM usage. When tasks are picked up by workers, the files are then scanned with FFprobe, Exiftool, MediaInfo and CCExtractor to make the full file data available temporarily for plugins etc. To see full file data for an individual file, just click on a file name on one of the sub-tabs on the 'Tdarr' tab. An on-demand file scan will then take place making it easy to see if your plugins are working as expected.

Below are the changes since the last update to the pro_latest container. Thanks for the support!

HGG

- Beta v1.206 release [17th September 2020]:

--------IMPORTANT----------------

- New lean database to reduce RAM usage.

- Files are only scanned with FFprobe (and optionally CC Checker) during a library scan - the initial scan time is now much faster

- This scan data is then stripped down to contain only the necessary data which Tdarr needs

- Files are then scanned more thoroughly before plugins are executed so that full FFprobe and ExifTool data is available

---------------------------------

- Revert queue building being in separate process (reduce RAM issues)

- Fixed bug whereby plugin validator wasn't running when toggling between "Community" and "Local" checkbox

- Now no need to refresh plugin stack page after adding a new Local/Community plugin to the plugins folder.

- Now no need to refresh page when saving an edited plugin on the "Plugins tab"

- Now no need to refresh page after pressing "Copy to Local" on a Community plugin

- Now no need to refresh page after creating new plugin in Plugin Creator

- Some styling changes to pages to make more use of space (For example the library nav bar has been moved above the settings to give more width for the plugin stack and schedule)

- Schedule now highlights current schedule block to make scheduling easier

- Added warning if library source folder or transcode cache are empty

---------v1.2061

- Click on file name on any of the tabs on the Tdarr tab to show full file details (on-demand file scan is carried out)

---------v1.2062

- Fixed bug with worker percentages during FFmpeg health check

---------v1.2063

- Fixed 'Scan on start' file scanner bug

- Fixed RAM displaying incorrectly

---------v1.2064

- Fixed slowness bug when 'Library prioritisation' enabled

- MediaInfo scan re-added

- Beta v1.2051 release [6th September 2020]:

- Add 2 second delay before deleting/renaming/moving files to prevent 'lock' permission errors after transcode

- Add worker spawn delay

- Prevent transcode workers from requesting item while queue is being built

- Beta v1.205 release [20th August 2020]:

- Clean-up/reduce size of files' log and history on start-up

- Add resource stats to dashboard for process and OS used/free memory as well as OS CPU%