Nearly there (Patreon)

Hi!

Just another progress update for Tdarr v2 as it's finally all coming together and is beginning to work quite well (for some, still a few bugs to fix).

I couldn't find an embedded database which could do what I need so I've spent the last week or so improving the existing one. It now has built-in live-backup, UI statuses and is non-blocking so Tdarr can start up straight away without needing to read the db (makes a big difference for those with large libraries). 

Performance-wise, it depends on your drive type/activity but on my SSD I've seen read speeds of up to 30,000 files/second which is a huge improvement over the old one which was around 4000 files/second. These new speeds are achieved by using up to 50 read threads on start-up and my million file test database has been loading up in about 100 seconds (10,000 files/second average - SSD thermal throttling in action after a few seconds I think). Even on my old 5400RPM HDD read speeds have been over 1000 files/second which isn't too bad. However, some on Discord have experienced non-ideal speeds so I'll be improving that this week.

Feature-wise,  there has been a fair bit added since the last post. You can now specify the hardware encoder which you'd like a node to use and workers will check all plugins in a plugin stack to see if they can process any of them before exiting. These two things together allow for situations like the following:

You can have CPU, QSV and NVENC h265 transcoding plugins in a *single* plugin stack. You can then start up a QSV and NVENC node, set 2 CPU workers, 2 QSV workers and 2 NVENC workers and the workers will process your files into h265. To set something up like this previously you'd need to have separate libraries and plugin stacks which was a lot of hassle. I'll write a tutorial on this closer to the release date. This setup has allowed me to use 100% CPU, Q and NVENC on my laptop at the same time which is pretty cool. 

I recently started creating changelogs so I've added the full v2 log so far below for those interested in the other new additions. I'm not planning on adding anything new in the short-term as I'm still trying to hit the release date of 24th of Jan. The next 2 weeks or so will be spent fixing bugs and optimising things. 

Welcome to those all those who have joined since the new year and thanks for the support!

HGG

- Beta v1.99.11 release [10th January 2021]:

- Vastly improved non-blocking database with automatic live backup, UI statuses and multi-threaded reading (up to 50 threads)

- Multi-threaded non-blocking queue builder

- Multi-threaded non-blocking search functionality

- Server now checks if node is same version

- Verbose logs now tagged with 'verbose'.Moved multiple logs to verbose

- Allow retry copying file from cache if error (e.g. if drive is unplugged)

- Absolute backup path is now shown on backups tab

- Auto-scroll to results at top when cliking on pie section on stats tab

- Fix worker hanging bug in some situations (error log too large)

- Fix 'Toggle all' schedule button

- Fix text not showing when editing plugin in WebUI

- Fix cache copy error when accepting a cache file after a plugin has failed

- Fix cache copy error crashing cache copy service

- Fix plugin update bug if default community plugin repo can't be fetched

- Fix library prioritisation not working

- Fix node limit bug whereby updating limits on one node would update others

- Fix worker task request issue if node is not yet registered

- Fix items in Transcode Success/Error and Health check Success/Error not being sorted newest first

- Fix scenario whereby newly transcoded files were being scanned with extra scanners instead of just FFprobe

- Fix folder watcher adding cache files in some situations

- Beta v1.99.10 release [2nd January 2021]:

- Fix folder watcher not detecting new files

- Fix on-demand file scan not showing extra file details in UI

- Fix path bug when using a windows Node with a unix Server

- Workers will check all plugins to see if they can process any of them. Previously, if a GPU transcode plugin was after a CPU transcode plugin, a GPU worker would requeue a file with 'queued:requireCPU' as it would only check up to the CPU plugin.

- Add Node option to specify what type of GPU Node should use. This means that if you have, for example, a QSV and NVENC plugin in the plugin stack, you can set one Node to process with QSV and another (on the same machine) to process with NVENC, maximising hardware usage.

- Stats now record transcode and health check counts + space saved

- Updated startup tests. Group tests together

- Fix Windows HandBrake startup test

- Tdarr will now cleanup any SRTs which CCExtractor makes while scanning.

- Staging section is now saved so the server can be restarted while Nodes are still running (although that is not recommended)

- Option added to requeue files in staging section

- Staging section is more organised and grouped.

- Fix Node path translators bug on Windows

- Fix cache copy service loop bug if error

- Beta v1.99.09 release [31st December 2020]:

- Add basic Node schedules

- Fix toPrecision bug on Tdarr tab

- Fix worker bug- invalid cache file

- Beta v1.99.08 release [30th December 2020]:

- Tdarr Server + WebUI for Arm and MacOS

- Add cache copy service to fix copy bug

- Save worker limits

- Use active instead of used RAM in resource stats

- Option to set plugin repository

- Add history to 'folder to folder conversion'

- Fix bug when cancelling remote workers

- Prevent Node registering if ID is already taken

- chmod binaries on start

- Put Nodes into tabs

- Show plugin stack status in workers section

- Beta v1.99.07 release [27th December 2020]:

- Add error log to failed items

- Add dark, medium theme

- Use memory db for all databases

- Fix GPU health check scan bug

- Fix WebUI URL bug

- Beta v1.99.06 release [26th December 2020]:

- Docker containers

- Add verbose logs to scanner

- Option to set file scanner thread count

- Fix health check worker bug (not processing items)

- Add folder to folder conversion

- Add progress bar to DB read on startup

- Add serverPort config option

- Improved Plugin tab (quicker and more reliable when loading plugins)

- New worker UI

- Option to set webUIPort

- Fix folder browser bug if at root folder

- Fix Search tab bug

- Fix Discord login

- Update default Node IPs to 0.0.0.0

- Beta v1.99.05 release [20th December 2020]:

- Option to auto accept successful transcodes

- Update HandBrake on Windows to 1.3.3

- Improve default paths for binaries

- Add transcode and scanner binary tests on start-up

- Move all module configs + logs into same folder

- Beta v1.99.04 release [17th December 2020]:

- v2 Preview base. New modular system comprised of Tdarr Updater, Server, Node, WebUI and DesktopUI