2025.1.12.0

YT YouTubeSettings: add 'FileAddChannelToFileName' property YouTubeMediaContainerBase: add channel name and video URL to info file; add channel name to file name SCrawler DownDetector: fix 403 error; add 'IDownDetector' interface and 'Checker' class; create an isolated environment API.Instagram: update 'SiteSettings' to the new 'DownDetector' environment; make 'PostKV' public; add static function 'LoadSavePostsKV' API.OnlyFans: add 'EnableCookiesUpdate' hidden property; add support for DRM keys; add the ability to disable cookie updates API.Pinterest: add 'x-pinterest-pws-handler' header API.Reddit: update 'SiteSettings' to the new 'DownDetector' environment API.ThisVid: fix subscription videos images API.Threads: change 'heic' extension to 'jpg' API.Twitter: add broadcasts download API.Xhamster: fix absolute M3U8 URLs API.YouTube: add support of personal API instances ('YouTube-operational-API') for download communities SiteEditorForm: add 'Ctrl+Enter' hotkey to force save settings, ignoring requirements PluginsEnvironment.Attributes: add 'UseDownDetectorAttribute' attribute SettingsHost: update to the new 'DownDetector' environment; add 'AvailableDownDetector' property SettingsHostCollection: update to the new 'DownDetector' environment; minor bugs in multiprofile SettingsCLS: add 'DownDetectorEnabled' property
AAndyProgram · Jan 12, 2025 · 4d74f52 · 4d74f52
1 parent b428327
commit 4d74f52
Show file tree

Hide file tree

Showing 30 changed files with 742 additions and 280 deletions.
diff --git a/Changelog.md b/Changelog.md
@@ -1,3 +1,28 @@
+# 2025.1.12.0
+
+*2025-01-12*
+
+- Added
+  - Sites:
+    - YouTube (standalone app):
+      - ability to add channel name to file name (`Add channel to file name`)
+      - adding channel name and video URL to info file
+    - OnlyFans: **built-in usage of DRM keys**
+    - Threads: automatically change `heic` extension to `jpg`
+    - Twitter: download broadcasts *(user option)*
+  - Minor improvements
+- Updated
+  - yt-dlp up to version **2024.12.23**
+  - gallery-dl up to version **1.28.3**
+  - **OF-Scraper** up to version **3.12.9** *(you must update it personally)*
+- Fixed
+  - Sites:
+    - DownDetector: fixed 403 error
+    - OnlyFans: **DRM videos not downloading**
+    - xHamster: some videos are not downloading
+    - YouTube: **communities are not downloading** *(see settings in wiki)*
+  - Minor bugs
+
 # 2024.11.21.0
 
 *2024-11-21*

diff --git a/FAQ.md b/FAQ.md
@@ -55,7 +55,7 @@ I strongly recommend you to **regularly** create backup copies of the settings f
     - [Video how to configure](#video-how-to-configure)
 - **Antivirus**
     - **Antivirus detects SCrawler as a virus** :arrow_forward: SCrawler doesn't contain any viruses at all. All code is posted on GitHub. You can review it. I have nothing to hide. SCrawler just downloads pictures and videos. That's all. If you trust SCrawler, you should just add it to the antivirus exceptions, as I did. Sometimes antiviruses identify SCawler as a virus. This is usually related to the number of files being edited (users' settings files) and the number of files being downloaded. In this case, the antivirus can also remove these files, which will damage users' settings. **If you don't trust SCrawler, just delete it.**
-    - **Antivirus detects gallery-dl as a virus** :arrow_forward: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it**. **But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself.
+    - **Antivirus detects gallery-dl as a virus** :arrow_forward: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it. But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself.
 
 ## Sites questions
 
@@ -67,7 +67,7 @@ I strongly recommend you to **regularly** create backup copies of the settings f
 - TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download.
 - Porn sites: **COOKIES**!
 - ThisVid: https://github.com/AAndyProgram/SCrawler/wiki/Settings#thisvid-faq
-- **OnlyFans**: cookies + **all fields** + [OF-Scraper (download the correct version that I pointed)](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper) & [mp4decrypt](https://www.bento4.com/downloads/) to download DRM protected videos. [OF-Scraper support](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper-support). Also read [this](https://github.com/AAndyProgram/SCrawler/wiki/Settings#onlyfans-faq)
+- **OnlyFans**: cookies + **all fields** + [OF-Scraper (download the correct version that I pointed)](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper) & [mp4decrypt](https://www.bento4.com/downloads/) & **DRM keys** to download DRM protected videos. [OF-Scraper support](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper-support). Also read [this](https://github.com/AAndyProgram/SCrawler/wiki/Settings#onlyfans-faq)
 - **JustForFans**: **THE VIDEO ISN'T DOWNLOADING AT THE MOMENT** ([Issue](https://discord.com/channels/1124032649682493462/1205547615199039551/1231349555132366870))
 
 ## Other questions

diff --git a/SCrawler.YouTube/Base/YouTubeSettings.vb b/SCrawler.YouTube/Base/YouTubeSettings.vb
@@ -191,7 +191,7 @@ Namespace API.YouTube.Base
         <Browsable(True), GridVisible, XMLVN({"Defaults"}), Category("Defaults"), DisplayName("Use cookies"),
             Description("By default, use cookies when downloading from YouTube.")>
         Public ReadOnly Property DefaultUseCookies As XMLValue(Of Boolean)
-        <Browsable(True), GridVisible, XMLVN({"Defaults"}, Protocols.Any), Category("Defaults"), DisplayName("Protocol"),
+        <Browsable(True), GridVisible, XMLVN({"Defaults"}, Protocols.https), Category("Defaults"), DisplayName("Protocol"),
             Description("Priority download protocol. Default: 'Any'")>
         Public ReadOnly Property DefaultProtocol As XMLValue(Of Protocols)
         <Browsable(True), GridVisible(False), XMLVN({"Defaults"}), Category("Defaults"),
@@ -267,6 +267,9 @@ Namespace API.YouTube.Base
         <Browsable(True), GridVisible, XMLVN({"Defaults"}), Category("Defaults"), DisplayName("Add date to title: video list"),
             Description("Add video upload date before video title (visual only) in the video list")>
         Public ReadOnly Property FileAddDateToFileName_VideoList As XMLValue(Of Boolean)
+        <Browsable(True), GridVisible, XMLVN({"Defaults"}, FileDateMode.None), Category("Defaults"), DisplayName("Add channel to file name"),
+            Description("Add channel name before/after the file name")>
+        Public ReadOnly Property FileAddChannelToFileName As XMLValue(Of FileDateMode)
 #End Region
 #Region "Defaults ChannelsDownload"
         <Browsable(True), GridVisible, XMLVN({"Defaults", "Channels"}), Category("Defaults"), DisplayName("Default download tabs for channels"),

diff --git a/SCrawler.YouTube/My Project/AssemblyInfo.vb b/SCrawler.YouTube/My Project/AssemblyInfo.vb
@@ -13,7 +13,7 @@ Imports System.Runtime.InteropServices
 <Assembly: AssemblyDescription("YouTube plugin environment")>
 <Assembly: AssemblyCompany("AndyProgram")>
 <Assembly: AssemblyProduct("SCrawler.YouTube")>
-<Assembly: AssemblyCopyright("Copyright © 2024")>
+<Assembly: AssemblyCopyright("Copyright © 2025")>
 <Assembly: AssemblyTrademark("AndyProgram")>
 
 <Assembly: ComVisible(False)>
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2024.10.24.0")>
-<Assembly: AssemblyFileVersion("2024.10.24.0")>
+<Assembly: AssemblyVersion("2025.1.12.0")>
+<Assembly: AssemblyFileVersion("2025.1.12.0")>
 <Assembly: NeutralResourcesLanguage("en")>
diff --git a/SCrawler.YouTube/Objects/YouTubeMediaContainerBase.vb b/SCrawler.YouTube/Objects/YouTubeMediaContainerBase.vb
@@ -685,10 +685,17 @@ Namespace API.YouTube.Objects
         Friend Sub FileDateUpdate()
             Dim n$ = _File.Name.StringTrim
             Dim s$ = IIf(n.IsEmptyString, String.Empty, " ")
+            Dim c$ = AccountName.IfNullOrEmpty(UserID)
             Select Case MyYouTubeSettings.FileAddDateToFileName.Value
                 Case FileDateMode.Before : n = $"[{DateAdded:yyyy-MM-dd}]{s}{n}"
                 Case FileDateMode.After : n = $"{n}{s}[{DateAdded:yyyy-MM-dd}]"
             End Select
+            If Not c.IsEmptyString Then
+                Select Case MyYouTubeSettings.FileAddChannelToFileName.Value
+                    Case FileDateMode.Before : n = $"[{c}] {n}"
+                    Case FileDateMode.After : n = $"{n} [{c}]"
+                End Select
+            End If
             _File.Name = n
         End Sub
         Public Property FileSettings As SFile
@@ -1214,6 +1221,9 @@ Namespace API.YouTube.Objects
                                     fileDesr.Extension = "txt"
                                     Using fileDesrText As New TextSaver(fileDesr)
                                         If .CreateDescriptionFiles_AddUploadDate Then fileDesrText.Append($"Uploaded: {DateAdded:yyyy-MM-dd HH:mm:ss}")
+                                        fileDesrText.AppendLine($"URL: {URL}")
+                                        fileDesrText.AppendLine($"Channel name: {AccountName}")
+                                        fileDesrText.AppendLine($"Channel ID: {UserID}")
                                         If Not Description.IsEmptyString Then
                                             If Not fileDesrText.IsEmptyString Then fileDesrText.AppendLine.AppendLine()
                                             fileDesrText.Append(Description)

diff --git a/SCrawler.YouTubeDownloader/My Project/AssemblyInfo.vb b/SCrawler.YouTubeDownloader/My Project/AssemblyInfo.vb
@@ -13,7 +13,7 @@ Imports System.Runtime.InteropServices
 <Assembly: AssemblyDescription("SCrawler YouTube downloader")>
 <Assembly: AssemblyCompany("AndyProgram")>
 <Assembly: AssemblyProduct("SCrawler.YouTubeDownloader")>
-<Assembly: AssemblyCopyright("Copyright © 2024")>
+<Assembly: AssemblyCopyright("Copyright © 2025")>
 <Assembly: AssemblyTrademark("AndyProgram")>
 
 <Assembly: ComVisible(False)>
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2024.10.24.0")>
-<Assembly: AssemblyFileVersion("2024.10.24.0")>
+<Assembly: AssemblyVersion("2025.1.12.0")>
+<Assembly: AssemblyFileVersion("2025.1.12.0")>
 <Assembly: NeutralResourcesLanguage("en")>
diff --git a/SCrawler/API/Base/DownDetector.vb b/SCrawler/API/Base/DownDetector.vb
@@ -6,8 +6,9 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports System.Net
+Imports SCrawler.Plugin
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports Download = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Base
     Friend NotInheritable Class DownDetector
         Private Shared ReadOnly Property Params As New RParams("x:.'([\S]+?)',.y:.(\d+)", -1, Nothing, RegexReturn.List)
@@ -34,34 +35,106 @@ Namespace API.Base
             Try
                 Dim l As List(Of Data) = Nothing
                 Dim l2 As List(Of Data) = Nothing
-                Using w As New WebClient
-                    Dim r$ = w.DownloadString($"https://downdetector.co.uk/status/{Site}/")
-                    If Not r.IsEmptyString Then
-                        l = RegexFields(Of Data)(r, {Params}, {1, 2})
-                        If l.ListExists(2) Then
-                            l.Sort()
-                            l2 = New List(Of Data)
-                            Dim d As Data
-                            Dim eDates As New List(Of Date)
-                            Dim MaxValue As Func(Of Date, Integer) = Function(dd) (From ddd In l Where ddd.Date = dd Select ddd.Value).DefaultIfEmpty(0).Max
-                            For i% = 0 To l.Count - 1
-                                If Not eDates.Contains(l(i).Date) Then
-                                    d = l(i)
-                                    d.Value = MaxValue(d.Date)
-                                    l2.Add(d)
-                                    eDates.Add(d.Date)
-                                End If
-                            Next
-                            eDates.Clear()
-                            l.Clear()
-                            l2.Sort()
-                        End If
+                Dim r$ = GetWebString($"https://downdetector.co.uk/status/{Site}/",, EDP.ThrowException)
+                If Not r.IsEmptyString Then
+                    l = RegexFields(Of Data)(r, {Params}, {1, 2})
+                    If l.ListExists(2) Then
+                        l.Sort()
+                        l2 = New List(Of Data)
+                        Dim d As Data
+                        Dim eDates As New List(Of Date)
+                        Dim MaxValue As Func(Of Date, Integer) = Function(dd) (From ddd As Data In l Where ddd.Date = dd Select ddd.Value).DefaultIfEmpty(0).Max
+                        For i% = 0 To l.Count - 1
+                            If Not eDates.Contains(l(i).Date) Then
+                                d = l(i)
+                                d.Value = MaxValue(d.Date)
+                                l2.Add(d)
+                                eDates.Add(d.Date)
+                            End If
+                        Next
+                        eDates.Clear()
+                        l.Clear()
+                        l2.Sort()
                     End If
-                End Using
+                End If
                 Return l2
             Catch ex As Exception
                 Return ErrorsDescriber.Execute(EDP.SendToLog + EDP.ReturnValue, ex, $"[DownDetector.GetData({Site})]")
             End Try
         End Function
+        Friend Interface IDownDetector
+            ReadOnly Property Value As Integer
+            ReadOnly Property AddToLog As Boolean
+            ReadOnly Property CheckSite As String
+            Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean
+        End Interface
+        Friend Class Checker(Of T As {ISiteSettings, IDownDetector})
+            Protected ReadOnly Property Source As T
+            Private ReadOnly NP As New ANumbers With {.FormatOptions = ANumbers.Options.GroupIntegral}
+            Friend Sub New(ByRef _Source As T)
+                Source = _Source
+            End Sub
+            Private ____AvailableChecked As Boolean = False
+            Private ____AvailableResult As Boolean = False
+            Friend Overridable Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean
+                If Settings.DownDetectorEnabled And Source.Value >= 0 Then
+                    If Not ____AvailableChecked Then
+                        ____AvailableResult = AvailableImpl(What, Silent)
+                        ____AvailableChecked = True
+                    End If
+                    Return ____AvailableResult
+                Else
+                    Return True
+                End If
+            End Function
+            Protected Overridable Function AvailableImpl(ByVal What As Download, ByVal Silent As Boolean) As Boolean
+                Try
+                    Source.AvailableText = String.Empty
+                    If Source.Value < 0 Then
+                        Return True
+                    Else
+                        Dim dl As List(Of Data) = GetData(Source.CheckSite)
+                        If dl.ListExists Then
+                            dl = dl.Take(4).ToList
+                            Dim avg% = dl.Average(Function(d) d.Value)
+                            If avg > Source.Value Then
+                                Source.AvailableText = $"Over the past hour, {Source.Site} has received an average of {avg.NumToString(NP)} outage reports:{vbCr}{dl.ListToString(vbCr)}"
+                                If Source.AddToLog Then MyMainLOG = Source.AvailableText
+                                If Silent Then
+                                    Return AvailableImpl_FALSE_SILENT()
+                                Else
+                                    If MsgBoxE({$"{Source.AvailableText}{vbCr}{vbCr}Do you want to continue parsing {Source.Site} data?",
+                                                $"There are outage reports on {Source.Site}"}, vbYesNo) = vbYes Then
+                                        Return AvailableImpl_FALSE_SILENT_NOT_MSG_YES()
+                                    Else
+                                        Return AvailableImpl_FALSE_SILENT_NOT_MSG_NO()
+                                    End If
+                                End If
+                            End If
+                        End If
+                        Return AvailableImpl_TRUE()
+                    End If
+                Catch ex As Exception
+                    Return ErrorsDescriber.Execute(EDP.SendToLog + EDP.ReturnValue, ex, $"[API.{Source.Site}.SiteSettings.Available([DownDetector])]", True)
+                End Try
+            End Function
+            Protected Overridable Function AvailableImpl_TRUE() As Boolean
+                Return True
+            End Function
+            Protected Overridable Function AvailableImpl_FALSE_SILENT() As Boolean
+                Return False
+            End Function
+            Protected Overridable Function AvailableImpl_FALSE_SILENT_NOT_MSG_YES() As Boolean
+                Return True
+            End Function
+            Protected Overridable Function AvailableImpl_FALSE_SILENT_NOT_MSG_NO() As Boolean
+                Return False
+            End Function
+            Friend Overridable Sub Reset()
+                ____AvailableChecked = False
+                ____AvailableResult = False
+                Source.AvailableText = String.Empty
+            End Sub
+        End Class
     End Class
 End Namespace