Package org.schabi.newpipe.extractor.services.youtube

Class YoutubeParsingHelper

  • public final class YoutubeParsingHelper
extends java.lang.Object

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static java.lang.String CONTENT_CHECK_OK
      A parameter sent by official clients named contentCheckOk.
      static java.lang.String CPN
      A parameter sent by official clients named contentPlaybackNonce.
      static java.lang.String DISABLE_PRETTY_PRINT_PARAMETER
      A parameter to disable pretty-printed response of InnerTube requests, to reduce response sizes.
      static java.lang.String RACY_CHECK_OK
      A parameter which may be sent by official clients named racyCheckOk.
      static java.lang.String VIDEO_ID  
      static java.lang.String YOUTUBEI_V1_GAPIS_URL
      The base URL of requests of non-web clients to the InnerTube internal API.
      static java.lang.String YOUTUBEI_V1_URL
      The base URL of requests of the WEB clients to the InnerTube internal API.

    • Field Detail

      • YOUTUBEI_V1_URL

        public static final java.lang.String YOUTUBEI_V1_URL
        The base URL of requests of the WEB clients to the InnerTube internal API.
        See Also:
        Constant Field Values

      • YOUTUBEI_V1_GAPIS_URL

        public static final java.lang.String YOUTUBEI_V1_GAPIS_URL
        The base URL of requests of non-web clients to the InnerTube internal API.
        See Also:
        Constant Field Values

      • DISABLE_PRETTY_PRINT_PARAMETER

        public static final java.lang.String DISABLE_PRETTY_PRINT_PARAMETER
        A parameter to disable pretty-printed response of InnerTube requests, to reduce response sizes.

        Sent in query parameters of the requests.

        See Also:
        Constant Field Values

      • CPN

        public static final java.lang.String CPN
        A parameter sent by official clients named contentPlaybackNonce.

        It is sent by official clients on videoplayback requests and InnerTube player requests in most cases.

        It is composed of 16 characters which are generated from this alphabet, with the use of strong random values.

        See Also:
        generateContentPlaybackNonce(), Constant Field Values

      • CONTENT_CHECK_OK

        public static final java.lang.String CONTENT_CHECK_OK
        A parameter sent by official clients named contentCheckOk.

        Setting it to true allows us to get streaming data on videos with a warning about what the sensible content they contain.

        See Also:
        Constant Field Values

      • RACY_CHECK_OK

        public static final java.lang.String RACY_CHECK_OK
        A parameter which may be sent by official clients named racyCheckOk.

        What this parameter does is not really known, but it seems to be linked to sensitive contents such as age-restricted content.

        See Also:
        Constant Field Values

    • Method Detail

      • isGoogleURL

        public static boolean isGoogleURL​(java.lang.String url)

      • isYoutubeURL

        public static boolean isYoutubeURL​(@Nonnull
                                   java.net.URL url)

      • isYoutubeServiceURL

        public static boolean isYoutubeServiceURL​(@Nonnull
                                          java.net.URL url)

      • isHooktubeURL

        public static boolean isHooktubeURL​(@Nonnull
                                    java.net.URL url)

      • isInvidiousURL

        public static boolean isInvidiousURL​(@Nonnull
                                     java.net.URL url)

      • isY2ubeURL

        public static boolean isY2ubeURL​(@Nonnull
                                 java.net.URL url)

      • parseDurationString

        public static int parseDurationString​(@Nonnull
                                      java.lang.String input)
                               throws ParsingException,
                                      java.lang.NumberFormatException
        Parses the duration string of the video expecting ":" or "." as separators
        Returns:
        the duration in seconds
        Throws:
        ParsingException - when more than 3 separators are found
        java.lang.NumberFormatException

      • getFeedUrlFrom

        @Nonnull
public static java.lang.String getFeedUrlFrom​(@Nonnull
                                              java.lang.String channelIdOrUser)

      • isYoutubeMixId

        public static boolean isYoutubeMixId​(@Nonnull
                                     java.lang.String playlistId)
        Checks if the given playlist id is a YouTube Mix (auto-generated playlist) Ids from a YouTube Mix start with "RD"
        Parameters:
        playlistId - the playlist id
        Returns:
        Whether given id belongs to a YouTube Mix

      • isYoutubeMyMixId

        public static boolean isYoutubeMyMixId​(@Nonnull
                                       java.lang.String playlistId)
        Checks if the given playlist id is a YouTube My Mix (auto-generated playlist) Ids from a YouTube My Mix start with "RDMM"
        Parameters:
        playlistId - the playlist id
        Returns:
        Whether given id belongs to a YouTube My Mix

      • isYoutubeMusicMixId

        public static boolean isYoutubeMusicMixId​(@Nonnull
                                          java.lang.String playlistId)
        Checks if the given playlist id is a YouTube Music Mix (auto-generated playlist) Ids from a YouTube Music Mix start with "RDAMVM" or "RDCLAK"
        Parameters:
        playlistId - the playlist id
        Returns:
        Whether given id belongs to a YouTube Music Mix

      • isYoutubeChannelMixId

        public static boolean isYoutubeChannelMixId​(@Nonnull
                                            java.lang.String playlistId)
        Checks if the given playlist id is a YouTube Channel Mix (auto-generated playlist) Ids from a YouTube channel Mix start with "RDCM"
        Returns:
        Whether given id belongs to a YouTube Channel Mix

      • isYoutubeGenreMixId

        public static boolean isYoutubeGenreMixId​(@Nonnull
                                          java.lang.String playlistId)
        Checks if the given playlist id is a YouTube Genre Mix (auto-generated playlist) Ids from a YouTube Genre Mix start with "RDGMEM"
        Returns:
        Whether given id belongs to a YouTube Genre Mix

      • extractVideoIdFromMixId

        @Nonnull
public static java.lang.String extractVideoIdFromMixId​(java.lang.String playlistId)
                                                throws ParsingException
        Parameters:
        playlistId - the playlist id to parse
        Returns:
        the PlaylistInfo.PlaylistType extracted from the playlistId (mix playlist types included)
        Throws:
        ParsingException - if the playlistId is null or empty, if the playlistId is not a mix, if it is a mix but it's not based on a specific stream (this is the case for channel or genre mixes)

      • extractPlaylistTypeFromPlaylistUrl

        public static PlaylistInfo.PlaylistType extractPlaylistTypeFromPlaylistUrl​(java.lang.String playlistUrl)
                                                                    throws ParsingException
        Parameters:
        playlistUrl - the playlist url to parse
        Returns:
        the PlaylistInfo.PlaylistType extracted from the playlistUrl's list param (mix playlist types included)
        Throws:
        ParsingException - if the playlistUrl is malformed, if has no list param or if the list param is empty

      • getClientVersion

        public static java.lang.String getClientVersion()
                                         throws java.io.IOException,
                                                ExtractionException
        Get the client version used by YouTube website on InnerTube requests.
        Throws:
        java.io.IOException
        ExtractionException

      • resetClientVersion

        public static void resetClientVersion()

        Only used in tests.

        Quick-and-dirty solution to reset global state in between test classes.

        This is needed for the mocks because in order to reach that state a network request has to be made. If the global state is not reset and the RecordingDownloader is used, then only the first test class has that request recorded. Meaning running the other tests with mocks will fail, because the mock is missing.

      • setNumberGenerator

        public static void setNumberGenerator​(java.util.Random random)

        Only used in tests.

      • isHardcodedYoutubeMusicClientVersionValid

        public static boolean isHardcodedYoutubeMusicClientVersionValid()
                                                         throws java.io.IOException,
                                                                ReCaptchaException
        Throws:
        java.io.IOException
        ReCaptchaException

      • getUrlFromNavigationEndpoint

        @Nullable
public static java.lang.String getUrlFromNavigationEndpoint​(@Nonnull
                                                            com.grack.nanojson.JsonObject navigationEndpoint)

      • getTextFromObject

        @Nullable
public static java.lang.String getTextFromObject​(com.grack.nanojson.JsonObject textObject,
                                                 boolean html)
        Get the text from a JSON object that has either a simpleText or a runs array.
        Parameters:
        textObject - JSON object to get the text from
        html - whether to return HTML, by parsing the navigationEndpoint
        Returns:
        text in the JSON object or null

      • getTextFromObjectOrThrow

        @Nonnull
public static java.lang.String getTextFromObjectOrThrow​(com.grack.nanojson.JsonObject textObject,
                                                        java.lang.String error)
                                                 throws ParsingException
        Throws:
        ParsingException

      • getTextFromObject

        @Nullable
public static java.lang.String getTextFromObject​(com.grack.nanojson.JsonObject textObject)

      • getUrlFromObject

        @Nullable
public static java.lang.String getUrlFromObject​(com.grack.nanojson.JsonObject textObject)

      • getTextAtKey

        @Nullable
public static java.lang.String getTextAtKey​(@Nonnull
                                            com.grack.nanojson.JsonObject jsonObject,
                                            java.lang.String theKey)

      • fixThumbnailUrl

        public static java.lang.String fixThumbnailUrl​(@Nonnull
                                               java.lang.String thumbnailUrl)

      • getImagesFromThumbnailsArray

        @Nonnull
public static java.util.List<Image> getImagesFromThumbnailsArray​(@Nonnull
                                                                 com.grack.nanojson.JsonArray thumbnails)
        Get images from a YouTube thumbnails JsonArray.

        The properties of the Images created will be set using the corresponding ones of thumbnail items.

        Parameters:
        thumbnails - a YouTube thumbnails JsonArray
        Returns:
        an unmodifiable list of Images extracted from the given JsonArray

      • getValidJsonResponseBody

        @Nonnull
public static java.lang.String getValidJsonResponseBody​(@Nonnull
                                                        Response response)
                                                 throws ParsingException,
                                                        java.net.MalformedURLException
        Throws:
        ParsingException
        java.net.MalformedURLException

      • getJsonPostResponse

        public static com.grack.nanojson.JsonObject getJsonPostResponse​(java.lang.String endpoint,
                                                                byte[] body,
                                                                Localization localization)
                                                         throws java.io.IOException,
                                                                ExtractionException
        Throws:
        java.io.IOException
        ExtractionException

      • getJsonAndroidPostResponse

        public static com.grack.nanojson.JsonObject getJsonAndroidPostResponse​(java.lang.String endpoint,
                                                                       byte[] body,
                                                                       @Nonnull
                                                                       Localization localization,
                                                                       @Nullable
                                                                       java.lang.String endPartOfUrlRequest)
                                                                throws java.io.IOException,
                                                                       ExtractionException
        Throws:
        java.io.IOException
        ExtractionException

      • getJsonIosPostResponse

        public static com.grack.nanojson.JsonObject getJsonIosPostResponse​(java.lang.String endpoint,
                                                                   byte[] body,
                                                                   @Nonnull
                                                                   Localization localization,
                                                                   @Nullable
                                                                   java.lang.String endPartOfUrlRequest)
                                                            throws java.io.IOException,
                                                                   ExtractionException
        Throws:
        java.io.IOException
        ExtractionException

      • prepareDesktopJsonBuilder

        @Nonnull
public static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject> prepareDesktopJsonBuilder​(@Nonnull
                                                                                                      Localization localization,
                                                                                                      @Nonnull
                                                                                                      ContentCountry contentCountry,
                                                                                                      @Nullable
                                                                                                      java.lang.String visitorData)
                                                                                               throws java.io.IOException,
                                                                                                      ExtractionException
        Throws:
        java.io.IOException
        ExtractionException

      • prepareAndroidMobileJsonBuilder

        @Nonnull
public static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject> prepareAndroidMobileJsonBuilder​(@Nonnull
                                                                                                            Localization localization,
                                                                                                            @Nonnull
                                                                                                            ContentCountry contentCountry)

      • prepareIosMobileJsonBuilder

        @Nonnull
public static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject> prepareIosMobileJsonBuilder​(@Nonnull
                                                                                                        Localization localization,
                                                                                                        @Nonnull
                                                                                                        ContentCountry contentCountry)

      • prepareTvHtml5EmbedJsonBuilder

        @Nonnull
public static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject> prepareTvHtml5EmbedJsonBuilder​(@Nonnull
                                                                                                           Localization localization,
                                                                                                           @Nonnull
                                                                                                           ContentCountry contentCountry,
                                                                                                           @Nonnull
                                                                                                           java.lang.String videoId)

      • createTvHtml5EmbedPlayerBody

        @Nonnull
public static byte[] createTvHtml5EmbedPlayerBody​(@Nonnull
                                                  Localization localization,
                                                  @Nonnull
                                                  ContentCountry contentCountry,
                                                  @Nonnull
                                                  java.lang.String videoId,
                                                  @Nonnull
                                                  java.lang.Integer sts,
                                                  @Nonnull
                                                  java.lang.String contentPlaybackNonce)

      • getAndroidUserAgent

        @Nonnull
public static java.lang.String getAndroidUserAgent​(@Nullable
                                                   Localization localization)
        Get the user-agent string used as the user-agent for InnerTube requests with the Android client.

        If the Localization provided is null, fallbacks to the default one.

        Parameters:
        localization - the Localization to set in the user-agent
        Returns:
        the Android user-agent used for InnerTube requests with the Android client, depending on the Localization provided

      • getIosUserAgent

        @Nonnull
public static java.lang.String getIosUserAgent​(@Nullable
                                               Localization localization)
        Get the user-agent string used as the user-agent for InnerTube requests with the iOS client.

        If the Localization provided is null, fallbacks to the default one.

        Parameters:
        localization - the Localization to set in the user-agent
        Returns:
        the iOS user-agent used for InnerTube requests with the iOS client, depending on the Localization provided

      • getYoutubeMusicHeaders

        @Nonnull
public static java.util.Map<java.lang.String,​java.util.List<java.lang.String>> getYoutubeMusicHeaders()
        Returns a Map containing the required YouTube Music headers.

      • getYouTubeHeaders

        public static java.util.Map<java.lang.String,​java.util.List<java.lang.String>> getYouTubeHeaders()
                                                                                                throws ExtractionException,
                                                                                                       java.io.IOException
        Returns a Map containing the required YouTube headers, including the CONSENT cookie to prevent redirects to consent.youtube.com
        Throws:
        ExtractionException
        java.io.IOException

      • getClientInfoHeaders

        public static java.util.Map<java.lang.String,​java.util.List<java.lang.String>> getClientInfoHeaders()
                                                                                                   throws ExtractionException,
                                                                                                          java.io.IOException
        Returns a Map containing the X-YouTube-Client-Name, X-YouTube-Client-Version, Origin, and Referer headers.
        Throws:
        ExtractionException
        java.io.IOException

      • getCookieHeader

        public static java.util.Map<java.lang.String,​java.util.List<java.lang.String>> getCookieHeader()
        Create a map with the required cookie header.
        Returns:
        A singleton map containing the header.

      • generateConsentCookie

        @Nonnull
public static java.lang.String generateConsentCookie()

      • extractCookieValue

        public static java.lang.String extractCookieValue​(java.lang.String cookieName,
                                                  @Nonnull
                                                  Response response)

      • defaultAlertsCheck

        public static void defaultAlertsCheck​(@Nonnull
                                      com.grack.nanojson.JsonObject initialData)
                               throws ParsingException
        Shared alert detection function, multiple endpoints return the error similarly structured.

        Will check if the object has an alert of the type "ERROR".

        Parameters:
        initialData - the object which will be checked if an alert is present
        Throws:
        ContentNotAvailableException - if an alert is detected
        ParsingException

      • extractCachedUrlIfNeeded

        public static java.lang.String extractCachedUrlIfNeeded​(java.lang.String url)
        Sometimes, YouTube provides URLs which use Google's cache. They look like https://webcache.googleusercontent.com/search?q=cache:CACHED_URL
        Parameters:
        url - the URL which might refer to the Google's webcache
        Returns:
        the URL which is referring to the original site

      • isVerified

        public static boolean isVerified​(com.grack.nanojson.JsonArray badges)

      • hasArtistOrVerifiedIconBadgeAttachment

        public static boolean hasArtistOrVerifiedIconBadgeAttachment​(@Nonnull
                                                             com.grack.nanojson.JsonArray attachmentRuns)

      • generateContentPlaybackNonce

        @Nonnull
public static java.lang.String generateContentPlaybackNonce()
        Generate a content playback nonce (also called cpn), sent by YouTube clients in playback requests (and also for some clients, in the player request body).
        Returns:
        a content playback nonce string

      • generateTParameter

        @Nonnull
public static java.lang.String generateTParameter()
        Try to generate a t parameter, sent by mobile clients as a query of the player request.

        Some researches needs to be done to know how this parameter, unique at each request, is generated.

        Returns:
        a 12 characters string to try to reproduce the parameter

      • isWebStreamingUrl

        public static boolean isWebStreamingUrl​(@Nonnull
                                        java.lang.String url)
        Check if the streaming URL is from the YouTube WEB client.
        Parameters:
        url - the streaming URL to be checked.
        Returns:
        true if it's a WEB streaming URL, false otherwise

      • isTvHtml5SimplyEmbeddedPlayerStreamingUrl

        public static boolean isTvHtml5SimplyEmbeddedPlayerStreamingUrl​(@Nonnull
                                                                java.lang.String url)
        Check if the streaming URL is a URL from the YouTube TVHTML5_SIMPLY_EMBEDDED_PLAYER client.
        Parameters:
        url - the streaming URL on which check if it's a TVHTML5_SIMPLY_EMBEDDED_PLAYER streaming URL.
        Returns:
        true if it's a TVHTML5_SIMPLY_EMBEDDED_PLAYER streaming URL, false otherwise

      • isAndroidStreamingUrl

        public static boolean isAndroidStreamingUrl​(@Nonnull
                                            java.lang.String url)
        Check if the streaming URL is a URL from the YouTube ANDROID client.
        Parameters:
        url - the streaming URL to be checked.
        Returns:
        true if it's a ANDROID streaming URL, false otherwise

      • isIosStreamingUrl

        public static boolean isIosStreamingUrl​(@Nonnull
                                        java.lang.String url)
        Check if the streaming URL is a URL from the YouTube IOS client.
        Parameters:
        url - the streaming URL on which check if it's a IOS streaming URL.
        Returns:
        true if it's a IOS streaming URL, false otherwise

      • setConsentAccepted

        public static void setConsentAccepted​(boolean accepted)
        Determines how the consent cookie that is required for YouTube, SOCS, will be generated.
        • false (the default value) will use CAE=;
        • true will use CAISAiAD.

        Setting this value to true is needed to extract mixes and some YouTube Music playlists in some countries such as the EU ones.

      • isConsentAccepted

        public static boolean isConsentAccepted()
        Get the value of the consent's acceptance.
        Returns:
        the consent's acceptance value
        See Also:
        setConsentAccepted(boolean)

      • extractAudioTrackType

        @Nullable
public static AudioTrackType extractAudioTrackType​(java.lang.String streamUrl)
        Extract the audio track type from a YouTube stream URL.

        The track type is parsed from the xtags URL parameter (Example: acont=original:lang=en).

        Parameters:
        streamUrl - YouTube stream URL
        Returns:
        AudioTrackType or null if no track type was found