org.schabi.newpipe.extractor.services.youtube

Class YoutubeParsingHelper

java.lang.Object

org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper

public final class YoutubeParsingHelper
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONTENT_CHECK_OK A parameter sent by official clients named contentCheckOk.
static String	CPN A parameter sent by official clients named contentPlaybackNonce.
static String	DISABLE_PRETTY_PRINT_PARAMETER A parameter to disable pretty-printed response of InnerTube requests, to reduce response sizes.
static String	RACY_CHECK_OK A parameter which may be send by official clients named racyCheckOk.
static String	VIDEO_ID
static String	YOUTUBEI_V1_GAPIS_URL The base URL of requests of non-web clients to the InnerTube internal API.
static String	YOUTUBEI_V1_URL The base URL of requests of the WEB clients to the InnerTube internal API.

Method Summary

Modifier and Type	Method and Description
static void	addClientInfoHeaders(Map<String,List<String>> headers) Add the X-YouTube-Client-Name, X-YouTube-Client-Version, Origin, and Referer headers.
static void	addCookieHeader(Map<String,List<String>> headers) Add the CONSENT cookie to prevent redirect to consent.youtube.com
static void	addYouTubeHeaders(Map<String,List<String>> headers) Add required headers and cookies to an existing headers Map.
static boolean	areHardcodedClientVersionAndKeyValid()
static byte[]	createDesktopPlayerBody(Localization localization, ContentCountry contentCountry, String videoId, String sts, boolean isTvHtml5DesktopJsonBuilder, String contentPlaybackNonce)
static void	defaultAlertsCheck(com.grack.nanojson.JsonObject initialData) Shared alert detection function, multiple endpoints return the error similarly structured.
static String	extractCachedUrlIfNeeded(String url) Sometimes, YouTube provides URLs which use Google's cache.
static String	extractCookieValue(String cookieName, Response response)
static PlaylistInfo.PlaylistType	extractPlaylistTypeFromPlaylistId(String playlistId)
static PlaylistInfo.PlaylistType	extractPlaylistTypeFromPlaylistUrl(String playlistUrl)
static String	extractVideoIdFromMixId(String playlistId)
static String	fixThumbnailUrl(String thumbnailUrl)
static String	generateConsentCookie()
static String	generateContentPlaybackNonce() Generate a content playback nonce (also called cpn), sent by YouTube clients in playback requests (and also for some clients, in the player request body).
static String	generateTParameter() Try to generate a t parameter, sent by mobile clients as a query of the player request.
static String	getAndroidUserAgent(Localization localization) Get the user-agent string used as the user-agent for InnerTube requests with the Android client.
static String	getClientVersion() Get the client version used by YouTube website on InnerTube requests.
static String	getFeedUrlFrom(String channelIdOrUser)
static String	getIosUserAgent(Localization localization) Get the user-agent string used as the user-agent for InnerTube requests with the iOS client.
static com.grack.nanojson.JsonObject	getJsonAndroidPostResponse(String endpoint, byte[] body, Localization localization, String endPartOfUrlRequest)
static com.grack.nanojson.JsonObject	getJsonIosPostResponse(String endpoint, byte[] body, Localization localization, String endPartOfUrlRequest)
static com.grack.nanojson.JsonObject	getJsonPostResponse(String endpoint, byte[] body, Localization localization)
static String	getKey() Get the internal API key used by YouTube website on InnerTube requests.
static List<MetaInfo>	getMetaInfo(com.grack.nanojson.JsonArray contents)
static String	getTextAtKey(com.grack.nanojson.JsonObject jsonObject, String theKey)
static String	getTextFromObject(com.grack.nanojson.JsonObject textObject)
static String	getTextFromObject(com.grack.nanojson.JsonObject textObject, boolean html) Get the text from a JSON object that has either a simpleText or a runs array.
static String	getThumbnailUrlFromInfoItem(com.grack.nanojson.JsonObject infoItem)
static String	getUrlFromNavigationEndpoint(com.grack.nanojson.JsonObject navigationEndpoint)
static String	getValidJsonResponseBody(Response response)
static String[]	getYoutubeMusicKey()
static boolean	isHardcodedYoutubeMusicKeyValid()
static boolean	isHooktubeURL(URL url)
static boolean	isInvidioURL(URL url)
static boolean	isVerified(com.grack.nanojson.JsonArray badges)
static boolean	isY2ubeURL(URL url)
static boolean	isYoutubeChannelMixId(String playlistId) Checks if the given playlist id is a YouTube Channel Mix (auto-generated playlist) Ids from a YouTube channel Mix start with "RDCM"
static boolean	isYoutubeGenreMixId(String playlistId) Checks if the given playlist id is a YouTube Genre Mix (auto-generated playlist) Ids from a YouTube Genre Mix start with "RDGMEM"
static boolean	isYoutubeMixId(String playlistId) Checks if the given playlist id is a YouTube Mix (auto-generated playlist) Ids from a YouTube Mix start with "RD"
static boolean	isYoutubeMusicMixId(String playlistId) Checks if the given playlist id is a YouTube Music Mix (auto-generated playlist) Ids from a YouTube Music Mix start with "RDAMVM" or "RDCLAK"
static boolean	isYoutubeMyMixId(String playlistId) Checks if the given playlist id is a YouTube My Mix (auto-generated playlist) Ids from a YouTube My Mix start with "RDMM"
static boolean	isYoutubeServiceURL(URL url)
static boolean	isYoutubeURL(URL url)
static OffsetDateTime	parseDateFrom(String textualUploadDate)
static int	parseDurationString(String input) Parses the duration string of the video expecting ":" or "." as separators
static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject>	prepareAndroidMobileJsonBuilder(Localization localization, ContentCountry contentCountry)
static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject>	prepareDesktopJsonBuilder(Localization localization, ContentCountry contentCountry)
static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject>	prepareIosMobileJsonBuilder(Localization localization, ContentCountry contentCountry)
static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject>	prepareTvHtml5EmbedJsonBuilder(Localization localization, ContentCountry contentCountry, String videoId)
static void	resetClientVersionAndKey() Only used in tests.
static void	setNumberGenerator(Random random) Only used in tests.
static String	unescapeDocument(String doc)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

YOUTUBEI_V1_URL

public static final String YOUTUBEI_V1_URL

The base URL of requests of the WEB clients to the InnerTube internal API.

See Also:

Constant Field Values

YOUTUBEI_V1_GAPIS_URL

public static final String YOUTUBEI_V1_GAPIS_URL

The base URL of requests of non-web clients to the InnerTube internal API.

See Also:

Constant Field Values

DISABLE_PRETTY_PRINT_PARAMETER

public static final String DISABLE_PRETTY_PRINT_PARAMETER

A parameter to disable pretty-printed response of InnerTube requests, to reduce response sizes.

Sent in query parameters of the requests, after the API key.

See Also:

Constant Field Values

CPN

public static final String CPN

A parameter sent by official clients named contentPlaybackNonce.

It is sent by official clients on videoplayback requests, and by all clients (except the WEB one to the player requests.

It is composed of 16 characters which are generated from this alphabet, with the use of strong random values.

See Also:

generateContentPlaybackNonce(), Constant Field Values

VIDEO_ID

public static final String VIDEO_ID

See Also:

Constant Field Values

CONTENT_CHECK_OK

public static final String CONTENT_CHECK_OK

A parameter sent by official clients named contentCheckOk.

Setting it to true allows us to get streaming data on videos with a warning about what the sensible content they contain.

See Also:

Constant Field Values

RACY_CHECK_OK

public static final String RACY_CHECK_OK

A parameter which may be send by official clients named racyCheckOk.

What this parameter does is not really known, but it seems to be linked to sensitive contents such as age-restricted content.

See Also:

Constant Field Values

Method Detail

isYoutubeURL

public static boolean isYoutubeURL(@Nonnull
                                   URL url)

isYoutubeServiceURL

public static boolean isYoutubeServiceURL(@Nonnull
                                          URL url)

isHooktubeURL

public static boolean isHooktubeURL(@Nonnull
                                    URL url)

isInvidioURL

public static boolean isInvidioURL(@Nonnull
                                   URL url)

isY2ubeURL

public static boolean isY2ubeURL(@Nonnull
                                 URL url)

parseDurationString

public static int parseDurationString(@Nonnull
                                      String input)
                               throws ParsingException,
                                      NumberFormatException

Parses the duration string of the video expecting ":" or "." as separators

Returns:

the duration in seconds

Throws:

ParsingException - when more than 3 separators are found

NumberFormatException

getFeedUrlFrom

@Nonnull
public static String getFeedUrlFrom(@Nonnull
                                             String channelIdOrUser)

parseDateFrom

public static OffsetDateTime parseDateFrom(String textualUploadDate)
                                    throws ParsingException

Throws:

ParsingException

isYoutubeMixId

public static boolean isYoutubeMixId(@Nonnull
                                     String playlistId)

Checks if the given playlist id is a YouTube Mix (auto-generated playlist) Ids from a YouTube Mix start with "RD"

Parameters:

playlistId - the playlist id

Returns:

Whether given id belongs to a YouTube Mix

isYoutubeMyMixId

public static boolean isYoutubeMyMixId(@Nonnull
                                       String playlistId)

Checks if the given playlist id is a YouTube My Mix (auto-generated playlist) Ids from a YouTube My Mix start with "RDMM"

Parameters:

playlistId - the playlist id

Returns:

Whether given id belongs to a YouTube My Mix

isYoutubeMusicMixId

public static boolean isYoutubeMusicMixId(@Nonnull
                                          String playlistId)

Checks if the given playlist id is a YouTube Music Mix (auto-generated playlist) Ids from a YouTube Music Mix start with "RDAMVM" or "RDCLAK"

Parameters:

playlistId - the playlist id

Returns:

Whether given id belongs to a YouTube Music Mix

isYoutubeChannelMixId

public static boolean isYoutubeChannelMixId(@Nonnull
                                            String playlistId)

Checks if the given playlist id is a YouTube Channel Mix (auto-generated playlist) Ids from a YouTube channel Mix start with "RDCM"

Returns:

Whether given id belongs to a YouTube Channel Mix

isYoutubeGenreMixId

public static boolean isYoutubeGenreMixId(@Nonnull
                                          String playlistId)

Checks if the given playlist id is a YouTube Genre Mix (auto-generated playlist) Ids from a YouTube Genre Mix start with "RDGMEM"

Returns:

Whether given id belongs to a YouTube Genre Mix

extractVideoIdFromMixId

@Nonnull
public static String extractVideoIdFromMixId(String playlistId)
                                               throws ParsingException

Parameters:

playlistId - the playlist id to parse

Returns:

the PlaylistInfo.PlaylistType extracted from the playlistId (mix playlist types included)

Throws:

ParsingException - if the playlistId is null or empty, if the playlistId is not a mix, if it is a mix but it's not based on a specific stream (this is the case for channel or genre mixes)

extractPlaylistTypeFromPlaylistId

@Nonnull
public static PlaylistInfo.PlaylistType extractPlaylistTypeFromPlaylistId(String playlistId)
                                                                            throws ParsingException

Parameters:

playlistId - the playlist id to parse

Returns:

the PlaylistInfo.PlaylistType extracted from the playlistId (mix playlist types included)

Throws:

ParsingException - if the playlistId is null or empty

extractPlaylistTypeFromPlaylistUrl

public static PlaylistInfo.PlaylistType extractPlaylistTypeFromPlaylistUrl(String playlistUrl)
                                                                    throws ParsingException

Parameters:

playlistUrl - the playlist url to parse

Returns:

the PlaylistInfo.PlaylistType extracted from the playlistUrl's list param (mix playlist types included)

Throws:

ParsingException - if the playlistUrl is malformed, if has no list param or if the list param is empty

areHardcodedClientVersionAndKeyValid

public static boolean areHardcodedClientVersionAndKeyValid()
                                                    throws IOException,
                                                           ExtractionException

Throws:

IOException

ExtractionException

getClientVersion

public static String getClientVersion()
                               throws IOException,
                                      ExtractionException

Get the client version used by YouTube website on InnerTube requests.

Throws:

IOException

ExtractionException

getKey

public static String getKey()
                     throws IOException,
                            ExtractionException

Get the internal API key used by YouTube website on InnerTube requests.

Throws:

IOException

ExtractionException

resetClientVersionAndKey

public static void resetClientVersionAndKey()

Only used in tests.

Quick-and-dirty solution to reset global state in between test classes.

This is needed for the mocks because in order to reach that state a network request has to be made. If the global state is not reset and the RecordingDownloader is used, then only the first test class has that request recorded. Meaning running the other tests with mocks will fail, because the mock is missing.

setNumberGenerator

public static void setNumberGenerator(Random random)

Only used in tests.

isHardcodedYoutubeMusicKeyValid

public static boolean isHardcodedYoutubeMusicKeyValid()
                                               throws IOException,
                                                      ReCaptchaException

Throws:

IOException

ReCaptchaException

getYoutubeMusicKey

public static String[] getYoutubeMusicKey()
                                   throws IOException,
                                          ReCaptchaException,
                                          Parser.RegexException

Throws:

IOException

ReCaptchaException

Parser.RegexException

getUrlFromNavigationEndpoint

@Nullable
public static String getUrlFromNavigationEndpoint(@Nonnull
                                                            com.grack.nanojson.JsonObject navigationEndpoint)
                                                     throws ParsingException

Throws:

ParsingException

getTextFromObject

@Nullable
public static String getTextFromObject(com.grack.nanojson.JsonObject textObject,
                                                 boolean html)
                                          throws ParsingException

Get the text from a JSON object that has either a simpleText or a runs array.

Parameters:

textObject - JSON object to get the text from

html - whether to return HTML, by parsing the navigationEndpoint

Returns:

text in the JSON object or null

Throws:

ParsingException

getTextFromObject

@Nullable
public static String getTextFromObject(com.grack.nanojson.JsonObject textObject)
                                          throws ParsingException

Throws:

ParsingException

getTextAtKey

@Nullable
public static String getTextAtKey(@Nonnull
                                            com.grack.nanojson.JsonObject jsonObject,
                                            String theKey)
                                     throws ParsingException

Throws:

ParsingException

fixThumbnailUrl

public static String fixThumbnailUrl(@Nonnull
                                     String thumbnailUrl)

getThumbnailUrlFromInfoItem

public static String getThumbnailUrlFromInfoItem(com.grack.nanojson.JsonObject infoItem)
                                          throws ParsingException

Throws:

ParsingException

getValidJsonResponseBody

@Nonnull
public static String getValidJsonResponseBody(@Nonnull
                                                       Response response)
                                                throws ParsingException,
                                                       MalformedURLException

Throws:

ParsingException

MalformedURLException

getJsonPostResponse

public static com.grack.nanojson.JsonObject getJsonPostResponse(String endpoint,
                                                                byte[] body,
                                                                Localization localization)
                                                         throws IOException,
                                                                ExtractionException

Throws:

IOException

ExtractionException

getJsonAndroidPostResponse

public static com.grack.nanojson.JsonObject getJsonAndroidPostResponse(String endpoint,
                                                                       byte[] body,
                                                                       @Nonnull
                                                                       Localization localization,
                                                                       @Nullable
                                                                       String endPartOfUrlRequest)
                                                                throws IOException,
                                                                       ExtractionException

Throws:

IOException

ExtractionException

getJsonIosPostResponse

public static com.grack.nanojson.JsonObject getJsonIosPostResponse(String endpoint,
                                                                   byte[] body,
                                                                   @Nonnull
                                                                   Localization localization,
                                                                   @Nullable
                                                                   String endPartOfUrlRequest)
                                                            throws IOException,
                                                                   ExtractionException

Throws:

IOException

ExtractionException

prepareDesktopJsonBuilder

@Nonnull
public static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject> prepareDesktopJsonBuilder(@Nonnull
                                                                                                               Localization localization,
                                                                                                               @Nonnull
                                                                                                               ContentCountry contentCountry)
                                                                                                        throws IOException,
                                                                                                               ExtractionException

Throws:

IOException

ExtractionException

prepareAndroidMobileJsonBuilder

@Nonnull
public static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject> prepareAndroidMobileJsonBuilder(@Nonnull
                                                                                                                     Localization localization,
                                                                                                                     @Nonnull
                                                                                                                     ContentCountry contentCountry)

prepareIosMobileJsonBuilder

@Nonnull
public static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject> prepareIosMobileJsonBuilder(@Nonnull
                                                                                                                 Localization localization,
                                                                                                                 @Nonnull
                                                                                                                 ContentCountry contentCountry)

prepareTvHtml5EmbedJsonBuilder

@Nonnull
public static com.grack.nanojson.JsonBuilder<com.grack.nanojson.JsonObject> prepareTvHtml5EmbedJsonBuilder(@Nonnull
                                                                                                                    Localization localization,
                                                                                                                    @Nonnull
                                                                                                                    ContentCountry contentCountry,
                                                                                                                    @Nonnull
                                                                                                                    String videoId)

createDesktopPlayerBody

@Nonnull
public static byte[] createDesktopPlayerBody(@Nonnull
                                                      Localization localization,
                                                      @Nonnull
                                                      ContentCountry contentCountry,
                                                      @Nonnull
                                                      String videoId,
                                                      @Nonnull
                                                      String sts,
                                                      boolean isTvHtml5DesktopJsonBuilder,
                                                      @Nonnull
                                                      String contentPlaybackNonce)
                                               throws IOException,
                                                      ExtractionException

Throws:

IOException

ExtractionException

getAndroidUserAgent

@Nonnull
public static String getAndroidUserAgent(@Nullable
                                                  Localization localization)

Get the user-agent string used as the user-agent for InnerTube requests with the Android client. If the Localization provided is null, fallbacks to the default one.

Parameters:

localization - the Localization to set in the user-agent

Returns:

the Android user-agent used for InnerTube requests with the Android client, depending on the Localization provided

getIosUserAgent

@Nonnull
public static String getIosUserAgent(@Nullable
                                              Localization localization)

Get the user-agent string used as the user-agent for InnerTube requests with the iOS client. If the Localization provided is null, fallbacks to the default one.

Parameters:

localization - the Localization to set in the user-agent

Returns:

the iOS user-agent used for InnerTube requests with the iOS client, depending on the Localization provided

addYouTubeHeaders

public static void addYouTubeHeaders(Map<String,List<String>> headers)
                              throws IOException,
                                     ExtractionException

Add required headers and cookies to an existing headers Map.

Throws:

IOException

ExtractionException

See Also:

addClientInfoHeaders(Map), addCookieHeader(Map)

addClientInfoHeaders

public static void addClientInfoHeaders(@Nonnull
                                        Map<String,List<String>> headers)
                                 throws IOException,
                                        ExtractionException

Add the X-YouTube-Client-Name, X-YouTube-Client-Version, Origin, and Referer headers.

Parameters:

headers - The headers which should be completed

Throws:

IOException

ExtractionException

addCookieHeader

public static void addCookieHeader(@Nonnull
                                   Map<String,List<String>> headers)

Add the CONSENT cookie to prevent redirect to consent.youtube.com

Parameters:

headers - the headers which should be completed

See Also:

CONSENT_COOKIE

generateConsentCookie

@Nonnull
public static String generateConsentCookie()

extractCookieValue

public static String extractCookieValue(String cookieName,
                                        @Nonnull
                                        Response response)

defaultAlertsCheck

public static void defaultAlertsCheck(@Nonnull
                                      com.grack.nanojson.JsonObject initialData)
                               throws ParsingException

Shared alert detection function, multiple endpoints return the error similarly structured.

Will check if the object has an alert of the type "ERROR".

Parameters:

initialData - the object which will be checked if an alert is present

Throws:

ContentNotAvailableException - if an alert is detected

ParsingException

getMetaInfo

@Nonnull
public static List<MetaInfo> getMetaInfo(@Nonnull
                                                  com.grack.nanojson.JsonArray contents)
                                           throws ParsingException

Throws:

ParsingException

extractCachedUrlIfNeeded

public static String extractCachedUrlIfNeeded(String url)

Sometimes, YouTube provides URLs which use Google's cache. They look like https://webcache.googleusercontent.com/search?q=cache:CACHED_URL

Parameters:

url - the URL which might refer to the Google's webcache

Returns:

the URL which is referring to the original site

isVerified

public static boolean isVerified(com.grack.nanojson.JsonArray badges)

unescapeDocument

@Nonnull
public static String unescapeDocument(@Nonnull
                                               String doc)

generateContentPlaybackNonce

@Nonnull
public static String generateContentPlaybackNonce()

Generate a content playback nonce (also called cpn), sent by YouTube clients in playback requests (and also for some clients, in the player request body).

Returns:

a content playback nonce string

generateTParameter

@Nonnull
public static String generateTParameter()

Try to generate a t parameter, sent by mobile clients as a query of the player request.

Some researches needs to be done to know how this parameter, unique at each request, is generated.

Returns:

a 12 characters string to try to reproduce the parameter