Improve URL checks to reduce false-negatives
This commit improves the URL health checking mechanism to reduce false negatives. - Treat all 2XX status codes as successful, addressing issues with codes like `204`. - Improve URL matching to exclude URLs within Markdown inline code block and support URLs containing parentheses. - Add `forceHttpGetForUrlPatterns` to customize HTTP method per URL to allow verifying URLs behind CDN/WAFs that do not respond to HTTP HEAD. - Send the Host header for improved handling of webpages behind proxies. - Improve formatting and context for output messages. - Fix the defaulting options for redirects and cookie handling. - Update the user agent pool to modern browsers and platforms. - Add support for randomizing TLS fingerprint to mimic various clients better, improving the effectiveness of checks. However, this is not fully supported by Node.js's HTTP client; see nodejs/undici#1983 for more details. - Use `AbortSignal` instead of `AbortController` as more modern and simpler way to handle timeouts.
This commit is contained in:
@@ -2,13 +2,13 @@ export async function fetchWithTimeout(
|
||||
url: string,
|
||||
timeoutInMs: number,
|
||||
init?: RequestInit,
|
||||
): Promise<Response> {
|
||||
const controller = new AbortController();
|
||||
): ReturnType<typeof fetch> {
|
||||
const options: RequestInit = {
|
||||
...(init ?? {}),
|
||||
signal: controller.signal,
|
||||
signal: AbortSignal.timeout(timeoutInMs),
|
||||
};
|
||||
const promise = fetch(url, options);
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutInMs);
|
||||
return promise.finally(() => clearTimeout(timeout));
|
||||
return fetch(
|
||||
url,
|
||||
options,
|
||||
);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user