Improve URL checks to reduce false-negatives
This commit improves the URL health checking mechanism to reduce false negatives. - Treat all 2XX status codes as successful, addressing issues with codes like `204`. - Improve URL matching to exclude URLs within Markdown inline code block and support URLs containing parentheses. - Add `forceHttpGetForUrlPatterns` to customize HTTP method per URL to allow verifying URLs behind CDN/WAFs that do not respond to HTTP HEAD. - Send the Host header for improved handling of webpages behind proxies. - Improve formatting and context for output messages. - Fix the defaulting options for redirects and cookie handling. - Update the user agent pool to modern browsers and platforms. - Add support for randomizing TLS fingerprint to mimic various clients better, improving the effectiveness of checks. However, this is not fully supported by Node.js's HTTP client; see nodejs/undici#1983 for more details. - Use `AbortSignal` instead of `AbortController` as more modern and simpler way to handle timeouts.
This commit is contained in:
@@ -1,3 +1,4 @@
|
||||
import { indentText } from '@tests/shared/Text';
|
||||
import { logCurrentArgs, CommandLineFlag, hasCommandLineFlag } from './cli-args';
|
||||
import { log, die } from './utils/log';
|
||||
import { ensureNpmProjectDir, npmInstall, npmBuild } from './utils/npm';
|
||||
@@ -15,7 +16,6 @@ import {
|
||||
APP_EXECUTION_DURATION_IN_SECONDS,
|
||||
SCREENSHOT_PATH,
|
||||
} from './config';
|
||||
import { indentText } from './utils/text';
|
||||
import type { ExtractionResult } from './app/extractors/common/extraction-result';
|
||||
|
||||
export async function main(): Promise<void> {
|
||||
|
||||
Reference in New Issue
Block a user