Fix failing URL status checking integration tests

Implement following redirects over `fetch` supporting cookies.
`node-fetch` does not support sending cookies during redirect. However,
this is needed to not end-up in a redirect loop for a sign-in callback.

Fix integration tests failing due to redirects and 403 errors:
  - Many redirects from `answers.microsoft.com` was throwing: throwing
    `FetchError: maximum redirect reached` error. It was caused by not
    having cookies when following redirects therefore having an infinite
    sign-in callback for the webpage.
  - Fixes integration tests failing due to additional referer header being
    sent by the application. It adds support for making exceptions to
    additional header sending through a list of regexes.

Add in-depth documentation for URL status checking.

tests/integration/application/collections/StatusChecker/BatchStatusChecker.ts

@@ -6,6 +6,7 @@ import { groupUrlsByDomain } from './UrlPerDomainGrouper';
 export async function getUrlStatusesInParallelAsync(
     urls: string[],
     options?: IBatchRequestOptions): Promise<IUrlStatus[]> {
+    // urls = [ 'https://privacy.sexy' ]; // Here to comment out when testing
     const uniqueUrls = Array.from(new Set(urls));
     options = { ...DefaultOptions, ...options };
     console.log('Options: ', options); // tslint:disable-line: no-console