Problems with hstspreload.org #241

arcsector · 2023-05-12T23:47:46Z

🐛 Summary

The hstspreload.org dependency is resulting in the inability to actually scan anything without any real errors or notices. My experience for the past hour has been the inability to do anything with PSHTT due to the dependecy on hstspreload, which is currently exhibiting odd connection responses. This can also happen in isolated environments where one would like to scan without the ability to connect out to the internet.

To reproduce

Steps to reproduce the behavior:

Connect to an isolated network or run PSHTT during very high load times
Run PSHTT with debug flag
Observe indefinite hang at hstspreload connection like so:

pshtt myhosts.csv -j -d --timeout=5
-------------------------

Fetching Chrome preload list from source...
Starting new HTTPS connection (1): chromium.googlesource.com:443
https://chromium.googlesource.com:443 "GET /chromium/src/+/main/net/http/transport_security_state_static.json?format=TEXT HTTP/1.1" 200 None

-------------------------

Fetching hstspreload.org pending list...
Starting new HTTPS connection (1): hstspreload.org:443

Expected behavior

There should be a way to either forgo this mandatory connection, or the connection should respect the timeout flag even though the server hasn't presented flags that it's hanging on. This can be seen where CURLing the domain during this time will hang indefinitely as well, but if you specify a timeout with CURL, you'll get the homepage.

Any helpful log output or screenshots

Expected cURL result that only happens when using timeout flag around these times:

[root@host user]# curl -vm 3 https://hstspreload.org
* About to connect() to hstspreload.org port 443 (#0)
*   Trying 216.239.32.21...
* After 1498ms connect time, move on!
*   Trying 216.239.34.21...
* After 748ms connect time, move on!
*   Trying 216.239.38.21...
* Connected to hstspreload.org (216.239.38.21) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=hstspreload.org
*       start date: Apr 12 15:03:56 2023 GMT
*       expire date: Jul 11 15:55:28 2023 GMT
*       common name: hstspreload.org
*       issuer: CN=GTS CA 1D4,O=Google Trust Services LLC,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: hstspreload.org
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Fri, 12 May 2023 22:58:33 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 13379
< Vary: Accept-Encoding
< Accept-Ranges: bytes
< Last-Modified: Wed, 03 May 2023 16:27:17 GMT
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Via: 1.1 google
< 
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=0.5">
  <title>HSTS Preload List Submission</title>
  <link rel="shortcut icon" href="/favicon.ico">
  <link rel="apple-touch-icon" href="/static/app-icon.png">
  <link rel="search" href="/search.xml" type="application/opensearchdescription+xml">

  <link rel="stylesheet" href="/static/css/style.css">
  <link rel="stylesheet" href="/static/css/form.css">
  <link rel="stylesheet" href="/static/css/github.css">
  <script src="/static/js/base.js"></script>
  <script src="/static/js/view.js"></script>
  <script src="/static/js/form.js"></script>
  <script>
    window.addEventListener('load', function() {
      new Form(PreloadController);
    });
  </script>
</head>
<body class="theme-green">
<a class="github-fork-ribbon" href="https://github.com/chromium/hstspreload.org" data-ribbon="On GitHub" title="On GitHub">On GitHub</a>

<div class="content form">
  <form id="domain-form" class="hidden">
    <h2><label for="domain">
      Enter a domain:
    </label></h2>

    <input id="domain" name="domain" type="text" placeholder="example.com"
      autocorrect="off" autocapitalize="off" spellcheck="false">
    <br>
    <input id="check" type="submit" value="Check HSTS preload status and eligibility">

  </form>

  <!-- We un-hide the form using inline JS so that (when JS is enabled)
       it shows in the normal rendering order as if it was never hidden. -->
  <script>document.getElementById("domain-form").classList.remove("hidden");</script>
  <noscript>Submitting entries to the HSTS preload list via this site requires JavaScript.</noscript>

  <div id="result-waiting" class="hidden">
    <div id="spinner" class="spinner"></div>
    <br>
    <p id="checking"></p>
  </div>

  <div id="result" class="hidden">
    <p id="status"></p>
    <p id="summary"></p>
    <div id="issues-wrapper"></div>
  </div>

  <form id="submit-form" class="hidden">
  <hr>
    <h2>Submit</h2>
    <div id="checkboxes">
      <label>
        <input type="checkbox" id="checkbox-owner"><span>I am the site owner of <code><span class="domain-text">example.com</span></code> or have their permission to preload HSTS.</span>
      </label>
      <span id="oops">
        (If this is not the case, <code><span class="domain-text">example.com</span></code> may be sending the HSTS <code>preload</code> directive by accident. Please <a id="oops-mailto" href="mailto:[email protected]">contact [email protected]</a> to let us know.)
      </span>
      <br><br>
      <label>
        <input type="checkbox" id="checkbox-subdomains"><span>I understand that preloading <code><span class="domain-text">example.com</span></code> through this form will prevent <strong>all subdomains and nested subdomains</strong> from being accessed without a valid HTTPS certificate:
        <span class="subdomain-example"><code>*.<span class="domain-text">example.com</span></code></span>
        <span class="subdomain-example"><code>*.*.<span class="domain-text">example.com</span></code></span>
        <span class="subdomain-example"><code>...</code></span>
        </span></label>
    </div>
    <br>
    <input id="submit" type="submit" disabled value="Submit to the HSTS preload list">
    <div id="submit-success" class="submit-feedback hidden">
      <hr>
      <h2>Success</h2>
        <p><code class="domain-text">example.com</code> is now pending inclusion in the HSTS preload list!
        </p>
        <p>Please make sure that <code class="domain-text">example.com</code> <strong>continues</strong> to satisfy all preload requirement, or it will be removed. Please revisit this site over the next few weeks to check on the status of your domain.</p>
        <p>Also consider scanning for TLS issues <a id="ssl-labs-link" href="https://www.ssllabs.com/ssltest/analyze.html">using SSL Labs</a>.</p>
    </div>
    <div id="submit-failure" class="submit-feedback hidden">
      <hr>
      <h2>Failure</h2>
      An error occurred. Please start over.
    </div>
  </form>
</div>

<div class="content">
  <section id="information">
    <h2><a class="hash-link" href="#information">Information</a></h2>
    <p>
      This form is used to submit domains for inclusion in Chrome's <a href="https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security">HTTP Strict Transport Security (HSTS)</a> preload list.
      This is a list of sites that are hardcoded into Chrome as being HTTPS only.
    </p>
    <p>
      Most major browsers (Chrome, <a href="https://blog.mozilla.org/security/2012/11/01/preloading-hsts/">Firefox</a>, Opera, Safari, <a href="https://blogs.windows.com/msedgedev/2015/06/09/http-strict-transport-security-comes-to-internet-explorer-11-on-windows-8-1-and-windows-7/">IE 11 and Edge</a>) also have HSTS preload lists based on the Chrome list. (See the <a href="https://caniuse.com/#feat=stricttransportsecurity">HSTS compatibility matrix</a>.)
    </p>
  </section>

  <section id="submission-requirements">
    <h2><a class="hash-link" href="#submission-requirements">Submission Requirements</a></h2>
    <p>If a site sends the <code>preload</code> directive in an HSTS header, it is considered to be requesting inclusion in the preload list and may be submitted via the form on this site.</p>
    <p>In order to be accepted to the HSTS preload list through this form, your site must satisfy the following set of requirements:</p>
    <ol>
      <li>Serve a valid <strong>certificate</strong>.</li>
      <li><strong>Redirect</strong> from HTTP to HTTPS on the same host, if you are listening on port 80.</li>
      <li>Serve all <strong>subdomains</strong> over HTTPS.
        <ul>
          <li>In particular, you must support HTTPS for the <code>www</code> subdomain if a DNS record for that subdomain exists.</li>
        </ul>
      </li>
      <li>Serve an <strong>HSTS header</strong> on the base domain for HTTPS requests:
        <ul>
          <li>The <code>max-age</code> must be at least <code>31536000</code> seconds (1 year).</li>
          <li>The <code>includeSubDomains</code> directive must be specified.</li>
          <li>The <code>preload</code> directive must be specified.</li>
          <li>If you are serving an additional redirect from your HTTPS site, that redirect must still have the HSTS header (rather than the page it redirects to).</li>
        </ul>
      </li>
    </ol>
    <p>
      For more details on HSTS, please see <a href="https://tools.ietf.org/html/rfc6797">RFC 6797</a>.

      Here is an example of a valid HSTS header:
    </p>
    <p class="header-example main-header-example">
      <span><code>Strict-Transport-Security:</code></span> <span><code>max-age=63072000; includeSubDomains; preload</code></span>
    </p>

    <p>
      You can check the status of your request by entering the domain name again in the form above, or consult the current Chrome preload list by visiting <code>chrome://net-internals/#hsts</code> in your browser.
      Note that new entries are hardcoded into the Chrome source code and can take several months before they reach the stable version.
    </p>
  </section>

  <section id="continued-requirements">
    <h2><a class="hash-link" href="#continued-requirements">Continued Requirements</a></h2>
    <p>
      You must make sure your site continues to satisfy the submission requirements at all times. Note that removing the <code>preload</code> directive from your header will make your site immediately eligible for the <a href="https://hstspreload.org/removal/">removal form</a>, and that sites may be removed at any time for failing to keep up the requirements.
    </p>
    <p>
      In particular, the <a href="#submission-requirements">requirements above</a> apply to all domains submitted through <code>hstspreload.org</code> on or after <strong>October 11, 2017</strong> (i.e. preloaded after Chrome 63)
    </p>
    <p>
      The same requirements apply to earlier domains submitted on or after <strong>February 29, 2016</strong> (i.e. preloaded after Chrome 50), except that the required max-age for those domains is only <code>10886400</code> seconds.
    </p>
  </section>

  <section id="deployment-recommendations">
    <h2><a class="hash-link" href="#deployment-recommendations">Deployment Recommendations</a></h2>

    <p>
      If your site is committed to HTTPS and you want to preload HSTS, we suggest the following steps:
    </p>

    <ol>
      <li>Examine all subdomains (and nested subdomains) of your site and make sure that they work properly over HTTPS.</li>
      <li>Add the <code>Strict-Transport-Security</code> header to all HTTPS responses and ramp up the <code>max-age</code> in stages, using the following header values:
        <ul>
          <li>
            5 minutes:<br>
            <code class="header-example">max-age=300; includeSubDomains</code>
          </li>
          <li>
            1 week:<br>
            <code class="header-example">max-age=604800; includeSubDomains</code>
          </li>
          <li>
            1 month:<br>
            <code class="header-example">max-age=2592000; includeSubDomains</code>
          </li>
        </ul>
        During each stage, check for broken pages and monitor your site's metrics (e.g. traffic, revenue). Fix any problems that come up and then wait the full <code>max-age</code> of the stage before you move on. For example, wait a month in the last stage.
      <li>Once you're confident that there will be no more issues, increase the <code>max-age</code> to 2 years and submit your site to the preload list:
        <ul>
          <li>
            2 years, requesting to be preloaded:<br>
            <code class="header-example">max-age=63072000; includeSubDomains; preload</code>
          </li>
        </ul>
      </li>
    </ol>

    <p>
      If you have a group of employees or users who can beta test the deployment, consider trying the first few ramp-up stages on those users. Then make sure to go through all stages for all users, starting over from the beginning.
    </p>

    <p>
      Consult the <a href="https://wiki.mozilla.org/Security/Guidelines/Web_Security">Mozilla Web Security guidelines</a> and the <a href="https://developers.google.com/web/fundamentals/security/?hl=en">Google Web Fundamentals pages on security</a> for more concrete advice about HTTPS deployment.
    </p>

  </section>

  <section id="opt-in">
    <h2><a class="hash-link" href="#opt-in">Preloading Should Be Opt-In</a></h2>
    <p>
      If you maintain a project that provides HTTPS configuration advice or provides an option to enable HSTS, <strong>do not include the <code>preload</code> directive by default</strong>. We get regular emails from site operators who tried out HSTS this way, only to find themselves on the preload list by the time they find they need to remove HSTS to access certain subdomains. <a href="#removal">Removal</a> tends to be slow and painful for those sites.
    </p>
    <p>
      It's great to support HSTS preloading as a best practice, and for projects to provide a simple option to enable it. However, site operators who enable HSTS should know about the long-term consequences of preloading before they turn it on for a given domain. They should also be informed that they need to meet additional requirements and submit their site to <a href="https://hstspreload.org/">hstspreload.org</a> to ensure that it is successfully preloaded (i.e. to get the full protection of the intended configuration).
    </p>
  </section>

  <section id="removal">
    <h2><a class="hash-link" href="#removal">Removal</a></h2>
    <p>
      Be aware that inclusion in the preload list cannot easily be undone.
      Domains can be removed, but it takes months for a change to reach users with a Chrome update and we cannot make guarantees about other browsers.
      Don't request inclusion unless you're sure that you can support HTTPS for <strong>your entire site and all its subdomains</strong> in the long term.
    </p>

    <p>
      However, we will generally honor requests to be removed from Chrome's preload list if you find that you have a subdomain that you cannot serve over HTTPS for strong technical or cost reasons.
      To request removal, please visit the <a href="/removal/">removal form</a>.
    </p>
  </section>

  <section id="tld">
    <h2><a class="hash-link" href="#tld">TLD Preloading</a></h2>
    <p>
      Owners of gTLDs, ccTLDs, or any other <a href="https://publicsuffix.org/">public suffix</a> domains are welcome to preload HSTS across all their registerable domains. This ensures robust security for the whole TLD, and is much simpler than preloading each individual domain. Please <a href="#contact">contact us</a> if you're interested, or would like to learn more.
    </p>
  </section>

  <section id="contact">
    <h2><a class="hash-link" href="#contact">Contact</a></h2>
    <p>
      <strong>Want to remove your domain?</strong> Please visit the <a href="/removal/">removal form</a>.
    </p>
    <p>
      Else, if you have questions or requests that are not covered by this site, email us <a href="mailto:[email protected]">here</a> using an appropriate subject line and one of the preload list maintainers will be in contact soon.
    </p>
  </section>
</div>

</body>
</html>
* Connection #0 to host hstspreload.org left intact

The text was updated successfully, but these errors were encountered:

arcsector · 2023-05-25T21:44:32Z

Happening again now

jsf9k added the bug This issue or pull request addresses broken functionality label May 13, 2023

jsf9k self-assigned this May 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problems with hstspreload.org #241

Problems with hstspreload.org #241

arcsector commented May 12, 2023

arcsector commented May 25, 2023

Problems with hstspreload.org #241

Problems with hstspreload.org #241

Comments

arcsector commented May 12, 2023

🐛 Summary

To reproduce

Expected behavior

Any helpful log output or screenshots

arcsector commented May 25, 2023