KeyError sh256 #5093

terriko · 2025-05-19T17:38:49Z

 │ /home/runner/work/cve-bin-tool/cve-bin-tool/cve_bin_tool/data_sources/nvd_so │
│ urce.py:421 in fetch_cves                                                    │
│                                                                              │
│   418 │   │   │   )                                                          │
│   419 │   │   │                                                              │
│   420 │   │   │   tasks = [                                                  │
│ ❱ 421 │   │   │   │   self.cache_update(self.session, url, meta["sha256"])   │
│   422 │   │   │   │   for url, meta in nvd_metadata.items()                  │
│   423 │   │   │   │   if meta is not None                                    │
│   424 │   │   │   ]                                                          │
╰──────────────────────────────────────────────────────────────────────────────╯
KeyError: 'sha256'

Looks like we hit a problem with the update code over the weekend, so the github cache is currently broken. I'm re-running it to see if it's already resolved (last time it was a data issue) but I'll be back to take a look in an hour or so.

Current workaround

Current workaround is to use -n api2 and set an nvd_api_key, which allows you to get the data from nvd directly via their API. If you've already got most of a cache this should be great, if you're starting with no data you may discover that this is very slow (so if you're changing a job that has a timeout set, you may need to lengthen the timeout)

The text was updated successfully, but these errors were encountered:

Giving111 · 2025-05-19T23:13:32Z

Is this problem fixed?

YonderLolasaurus · 2025-05-20T02:34:39Z

Also encountering this, any update on a fix?

Giving111 · 2025-05-20T22:19:21Z

Is there a work around for this problem?

Pzx-Doctom · 2025-05-21T05:18:13Z

Is this problem fixed?

c-mehring · 2025-05-21T06:28:59Z

Did some investigation on this:

We are running cve-bin-tool based on official python:3.11 docker image. Builds are failing since the beginning of this week. I compared used pip dependencies between our last successful and the failing builds. No differences found. Switching to latest python:3 image did not solve the issue.

I guess the problem arises from latest definition updates?

raoulbhatia · 2025-05-21T08:27:54Z

We started to use the following as a temporary workaround.
It looks fine right now, but we also have a comparatively low volume use case.

cve-bin-tool -n json-mirror --sbom cyclonedx --sbom-file "..." -f html -o "..." ||\
  cve-bin-tool -n json-nvd --sbom cyclonedx --sbom-file "..." -f html -o "..."

anthonyharrison · 2025-05-21T10:01:48Z

I have the seen the same problem (running Python 3.10).

A bit of digging has shown that some of the files have missing metadata

https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2002.json.gz {'lastModifiedDate': '2025-04-24T03:02:43-04:00', 'size': '148808', 'zipSize': '8522', 'gzSize': '8386', 'sha256': '62F39E12301955E753B38D606483D5AE3A6724B057A7F920DDE860EF812B77CC'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2003.json.gz {}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2004.json.gz {'lastModifiedDate': '2025-05-03T03:02:41-04:00', 'size': '99013', 'zipSize': '6133', 'gzSize': '5997', 'sha256': '4E15E47B907C5E38C5AF2D06D1D6FB5647363131B5566804F4C8E1E30DA0A3CE'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2005.json.gz {}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2006.json.gz {}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2007.json.gz {'lastModifiedDate': '2025-05-13T03:03:15-04:00', 'size': '191701', 'zipSize': '11995', 'gzSize': '11859', 'sha256': '71CD008F025BB7BA80FFC9E2E44D1DDEBEA4A7FC53D40D04306A5719840268E3'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2008.json.gz {'lastModifiedDate': '2025-05-02T03:02:23-04:00', 'size': '239247', 'zipSize': '13376', 'gzSize': '13240', 'sha256': '29E3174B5E7B37AA466FF368D014147DDC84D048092C1BC1C89F39A7C502F8CF'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2009.json.gz {}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2010.json.gz {'lastModifiedDate': '2025-04-20T03:01:42-04:00', 'size': '687163', 'zipSize': '37040', 'gzSize': '36904', 'sha256': '8424076CC7C94C29A24AD750B042CDA7B2304D2BA642B5CE65F329D73B2D9704'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2011.json.gz {'lastModifiedDate': '2025-04-20T03:01:41-04:00', 'size': '1283271', 'zipSize': '71337', 'gzSize': '71201', 'sha256': 'B51728E32F2C7BBC220C4AF7BBBFE83ABC0BE1F05128118456AC14727DDA9443'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2012.json.gz {}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2013.json.gz {}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2014.json.gz {'lastModifiedDate': '2025-04-20T03:01:38-04:00', 'size': '3918378', 'zipSize': '207350', 'gzSize': '207214', 'sha256': '38ED00C842E2975EDF7A61F255E5EBEB668CCD0F1BCFBFDED8CC7118E7A8414B'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2015.json.gz {}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2016.json.gz {}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2017.json.gz {'lastModifiedDate': '2025-05-15T03:01:59-04:00', 'size': '25484958', 'zipSize': '1109281', 'gzSize': '1109145', 'sha256': '2C6F4AD7CC1A333D6ACB1A60DBC094B2C260362C4E2EA4E99B7641E95FF3D86F'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2018.json.gz {'lastModifiedDate': '2025-05-17T03:02:45-04:00', 'size': '89692912', 'zipSize': '4350260', 'gzSize': '4350124', 'sha256': 'A8FA94669A982990B6A54AFD0A1E4636AEC4D527DBC63182199B1A7B979162EF'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2019.json.gz {}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2020.json.gz {'lastModifiedDate': '2025-05-17T03:02:26-04:00', 'size': '128079274', 'zipSize': '6049495', 'gzSize': '6049359', 'sha256': '0C3129B86DD9595983D4D97D89E8CAD4FD0872ED1EE08BEE4E94C362E5018590'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2021.json.gz {'lastModifiedDate': '2025-05-17T03:01:57-04:00', 'size': '149495731', 'zipSize': '7160465', 'gzSize': '7160329', 'sha256': '334FBA98DE5326B3C385824230E4E6FE8955B0B9C19028E75745C0A4C05BD6D7'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2022.json.gz {'lastModifiedDate': '2025-05-20T03:01:30-04:00', 'size': '145443204', 'zipSize': '7578701', 'gzSize': '7578565', 'sha256': 'A917BE0D1845E22FB47B0A77EB97D64A62A20210CAA73D58E53341F900CF1389'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2023.json.gz {'lastModifiedDate': '2025-05-16T03:02:08-04:00', 'size': '155584741', 'zipSize': '8017572', 'gzSize': '8017436', 'sha256': 'F1DA6DDDDCB90C2E2C3D27B2A65C811F9AE0833EFAA9518E1BA95A7BA2ABF286'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2024.json.gz {'lastModifiedDate': '2025-05-16T03:01:03-04:00', 'size': '134964022', 'zipSize': '9809852', 'gzSize': '9809716', 'sha256': '5C0CC3FBECC53C4CC17DDD0105D5544B23E2BFFC587B03A141C0B9A66F277A0F'}
https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2025.json.gz {'lastModifiedDate': '2025-05-16T03:00:10-04:00', 'size': '28830411', 'zipSize': '2443138', 'gzSize': '2443002', 'sha256': 'F278D9C1DB6FD0B303E483B79543AB6F300DE3D02779F58E6ADBA1899DD77087'}

However, after modifying the code to add a check that the sha256 field was present, I still encountered an error with the data for 2022. It looks like the meta data has a SHA value but as there is no file avaialble, so there is a SHA mismatch

cve_bin_tool.CVEDB - SHAMismatch: https://mirror.cveb.in/nvd/json/cve/1.1/nvdcve-1.1-2022.json.gz (have: E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855, want: A917BE0D1845E22FB47B0A77EB97D64A62A20210CAA73D58E53341F900CF1389)

Looking at the mirror site you can see two years have 0 length gz files.

Adding a check to detect a zeo length file, prevents the SHA mismatch and allows the database update to proceed.

Re54ik · 2025-05-21T11:03:09Z

Workaround (removed sha256 check).

nvd_source.py
before
self.cache_update(self.session, url, meta["sha256"])

after
self.cache_update(self.session, url, meta.get("sha256", ""))

before
async def cache_update(...)

after

    async def cache_update(
        self,
        session: RateLimiter,
        url: str,
        sha: str,
        chunk_size: int = 16 * 1024,
        max_retries: int = 3,
        skip_sha_check: bool = False,
    ) -> None:

        filename = url.split("/")[-1]
        cache_path = Path(self.cachedir).resolve()
        filepath = (cache_path / filename).resolve()
        
        if not str(filepath).startswith(str(cache_path)):

                raise AttemptedToWriteOutsideCachedir(filepath)

        if filepath.is_file():

                calculate = hashlib.sha256()
                async with GzipFile(filepath, "rb") as f:
                        while chunk := await f.read(chunk_size):

                                calculate.update(chunk)

                gotsha = calculate.hexdigest().upper()
                if gotsha == sha:
                        self.LOGGER.debug(f"File {filename} is up-to-date")
                        return
                filepath.unlink()

        for attempt in range(max_retries):
                try:

                        async with await session.get(url) as response:

                                if response.status == 403:
                                        raise NVDRateLimit(f"Rate limited: {url}")

                                response.raise_for_status()
                                gzip_data = await response.read()
                                
                                if not gzip_data:
                                        raise ValueError("Empty response")
                                
                                gotsha = hashlib.sha256(gzip_data).hexdigest().upper()
                                if sha and not skip_sha_check and gotsha != sha:
                                    print(f"Warning: SHA mismatch - {gotsha} vs {sha}")  
                                
                                async with FileIO(filepath, "wb") as f:
                                        await f.write(gzip_data)
                                return
                                
                except (ClientError, asyncio.TimeoutError) as e:
                        if attempt == max_retries - 1:
                                raise
                        await asyncio.sleep(5 * (attempt + 1))

terriko · 2025-05-21T15:29:39Z

This is not fixed. It's going to be at least a few more hours before we can get the mirror updated.

Current workaround is to use -n api2 and set an nvd_api_key, which allows you to get the data from nvd directly.

In theory changing data sources is safer than turning off signature verification, but the signatures aren't really the greatest validation at the best of times so I wouldn't blame folk for using that workaround.

terriko · 2025-05-21T15:32:05Z

pinging @warthog9 so he can see the details before he goes to mess with the mirror

warthog9 · 2025-05-21T17:00:21Z

Ok doing some poking and digging, looks like something weird is going on upstream of me, which is not exactly helpful. When I'm mirroring out the files from NVD itself I'm apparently getting 0 length files on occasion now.

LOOKS like if I mirror it, then mirror it again, I get things sorted out and no longer get 0 length files.

Not ideal, and it makes me wonder what's changed somewhat recently to be getting those kinds of issues.

Note: Should be fixed on my end, I'm running a couple of checks now though

Giving111 · 2025-05-21T18:50:39Z

Thank you all!

terriko · 2025-05-21T21:11:43Z

In case anyone's wondering: our cache in github did not update correctly. I'm re-running it in case it was just a mirror propagation issue, but we may still have a problem.

anthonyharrison added a commit to anthonyharrison/cve-bin-tool that referenced this issue May 21, 2025

fix: improved metadata handling for NVD mirror (fixes intel#5093)

a33b4ae

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KeyError sh256 #5093

KeyError sh256 #5093

KeyError sh256 #5093

KeyError sh256 #5093

Comments

Current workaround