Overview¶
The HLS STAC Geoparquet Archive is an unofficial copy of the HLS 2.0 granule STAC item metadata that is generated in the HLS pipeline. The data are stored in two hive-partitioned parquet datasets (one per collection, partitioned by year and month). The parquet files are updated every 5 days from CMR API Granule queries, covering both the previous month (straggler catch-up) and the current month (incremental build).
Warning: This archive is not guaranteed to contain all of the records available in CMR, particularly for the most recent months. If you need the most recent granules do not use this archive!
The parquet files can be accessed from the nasa-maap-data-store bucket in AWS S3 (us-west-2):
s3://nasa-maap-data-store/file-staging/nasa-map/hls-stac-geoparquet-archive/v2/{collection}/year={year}/month={month}/{collection}-{year}-{month}.parquet
where collection is either HLSL30_2.0 (Landsat) or HLSS30_2.0 (Sentinel-2).
Static Iceberg metadata is published for each collection at:
s3://nasa-maap-data-store/file-staging/nasa-map/hls-stac-geoparquet-archive/v2/{collection}/iceberg/metadata/latest.metadata.json
Usage¶
DuckDB Iceberg metadata¶
Use this path when the objects are anonymously readable but the S3 bucket cannot be listed. The query reads exact file locations from Iceberg metadata, not a recursive parquet glob, so it only needs GetObject access.
Note: The HLSL30_2.0 and HLSS30_2.0 collections must be queried separately because the STAC items have slightly different parquet schemas.
import duckdb
con = duckdb.connect()
con.execute("INSTALL iceberg")
con.execute("LOAD iceberg")
con.execute("SET enable_global_s3_configuration=false")
con.execute("SET s3_region='us-west-2'")
iceberg_metadata_href = "s3://nasa-maap-data-store/file-staging/nasa-map/hls-stac-geoparquet-archive/v2/{collection}/iceberg/metadata/latest.metadata.json"
hls_l30_count = con.execute(
f"""
SELECT count(*)
FROM iceberg_scan('{iceberg_metadata_href.format(collection="HLSL30_2.0")}')
WHERE datetime BETWEEN '2025-05-01T00:00:00Z' AND '2025-05-31T23:59:59Z'
"""
).fetchone()[0]
print(f"found {hls_l30_count} HLSL30_2.0 items")
found 171586 HLSL30_2.0 items
from rustac import DuckdbClient
client = DuckdbClient(use_hive_partitioning=True)
# configure duckdb to find S3 credentials for listing/reading the files in S3
# on the MAAP HUB
# aws_session = boto3.Session()
# creds = aws_session.get_credentials().get_frozen_credentials()
# client.execute(
# f"""
# CREATE OR REPLACE SECRET secret (
# TYPE S3,
# REGION '{aws_session.region_name}',
# KEY_ID '{creds.access_key}',
# SECRET '{creds.secret_key}',
# SESSION_TOKEN '{creds.token}'
# );
# """
# )
# on the MAAP ADE
client.execute(
"""
CREATE OR REPLACE SECRET secret (
TYPE S3,
PROVIDER credential_chain
);
"""
)
parquet_href = "s3://nasa-maap-data-store/file-staging/nasa-map/hls-stac-geoparquet-archive/v2/{collection}/**/*.parquet"
datetime = "2025-05-01T00:00:00Z/2025-05-31T23:59:59Z"
bbox = (-90, 45, -85, 50)
hls_l30_items = client.search(
href=parquet_href.format(collection="HLSL30_2.0"),
datetime=datetime,
bbox=bbox,
)
print(f"found {len(hls_l30_items)} HLSL30_2.0 items")
hls_s30_items = client.search(
href=parquet_href.format(collection="HLSS30_2.0"),
datetime=datetime,
bbox=bbox,
)
print(f"found {len(hls_s30_items)} HLSS30_2.0 items")
found 292 HLSL30_2.0 items found 394 HLSS30_2.0 items
Example item¶
The items in the HLS STAC Geoparquet Archive were copied directly from the STAC item JSON files that are produced for every HLS granule (e.g. https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSS30.020/HLS.S30.T21JXN.2025341T134221.v2.0/HLS.S30.T21JXN.2025341T134221.v2.0_stac.json).
from pystac import Item
Item.from_dict(hls_s30_items[0])
Comparison to CMR API granules¶
This archive is generated by running granule queries for the HLS collections from the CMR API and represents a snapshot of a dynamic catalog. The archive is updated every 5 days, covering both the previous month (to catch stragglers) and the current month (incremental updates as new granules are published). This is bound to be a partially incomplete copy of the canonical source, but it should have 99% of the full set of granules.
Loading ITables v2.9.1 from the init_notebook_mode cell...
(need help?)
|
| ⓘ | collection | HLSL30_2.0 | HLSS30_2.0 | total | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| source | CMR | Geoparquet | difference | CMR | Geoparquet | difference | CMR | Geoparquet | difference | |
| year | month | |||||||||
| 2013 | 4 | 38836 | 38836 | 0 | 0 | 0 | 0 | 38836 | 38836 | 0 |
| 2013 | 5 | 60772 | 60772 | 0 | 0 | 0 | 0 | 60772 | 60772 | 0 |
| 2013 | 6 | 72006 | 72006 | 0 | 0 | 0 | 0 | 72006 | 72006 | 0 |
| 2013 | 7 | 77981 | 77981 | 0 | 0 | 0 | 0 | 77981 | 77981 | 0 |
| 2013 | 8 | 80416 | 80416 | 0 | 0 | 0 | 0 | 80416 | 80416 | 0 |
| 2013 | 9 | 69496 | 69496 | 0 | 0 | 0 | 0 | 69496 | 69496 | 0 |
| 2013 | 10 | 60854 | 60854 | 0 | 0 | 0 | 0 | 60854 | 60854 | 0 |
| 2013 | 11 | 51152 | 51152 | 0 | 0 | 0 | 0 | 51152 | 51152 | 0 |
| 2013 | 12 | 48766 | 48766 | 0 | 0 | 0 | 0 | 48766 | 48766 | 0 |
| 2014 | 1 | 51100 | 51100 | 0 | 0 | 0 | 0 | 51100 | 51100 | 0 |
| 2014 | 2 | 52247 | 52247 | 0 | 0 | 0 | 0 | 52247 | 52247 | 0 |
| 2014 | 3 | 69447 | 69447 | 0 | 0 | 0 | 0 | 69447 | 69447 | 0 |
| (149 more rows not shown) | ||||||||||