Export any web page to OKF markdown with --content and…
0 reactions
Export any web page to OKF markdown with --content and --technical layers
I built a command that fetches any web page server-side and writes an as-is snapshot of both (or split) content and technical layers.
The output uses OKF (Open Knowledge Format), an open format from Google Cloud's knowledge-catalog repo. Here's some info on the topic on Google blog https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing .
npm install -g
sleepwalker okf export https://your-site.com
As said, this contains the content layer (clean markdown with headings, paragraphs, links) and a technical snapshot (HTTP headers, meta tags, JSON-LD, hreflang, robots directives, image alt coverage). Use --content or --technical for a focused fetch.
Runs locally, happy with any feedback!
Repo: https://github.com/followanton/sleepwalker this command is free and open source, please see if you can give it a "star". I would really appreciate it! <3
Example (technical snapshot):
---
type: "TechnicalSnapshot"
title: "Technical snapshot: Apple"
description: "Meta tags, structured data, headers and robots directives as served for www.apple.com ."
resource: " https://www.apple.com/ "
tags: ["technical"]
timestamp: 2026-07-31T17:37:16.499Z
---
# Technical snapshot: Apple
## Fetch
Redirect chain:
1. HTTP 301 https://apple.com
2. HTTP 200 https://www.apple.com/
HTML size: 251 KB (257390 bytes).
## HTTP headers
```http
# Content
content-type: text/html; charset=utf-8
# Security
content-security-policy: default-src 'self' blob: data: *. akamaized.net *. apple.com *. apple-mapkit.com *. cdn-apple.com *. organicfruitapps.com ; child-src blob: mailto: embed.music.apple.com embed.podcasts.apple.com https://recyclingprogram.apple.com https://smb.apple.com https://nova.apple.com swdlp.apple.com www.apple.com www.instagram.com platform.twitter.com www.youtube-nocookie.com ; img-src 'unsafe-inline' blob: data: *. apple.com *. apple-mapkit.com *. cdn-apple.com *. mzstatic.com ; script-src 'unsafe-inline' 'unsafe-eval' blob: *. apple.com *. apple-mapkit.com www.instagram.com platform.twitter.com ; style-src 'unsafe-inline' *. apple.com
referrer-policy: no-referrer-when-downgrade
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
# Cache
cache-control: max-age=8
expires: Fri, 31 Jul 2026 17:37:24 GMT
vary: Accept-Encoding
# Server
server: Apple
```
## Meta tags
```html
<html lang="en-US" dir="ltr">
<title>Apple</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<meta name="Description" content="Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.">
<meta property="analytics-track" content="apple - index/tab">
<meta property="analytics-s-channel" content="homepage">
<meta property="analytics-s-bucket-0" content="applestoreww">
<meta property="analytics-s-bucket-1" content="applestoreww">
<meta property="analytics-s-bucket-2" content="applestoreww">
<meta name="globalnav-store-key" content="SFX9YPYY9PPXCU9KH">
<link rel="canonical" href="https://www.apple.com/">
```
## Headings (23)
```html
<h1>Apple</h1>
<h2>iPhone</h2>
<h2>College, sorted.</h2>
<h2>MacBook Air</h2>
<h3>Apple Upgrade</h3>
<h3>iPad Air</h3>
<h3>Apple Watch Series 11</h3>
<h3>App Store</h3>
<h3>Apple Trade In</h3>
<h3>Apple Card</h3>
<h2>Endless entertainment.</h2>
<h2>Apple Footer</h2>
<h3>Shop and Learn Shop and Learn</h3>
<h3>Apple Wallet Apple Wallet</h3>
<h3>Account Account</h3>
<h3>Entertainment Entertainment</h3>
<h3>Apple Store Apple Store</h3>
<h3>For Business For Business</h3>
<h3>For Education For Education</h3>
<h3>For Healthcare For Healthcare</h3>
<h3>For Government For Government</h3>
<h3>Apple Values Apple Values</h3>
<h3>About Apple About Apple</h3>
```
#dev #programming #technology
I built a command that fetches any web page server-side and writes an as-is snapshot of both (or split) content and technical layers.
The output uses OKF (Open Knowledge Format), an open format from Google Cloud's knowledge-catalog repo. Here's some info on the topic on Google blog https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing .
npm install -g
sleepwalker okf export https://your-site.com
As said, this contains the content layer (clean markdown with headings, paragraphs, links) and a technical snapshot (HTTP headers, meta tags, JSON-LD, hreflang, robots directives, image alt coverage). Use --content or --technical for a focused fetch.
Runs locally, happy with any feedback!
Repo: https://github.com/followanton/sleepwalker this command is free and open source, please see if you can give it a "star". I would really appreciate it! <3
Example (technical snapshot):
---
type: "TechnicalSnapshot"
title: "Technical snapshot: Apple"
description: "Meta tags, structured data, headers and robots directives as served for www.apple.com ."
resource: " https://www.apple.com/ "
tags: ["technical"]
timestamp: 2026-07-31T17:37:16.499Z
---
# Technical snapshot: Apple
## Fetch
Redirect chain:
1. HTTP 301 https://apple.com
2. HTTP 200 https://www.apple.com/
HTML size: 251 KB (257390 bytes).
## HTTP headers
```http
# Content
content-type: text/html; charset=utf-8
# Security
content-security-policy: default-src 'self' blob: data: *. akamaized.net *. apple.com *. apple-mapkit.com *. cdn-apple.com *. organicfruitapps.com ; child-src blob: mailto: embed.music.apple.com embed.podcasts.apple.com https://recyclingprogram.apple.com https://smb.apple.com https://nova.apple.com swdlp.apple.com www.apple.com www.instagram.com platform.twitter.com www.youtube-nocookie.com ; img-src 'unsafe-inline' blob: data: *. apple.com *. apple-mapkit.com *. cdn-apple.com *. mzstatic.com ; script-src 'unsafe-inline' 'unsafe-eval' blob: *. apple.com *. apple-mapkit.com www.instagram.com platform.twitter.com ; style-src 'unsafe-inline' *. apple.com
referrer-policy: no-referrer-when-downgrade
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
# Cache
cache-control: max-age=8
expires: Fri, 31 Jul 2026 17:37:24 GMT
vary: Accept-Encoding
# Server
server: Apple
```
## Meta tags
```html
<html lang="en-US" dir="ltr">
<title>Apple</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<meta name="Description" content="Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.">
<meta property="analytics-track" content="apple - index/tab">
<meta property="analytics-s-channel" content="homepage">
<meta property="analytics-s-bucket-0" content="applestoreww">
<meta property="analytics-s-bucket-1" content="applestoreww">
<meta property="analytics-s-bucket-2" content="applestoreww">
<meta name="globalnav-store-key" content="SFX9YPYY9PPXCU9KH">
<link rel="canonical" href="https://www.apple.com/">
```
## Headings (23)
```html
<h1>Apple</h1>
<h2>iPhone</h2>
<h2>College, sorted.</h2>
<h2>MacBook Air</h2>
<h3>Apple Upgrade</h3>
<h3>iPad Air</h3>
<h3>Apple Watch Series 11</h3>
<h3>App Store</h3>
<h3>Apple Trade In</h3>
<h3>Apple Card</h3>
<h2>Endless entertainment.</h2>
<h2>Apple Footer</h2>
<h3>Shop and Learn Shop and Learn</h3>
<h3>Apple Wallet Apple Wallet</h3>
<h3>Account Account</h3>
<h3>Entertainment Entertainment</h3>
<h3>Apple Store Apple Store</h3>
<h3>For Business For Business</h3>
<h3>For Education For Education</h3>
<h3>For Healthcare For Healthcare</h3>
<h3>For Government For Government</h3>
<h3>Apple Values Apple Values</h3>
<h3>About Apple About Apple</h3>
```
#dev #programming #technology
| 0 mlx | |
| 0 | |
| 0 |