raspar
master
A simple to use Promise-based web scraper with local caching.
A simple to use Promise-based web scraper with local caching.
Request content from URL.
| Name | Type | Description | |
|---|---|---|---|
| url |
String
|
A URL string. |
|
| options |
Object
|
Options object. |
Optional |
| options.cacheDirectory |
String
|
Directory to store cache. Default is |
Optional |
| options.requestOptions |
Object
|
Custom request options object. Default is |
Optional |
| options.ttl |
Integer
|
TTL (Time to live) in seconds. Default is 1800 |
Optional |
fetchFromUrl('http://www.google.com/humans.txt').then(content => console.log(content));
Object
Promise
Request content from URL or array of URLs.
| Name | Type | Description | |
|---|---|---|---|
| url |
String
Array
|
A URL or an array of URL strings. |
|
| options |
Object
|
Options object. |
Optional |
| options.cacheDirectory |
String
|
Directory to store cache. Default is |
Optional |
| options.requestOptions |
Object
|
Custom request options object. Default is |
Optional |
| options.ttl |
Integer
|
TTL (Time to live) in seconds. Default is 1800 |
Optional |
fetch('http://www.google.com/humans.txt').then(content => console.log(content));fetch(['http://www.google.com/humans.txt']).then(contents => console.log(content[0]));
Object
Promise
Check to see if file has expired based on the given TTL.
| Name | Type | Description | |
|---|---|---|---|
| path |
String
|
File path to check. |
|
| ttl |
Integer
|
TTL (Time to live) in seconds. |
checkCacheExpiry('temp/cache.txt', 1800).catch(() => console.log('Cache has expired.'));
Object
Promise
Generate unique identifier from string.
| Name | Type | Description | |
|---|---|---|---|
| content |
String
|
String to generate unique identifier. |
console.log(generateUUID('http://www.google.com/humans.txt'));
String
Read cache from file only if cache hasn't expired.
| Name | Type | Description | |
|---|---|---|---|
| path |
String
|
File path to read cache. |
|
| ttl |
Integer
|
TTL (Time to live) in seconds. |
readCache('temp/cache.txt', 1800).then(contents => console.log(contents))
Object
Promise
Write cache contents to file. Will create directories if they don't exist.
| Name | Type | Description | |
|---|---|---|---|
| path |
String
|
File path to store cache. |
|
| content |
String
|
Contents of cache. |
writeCache('temp/cache.txt', 'cached contents').then(content => console.log(content));
Object
Promise