A simple to use Promise-based web scraper with local caching.
Request URL from remote resource.
Name | Type | Description | |
---|---|---|---|
url |
String
|
A URL string. |
|
options |
Object
|
Options object. |
Optional |
options.cacheDirectory |
Boolean
|
Directory to store cache. Defaults to |
Optional |
options.cacheMemory |
Boolean
|
Store cache in memory instead of local files. Defaults to |
Optional |
options.requestOptions |
Object
|
Custom request options object. |
Optional |
options.ttl |
Boolean
|
TTL (Time to live) in seconds for local/memory based cache. Default: 1800 |
Optional |
raspar.requestFromUrl('http://www.google.com/humans.txt').then(res => {});
Object
Promise
Parses local cache and adds cached property to the returned result.
Name | Type | Description | |
---|---|---|---|
res |
Object
|
Stored cache response. |
Optional |
raspar.formatCache({}).then(res => { console.log(res.cached); });
Object
Modified object with new cached property.
Requests a URL from cache.
Name | Type | Description | |
---|---|---|---|
url |
String
|
A URL string. |
|
options |
Object
|
Options object. |
Optional |
options.cacheDirectory |
Boolean
|
Directory to store cache. Defaults to |
Optional |
options.cacheMemory |
Boolean
|
Store cache in memory instead of local files. Defaults to |
Optional |
options.requestOptions |
Object
|
Custom request options object. |
Optional |
options.ttl |
Boolean
|
TTL (Time to live) in seconds for local/memory based cache. Default: 1800 |
Optional |
raspar.requestFromCache('http://www.google.com/humans.txt').then(res => {});
Object
Promise
Requests a URL from either local cache or remote resource.
Name | Type | Description | |
---|---|---|---|
url |
String
|
A URL string. |
|
options |
Object
|
Options object. |
Optional |
options.cacheDirectory |
Boolean
|
Directory to store cache. Defaults to |
Optional |
options.cacheMemory |
Boolean
|
Store cache in memory instead of local files. Defaults to |
Optional |
options.requestOptions |
Object
|
Custom request options object. |
Optional |
options.ttl |
Boolean
|
TTL (Time to live) in seconds for local/memory based cache. Default: 1800 |
Optional |
raspar.requestFromUrlorCache('http://www.google.com/humans.txt').then(res => {});
Object
Promise
Requests a URL or an array of URLs.
Name | Type | Description | |
---|---|---|---|
url |
String
Array
|
Either a URL string or an array of URLs. |
|
options |
Object
|
Options object. |
Optional |
options.cacheDirectory |
Boolean
|
Directory to store cache. Defaults to |
Optional |
options.cacheMemory |
Boolean
|
Store cache in memory instead of local files. Defaults to |
Optional |
options.requestOptions |
Object
|
Custom request options object. |
Optional |
options.ttl |
Boolean
|
TTL (Time to live) in seconds for local/memory based cache. Default: 1800 |
Optional |
raspar.fetch('http://www.google.com/humans.txt').then(res => {});
raspar.fetch(['http://www.google.com/humans.txt']).then(res => {});
Object
Promise