Back to Top

raspar master

A simple to use Promise-based web scraper with local caching.

fetchFromUrl(url[, options])

Request content from URL.

Parameters

Name Type Description
url String

A URL string.

options Object

Options object.

Optional
options.cacheDirectory String

Directory to store cache. Default is temp/cache/.

Optional
options.requestOptions Object

Custom request options object. Default is {}.

Optional
options.ttl Integer

TTL (Time to live) in seconds. Default is 1800

Optional

Examples

fetchFromUrl('http://www.google.com/humans.txt').then(content => console.log(content));

Returns

Object

Promise

fetch(url[, options])

Request content from URL or array of URLs.

Parameters

Name Type Description
url String Array

A URL or an array of URL strings.

options Object

Options object.

Optional
options.cacheDirectory String

Directory to store cache. Default is temp/cache/.

Optional
options.requestOptions Object

Custom request options object. Default is {}.

Optional
options.ttl Integer

TTL (Time to live) in seconds. Default is 1800

Optional

Examples

fetch('http://www.google.com/humans.txt').then(content => console.log(content));
fetch(['http://www.google.com/humans.txt']).then(contents => console.log(content[0]));

Returns

Object

Promise

checkCacheExpiry(path, ttl)

Check to see if file has expired based on the given TTL.

Parameters

Name Type Description
path String

File path to check.

ttl Integer

TTL (Time to live) in seconds.

Examples

checkCacheExpiry('temp/cache.txt', 1800).catch(() => console.log('Cache has expired.'));

Returns

Object

Promise

generateUUID(content)

Generate unique identifier from string.

Parameters

Name Type Description
content String

String to generate unique identifier.

Examples

console.log(generateUUID('http://www.google.com/humans.txt'));

Returns

String

readCache(path, ttl)

Read cache from file only if cache hasn't expired.

Parameters

Name Type Description
path String

File path to read cache.

ttl Integer

TTL (Time to live) in seconds.

Examples

readCache('temp/cache.txt', 1800).then(contents => console.log(contents))

Returns

Object

Promise

writeCache(path, content)

Write cache contents to file. Will create directories if they don't exist.

Parameters

Name Type Description
path String

File path to store cache.

content String

Contents of cache.

Examples

writeCache('temp/cache.txt', 'cached contents').then(content => console.log(content));

Returns

Object

Promise