Back to Top

raspar master

A simple to use Promise-based web scraper with local caching.

requestFromUrl(url[, options]) private declaration

Request URL from remote resource.

Parameters

Name Type Description
url String

A URL string.
options Object

Options object.

 Optional
options.cacheDirectory String

Directory to store cache. Defaults to temp/cache/.

 Optional
options.cacheMemory Boolean

Store cache in memory instead of local files. Defaults to false.

 Optional
options.requestOptions Object

Custom request options object.

 Optional
options.ttl Integer

TTL (Time to live) in seconds for local/memory based cache. Default: 1800

 Optional

Examples

raspar.requestFromUrl('http://www.google.com/humans.txt').then(res => {});

Returns

Object

Promise

formatCache([res]) private declaration

Parses local cache and adds cached property to the returned result.

Parameters

Name Type Description
res Object

Stored cache response.

 Optional

Examples

raspar.formatCache({}).then(res => { console.log(res.cached); });

Returns

Object

Modified object with new cached property.

requestFromCacheMemory(hash) private declaration

Returns cache from memory based on a specific hash.

Parameters

Name Type Description
hash String

Hash generated based on the URL.

Returns

Object

Promise

requestFromCacheDirectory(hash, settings) private declaration

Returns cache from local directory based on a specific hash.

Parameters

Name Type Description
hash String

Hash generated based on the URL.
settings Object

Settings object.
settings.cacheDirectory String

Directory to store cache.
settings.ttl Integer

TTL (Time to live) in seconds for local/memory based cache.

Returns

Object

Promise

requestFromCache(url[, options]) private declaration

Requests a URL from cache.

Parameters

Name Type Description
url String

A URL string.
options Object

Options object.

 Optional
options.cacheDirectory String

Directory to store cache. Defaults to temp/cache/.

 Optional
options.cacheMemory Boolean

Store cache in memory instead of local files. Defaults to false.

 Optional
options.requestOptions Object

Custom request options object.

 Optional
options.ttl Integer

TTL (Time to live) in seconds for local/memory based cache. Default: 1800

 Optional

Examples

raspar.requestFromCache('http://www.google.com/humans.txt').then(res => {});

Returns

Object

Promise

requestFromUrlorCache(url[, options]) private declaration

Requests a URL from either local cache or remote resource.

Parameters

Name Type Description
url String

A URL string.
options Object

Options object.

 Optional
options.cacheDirectory Boolean

Directory to store cache. Defaults to temp/cache/.

 Optional
options.cacheMemory Boolean

Store cache in memory instead of local files. Defaults to false.

 Optional
options.requestOptions Object

Custom request options object.

 Optional
options.ttl Integer

TTL (Time to live) in seconds for local/memory based cache. Default: 1800

 Optional

Examples

raspar.requestFromUrlorCache('http://www.google.com/humans.txt').then(res => {});

Returns

Object

Promise

fetch(url[, options])

Requests a URL or an array of URLs.

Parameters

Name Type Description
url String Array

Either a URL string or an array of URLs.
options Object

Options object.

 Optional
options.cacheDirectory String

Directory to store cache. Defaults to temp/cache/.

 Optional
options.cacheMemory Boolean

Store cache in memory instead of local files. Defaults to false.

 Optional
options.requestOptions Object

Custom request options object.

 Optional
options.ttl Integer

TTL (Time to live) in seconds for local/memory based cache. Default: 1800

 Optional

Examples

raspar.fetch('http://www.google.com/humans.txt').then(res => {});
raspar.fetch(['http://www.google.com/humans.txt']).then(res => {});

Returns

Object

Promise