> ## Documentation Index
> Fetch the complete documentation index at: https://www.forager.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Forager is a REST API for B2B people, company and job post data: contact lookups (phone numbers, personal emails, work emails), person, company and job post search, and bulk data feeds.
> Server: https://api-v2.forager.ai. Send the API key in the X-API-KEY header. Most paths include {account_id}. Read FORAGER_API_KEY and FORAGER_ACCOUNT_ID from the environment; never put the key in client-side code, logs or prompts.
> To add Forager to a project, follow https://www.forager.ai/docs/guides/agent-quickstart.md. To pick an endpoint, read https://www.forager.ai/docs/guides/choose-an-endpoint.md. For end-to-end builds, see https://www.forager.ai/docs/recipes/overview.md.
> Most requests spend credits. Do not guess costs; use https://www.forager.ai/docs/api-overview/credit-pricing.md and ask the user before spending credits.
> Use only endpoints and fields defined in https://www.forager.ai/docs/openapi.yaml. Forager has no MCP server for the API yet; the /mcp endpoint on this site searches the documentation only.

# Nightly sync of job postings

> Sample app: a Node.js script that runs nightly, resolves company domains, pulls their new job postings from the Forager API, dedupes and writes NDJSON.

```text Setup prompt theme={"theme":{"light":"github-light","dark":"github-dark"}}
Implement https://www.forager.ai/docs/samples/nightly-job-postings-sync.md in my project: a Node.js script, run by cron every night, that resolves a list of company domains to Forager organization IDs once, searches their active job postings featured since the previous run, pages through the results with a hard page cap, dedupes by post id against a state file, and appends new posts as NDJSON. Keep the API key in FORAGER_API_KEY with FORAGER_ACCOUNT_ID and add tests with stubbed fetch.
```

This sample is one Node.js file, `sync-jobs.mjs`, that keeps a local file of job postings for the companies you watch. Each run uses the free [organizations autocomplete](https://www.forager.ai/docs/openapi/autocomplete/v1_organization_lookup.md) to turn new domains into organization IDs, then the [job post search](https://www.forager.ai/docs/openapi/job-posts/datastorage_job_search_create.md) (2 credits per request) to fetch active postings featured since the last run. New posts go to an NDJSON file you can load anywhere; a state file remembers the IDs, the last run date and every post already seen.

## What you need

* Node.js 18 or newer (for the built-in `fetch`). No packages.
* `FORAGER_API_KEY` and `FORAGER_ACCOUNT_ID` in the environment, from the [Web App](https://www.forager.ai/docs/api-overview/create-an-account.md).
* `companies.txt`, one domain per line. Fictional sample:

```text companies.txt theme={"theme":{"light":"github-light","dark":"github-dark"}}
acme.com
example.com
smith-example.co
```

## The script

**1. Call Forager with retries and keep state on disk**

The helper treats `403` as fatal, retries `429` with backoff (autocomplete allows 10 requests per second per account) and returns everything else to the caller. State is a JSON file next to the script.

```javascript sync-jobs.mjs (part 1 of 3) theme={"theme":{"light":"github-light","dark":"github-dark"}}
#!/usr/bin/env node
// Nightly sync of job postings for a list of company domains.
// usage: node sync-jobs.mjs [companies.txt] [out-dir]
import fs from "node:fs";
import path from "node:path";

const BASE = `https://api-v2.forager.ai/api/${process.env.FORAGER_ACCOUNT_ID}/datastorage`;
const HEADERS = { "X-API-KEY": process.env.FORAGER_API_KEY, "Content-Type": "application/json" };
const JOB_SEARCH_CREDITS = 2;   // per request, https://www.forager.ai/docs/api-overview/credit-pricing
const MAX_PAGES_PER_RUN = 25;   // hard cap: 50 credits per night at most

const [companiesFile = "companies.txt", outDir = "."] = process.argv.slice(2);
const statePath = path.join(outDir, "sync-state.json");
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function forager(method, url, json) {
  let wait = 2000;
  for (let attempt = 0; attempt < 4; attempt++) {
    const resp = await fetch(url, { method, headers: HEADERS, body: json && JSON.stringify(json) });
    if (resp.status === 403) throw new Error(`Forager rejected the API key or Account ID: ${await resp.text()}`);
    if (resp.status === 429 && attempt < 3) {
      const retryAfter = Number(resp.headers.get("retry-after"));
      await sleep(retryAfter > 0 ? retryAfter * 1000 : wait);
      wait *= 2;
      continue;
    }
    if (!resp.ok) throw new Error(`Forager ${method} ${url} -> HTTP ${resp.status}: ${await resp.text()}`);
    return resp.json();
  }
}

function loadState() {
  try {
    return JSON.parse(fs.readFileSync(statePath, "utf8"));
  } catch {
    return { orgIds: {}, seenPostIds: [], lastRunDate: null };
  }
}

const saveState = (state) => fs.writeFileSync(statePath, JSON.stringify(state, null, 2));
```

**2. Resolve domains to organization IDs once**

Autocomplete is free and returns `text` as "name - domain", so the script keeps the first result whose text ends with the domain you asked for and stores the ID in the state file. A domain that stays unresolved is reported, not guessed.

```javascript sync-jobs.mjs (part 2 of 3) theme={"theme":{"light":"github-light","dark":"github-dark"}}
async function resolveOrganizations(domains, state) {
  const unresolved = [];
  for (const domain of domains) {
    if (state.orgIds[domain]) continue;
    const data = await forager("GET", `${BASE}/autocomplete/organizations/?q=${encodeURIComponent(domain)}`);
    const hit = data.results.find((r) => r.text.toLowerCase().endsWith(` - ${domain.toLowerCase()}`)) ?? data.results[0];
    if (hit) state.orgIds[domain] = Number(hit.id);
    else unresolved.push(domain);
  }
  return unresolved;
}
```

**3. Fetch new postings and write them out**

One search covers every watched company, because `organization_ids` takes a list. `date_featured_start` is the previous run's date (the first run uses today), `is_active` keeps closed postings out, and `page` walks the results until `total_search_results` is reached or the page cap stops the run. Posts whose `id` is already in the state file are dropped.

```javascript sync-jobs.mjs (part 3 of 3) theme={"theme":{"light":"github-light","dark":"github-dark"}}
async function fetchNewPosts(orgIds, since, seen) {
  const fresh = [];
  let total = Infinity;
  let fetched = 0;
  let pages = 0;
  for (let page = 1; page <= MAX_PAGES_PER_RUN && fetched < total; page++) {
    pages++;
    const data = await forager("POST", `${BASE}/job_search/`, {
      organization_ids: orgIds,
      is_active: true,
      date_featured_start: since,
      page,
    });
    total = data.total_search_results;
    const posts = data.search_results;
    if (!posts.length) break;
    fetched += posts.length;
    for (const post of posts) {
      if (seen.has(post.id)) continue;
      seen.add(post.id);
      fresh.push({
        id: post.id,
        title: post.title,
        url: post.url,
        source: post.source,
        date_featured: post.date_featured,
        is_remote: post.is_remote,
        location: post.location?.name ?? null,
        organization: post.organization?.name ?? null,
        organization_domain: post.organization?.domain ?? null,
      });
    }
  }
  return { fresh, total, fetched, pages };
}

async function main() {
  const domains = fs.readFileSync(companiesFile, "utf8").split("\n").map((l) => l.trim()).filter(Boolean);
  const state = loadState();
  const unresolved = await resolveOrganizations(domains, state);
  saveState(state);
  if (unresolved.length) console.error(`no organization found for: ${unresolved.join(", ")}`);

  const today = new Date().toISOString().slice(0, 10);
  const since = state.lastRunDate ?? today;
  const seen = new Set(state.seenPostIds);
  const orgIds = domains.map((d) => state.orgIds[d]).filter(Boolean);
  const { fresh, total, fetched, pages } = await fetchNewPosts(orgIds, since, seen);

  if (fresh.length) {
    const outFile = path.join(outDir, `new-jobs-${today}.ndjson`);
    fs.appendFileSync(outFile, fresh.map((p) => JSON.stringify(p)).join("\n") + "\n");
    console.log(`wrote ${fresh.length} new posts to ${outFile}`);
  }
  console.log(`${orgIds.length} companies, ${total} active posts since ${since}, ${fresh.length} new, ${pages * JOB_SEARCH_CREDITS} credits spent`);
  if (fetched < total) console.warn(`stopped at the ${MAX_PAGES_PER_RUN}-page cap; ${total - fetched} posts not fetched`);

  state.lastRunDate = today;
  state.seenPostIds = [...seen];
  saveState(state);
}

main().catch((e) => { console.error(e.message); process.exit(1); });
```

**4. Schedule it**

Run it once by hand, then add it to cron. The environment file holds the two variables and nothing else.

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
node sync-jobs.mjs companies.txt ./jobs
```

```text crontab (02:15 every night) theme={"theme":{"light":"github-light","dark":"github-dark"}}
15 2 * * *  cd /srv/forager-jobs && set -a && . ./forager.env && set +a && node sync-jobs.mjs companies.txt ./jobs >> sync.log 2>&1
```

```text Output (illustrative) theme={"theme":{"light":"github-light","dark":"github-dark"}}
wrote 4 new posts to jobs/new-jobs-2026-11-01.ndjson
3 companies, 11 active posts since 2026-10-31, 4 new, 4 credits spent
```

```json One line of new-jobs-2026-11-01.ndjson (illustrative) theme={"theme":{"light":"github-light","dark":"github-dark"}}
{"id":9183021,"title":"Senior Data Engineer","url":"https://example.com/careers/senior-data-engineer","source":"linkedin","date_featured":"2026-10-31","is_remote":true,"location":"Austin, Texas, United States","organization":"Acme Example","organization_domain":"acme.com"}
```

## Cost and limits

* Each search request costs 2 credits, so a night costs 2 credits per page fetched: at most 2 times `MAX_PAGES_PER_RUN` (50 with the default). Autocomplete and the state file are free.
* The page size is not documented, so the script stops on `total_search_results`, on an empty page, or at the page cap, whichever comes first.&#x20;
* Autocomplete is limited to 10 requests per second per account; the script resolves each domain once and never again.
* A domain that resolves to the wrong company is the most common mistake. Check `sync-state.json` after the first run and correct an ID by hand if needed.

## Adapt it

* Watch a team instead of a company list: add `title` with a boolean text query such as `"data engineer" OR "analytics engineer"` to the search body. See [Search job posts](https://www.forager.ai/docs/guides/search-job-posts.md).
* Send alerts: group `fresh` by `organization` and post a summary to chat or email, as in the [hiring signals recipe](https://www.forager.ai/docs/recipes/hiring-signals.md).
* Thousands of companies, or history back more than a few weeks? The [job post data feed](https://www.forager.ai/docs/data-license/v2/data-feed-overview.md) delivers the whole index as files.

## Related

> [card] [Node.js quickstart](https://www.forager.ai/docs/quickstarts/node.md)
>
> The client this script is built on, step by step.

> [card] [Hiring signals recipe](https://www.forager.ai/docs/recipes/hiring-signals.md)
>
> The same data, framed as alerts for sales.

> [card] [Job post search reference](https://www.forager.ai/docs/openapi/job-posts/datastorage_job_search_create.md)
>
> Every filter and response field.

> [card] [Find filter values](https://www.forager.ai/docs/guides/find-filter-values.md)
>
> Which autocomplete feeds which filter.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.