Tweet Harvest (Twitter Crawler)

Common errors

• Error: Playwright browser has not been installed.

  cause The Playwright browser binaries (e.g., Chromium) required by `tweet-harvest` have not been downloaded.
  fix Run `npx playwright install` in your project directory to download the necessary browser binaries for Playwright.

• Error: Auth token is not valid. Please make sure you enter a valid Twitter auth token.

  cause The provided Twitter `auth_token` is either expired, invalid, or incorrectly formatted, preventing successful authentication with Twitter.
  fix Log into Twitter in a browser, extract a fresh `auth_token` cookie, and update your configuration or environment variable. Ensure there are no leading/trailing spaces or other characters.

• Error: Cannot read properties of undefined (reading 'page')

  cause This often indicates that Playwright failed to launch the browser or navigate to the Twitter page, potentially due to network issues, an unsupported environment, or conflicting browser processes.
  fix Check your internet connection, ensure no other processes are interfering with Playwright, and try running in a headful mode (if available via options) to debug browser launch issues. Ensure your Node.js version is compatible with Playwright.

• CSV output does not match expected format / Missing columns.

  cause Breaking changes in CSV header order (v2.5.3), delimiter (v2.4.2), or added fields can alter the structure of the output CSV.
  fix Check the `tweet-harvest` changelog for recent versions to identify changes in export format. Adjust your CSV parsing logic to account for new delimiters, header order, or additional columns.

Warnings

• breaking The short option for the `--to` flag (`-t`) was removed due to ambiguity with other short options. Users relying on `-t` for the 'to date' will need to update their scripts.
  Fix: Use the full `--to` flag instead of the `-t` short option, or consult CLI help for current short options.
• gotcha Tweet Harvest requires a valid Twitter `auth_token` cookie for authentication. This token can expire or become invalid, leading to failed scrapes. Twitter actively prohibits unauthenticated search, making this token essential.
  Fix: Regularly obtain an updated `auth_token` by logging into Twitter in your browser and extracting the `auth_token` cookie. Pass it via the `--auth-token` CLI option or `auth_token` option in programmatic usage.
• breaking The default CSV delimiter was changed from `;` to `,`. This will affect any scripts or tools parsing the output that expected the semicolon delimiter.
  Fix: Update CSV parsing logic to expect comma-separated values, or check for an option to explicitly set the delimiter if one is introduced in later versions.
• gotcha Changes in Twitter's cookie domains or internal structure can cause authentication or scraping failures. Version 2.6.1 specifically addressed a fix for 'cookie domain changes'.
  Fix: Ensure you are using the latest version of `tweet-harvest` to benefit from upstream fixes for Twitter's evolving platform. If issues persist, your `auth_token` might be outdated or Twitter may have implemented new anti-scraping measures.
• breaking Consistency of CSV headers order was fixed, and support for Gephi format was added. Existing scripts that rely on a specific, potentially inconsistent, header order might break or yield incorrect data.
  Fix: Review CSV output and update any downstream processing scripts to be resilient to changes in header order or to specifically match headers by name rather than by index.

Install

• npm install tweet-harvest npm
• yarn add tweet-harvest yarn
• pnpm add tweet-harvest pnpm

Imports

• harvest
  wrong:

  const { harvest } = require('tweet-harvest');

  correct:

  import { harvest } from 'tweet-harvest';

  Primary programmatic function to initiate the tweet scraping process. Designed for ESM usage. CommonJS `require` is generally not supported without transpilation.
• Options

  import { type Options } from 'tweet-harvest';

  TypeScript type definition for the configuration object passed to the `harvest` function. Essential for type-safe programmatic usage.
• cleanTweet
  wrong:

  const cleanTweet = require('tweet-harvest').cleanTweet;

  correct:

  import { cleanTweet } from 'tweet-harvest';

  Utility function for processing and cleaning individual tweet data. Primarily designed for ESM.

Quickstart

Demonstrates programmatic usage of `tweet-harvest` to scrape tweets with specified keywords and date ranges, requiring a Twitter authentication token.

import { harvest } from 'tweet-harvest';
import type { Options } from 'tweet-harvest';

const twitterAuthToken = process.env.TWITTER_AUTH_TOKEN ?? ''; // Get this from your browser cookies

if (!twitterAuthToken) {
  console.error('TWITTER_AUTH_TOKEN environment variable is not set. Please provide a valid Twitter auth token from your browser cookies.');
  process.exit(1);
}

const options: Options = {
  keyword: 'AI ethics',
  from: '2023-01-01',
  to: '2023-12-31',
  filename: 'ai-ethics-tweets',
  limit: 100, // Limit to 100 tweets for this example
  exportFormat: 'csv',
  auth_token: twitterAuthToken,
  withReplies: false,
  withImages: false,
  withVideos: false
};

async function runHarvest() {
  console.log('Starting tweet harvest...');
  try {
    await harvest(options);
    console.log(`Successfully harvested tweets to ${options.filename}.csv`);
  } catch (error) {
    console.error('Error during tweet harvest:', error);
    if (error instanceof Error && error.message.includes('auth_token')) {
        console.error('Ensure your TWITTER_AUTH_TOKEN is valid and up-to-date.');
    }
  }
}

runHarvest();