Gibberish Detector

0.1.1 · maintenance · verified Thu Apr 16

The `gibberish-detector` Python library, currently at version 0.1.1, identifies nonsensical strings using a Markov Chain-based model. It's an adaptation of an earlier project, updated for Python 3. Users first train a model on a corpus of 'good' text to understand character transition probabilities, and then use this model to determine if new input strings are gibberish. The library is primarily maintained as a utility for text validation and spam filtering, with updates occurring on an infrequent basis.

Common errors

```
FileNotFoundError: [Errno 2] No such file or directory: 'big.model'
```
cause The `detector.create_from_model()` function was called with a model file path that does not exist or is incorrect.

fix Ensure you have trained a model and provided the correct path to the `.model` file. Example training: `gibberish-detector train examples/big.txt > big.model`.
```
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```
cause The model file specified is either empty, corrupted, or not a valid JSON format that the `gibberish-detector` expects.

fix Verify the integrity of your `.model` file. Retrain the model if necessary using `gibberish-detector train <path_to_good_text_file> > your_model.model` to ensure a correctly formatted model file is generated.

Warnings

gotcha The library requires a pre-trained model file to detect gibberish. Simply installing the package does not provide a functional model out-of-the-box. Attempting to use `create_from_model` without a valid model file will result in errors.
Fix: First, train a model using the command-line interface: `gibberish-detector train <path_to_good_text_file> > your_model.model`. Then, load this generated model file using `detector.create_from_model('your_model.model')`.
gotcha The effectiveness of gibberish detection heavily depends on the quality and size of the training data. A model trained on a small or unrepresentative dataset may produce inaccurate results.
Fix: Use a large corpus of relevant, non-gibberish text (e.g., several megabytes of English text) to train your model for optimal performance. The GitHub README suggests `examples/big.txt` for training.

Install

pip install gibberish-detector Install latest version

Imports

detector
```
from gibberish_detector import detector
```

Quickstart

This quickstart demonstrates how to import the `gibberish_detector` and use a trained model to detect gibberish. Note that a valid trained model file is essential for the `create_from_model` function to work correctly. The provided code creates a dummy model file, which will likely cause an error upon loading but illustrates the API usage. For actual detection, you must first train a model using the `gibberish-detector train` command-line tool, providing a large text file of 'good' (non-gibberish) text, and then point `create_from_model` to your generated model file.

import os
import tempfile

# NOTE: In a real scenario, you would train a model on a large text file.
# For this quickstart, we'll create a dummy model file for demonstration.
# A proper model training example would be:
#   gibberish-detector train examples/big.txt > big.model

# Create a dummy model file for demonstration purposes
# This content is NOT a valid gibberish-detector model and will likely fail.
# It's purely to show the API usage. A real model is a JSON file.
model_content = "{}"

with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.model', encoding='utf-8') as tmp_model_file:
    tmp_model_file.write(model_content)
    model_path = tmp_model_file.name

try:
    from gibberish_detector import detector

    # Attempt to load from the dummy model file
    # This will likely fail with a JSONDecodeError or similar since it's an empty dict string.
    # In a real application, ensure your model_path points to a valid, trained model file.
    print(f"Attempting to load model from: {model_path}")
    my_detector = detector.create_from_model(model_path)

    # Example usage with a loaded detector
    print(f"'superman' is gibberish: {my_detector.is_gibberish('superman')}")
    print(f"'ertrjiloifdfyyoiu' is gibberish: {my_detector.is_gibberish('ertrjiloifdfyyoiu')}")

except Exception as e:
    print(f"Could not run quickstart due to an error. This is expected if the model_path is not a valid trained model. Error: {e}")
    print("To run properly, first train a model using the command line tool:")
    print("  gibberish-detector train <path_to_good_text_file> > your_model.model")
    print("Then, replace 'model_path' above with 'your_model.model'.")
finally:
    # Clean up the dummy model file
    if os.path.exists(model_path):
        os.remove(model_path)

view raw JSON →