Website : rimsha.abasa.com
backdoor
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
var
/
www
/
talha_silentcontent
/
Filename :
README.md
back
Copy
# SilentContent An AI-powered autoblogging platform that generates SEO-optimized articles using OpenAI/DeepSeek APIs, web scraping, and NLP processing. Built with Next.js 14, TypeScript, MongoDB, and Redis. ## 🚀 Quick Start ### Prerequisites - Node.js 20+ - MongoDB instance - Redis instance - OpenAI or DeepSeek API key ### Installation 1. **Clone the repository** ```bash git clone <repository-url> cd talha_silentcontent ``` 2. **Install dependencies** ```bash npm install ``` If you encounter peer dependency conflicts, use: ```bash npm install --legacy-peer-deps ``` 3. **Environment Variables** Create a `.env` file in the root directory with the following variables: ```env # Database MONGO_URI=your_mongodb_connection_string # Authentication TOKEN_SECRET=your_jwt_secret_key # Redis (for job queues) REDIS_HOST=your_redis_host REDIS_PORT=your_redis_port REDIS_PASSWORD=your_redis_password REDIS_USERNAME=your_redis_username # Email (for password reset) EMAIL_HOST=your_smtp_host EMAIL_PORT=your_smtp_port EMAIL_USER=your_email EMAIL_PASS=your_email_password # AWS (optional, for file storage) AWS_ACCESS_KEY_ID=your_aws_key AWS_SECRET_ACCESS_KEY=your_aws_secret AWS_REGION=your_aws_region ``` 4. **Run the development server** ```bash npm run dev ``` 5. **Start the queue worker** (in a separate terminal) The application uses BullMQ for background job processing. The queue worker is automatically initialized when the app starts, but ensure Redis is running. 6. **Open your browser** Navigate to [http://localhost:3000](http://localhost:3000) ## 📦 Tech Stack ### Frontend - **Next.js 14** - React framework with App Router - **TypeScript** - Type safety - **Tailwind CSS** - Utility-first styling - **Shadcn UI** - Component library built on Radix UI - **React Hook Form + Zod** - Form management and validation - **TanStack Query** - Server state management - **TipTap** - Rich text editor - **TanStack Table** - Data tables ### Backend - **Next.js API Routes** - Serverless API endpoints - **MongoDB + Mongoose** - Database and ODM - **Redis + BullMQ** - Job queue system - **JWT** - Authentication - **bcryptjs** - Password hashing - **OpenAI/DeepSeek** - AI content generation - **Cheerio** - Web scraping - **Natural.js** - NLP processing (TF-IDF, tokenization) - **Trigger.dev** - Background job orchestration (configured) ## 🏗️ Project Structure ``` talha_silentcontent/ ├── src/ │ ├── app/ │ │ ├── (application)/ # Protected routes (dashboard, articles, etc.) │ │ │ ├── dashboard/ # Main content generation interface │ │ │ ├── articles/ # Article management │ │ │ ├── batches/ # Batch processing view │ │ │ ├── organizations/ # Organization management │ │ │ └── settings/ # User settings │ │ ├── (auth)/ # Public auth routes │ │ │ ├── signin/ │ │ │ ├── signup/ │ │ │ └── reset-password/ │ │ ├── api/ # API routes │ │ │ ├── articles/ # Article CRUD │ │ │ ├── batches/ # Batch management │ │ │ ├── organizations/ # Org management │ │ │ ├── signin/ # Authentication │ │ │ └── ... │ │ └── dbConfig/ # Database utilities │ │ ├── dbConfig.ts # MongoDB connection │ │ ├── aigenerate.ts # AI API wrapper │ │ ├── scrapeData.ts # Web scraping logic │ │ └── longGenerate.ts # Long-form content generation │ ├── components/ # React components │ │ ├── ui/ # Shadcn UI components │ │ ├── Articles/ # Article-related components │ │ ├── Batches/ # Batch components │ │ └── ... │ ├── models/ # Mongoose schemas │ │ ├── userModel.js │ │ ├── batchModel.js │ │ ├── keywordModel.js │ │ └── ... │ ├── queues/ # Job queue configuration │ │ ├── redisClient.ts # Redis connection │ │ └── longArticleQueue.ts # Article generation queue │ └── lib/ # Utilities │ ├── validators/ # Form validation schemas │ └── ... ├── public/ # Static assets └── package.json ``` ## 🔄 Architecture Overview ### Frontend Flow 1. User submits keywords (CSV or text) on `/dashboard` 2. Form data validated with Zod schemas 3. POST request to `/api/articles` creates batch and keywords 4. Keywords queued for processing via BullMQ 5. Real-time status updates via polling ### Backend Flow 1. **API Route** (`/api/articles`) receives request 2. Creates `Batch` and `Keyword` documents in MongoDB 3. Adds jobs to `longArticlesQueue` (BullMQ) 4. **Queue Worker** processes jobs: - Web scraping (if enabled) with Cheerio - NLP processing with Natural.js (TF-IDF, tokenization) - AI content generation via OpenAI/DeepSeek - Updates keyword status in database ### Key Libraries Usage - **BullMQ**: Background job processing for article generation - **Cheerio**: Scrapes top search results for content research - **Natural.js**: Extracts keywords and analyzes content relevance - **TipTap**: Rich text editing for article customization - **React Query**: Manages server state, caching, and polling - **Zod**: Runtime validation for forms and API requests ## 🛠️ Development ### Available Scripts ```bash npm run dev # Start development server npm run build # Build for production npm run start # Start production server npm run lint # Run ESLint ``` ### Queue Processing The application uses BullMQ workers that automatically process jobs from Redis. Ensure Redis is running before starting the app. Workers handle: - Article generation jobs - Status updates - Error handling and retries ### Database Models - **User**: Authentication and user profiles - **Batch**: Groups of keywords/articles - **Keyword**: Individual article generation tasks - **Prompt**: Reusable prompt templates - **Organization**: Team collaboration - **OrganizationJob**: Scheduled content generation jobs ## 🐳 Docker A Dockerfile is included for containerized deployment. The app runs on port 3034 in production: ```bash docker build -t silentcontent . docker run -p 3034:3034 silentcontent ``` Note: Update the port in `Dockerfile` if you need a different port. ## 📝 Notes - TypeScript build errors are ignored in production (`ignoreBuildErrors: true` in `next.config.js`) - The app uses Next.js middleware for route protection - JWT tokens stored in httpOnly cookies - Queue workers use rate limiting (1 job per 500ms) ## 🔒 Security - Passwords hashed with bcryptjs - JWT-based authentication - httpOnly cookies for token storage - Route protection via Next.js middleware ## 📄 License Private project - All rights reserved