StoryForge AI Studio: Multilingual Story-to-Video Pipeline
StoryForge AI Studio PipelineStoryForge turns a handful of keywords into a complete narrated video — story, scene images, voice narration, and compiled MP4 — through a 7-stage generative pipeline with a Streamlit interface. Supports English, Hindi, French, German, Spanish, Italian, and Telugu.
Pipeline Architecture
Each stage is a dedicated Python module; they can be run individually or end-to-end through the Streamlit UI.
| # | Stage | Module | Model / Tool |
|---|---|---|---|
| 1 | Story generation | story_generation.py | Llama 3 via Ollama |
| 2 | Scene prompt extraction | prompt_generator.py | Llama 3 |
| 3 | Image synthesis | image_generator.py | Stable Diffusion 2.1 + LoRA |
| 4 | Consistency verification | consistancy_check.py | BLIP + CLIP |
| 5 | Audio narration | audio_generator.py | gTTS |
| 6 | Video compilation | movie_creater.py | MoviePy |
| 7 | Interactive interface | streamlit.py | Streamlit |
Character consistency across scenes is enforced with SameFace_Fix LoRA fine-tuning on the diffusion model. Semantic alignment between generated images and story prompts is measured using CLIP similarity scores, averaging >25.0 in test runs.
Technical Stack
- LLM: Llama 3 (Ollama) for story and prompt generation
- Image generation: Stable Diffusion 2.1, custom LoRA fine-tuning, VAE
- Vision-language: BLIP (captioning), CLIP (semantic similarity)
- Audio: gTTS with automatic language detection
- Video: MoviePy (image transitions + audio sync, MP4/AVI output)
- Framework: PyTorch, HuggingFace Transformers
- UI: Streamlit with real-time progress tracking and ZIP export
Demo Video
Demo video coming soon.
Project Report
Project report coming soon.
Course: ES667 — Deep Learning, IIT Gandhinagar (Jan–Apr 2025) · 4-member team · Supervised by Prof. Anirban Dasgupta
