StoryForge AI Studio: Multilingual Story-to-Video Pipeline

Wed, 01 Jan 2025 00:00:00 +0000

StoryForge turns a handful of keywords into a complete narrated video — story, scene images, voice narration, and compiled MP4 — through a 7-stage generative pipeline with a Streamlit interface. Supports English, Hindi, French, German, Spanish, Italian, and Telugu.

Pipeline Architecture

Each stage is a dedicated Python module; they can be run individually or end-to-end through the Streamlit UI.

#	Stage	Module	Model / Tool
1	Story generation	`story_generation.py`	Llama 3 via Ollama
2	Scene prompt extraction	`prompt_generator.py`	Llama 3
3	Image synthesis	`image_generator.py`	Stable Diffusion 2.1 + LoRA
4	Consistency verification	`consistancy_check.py`	BLIP + CLIP
5	Audio narration	`audio_generator.py`	gTTS
6	Video compilation	`movie_creater.py`	MoviePy
7	Interactive interface	`streamlit.py`	Streamlit

Character consistency across scenes is enforced with SameFace_Fix LoRA fine-tuning on the diffusion model. Semantic alignment between generated images and story prompts is measured using CLIP similarity scores, averaging >25.0 in test runs.

Technical Stack

LLM: Llama 3 (Ollama) for story and prompt generation
Image generation: Stable Diffusion 2.1, custom LoRA fine-tuning, VAE
Vision-language: BLIP (captioning), CLIP (semantic similarity)
Audio: gTTS with automatic language detection
Video: MoviePy (image transitions + audio sync, MP4/AVI output)
Framework: PyTorch, HuggingFace Transformers
UI: Streamlit with real-time progress tracking and ZIP export

Demo Video

Demo video coming soon.

Project Report

Project report coming soon.

Course: ES667 — Deep Learning, IIT Gandhinagar (Jan–Apr 2025) · 4-member team · Supervised by Prof. Anirban Dasgupta

Semantic Segmentation of Aerial Drone Images

Wed, 01 Nov 2023 00:00:00 +0000

A systematic benchmark of four segmentation architectures on the — 400 high-resolution aerial images (up to 4000×6000 px) with pixel-level labels across 23 urban classes. The goal: understand the accuracy–cost tradeoff across CNN and transformer approaches under real GPU memory constraints.

Best result: UNet at 86.4% accuracy, outperforming DeepLabV3+ (~80%), Swin Transformer (~82%), and SegNet (~75%).

Dataset

Semantic Drone Dataset — 400 images, 350 train / 50 test, 23 semantic classes:

Structures: building, wall, fence, bridge
Nature: tree, vegetation, grass, water
Infrastructure: road, sidewalk, parking, rail track
Objects: car, truck, person, bicycle
Environment: sky, ground, terrain

Images were downscaled to 384×512 for training due to GPU memory constraints (original 4000×6000 px).

Architectures

Model	Params	Accuracy	Inference
UNet	~34.6M	86.4%	Fast
Swin Transformer	—	~82%	Slow
DeepLabV3+ (ResNet-50)	—	~80%	Medium
SegNet (VGG)	—	~75%	Fast

All models trained with Adam (lr=0.001), sparse categorical crossentropy, up to 500 epochs with early stopping; convergence around epoch 300–400.

Key tradeoffs:

UNet’s encoder–decoder with skip connections gave the best overall accuracy with fast inference.
Swin Transformer captured finer details thanks to global attention but at high compute cost.
SegNet was the most memory-efficient via pooling-index upsampling.
DeepLabV3+ with ASPP is best suited for full-resolution inference when memory allows.

Known limitation: all models struggled with small objects (vehicles, pedestrians <60% accuracy) due to downscaling; patch-based inference at full resolution would close this gap.

Technical Stack

TensorFlow 2.x / Keras · NumPy · OpenCV · Google Colab (T4 GPU) · TensorBoard

Project Report

Course: CS5480 — Deep Learning for Vision, IIT Hyderabad (Aug–Nov 2023) · Supervised by Prof. Summohana Channapayya

Computer Vision | Abhiroop Chintalapudi - AI Researcher

StoryForge AI Studio: Multilingual Story-to-Video Pipeline

Pipeline Architecture

Technical Stack

Demo Video

Project Report

Semantic Segmentation of Aerial Drone Images

Dataset

Architectures

Technical Stack

Project Report