VidSense: My Final Year Project, or How I Got Into AI

VidSense was my final year project at IBA, built with four teammates and advised by Dr. Muhammad Saeed and Umair Nazir. Honestly, this is the project that pulled me toward AI engineering as a career direction rather than a side interest.

The problem we picked

Video is the format everyone consumes and nobody can search. You can't Ctrl+F a lecture recording, you can't skim a two hour podcast for the ten minutes that matter, and if you want a summary you either watch the whole thing or trust someone else's. We wanted something that understood video well enough to answer questions about it, produce real timestamped clips, and generate genuinely useful derivative content without a human editing it.

Transcription, the boring layer everything depends on

We used a cascading strategy. Try YouTube's native transcript API first, since it's fast and often more accurate when it exists, then fall back to Whisper Large-v3-Turbo running locally on GPU when it doesn't. Local uploads, disabled transcripts, whatever the case. Getting that fallback logic right mattered more than any of the flashier features, because every downstream feature is only as good as the transcript underneath it.

Building the retrieval layer

We chunked transcripts along semantic boundaries rather than fixed time windows, embedded them with sentence-transformers, and built three tiers of retrieval. Semantic similarity first, then weighted text matching where exact phrases and proper nouns get priority, then broader chunk retrieval if both come up short. That tiering came out of trial and error, watching the system fail on short specific questions where pure embedding similarity wasn't precise enough.

For generation we ran DeepSeek-R1 locally through Ollama rather than calling an external API for everything. Partly for privacy on sensitive content, partly because it let us iterate without burning API credits, which matters more than it sounds when you're a student team.

The feature I'm proudest of

Highlight reel generation. It isn't one model doing everything, it's a hybrid. Fast algorithmic scoring using word frequency, structural cues and speech dynamics picks the candidate segments, then the LLM writes titles and descriptions once the boundaries are already locked. That split gave us speed and quality at once. Pure LLM approaches were slow and sometimes chose bad boundaries. Pure algorithmic approaches chose fine boundaries and wrote terrible copy.

What I took from it

This was where I learned that AI engineering isn't really about the model, it's about the plumbing around it. Cascading fallbacks, caching so you aren't re-running expensive calls, validating that generated timestamps actually exist in the source transcript, keeping quality consistent across languages. The model call is maybe twenty percent of the real engineering. That lesson has shaped everything I've built since.