Essentially this is an advanced prompt engineering system with a specific purpose - to help AI models better understand and work with my thinking patterns. At its core, it's a middleware layer between my thoughts and AI assistants, built using specialized datasets that capture how I think, what I care about, and my ongoing initiatives. The system starts with voice-to-text captures of my daily thoughts, runs them through several processing layers (like transcription correction and thought classification), and ultimately generates actionable recommendations and discussion agendas. While I've built in ways to measure and improve the system's effectiveness, it's not really about continuous self-improvement - rather, I'm aiming to create a stable, reliable interface that consistently helps AI models understand my context and goals. Think of it as a translation layer that helps AI assistants function more like a COO to my CEO, taking my raw thoughts and turning them into structured, actionable insights that align with my objectives. Yes, you could call it a thought processor, but it's really just sophisticated prompt engineering with extra steps to ensure the AI understands my specific context and patterns. expectations of success - 4.0: 70-80% - o1-mini: 75% - sonnet: 40% ### Proof of Concept #### Build Datasets - [ ] manually correct transcription errors and have AI create [[#Common Transcription Error Data Point]] by comparing the difference between original and correction etc:30 - [ ] manually tag 100 [[#Flow Classification Data Point]]s etc:180 - during flow: mention concepts, ideas, initiatives - after flow: mention emotion, importance(1-4) - during review: tag action - [ ] create 20 [[#Concepts Data Point]]s etc:45 - [ ] create all current [[#Initiative Data Point]]s etc:20 #### Create Agenda, Follow-ups, and Action Recommendations #### Test Ideation Partner ### Process #### Idea Capture Throughout the day, as you come up with new ideas or thoughts, you simply speech-to-text a new line into your second brain. #### Process Ideas ##### Speech-to-Text Correction Improves transcription accuracy by identifying and correcting common misheard words based on your speech patterns - input: [[#Common Transcription Error Data Point]] - output: corrected flows ###### Rules | Category | Rule | Example | Rationale | | -------- | -------------------------- | ------------------------------------------------------------------------- | ---------------------------------- | | DO | Fix transcription errors | "phil" -> "Theo" | Maintains accuracy | | DO | Fix disfluencies | "I I think" -> "I think" | Removes speech artifacts | | DO | Fix basic punctuation | "she ran she jumped" -> "she ran. She jumped." | Improves readability | | DO | Fix incorrect articles | "a apple" -> "an apple" | Grammar correction | | DO | Fix verb agreement | "they is" -> "they are" | Basic grammar | | DO | Fix obvious typos in names | "jsutine" -> "Justine" | Maintains clarity | | DON'T | Reorganize sequence | Keep "went upstairs, tired, got book" order | Preserves thought flow | | DON'T | Change temporal flow | Keep "going to store...at store now" | Maintains timeline | | DON'T | Remove observations | Keep "let's see...oh nothing there" | Preserves context | | DON'T | Change word choice | Keep "really awesome" vs "excellent" | Maintains voice | | DON'T | Add context | Keep original level of detail | Preserves authenticity | | DON'T | Break up flows | Keep "went to store got milk saw friend talked about work" as single flow | Preserves thought unit | | DON'T | Combine flows | Keep separate entries even if about same topic | Maintains discrete thought capture | ###### Challenges - Determining what constitutes a "disfluency" vs intentional repetition - Maintaining the right balance of fixing vs preserving stream-of-consciousness style - Consistent handling of temporal markers - Understanding context-specific importance of certain repetitions or phrases ###### Testing 1. prompt AI to process `Uncorrected Test Flow` given [[#Transcription Correction Data Point]]s 2. collect `AI corrected test flow` with confidence metrics 3. score each `AI corrected test flow` against `manual corrected test flow` following a rubrik 4. create table of flows with columns original, manual correction, AI correction, AI confidence, AI score, possible improvments 5. calculate overall test score by averaging all flow's AI scores ##### Stream of Consciousness Classifier Organizes raw thoughts into actionable categories like emotions, actions, and priorities for easy processing. - input: corrected flows, [[#Flow Classification Data Point]], [[#Initiative Data Point]], [[#Concepts Data Point]] - output: classified flows ##### Action Recommendations and Conversation Agenda Creation Analyzes thoughts to suggest actionable items and create structured, prioritized agendas for discussion. - inputs: classified flows, [[#flow to action item data point]], [[#flow to agenda item data point]], [[#Flow to Follow Up Questions Data Point]], [[#Initiative Data Point]], [[#Concepts Data Point]] - outputs: agenda, follow up questions, action items #### Daily Discussion Engages in fluid, natural conversations to help refine and expand ideas with insightful feedback. - inputs: [[#Initiative Data Point]], [[#Concepts Data Point]], agenda, action items, follow up questions - output: updated action items ### Definitions - flow: a single thought captured in a bullet point. each flow contains multiple related thoughts, which form a coherent unit in the context of your thinking process. - personal knowledge management system (PKM): is a set of tools and processes individuals use to capture, organize, and apply information for personal growth and decision-making. It helps manage knowledge, integrate insights, and support learning and goal achievement. ### Datasets #### Flow Classification Data Point - expect ~100 ``` { "flow": "I'm thinking about my financial goals and how I might need to adjust them after the unexpected expenses last month.", "tags": ["financial goals", "adjustment", "short-term objectives"], "emotion": "concerned", "importance": "high", "action": "evaluate budget" } ``` #### Transcription Correction Data Point - expect ~100 - includes disfluencies ``` { "original": "I me and my attitude was the main issue." "correction": "my attitude was the main issue.", "notes": "removed disfluency" } ``` #### Flow to Agenda Item Data Point - expect <50 ``` { "agenda_item": "Discuss project progress with the team", "flows": [ "I need to talk to the team about the project.", "Review project deadlines."] "rationale": "..." } ``` #### Flow to Action Item Data Point - expect <50 ``` { "action": "Schedule a meeting", "flows": ["I need to talk to the team about the project.",... ] "rationale": "....." } ``` #### Flow to Follow Up Questions Data Point - expect <50 ``` { "question": "what do you mean by brocoli", "flows": ["I never want to go to brocoli",... ] "rationale": "brocoli is more of a metaphor her to describe ..." } ``` #### Initiative Data Point - expect <10 action initiatives at any time - I will update data when adding/removing initiative ``` [ "easy family": "Our family is implementing a system to reduce parental burden by encouraging our 5-year-old twins to take on more responsibility. Through daily meetings and AI-assisted tracking, we’ll set clear expectations and celebrate each person’s contributions. The system includes a quick morning huddle to plan the day, an evening recap to celebrate progress, and weekly feedback from AI to guide improvements. The goal is to turn daily struggles into growth opportunities, fostering family unity and teaching the kids about the importance of helping out, all while reducing the management load on parents." ] ``` #### Concepts Data Point - expect <10 action initiatives at any time - I will periodically update data ``` [ "liberalism": "a political philosophy that promotes individual freedom, equality, and democracy, emphasizing the protection of personal rights and liberties. It advocates for a government that ensures fairness, provides opportunities for all, and fosters social progress. At its core, liberalism seeks to create a society where individuals can pursue their own happiness and potential, while respecting the rights of others. It supports policies that protect freedoms, promote economic opportunity, and ensure equality before the law, all while encouraging active participation in a democratic system." ] ``` ### Desired Relationship Examples - COO/CEO: You, as the CEO, set the direction, and the AI (COO) **coordinates, manages, and executes** the operational elements of the plan. You focus on high-level decisions while the AI ensures everything is carried out efficiently and on schedule. - Air Traffic Controller/Pilot: You maintain control and make decisions while the AI coordinates, provides guidance, monitors conditions, and ensures smooth operations. - Spock/Kirk: You lead with intuition and take action while the AI provides logical analysis, data, and objective recommendations without overriding your authority. - idea manager: tagging and initiative alignment ### Future Work - Relationship Management Tool - RAG (Retrieval-Augmented Generation): Use AI to retrieve relevant information from your second brain or external sources, augmenting your conversations. - Conversation Prioritization: Tools to help prioritize and manage which topics to discuss or work on based on your ongoing needs. ### Notes - Yes, you're building a personalized system through **prompt engineering** where these datasets play distinct roles: - You're building an adapter between your second brain (your knowledge management system) and conversational AI to help organize and process your thoughts, test them against relevant initiatives, and generate actionable insights or decisions based on those inputs. You’re also trying to make this process as streamlined and useful as possible by tagging and evaluating how well the AI interprets and organizes your ideas. - hope some AI can generate value aligned with your needs, goals, and methods - creating a highly structured interaction system between your thoughts and the AI that not only aids in organizing ideas but also optimizes the way the AI interacts with and interprets your input. It’s about engineering artifacts and processes that guide the AI to generate insights or responses. - translator and enhancer for your thoughts - personalized prompt engineering - fine-tuning required to make LLMs understand your unique flow - The adapter is evaluating the prompt engineering as much as the LLM